Got feedback or spotted a mistake?

Leave a comment at the end of this page or email contact@krishagni.com

Data Validations (Edit Checks)

This feature is part of a paid plugin that is not part of EE or CE. Email contact@krishagni.com for more details.

Introduction

Data validation is used to enter high-quality data. This ensures that while entering the data, the user does not violate any data entry rules.

Data validation can be defined either at the system level or at the CP level. The data validations defined at the CP level take precedence over the system-level data validations.

What is Data Validation or Edit Check?

  1. Data Validation is a series of constraints designed to check the validity of the input data.

  2. Data Validation constraints can be made of variables of a single record or multiple records.

  3. Data Validation is like database check constraints designed to ensure high-quality specimen data is collected.

Examples

  1. Participant age at the time of registration should be more than 18 years

  2. Only male participant registrations are allowed

  3. 'Male' participant clinical diagnoses should not be anything but 'Adenolymphoma'.

  4. Frozen event timestamp should be later than the specimen creation time.

  5. Cryostat is the preferred method for freezing tissue samples.

How to define data validation?

Data validations can be defined using JSON. The code has three attributes - records, forms and rules

  1. records - Whenever any of these specified records are modified; the constraint rules should be evaluated/enforced. This refers to the data entry level at which the fields are present for which the edit check is to be applied. The value for records can be cpr, specimen, visit or a combination of these.

  2. forms - Optional field. If present, specifies the list of forms whose data is used to enforce the constraints.

  3. rules - List of integrity rules that should be satisfied by the records and forms.

A ‘rule’ is made up of three attributes when, expr, and description.

when: The attribute 'when' is optional and specifies when the rule is applicable. For example, the rule is applicable only for primary specimens

expr: The attribute 'expr' specifies the check constraint that should be satisfied.

description: The attribute 'description' describes the constraint in user-friendly language. This is also used in error messages when the rule is broken or not satisfied.

The aliases available for use in the rules are - “cpr, visit, primarySpecimen, specimen”, which are self-explanatory.

The aliases used for accessing the form records are cprForms, visitForms, primarySpecimenForms, specimenForms. These are maps that can be indexed by the form name to access the relevant form data. For example, specimenForms['SpecimenFrozenEvent'] can be used to access the latest frozen event. Similarly, specimenForms['SpecimenFrozenEvent$Array'] can be used to access the list of all the frozen events for the specimen in question.

The ‘editChecks’ section of the code needs to be inserted after the ‘dictionary’ section in the workflows JSON.

Example JSON

 

{ "name" : "editChecks", "data" : { "constraints" : [ { "records" : [ "cpr", "visit" ], "rules" : [ { "expr" : "#cpr.registrationDate != null && #visit.visitDate != null && !#cpr.registrationDate.after(#visit.visitDate)", "description" : "Visit date should be same or later than the registration date!" } ] }, { "records" : [ "cpr", "primarySpecimen" ], "rules" : [ { "when" : "#specimen.status == 'Collected'", "expr" : "!#cpr.registrationDate.after(#specimen.collectionEvent.time)", "description" : "The specimen #specimen.label (#specimen.type) collection date should be same or later than the registration date!" }, { "when" : "#specimen.status == 'Collected'", "expr" : "#cpr.participant.gender != 'Male Gender' || #specimen.extensionDetail?.getAttrsMap()?.get('DD6') == 'Room Temperature'", "description" : "Specimen #specimen.label should be stored at room temperature!" } ] }, { "records" : [ "visit", "primarySpecimen" ], "rules" : [ { "when" : "#primarySpecimen.status == 'Collected'", "expr" : "!#containsAny(#visit.clinicalDiagnoses, {'Liver regeneration (disorder)', 'Liver cell adenoma'}) || #primarySpecimen.anatomicSite == 'Liver'", "description" : "Specimen #specimen.label anatomic site should be Liver" } ] }, { "records": ["specimen"], "forms": { "specimen": ["SpecimenFrozenEvent", "SpecimenTissueReviewEvent"] }, "rules": [ { "when": "#specimen.specimenClass == 'Tissue'", "expr": "#collFns.forEvery(#specimenForms['SpecimenFrozenEvent$Array'], 'fe', \"#f['method'] == 'Cryostat'\")", "description": "Tissue specimen #specimen.label freezing method is not Cryostat!" }, { "expr": "#collFns.forEvery(#specimenForms['SpecimenFrozenEvent$Array'], 'fe', \"#f['time'].after(#specimen.createdOn)\")", "description": "Specimen #specimen.label freezing time should be later than the creation time" }, { "when": "#specimen.specimenClass == 'Tissue'", "expr": "#specimenForms['SpecimenTissueReviewEvent'].histologicalQuality != 'Poor- No Definable Features'", "description": "Poor quality tissue specimens not allowed!" } ] } ] } }

Attribute Types and Examples

Below are the examples of edit checks implemented in 4 cases:

  1. Core fields

  2. Custom fields

  3. Custom form fields

  4. Combination of the core fields, custom fields, custom form fields.

Core Fields

Example: Making fields mandatory

Data entry level: Participant

Use case: Patient first name, last name should not be null

{ "records" : [ "cpr" ], "rules" : [ { "expr" : "#cpr.participant.firstName != null && #cpr.participant.firstName.length() > 0 && #cpr.participant.lastName != null && #cpr.participant.lastName.length() > 0", "description" : "First name and last name should not be null" } ] }

If there are more than one rules to be applied in the case of the same set of records, they can be added under the same records.

Example: Restrict value selected in one field based on values selected in another field at the participant level

Data entry level: Participant

Use case 1: Only male participant registrations are allowed

Use case 2: If Race is one of (White or Asian), then Ethnicity should be 'American.'

{ "records" : [ "cpr" ], "rules" : [ { "expr" : "#cpr.participant.gender != null && !#containsAny(#cpr.participant.gender, {'Female', 'Unknown', 'Undifferentiated'})", "description" : "Only Male patient should be registered" } ] }, { "records" : [ "cpr" ], "rules" : [ { "when" : "#containsAny(#cpr.participant.races, {'White', 'Asian'})", "expr" : "#cpr.participant.ethnicities.size() == 1 && #cpr.participant.ethnicities.contains('American')", "description" : "For White/Asian races, ethnicity should be American" } ] }

It is possible to apply edit check on one data entry level based upon a field in another data entry level. This can be achieved by specifying the respective data entry levels in ‘records.’


Example: Comparison of dates at different levels

Data entry level: Visit, Participant

Use case: The participant registration date should be the same or greater than the participant visit date.


Example: Restrict values that can be selected for a particular field

Data entry level: Visit

Use case: The values for clinical diagnoses should be one of ‘Cholera’, ‘Liver cell carcinoma.’

 

Use case: The values for clinical diagnoses should be ‘Breast implant status’ and ‘Other disorders of breast.’


Example: Using RegEx for validation of a particular custom field

Data entry level: Participant

Use case: The email id of the participant should be of the form ‘example@something.com’


Example: Make field mandatory if a specific value selected for another field

Data entry level: Participant

Use case: If the patient is ‘Dead’ then, specify the value ‘Cause of death’ (participant level custom field).


Example: Based on values selected in the core field, restrict values that can be selected in a custom field

Data entry level: Visit (Core field and custom field)

Use case: If the clinical diagnosis is one of 'Cholera', 'Liver cell carcinoma', then the Diagnosis subtype id 'Negative.'


Custom Form Fields

Example: Restrict selection of certain values under custom form field

Data entry level: Participant custom form

Use case: The Smoking History Form is attached at the participant custom forms level. The value of the ‘Have you ever smoked?' field should be ‘No’.


Combinations of Core & Custom Fields

Example: Based on value selected in one field in custom form, restrict the list of values that can be selected in another field at the visit level

Data entry level: Visit (Custom form field and Core field)

Use case: If the 'Primary Site' values(Custom Form) is in {'Extrahepatic bile duct', 'Intrahepatic bile duct', 'Liver'}, allowed value for Clinical Diagnosis(Core) should be in {'10 weeks gestation of pregnancy', '11 weeks gestation of pregnancy', '12 weeks gestation of pregnancy', '13 weeks gestation of pregnancy', '14 weeks gestation of pregnancy'}


Example: Based on values selected in one field at the participant level, restrict the list of values that can be selected in another field at the specimen level

Data entry level: Participant, Specimen, Visit

Use case: In case a person is of ‘Male’ gender, the anatomic site should not include values in 'Female genital tract, NOS', 'Overlapping lesion of female genital organs', 'Other specified parts of female genital organs', 'Breast, NOS'.

Example: Add a min/max range on a numeric field

Data entry level: Participant

Use case: The participant’s age registered under the COVID protocol should be between 18-55

Error Report

A report of all the data validation errors can be downloaded from the collection protocol overview page > ‘More’ > ‘Export Errors’

Clicking on ‘Export Errors’ redirects to a page as shown below:

Report Interval: You can define the date range for which the report is to be downloaded.

User: This is used to define the user for which the report is to be downloaded.

The report is downloaded when clicked on ‘Export’. In case no value is entered for these fields, an entire report is downloaded, including all errors shown for all users.

The downloaded zip has a CSV file with the below columns:

Column

Description

Column

Description

Identifier

System-generated unique number

User#First Name

First name of the user who was logged in while performing data entry

User#Last Name

Last name of the user who was logged in while performing data entry

User#Email Address

Email address of the user who was logged in while performing data entry

Collection Protocol#Short Title

Collection protocol on which the data validation showed an error

Record Type

The data entry level on which the error occurred. It can have the values:

Registration

Visit

Specimen

Form name of a custom form

PPID

The participant protocol ID for which data entry gave an error

Visit Name

Visit for which the error was thrown. This column will have values only when the error is thrown at the ‘Visit’ level of a custom form attached at visit level

Error Time

The timestamp of the error thrown

Error Message

The display message for the error thrown. This is the same message that is displayed on the screen when an edit check fails. It is defined in JSON.

Error: ‘Cannot index into a null value’

While adding multiple edit checks on non-mandatory fields/forms, it is necessary to have a null check on all the fields involved in the edit check before accessing it. This can be done by checking whether the form/field is not equal to ‘null’ in the when part of the edit check.

Refer to ‘How to solve ‘Cannot index into a null value’ error’ for more details.

Adding Multiple Edit Checks in Same Workflow

In case there are many edit checks to be performed, here is a tip that could help in avoiding errors:

Check and test the data entry workflow after adding each edit check. This will allow you to identify if any edit check fails and will be useful in rectifying the failed one. In case all the edit checks are added together, it might be difficult to identify the edit check in case the above error ‘Cannot index into a null value’ occurs.

Got feedback or spotted a mistake?

Leave a comment at the end of this page or email contact@krishagni.com