...

Introduction

Data validation is can be used to enter add validation rules to fields to ensure high-quality data. This ensures that while entering the data, the user does not violate any data entry rules. Data validation These rules can be defined either at the system level or at the CP level. The data validations defined at the CP level take precedence over the system-level data validations.

What is Data Validation or Edit Check?

1. Data Validation is a series of constraints designed to check the validity of the input data.

2. Data Validation constraints can be made of variables of a single record or multiple records.

3. Data Validation is like database check constraints designed to ensure high-quality specimen data is collected.

...

This ensures that the user does not violate any data entry rules while entering the data.

Examples:

1. Participant age at the time of registration should be more than 18 years

...

Only male participant registrations are allowed

...

2. Frozen event timestamp should be later than the specimen creation time.

...

Cryostat is the preferred method for freezing tissue samples.

How to define data validation?

Data validations can be defined using JSON. The code has three attributes - records, forms and rules

1. records - Whenever any of these specified records are modified; the constraint rules should be evaluated/enforced. This refers to the data entry level at which the fields are present for which the edit check is to be applied. The value for records can be cpr, specimen, visit or a combination of these.

2. forms - Optional field. If present, specifies the list of forms whose data is used to enforce the constraints.

3. rules - List of integrity rules that should be satisfied by the records and forms.

A ‘rule’ is made up of three attributes when, expr, and description.

when: The attribute 'when' is optional and specifies when the rule is applicable. For example, the rule is applicable only for primary specimens

expr: The attribute 'expr' specifies the check constraint that should be satisfied.

description: The attribute 'description' describes the constraint in user-friendly language. This is also used in error messages when the rule is broken or not satisfied.

The aliases available for use in the rules are - “cpr, visit, primarySpecimen, specimen”, which are self-explanatory.

The aliases used for accessing the form records are cprForms, visitForms, primarySpecimenForms, specimenForms. These are maps that can be indexed by the form name to access the relevant form data. For example, specimenForms['SpecimenFrozenEvent'] can be used to access the latest frozen event. Similarly, specimenForms['SpecimenFrozenEvent$Array'] can be used to access the list of all the frozen events for the specimen in question.

The ‘editChecks’ section of the code needs to be inserted after the ‘dictionary’ section in the workflows JSON.

Example JSON

{
  "name" : "editChecks",
  "data" : {
    "constraints" : [ 
      {
        "records" : [ "cpr", "visit" ],
        "rules" : [
          {
            "expr" : "#cpr.registrationDate != null && #visit.visitDate != null && !#cpr.registrationDate.after(#visit.visitDate)",
            "description" : "Visit date should be same or later than the registration date!"
          } 
        ]
      },
      {
        "records" : [ "cpr", "primarySpecimen" ],
        "rules" : [
          {
            "when" : "#specimen.status == 'Collected'",
            "expr" : "!#cpr.registrationDate.after(#specimen.collectionEvent.time)",
            "description" : "The specimen #specimen.label (#specimen.type) collection date should be same or later than the registration date!"
          },
          {
            "when" : "#specimen.status == 'Collected'",
            "expr" : "#cpr.participant.gender != 'Male Gender' || #specimen.extensionDetail?.getAttrsMap()?.get('DD6') == 'Room Temperature'",
            "description" : "Specimen #specimen.label should be stored at room temperature!"
          }
        ]
      },
      {
        "records" : [ "visit", "primarySpecimen" ],
        "rules" : [
          {
            "when" : "#primarySpecimen.status == 'Collected'",
            "expr" : "!#containsAny(#visit.clinicalDiagnoses, {'Liver regeneration (disorder)', 'Liver cell adenoma'}) || #primarySpecimen.anatomicSite == 'Liver'",
            "description" : "Specimen #specimen.label anatomic site should be Liver"
          }
        ]
      },
      {
        "records": ["specimen"],
        "forms": {
          "specimen": ["SpecimenFrozenEvent", "SpecimenTissueReviewEvent"]
        },
        "rules": [
          {
            "when": "#specimen.specimenClass == 'Tissue'",
            "expr": "#collFns.forEvery(#specimenForms['SpecimenFrozenEvent$Array'], 'fe', \"#f['method'] == 'Cryostat'\")",
            "description": "Tissue specimen #specimen.label freezing method is not Cryostat!"
          },
          {
            "expr": "#collFns.forEvery(#specimenForms['SpecimenFrozenEvent$Array'], 'fe', \"#f['time'].after(#specimen.createdOn)\")",
            "description": "Specimen #specimen.label freezing time should be later than the creation time"
          },
          {
            "when": "#specimen.specimenClass == 'Tissue'",
            "expr": "#specimenForms['SpecimenTissueReviewEvent'].histologicalQuality != 'Poor- No Definable Features'",
            "description": "Poor quality tissue specimens not allowed!"
          } 
        ]
      } 
    ]
  }
}

Attribute Types and Examples

Below are the examples of edit checks implemented in 4 cases:

1. Core fields

2. Custom fields

3. Custom form fields

4. Combination of the core fields, custom fields, custom form fields.

Core Fields

Example: Making fields mandatory

Data entry level: Participant

Use case: Patient first name, last name should not be null

{
           "records" : [ "cpr" ],
           "rules" : [
             {
               "expr" : "#cpr.participant.firstName != null && #cpr.participant.firstName.length() > 0 && #cpr.participant.lastName != null && #cpr.participant.lastName.length() > 0",
               "description" : "First name and last name should not be null"
             }
           ]
         }

If there are more than one rules to be applied in the case of the same set of records, they can be added under the same records.

Example: Restrict value selected in one field based on values selected in another field at the participant level

Data entry level: Participant

Use case 1: Only male participant registrations are allowed

Use case 2: If Race is one of (White or Asian), then Ethnicity should be 'American.'

{
           "records" : [ "cpr" ],
           "rules" : [
             {
               "expr" : "#cpr.participant.gender != null && !#containsAny(#cpr.participant.gender, {'Female', 'Unknown', 'Undifferentiated'})",
               "description" : "Only Male patient should be registered"
             }
           ]
         },
         {
           "records" : [ "cpr" ],
           "rules" : [
             {
               "when" : "#containsAny(#cpr.participant.races, {'White', 'Asian'})",
               "expr" : "#cpr.participant.ethnicities.size() == 1 && #cpr.participant.ethnicities.contains('American')",
               "description" : "For White/Asian races, ethnicity should be American"
             }
           ]
         }

It is possible to apply edit check on one data entry level based upon a field in another data entry level. This can be achieved by specifying the respective data entry levels in ‘records.’

Example: Comparison of dates at different levels

Data entry level: Visit, Participant

Use case: The participant registration date should be the same or greater than the participant visit date.

{
  "records" : [ "cpr", "visit" ],
  "rules" : [
    {
      "expr" : "#cpr.registrationDate != null && #visit.visitDate != null && !#cpr.registrationDate.after(#visit.visitDate)",
      "description" : "Visit date should be same or later than the registration date!"
      }
      ]
    }

Example: Restrict values that can be selected for a particular field

Data entry level: Visit

Use case: The values for clinical diagnoses should be one of ‘Cholera’, ‘Liver cell carcinoma.’

{
      "records" : [ "visit"],
      "rules" : [
        {
          "when" : "#visit.status == 'Complete'",
          "expr" : "#containsAny(#visit.clinicalDiagnoses, {'Cholera', 'Liver cell carcinoma'})",
          "description" : "Clinical diagnosis should be Cholera/Liver cell carcinoma"
        }
      ]
    }

Use case: The values for clinical diagnoses should be ‘Breast implant status’ and ‘Other disorders of breast.’

{
      "records" : [ "visit" ],
      "rules" : [ {
        "when" : "#visit.status == 'Complete'",
        "expr" : "#visit.clinicalDiagnoses.size() == 2 && #visit.clinicalDiagnoses.contains('Breast implant status') && #visit.clinicalDiagnoses.contains('Other disorders of breast')",
        "description" : "Clinical diagnosis should be Breast implant status and Other disorders of breast"
      } ]
    },

Example: Using RegEx for validation of a particular custom field

Data entry level: Participant

Use case: The email id of the participant should be of the form ‘example@something.com’

{
      "records" : [ "cpr" ],
      "rules" : [
        {
        "when" : "#cpr.registrationDate != null && #cpr.participant.extensionDetail?.getAttrsMap()?.get('ST6')!= null",
        "expr" : "#cpr.participant.extensionDetail?.getAttrsMap()?.get('ST6') matches '^\\w+(\\.\\w+)*@\\w+(\\.\\w+)*(\\.[A-Za-z]{2,})$'",
        "description" : "Please enter a valid email address"
      } ]
    }

Example: Make field mandatory if a specific value selected for another field

Data entry level: Participant

Use case: If the patient is ‘Dead’ then, specify the value ‘Cause of death’ (participant level custom field).

{
         "records" : [ "cpr" ],
         "rules" : [
           {
             "when" : "#cpr.participant.vitalStatus == 'Dead'",
             "expr" : "#cpr.participant.extensionDetail?.getAttrsMap()?.get('ST2') != null ",
             "description" : "Specify Death Cause"
           }
         ]
       }

Example: Based on values selected in the core field, restrict values that can be selected in a custom field

Data entry level: Visit (Core field and custom field)

Use case: If the clinical diagnosis is one of 'Cholera', 'Liver cell carcinoma', then the Diagnosis subtype id 'Negative.'

{
      "records" : [ "visit"],
      "rules" : [
        {
          "when" : "#containsAny(#visit.clinicalDiagnoses, {'Cholera', 'Liver cell carcinoma'})",
          "expr" : "#visit.extensionDetail?.getAttrsMap()?.get('DD2') == 'Negative' ",
          "description" : "Clinical Subtype should be Negative"
        }
      ]
    },

Custom Form Fields

Example: Restrict selection of certain values under custom form field

Data entry level: Participant custom form

Use case: The Smoking History Form is attached at the participant custom forms level. The value of the ‘Have you ever smoked?' field should be ‘No’.

{
        "records": ["cpr"],
        "forms": {
          "cpr": [ "smokingHistoryForm" ]
        },
        "rules": [
          {
            "when": "#cprForms != null && #cprForms['smokingHistoryForm'] != null",
            "expr": "#cprForms['smokingHistoryForm']['haveYouEverSmoked'] != 'Yes'",
            "description": "SH:Smokers not allowed!"
          }
         ]
       }

Combinations of Core & Custom Fields

Example: Based on value selected in one field in custom form, restrict the list of values that can be selected in another field at the visit level

Data entry level: Visit (Custom form field and Core field)

Use case: If the 'Primary Site' values(Custom Form) is in {'Extrahepatic bile duct', 'Intrahepatic bile duct', 'Liver'}, allowed value for Clinical Diagnosis(Core) should be in {'10 weeks gestation of pregnancy', '11 weeks gestation of pregnancy', '12 weeks gestation of pregnancy', '13 weeks gestation of pregnancy', '14 weeks gestation of pregnancy'}

{
      "records" : [ "visit" ],
      "forms" : {
        "visit" : [ "newPathologyForm" ]
      },
      "rules" : [ {
        "when" : "#visitForms != null && #visitForms['newPathologyForm'] != null && {'Extrahepatic bile duct', 'Intrahepatic bile duct', 'Liver'}.contains(#visitForms['newPathologyForm']['tumourSiteAnatomicSite'])",
        "expr" : "#containsAny(#visit.clinicalDiagnoses, {'10 weeks gestation of pregnancy', '11 weeks gestation of pregnancy', '12 weeks gestation of pregnancy', '13 weeks gestation of pregnancy', '14 weeks gestation of pregnancy'})",
        "description" : "Check the combination of Primary Site in form and Clinical Diagnoses in core"
      } ]
    }

Example: Based on values selected in one field at the participant level, restrict the list of values that can be selected in another field at the specimen level

Data entry level: Participant, Specimen, Visit

Use case: In case a person is of ‘Male’ gender, the anatomic site should not include values in 'Female genital tract, NOS', 'Overlapping lesion of female genital organs', 'Other specified parts of female genital organs', 'Breast, NOS'.

{
      "records" : [ "specimen", "cpr", "visit" ],
      "rules" : [
        {
          "when" : "#cpr.participant.gender == 'Male' && #specimen.status == 'Collected' && #visit.eventLabel == 'Baseline'",
          "expr" : "!{'Female genital tract, NOS', 'Overlapping lesion of female genital organs', 'Other specified parts of female genital organs', 'Breast, NOS'}.contains(#specimen.anatomicSite)",
          "description" : "CORE: Check the combination of values from Gender and Anatomic Site"
        }
     ]
    }

Example: Add a min/max range on a numeric field

Data entry level: Participant

Use case: The participant’s age registered under the COVID protocol should be between 18-55

{
  "name" : "editChecks",
  "view" : null,
  "ctrl" : null,
  "data" : {
    "constraints" : [ {
      "records" : [ "cpr" ],
      "rules" : [ {
        "when" : "#cpr.participant.extensionDetail?.getAttrsMap()?.get('min_max_age') != null && !#cpr.participant.extensionDetail?.getAttrsMap()?.get('min_max_age').isEmpty()",
        "expr" : "T(java.lang.Integer).parseInt(#cpr.participant.extensionDetail?.getAttrsMap()?.get('min_max_age')) >= 18 && T(java.lang.Integer).parseInt(#cpr.participant.extensionDetail?.getAttrsMap()?.get('min_max_age')) <= 55",
        "description" : "Partiicpant's age should be between 18 and 55"
      } ]
    } ]
  }
}

Error Report

A report of all the data validation errors can be downloaded from the collection protocol overview page > ‘More’ > ‘Export Errors’

...

Clicking on ‘Export Errors’ redirects to a page as shown below:

...

Report Interval: You can define the date range for which the report is to be downloaded.

User: This is used to define the user for which the report is to be downloaded.

The report is downloaded when clicked on ‘Export’. In case no value is entered for these fields, an entire report is downloaded, including all errors shown for all users.

The downloaded zip has a CSV file with the below columns:

...

Column

...

Description

...

Identifier

...

System-generated unique number

...

User#First Name

...

First name of the user who was logged in while performing data entry

...

User#Last Name

...

Last name of the user who was logged in while performing data entry

...

User#Email Address

...

Email address of the user who was logged in while performing data entry

...

Collection Protocol#Short Title

...

Collection protocol on which the data validation showed an error

...

Record Type

...

The data entry level on which the error occurred. It can have the values:

Registration

Visit

Specimen

Form name of a custom form

...

PPID

...

The participant protocol ID for which data entry gave an error

...

Visit Name

...

Visit for which the error was thrown. This column will have values only when the error is thrown at the ‘Visit’ level of a custom form attached at visit level

...

Error Time

...

The timestamp of the error thrown

...

Error Message

...

The display message for the error thrown. This is the same message that is displayed on the screen when an edit check fails. It is defined in JSON.

Error: ‘Cannot index into a null value’

While adding multiple edit checks on non-mandatory fields/forms, it is necessary to have a null check on all the fields involved in the edit check before accessing it. This can be done by checking whether the form/field is not equal to ‘null’ in the when part of the edit check.

Refer to ‘How to solve ‘Cannot index into a null value’ error’ for more details.

Adding Multiple Edit Checks in Same Workflow

In case there are many edit checks to be performed, here is a tip that could help in avoiding errors:

...

1. When value for ‘Have you ever smoked’ in ‘Smoking History’ form is ‘Yes’, the error ‘Smokers not allowed!’ should be thrown.

Data Validations Rulesare supported for the below:

• Participant Fields, Custom Form, and Custom Fields

• Visit Fields, Custom Form, and Custom Fields

• Specimen Fields, Events, Custom Forms, and Custom Fields

• Shipment

• Order

More details