Introduction
caTissue Plus contains the following improvements to the Bulk Operation feature:
- Upload Dynamic Extensions (DE) data via BO
- Remove dependency on system identifiers (like participant id, specimen id, etc)
- BO rerun
- Better handling missing columns
- Default value
DE BO Support
In NCI 1.2, bulk upload was only possible for static model data. In Plus, user will be able to insert data using BO for DE forms. Sample templates for default DE annotations at every level - participant, SCG & specimen is listed in the table below
Remove dependency on system identifiers
In NCI 1.2, when migrating data for Participant, SCG, and Specimens, one needed to perform certain manual operations in between uploading the files. I.e., upload participants, copy the participant system identifier into SCG CSV, upload SCG CSV, copy SCG system identifiers into Specimen CSV. This can long and tedious, as well as error prone due to manual intervention.
Plus fixes this issue by removing the dependency on system identifiers. So the work flow would be:
- Participant CSV should contain PPID
- SCG CSV should contain SCG Label, PPID & CPID
- Specimen CSV should contain SCG Label
- SCG upload with MRN + Site name & CP title
With this, you can load all the CSVs one after the other without any manual intervention
Someone who has done legacy data migration in caTissue will appreciate this enhancement better since they have gone through the pain!
Note: there are some cases where you still need system ids and being resolved as we move along in Plus. Please feel free to report places where you find them.
BO rerun
When uploading data via BO, chances of failure is common due to the data errors in the uploaded file. The errors might be due to incorrect data specified for enumerated fields or format. After user runs BO upload, system generates a report and displays in the UI dashboard. In previous version, when the such error occurs, user had to correct the data and recreate the data file with the failed records only and upload again. With new version of BO, user can correct the data within the BO report generated by the system and upload it back. User does not have to remove the success records or remove the additional status column. System will ignore the records and process only the previously failed records.
Setting default value in template
In Plus, user can specify default values in the BO XML template at attribute level. User can skip such columns from CSV when loading data. Below are some use cases and examples how this can be used:
- For the fields that are not used in particular biobank but are mandatory in caTissue, specify default value like Not Specified - Examples:
- SpecimenCollectionGroup.ClinicalStatus - Not Specified
- Specimen.TissueSite - Not Specified
- For the fields that are mandatory and value will be consistent for all protocols, set the value in XML - Examples:
- SpecimenCollectionGroup.CollectionSite - ABC Hospital (If specimens collected from same collection site)
- Specimen.Type - Whole Blood (If only whole blood collected)
In above cases, when loading Specimen Collection Group or Specimens, user can skip the attributes from CSV where default value is specified. If value is specified in CSV, the default value is overwritten.
Better handling for missing columns
If the BO XML template contains a certain set of columns, then it was mandatory to have those columns in the CSV file too. However, in case of optional columns or in case of "Edit", it is not necessary to have all the columns in the CSV file. In Plus, BO ignores the missing columns unless any of the missing column was mandatory. Even the mandatory columns can be skipped if default value is specified in the template.
Delete existing Bulk Operation templates
Use below SQLs to delete existing BO templates:
NOTE: Please take backup of the database before running these commands.
Delete all templates:
DELETE FROM catissue_bulk_operation;
Delete specific template:
DELETE FROM catissue_bulk_operation WHERE operation LIKE 'OperationNameHere';
Load standard Bulk Operation templates
You can load standard BO templates by running below ant target from CATISSUE_HOME directory.
ant load_bulk_operator_templates
Ant target to load Bulk Operation templates
Run below ant target from CATISSUE_HOME directory by changing respective parameters depending upon your template.
ant add_bulk_operation_template -DoperationName="Operation Name Here" -DdropdownName="Drop-down Name Here" -DcsvFile="CSV File Path" -DxmlFile="XML File Path"
Note: Path must be separated by Unix style path separator "/".
caTissue Suite 3.0 Bulk Operation Templates Library:
Template Download | Function | Comments | Supported Version |
---|---|---|---|
Create Collection Protocol | Create Collection Protocol | Switch off the specimen label generator to specify user defined Specimen, Aliquot and Derivative label format. | Plus v3.1, Plus v3.0, v1.2 |
Register multiple participants based on protocol title and PPI |
| Plus v3.1, Plus v3.0, v1.2 | |
Add SCG to the existing collection protocol | Use combination of MRN and Site Name to create SCG. Note: If SCG name generator is on, system will ignore the name specified in template. Switch off the name generator to specify user defined name. | Plus v3.1, Plus v3.0, v1.2 | |
Create Parent Specimen | To add specimens as virtual, leave container name, position one and two as blank. | Plus v3.0 | |
Create Derivatives | Create Derivatives | To add specimens as virtual, leave container name, position one and two as blank. | Plus v3.0 |
Create Site | Create Site | Plus v3.0, v1.2 | |
Create Biohazard | Create Biohazard | Plus v3.0, v1.2 | |
Create Storage Type | Create Storage Type | Plus v3.0, v1.2 | |
Add Parent Container | Add Parent Container | Plus v3.0, v1.2 | |
Add Child Container | Add Child Container | Plus v3.0, v1.2 | |
Create CheckIn CheckOut Event Parameters | Create CheckIn CheckOut Event Parameters | Plus v3.0, v1.2 | |
Create Embedded Event Parameters | Create Embedded Event Parameters | Plus v3.0, v1.2 | |
Create Frozen Event Parameters | Create Frozen Event Parameters | Plus v3.0, v1.2 | |
Create Spun Event Parameters | Create Spun Event Parameters | Plus v3.0, v1.2 | |
Create Thaw Event Parameters | Create Thaw Event Parameters | Plus v3.0, v1.2 | |
Create Molecular Specimen Review Parameters | Create Molecular Specimen Review Parameters | Plus v3.0, v1.2 | |
Create Tissue Specimen Review Parameters | Create Tissue Specimen Review Parameters | Plus v3.0 | |
Create Transfer Event | Create Transfer Event | Plus v3.0, v1.2 | |
Create Disposal Event Parameters | Create Disposal Event | Plus v3.0, v1.2 | |
Participant DE BO Sample | Add DE form data at participant level | This template adds the data for Lab Annotation form(default annotation at participant level) | Plus v3.0 |
Add DE form data at Specimen Collection Group level | This template adds the data for Base Haematology Pathology Annotation form(default annotation at SCG level) | Plus v3.0 | |
Add form data at Specimen Collection Group level | This template adds the data for Base Pathology Annotation form(default annotation at SCG level) | Plus v3.0 | |
Add form data at specimen level | This template adds the data for Melanoma Specimen Pathology Annotation form(default annotation at specimen level) | Plus v3.0 | |
Add Container | Add Parent and Child container | This template will work for parent and child container. | Plus v3.0 |
Create Specimen | Create Parent and Child specimen | This template will work for parent and child specimen. | Plus v3.0 |
Edit Specimen | Edit specimen | This template will work for edit specimen based on id | Plus v3.0 |
Create Participant with anticipated SCG | This templates create participant with anticipated SCG | Plus v3.0,v3.1 | |
Create Aliquot | Create the aliquots specimen | Plus v3.1 | |
Edit Specimen | Edit child and parent specimen | This template edit the child and parent specimen on the basis of specimen label | Plus v3.0.6, v3.1 |
Create Procedure Event Parameter | Create Procedure Event Parameter | This template create procedure event parameter on the basis of specimen label | Plus v3.0.6, v3.1 |