Introduction

OpenSpecimen v3.2 supports a new feature to automatically monitor a folder to consume bulk import files on a continuous basis. This feature is very useful to integrate OpenSpecimen with other systems like REDCap, OpenClinica etc.

With this feature, the external sources do not need to learn OS REST APIs to integrate with OS. They just need to generate OS BO compatible CSV files and store it in a pre-defined folder. OS continuously monitors this folder and consumes the BO file when it finds a new file. The CSV files are processed in ascending order of timestamp specified in the file name. This helps maintain the order of the files imported.

Steps

1. Create a "scheduled-bulk-import" directory in-app data (os-data) directory if not present. The location of the os-data directory can be retrieved from $Tomcat_Home/conf/openspecimen.properties file. Check for app.data_dir property.
2. Copy the bulk import CSV files in 'scheduled-bulk-import' directory. You can do it via scripts in case of integrations.

3. The file name should be in a format as mentioned in the table below

   Data	Format	Description
   Standard entities (e.g. participant, specimen, etc)	<object_type>_<operation>_<timestamp>_[<csv_type>].csv	object_type: Entity name specified in bulk import schema file (see list below) operation: Operation to perform, valid values are "create" or "update". timestamp: In yyyyMMddHHmmss:SSS". csv_type (optional): Specify "m" in case of "Order" and "Shipment" Examples cp_create_20160511162033124.csv distributionOrder_create_20160511162033124_m.csv
   Custom fields	<entity>_<operation>_<timestamp>_cpId_<cpId>.csv	entity: cpr, visit, specimen. operation - Operation to be performed, "create" or "update" timestamp - In "yyyyMMddHHmmssSSS" format cpId - Specifying Identifier of collection protocol.  <cpId> - Identifier of collection protocol. You can get this identifier via DB or from the browser URL on the collection protocol overview page. Example: Specimen custom field level update file specimen_update_20200925115950000_cpId_4292.csv
   Custom forms	extensions_<attached_level><form_name><operation>_<timestamp>.csv	extensions: Static word to identify a custom form. attached_level: Level at which form is attached.  Participant SpecimenCollectionGroup (i.e Visit) Specimen SpecimenEvent form_name - System generated 'Form Name' of the custom form. operation - Operation to be performed, "create" or "update" timestamp - In "yyyyMMddHHmmssSSS" format Example: extensions_Participant_familyHistoryAnnotation_create_20160511162246252.csv

   
   

4. Once the files are processed (uploaded into OpenSpecimen), they are moved to a folder named - 'processed-bulk-import'
5. If there are any issues with the file name format, then the file is not uploaded and moved to a folder name 'unprocessed-bulk-import'
6. The file name needs to be updated and the file must be moved to the 'scheduled-bulk-import' from where it can be taken up for upload.
7. To view the bulk import jobs, use the URL <https://IP address/openspecimen/#/bulk-import-jobs>.
8. Following is the list of object types for OpenSpecimen entities

...