Introduction
OpenSpecimen supports bulk/batch importing of many entities such as Collection Protocols, Specimen, Visits, Users, etc. Bulk Importing is used in high throughput biobanks or when the user needs to add/update multiple entities to/from OpenSpecimen. (Follow this page for an overview on the BulkImport module)
This page documents the details of bulk import API. Follow this wiki-page for a working example of CP import via API.
Goals for Bulk Import API
- Provide an easy and efficient way to perform a single operation (Insert/Update) on multiple entities.
Steps for Bulk importing
- Download the CSV template file
- Upload the CSV file
- Create and run the import job
- Check the job status
- Download the job report
Download the CSV template file
Follow the below steps to download the template file:
Login to OpenSpecimen → Navigate To → Collection Protocols → More → Import Biospecimen Data (CSV) → Choose the Record Type → Click on Download Template
Upload the CSV file
| Attribute | Value |
|---|---|
| URL | http://<host>:<port>/openspecimen/rest/ng/import-jobs/input-file |
| Method | POST |
| Request Type | form-data |
| Response | {
"fileId": "fa8e7401-0495-443b-bedc-99a43e0738f5"
}
|
Note down the 'fileId' as this will be used in step #3
Create and run the import job
| Attribute | Value |
|---|---|
| URL | http://<host>:<port>/openspecimen/rest/ng/import-jobs |
| Method | POST |
| Request Type | application/json |
| Request | {
"objectType" : <objectType>,
"importType" : <operationType>,
"inputFileId" : <fileId>,
"dateFormat": <dateFormat>,
"timeFormat": <timeFormat>
}
|
- Replace '<objectType>' with the schemaName for the corresponding Bulk Import entity. (Find here the schema name to entity mapping)
- Replace '<operationType>' with 'CREATE' or 'UPDATE' for bulk creating or bulk updating entities respectively.
- Replace '<fileId>' with the fileId retrieved from the step #2
- Date and time formats are optional. When not specified the system configured formats are used. Specify dateFormat and timeFormat when your input CSV uses date and times that are incompatible with the system configuration.
Check the job status
| Attribute | Value |
|---|---|
| URL | http://<host>:<port>/openspecimen/rest/ng/import-jobs/{job-id} |
| Method | GET |
| Request Type | application/json |
Download the job report/output
| Attribute | Value |
|---|---|
| URL | http[s]://<host>:<port>/openspecimen/rest/ng/import-jobs/{job-id}/output |
| Method | GET |
Example of Bulk Importing CP(s) using API
Follow this wiki-page to see an example of bulk importing CP(s) using bulk import API.
Error Cases
| Code | Applies to | Status Message |
|---|---|---|
| 200 | All requests | Bulk Import request was successfully processed. |
| 401 | All requests | Authorisation failed, user doesn’t have the authority. |
| 500 | All requests | Internal server error, encountered server error while performing operations. |
Sample Code
The following is the sample python code to do bulk import for Specimen Events:
How to run?
python3 bulk_import.py <CSV_FILE_NAME>