Where is the option to bulk upload data through CSV files in OpenSpecimen?

Under every data, there is an import option in the top-left corner of the screen.

Updating CP-specific custom fields using bulk import

When updating custom field values, perform the bulk import within the Collection Protocol, not from the CP list page. This is important because custom field values specific to a CP cannot be updated from the CP list view.

How to Download the Data Dictionary in OpenSpecimen?

Go to the collection protocols list page, click on the 'More' button, and select the option 'Import Biospecimen Data (CSV)' OR
Go to the participants' list page of the CP, click on the 'More' button, and select 'Import Biospecimen Data (CSV)'.
Select 'Specimens' from the 'Select Record Type' dropdown and click on 'Download Template File' to download the template file.

You can repeat the above steps for downloading templates for other entities, like Participants, Visits, Collection Protocol, etc.

Why are templates different when inside a CP vs. the CP list view?

The difference in template content is due to custom fields. When you download a template, OpenSpecimen decides whether to include custom fields based on whether it knows which Collection Protocol (CP) you are working with.

1. Downloading from the CP List View

Result: You receive a Generic Template.
Why: At the list level, the system doesn't know which specific CP or Collection Protocol Group (CPG) you intend to use. Since different CPs have different custom fields, the system defaults to a generic template.

2. Downloading from within a Specific CP

Result: You receive a CP specific Template.
Why: OpenSpecimen has the necessary context since you are already "inside" the C.

Bottomline: Always download your templates from the CP Detail View to ensure all required columns are present.

Which file types can be imported?

The data import file has to be in CSV (Comma Separated Values) format.

From v6.1, OpenSpecimen supports semicolon-separated input files for bulk import. This is useful for files being imported from external devices(E.g. Liquid handling systems) which might be outputting data in different formats.

To check what field separator is used in your file, open the file in a text editor.

Can I import any CSV files?

The data in the CSV file should be in specific template formats. The templates can be downloaded from the application under the import option for every data.

Can I import participants, visits, specimens, etc. in one go?

Yes, using the 'Master Specimens' template you can import participants, visits, and specimens in one go. For more details on import, refer to 'Master Template'

Does the order of columns in the CSV matter?

No

Email Notifications

Email notifications are sent after the bulk import job is completed, failed, or aborted to the user who performed the bulk import. The email is also CCed to 'Administrator Email Address' set under Settings → Email.

Can the Super Admin view the import jobs of all users?

Yes, the "Super Administrator" can view the import jobs of all the users. Please note that the user can bulk upload the data from two places.

Outside individual CP: i.e. from the Collection protocol list page (Collection Protocols → More → Import)
For specific collection protocol (Collection Protocols → Participant List → More → Import)

The jobs will be visible to the Super Admin based on how the user uploaded the file. In other words, jobs uploaded at the global level won't be visible under specific CP and vice versa.

Users other than Super Admin can download or see only their imports. The system doesn’t know what is present in the file, so it restrict the access to only the users who have created the job or super admin.

Identifiers and Updating Existing Records through Bulk Import

In OpenSpecimen, each object has two identifiers:

Examples:

A participant in a collection protocol has an ‘Identifier’ and a 'PPID’.
A Box or Container has an ‘Identifier’ and a 'Unique Name'.

Check the individual CSV wiki pages to see which identifiers are associated with each object.
The 'Identifier' is a sequential number generated by the OpenSpecimen Database.

When updating a record, include one of the identifiers in your CSV file.

Example:

To update ‘Visit’ records, include either ‘Visit Name’ or its ‘Identifier’. You do not need to include the 'PPID’ for this import.
Similarly, if you want to update the specimen, ‘Labels' is enough for you to update the records, no need to include ‘Visit Name’, ‘PPIDs’, etc.

How do I check the status of the import?

Once the CSV file is imported, a report is generated under 'View Past Imports' in the import feature. The dashboard will show the status of every data import.

From the list view page, click on the 'Import' button → View Past Imports

If you are using bulk import with a CP, you can click on 'More' → View Past Imports

How to avoid errors?

Watch the video below to learn about the common mistakes made during CSV import and how to avoid errors.

How will I know if there are errors?

Reports are generated for every data upload. These are available under 'View Past Imports'.

The report contains details about if a record upload was successful or failed. In case of failure, you can download the bulk import job file showing the import status and error showing the reason of failure.

What do I do in case of errors?

The actual data in the application or in the data file needs to be corrected for the failed records, and they can be re-uploaded.

Can I abort a bulk import job?

If you want to abort the bulk import job, click on the 'Abort' icon on the jobs page for the specific import job:

This will help if you have uploaded large files like 10k records and realize that there is a mistake in the records and would like to abort the bulk import job instead of waiting for the whole file to process.

If the file is imported without validation, the data records are processed until the job is aborted and those records are not rolled back.

Frequently Asked Questions