How to export data from OpenSpecimen to another system?

Often, institutes need to pull data from OpenSpecimen into an external database like the institute's data warehouse, LIMS, or other clinical systems. This can be achieved in two ways:
  1. Using APIs
  2. Use CSV export using query module

Use APIs

OpenSpecimen is a 100% REST API enabled application, which means each and every object can be created, read, updated or deleted (CRUD) using an API. The APIs are documented here: /wiki/spaces/CAT/pages/1116035.

Is there an ability to provide authentication by tokens or certificates?

Yes, the APIs have a mechanism to authenticate the API invokers/callers based on the token included in every request. The auth token is included in the X-OS-API-TOKEN header. More details about authentication can be found here.

The token is invalidated if it is not used for more than the configured session timeout minutes.

Unlike other social media applications (notorious for compromising users' data), OpenSpecimen does not support persistent CP or user-based tokens. 

Can we do read-only API credentialing per Collection Protocol?

Yes. APIs are accessed via an OpenSpecimen user account. The CPs accessible to that user via the UI are the CPs that are accessible via the APIs. You can configure it by going to the Users→Roles tab.

Using the Query module

OpenSpecimen's Query module supports the automated export of data into a CSV file. This CSV file can then be picked by an external program and processed.

Steps:

  1. Via GUI, create a query with the required filters (e.g. filter by CP) and add the columns needed by the external system
  2. Save the query
  3. Schedule the query to run at required intervals (E.g. nightly, hourly, etc) (For details: Scheduling Queries)
  4. In every query run, the data satisfying the filters are exported to a CSV file. The exported CSV file is stored in the directory $OS_DATA_DIR/query-exported-data using the following name format - scheduled_query_<query_id>_<job_run_id>.csv
  5. Write an external program to monitor the query export directory and process the required CSV files (using the filename scheduled_query_<query_id>_*.csv).

The advantage of this approach is that external developers do not have to spend time learning OpenSpecimen APIs.