Script: Auto Bulk Import

1 Prerequisites
2 Overview
- 2.1 Folder Structure
3 Configuration
4 How to run?

Use the API-based script to automate bulk imports on Krishagni-hosted servers. This script continuously monitors a configured folder for bulk import CSV files and submits them to OpenSpecimen using REST API.

Prerequisites

Python3 need to be installed on the machine from where you are going to execute the script.

Overview

This script automates the following:

Monitors a directory input-file for new .csv files every minute.
Submits them one-by-one to OpenSpecimen bulk import API.
Waits for the import job to finish (COMPLETED / FAILED).
Downloads the output report to output-file directory.
Logs all activity in a dedicated log file

Folder Structure

/path/to/import-automation/
│
├── monitor_import.py        # Main script
├── config.json              # Configuration file
├── import_job.log           # Log file (auto-created)
├── input_files/             # Place new .csv files here
└── output_files/            # Output reports saved here

Configuration

Replace actual values in the config.json file:

{
  "username": "<api username>",
  "password": "<password>",
  "serverUrl": "<server_url>",
  "objectType": "<object_type>",    # You can find the entity mapping below.
  "importType": "<operation_type>", # Possible values: 'CREATE','UPDATE','UPSERT'
  "dateFormat": "",                 # If not specified takes the system default. 
  "timeFormat": "",                 # If not specified takes the system default. 
  "inputDir": "./input_files",      # Script creates directory automatically.
  "outputDir": "./output_files",    # Script creates directory automatically.
  "logFile": "import_job.log",      # Script generates the log file.
  "pollIntervalSeconds": 60
}

Replace '<object_type>' with the schemaName for the corresponding Bulk Import entity. (Find here the schema name to entity mapping)
Replace ‘<operation_type>' with ‘CREATE' or ‘UPDATE' or 'UPSERT' for bulk creating or bulk updating or create else update entities, respectively.

How to run?

Download the below zip file and extract.

Source Code
Execution Command	`python3 auto_bulk_import.py config.json`