FAQs

Deploying

...

Important

Which version of REDCap is required for the plugin to work?

Ans: This is by design. The REDCap’s export records API gives a snapshot of ALL records of the project. This is undesirable for the following reasons - 

1.  Fetching all project records for a medium to a large project can be quite taxing on REDCap and OpenSpecimen server resources.

2.  The plugin (frequently) synchronizes records between REDCap and OpenSpecimen. The plugin requires only those changes that have occurred in the REDCap project since the last synchronization run. There is no need to fetch all records every time the plugin run synchronization.

3. The plugin has been designed to fetch changes in the REDCap project in manageable chunks of 100 changes. It ensures that there is no undue memory pressure either on the REDCap server or OpenSpecimen server.

The redcap_log_events table will have no changes recorded for copied REDCap projects. How is this scenario handled?

Ans: Although REDCap provides an option to copy records, we believe users copy REDCap projects mostly to replicate the metadata (instruments, arms, events, etc.) and not the data records. Also, please note, by default, REDCap does not copy records when a project is copied. That being said, we will consider enhancing the plugin to provide an option to import the REDCap records CSV files.

Which version of REDCap is required for the plugin to work?

Ans: The plugin has been developed and tested using REDCap v7.0.7.

Why is a custom REDCap PHP plugin required?

Ans: Refer above question for the preamble. The custom REDCap PHP plugin is required for the following reasons:

1. There is no REDCap API to fetch changes in data that have occurred in a project since a given time point. The API always responds with ‘ALL changes’. The custom PHP plugin solves the problem by implementing an API that provides options like time point from when the changes need to be picked, how many changes, etc.

2. For longitudinal projects, the REDCap events API does not provide event IDs. It poses a problem in enforcing mapping between REDCap and OpenSpecimen events when there is a change in the REDCap event name (including unique_event_name). The custom PHP plugin implements an API to export project events along with their IDs. The OpenSpecimen plugin then uses these event IDs to create events mapping between REDCap and OpenSpecimen. The event IDs remain the same even when there is a change of event name. Significantly, data change records fetched from REDCap contain references to event ID.

Synchronization Run

Will the interface be real-time? Can the data entered into REDCap will be available in OpenSpecimen right away?

Ans: You can configure the auto-synchronization run on a nightly, hourly, or minutely basis. You will find details about it here.

How are visits synchronized between REDCap and OpenSpecimen?

>>If the user has multiple visit dates in redcap, and we mapped them into OS, but

>>there is no visit names (required field in OS) in redcap.  After we map the visit date into

>>OpenSpecimen, can OS automatically generate visit names and required visit fields in

>>OpenSpecimen to be able to let the user to start to enter the specimen information?

Ans: Yes, visits will be created automatically using the default visit naming scheme. You can then collect specimens under that visit.

How to find the participant and visit records added from REDCap project synchronization?

Ans: Below, SQL can be used to prepare the participant list and visit records that were added from REDCap project synchronization.

Below SQLs can be used to prepare the list of participant and visit records that were added from REDCap project synchronisation.

select
  cp.short_title as "CP", p.name as "Project", a.protocol_participant_id as "PPID"
from 
  os_redcap_records r
  inner join os_redcap_projects p 
    on p.identifier = r.project_id
  inner join catissue_coll_prot_reg_aud a 
    on a.identifier = r.cpr_id
  inner join catissue_collection_protocol cp 
    on cp.identifier = a.collection_protocol_id
where
  -- 0 for insert, 1 for update 
  a.revtype = 0
order by
  r.identifier;

select
  cp.short_title as "CP", p.name as "Project", a.identifier as "Visit ID", a.name as "Visit Name"
from 
  os_redcap_records r
  inner join os_redcap_record_visits vr
    on vr.record_id = r.identifier
  inner join os_redcap_projects p 
    on p.identifier = r.project_id
  inner join cat_specimen_coll_group_aud a 
    on a.identifier = vr.visit_id
  inner join catissue_coll_prot_reg reg 
    on reg.identifier = a.collection_protocol_reg_id
  inner join catissue_collection_protocol cp 
    on cp.identifier = reg.collection_protocol_id
where 
  -- 0 for insert, 1 for update
  a.revtype = 0 
order by
  r.identifier;

Is it possible to get a subset of participants from REDCap?

Ans: No. Example case: There are 1000 participants in REDCap, but only 100 are to be registered to OpenSpecimen. It is not possible to fetch a subset of participants from REDCap. But if you want to register only specific type of participants such as participants who are female and above 18 years old, then you can use 'Record Inclusion Rule' as shown in section here.

Is it possible to prefix values fetched from REDCap?

Ans: No. Example case: Append RC project ID to the PPID/MPI in OpenSpecimen.

...

The plugin supports most of the latest versions of REDCap. We usually keep up to date with the new releases of REDCap.

Will the interface be real-time? Is the data entered into REDCap will be available in OpenSpecimen right away?

Ans: You can configure the auto-synchronization run on a nightly, hourly, or minutely basis. You will find details about it here.

What is the difference in handling classical vs longitudinal REDCap projects?

Ans: For classical REDCap projects, there is no concept of visits. Therefore all REDCap records are tied at the participant level in OpenSpecimen.

For longitudinal projects, users are provided with an option to map the REDCap project events to corresponding events in OpenSpecimen. As a result, all the instruments and their data from REDCap are duplicated at the visit level in OpenSpecimen.

Deploying

Why does the plugin use the redcap_log_events table instead of REDCap’s export records API?

Ans: This is by design. The REDCap’s export records API gives a snapshot of ALL records of the project. This is undesirable for the following reasons - 

1.  Fetching all project records for a medium to a large project can be quite taxing on REDCap and OpenSpecimen server resources.

2.  The plugin (frequently) synchronizes records between REDCap and OpenSpecimen. The plugin requires only those changes that have occurred in the REDCap project since the last synchronization run. There is no need to fetch all records every time the plugin run synchronization.

3. The plugin has been designed to fetch changes in the REDCap project in manageable chunks of 100 changes. It ensures that there is no undue memory pressure either on the REDCap server or OpenSpecimen server.

The redcap_log_events table will have no changes recorded for copied REDCap projects. How is this scenario handled?

Ans: Although REDCap provides an option to copy records, we believe users copy REDCap projects mostly to replicate the metadata (instruments, arms, events, etc.) and not the data records. Also, please note, by default, REDCap does not copy records when a project is copied. That being said, we will consider enhancing the plugin to provide an option to import the REDCap records CSV files.

Why is a custom REDCap PHP plugin required?

Ans: Refer above question for the preamble. The custom REDCap PHP plugin is required for the following reasons:

1. There is no REDCap API to fetch changes in data that have occurred in a project since a given time point. The API always responds with ‘ALL changes’. The custom PHP plugin solves the problem by implementing an API that provides options like time point from when the changes need to be picked, how many changes, etc.

2. For longitudinal projects, the REDCap events API does not provide event IDs. It poses a problem in enforcing mapping between REDCap and OpenSpecimen events when there is a change in the REDCap event name (including unique_event_name). The custom PHP plugin implements an API to export project events along with their IDs. The OpenSpecimen plugin then uses these event IDs to create events mapping between REDCap and OpenSpecimen. The event IDs remain the same even when there is a change of event name. Significantly, data change records fetched from REDCap contain references to event ID.

Synchronization Run

If PHI is marked as "Disabled", can record matching occur on MRN or DOB or Name ?

Ans: If "Save PHI data" is set to "Disabled," record matching can still occur on fields like MRN, eMPI, External Subj ID, etc if the mapping is added to participant identifier fields. OpenSpecimen will prioritize the participant mapping over the "Save PHI data" setting, ignoring the "Disabled" option for the mapped fields and performing record matching based on the identifiers.

How are visits synchronized between REDCap and OpenSpecimen?

>>If the user has multiple visit dates in redcap, and we mapped them into OS, but

>>there is no visit names (required field in OS) in redcap.  After we map the visit date into

>>OpenSpecimen, can OS automatically generate visit names and required visit fields in

>>OpenSpecimen to be able to let the user start to enter the specimen information?

Ans: Yes, visits will be created automatically using the default visit naming scheme. You can then collect specimens under that visit.

How to find the participant and visit records added from REDCap project synchronization?

Ans: Below, SQL can be used to prepare the participant list and visit records that were added from REDCap project synchronization.

Below SQLs can be used to prepare the list of participants and visit records that were added from REDCap project synchronization.

select
  cp.short_title as "CP", p.name as "Project", a.protocol_participant_id as "PPID"
from 
  os_redcap_records r
  inner join os_redcap_projects p 
    on p.identifier = r.project_id
  inner join catissue_coll_prot_reg_aud a 
    on a.identifier = r.cpr_id
  inner join catissue_collection_protocol cp 
    on cp.identifier = a.collection_protocol_id
where
  -- 0 for insert, 1 for update 
  a.revtype = 0
order by
  r.identifier;

select
  cp.short_title as "CP", p.name as "Project", a.identifier as "Visit ID", a.name as "Visit Name"
from 
  os_redcap_records r
  inner join os_redcap_record_visits vr
    on vr.record_id = r.identifier
  inner join os_redcap_projects p 
    on p.identifier = r.project_id
  inner join cat_specimen_coll_group_aud a 
    on a.identifier = vr.visit_id
  inner join catissue_coll_prot_reg reg 
    on reg.identifier = a.collection_protocol_reg_id
  inner join catissue_collection_protocol cp 
    on cp.identifier = reg.collection_protocol_id
where 
  -- 0 for insert, 1 for update
  a.revtype = 0 
order by
  r.identifier;

Is it possible to get a subset of participants from REDCap?

Ans: No. Example case: There are 1000 participants in REDCap, but only 100 are to be registered to OpenSpecimen. It is not possible to fetch a subset of participants from REDCap. But if you want to register the only specific types of participants such as participants who are female and above 18 years old, then you can use the 'Record Inclusion Rule' as shown in the section here.

Is it possible to prefix values fetched from REDCap?

Ans: No. Example case: Append RC project ID to the PPID/MPI in OpenSpecimen.

Forms and Mapping

What is the difference between “Record_id=participant.extensionDetail.attrsMap.redcap_record_id” and “record_id=ppid” mapping?

Ans: The first one is to map the "RC record ID" to a custom field in the OS. Here you will use the option to auto-generate PPID in OS. The next is to directly map "RC record ID" to a PPID field in OS and hence PPID are not auto-generated.

That being said, the purpose of using one option over another depends upon your specific requirements.

What if the fields are not available in OpenSpecimen's default setting? Does it mean that we first need to create the custom form to store these fields?

Ans: No, the missing fields are automatically created as custom forms/fields by the plugin.

Can the plugin map the REDCap fields into existing custom form

...

Ans: No, the plugin cannot map the REDCap fields into existing custom forms.

What is the difference in handling classical vs longitudinal REDCap projects?

Ans: For classical REDCap projects, there is no concept of visits. Therefore all REDCap records are tied at the participant level in OpenSpecimen.

...

fields? 

Ans: No, the plugin cannot map the REDCap fields into existing custom forms. The plugin will create new forms for each REDCap instrument being synced.

Why are events mapping required?

Ans: Events mapping is exclusively for longitudinal projects. It specifies the anticipated or existing visits to which the instrument records from the REDCap are linked.

Is it mandatory to specify events mapping? What happens when events mapping is not specified?

Ans: No, it is not required to specify events mapping. Unplanned visits are created in the absence of such mapping, and the records are linked to those visits.

Is it mandatory to map all REDCap events?

Ans: No, it is not required to map all REDCap events to the CP visits. However, unplanned visits are created for each unmapped REDCap event.

Is it mandatory to specify the visit fields mapping?

Ans: No, it is not mandatory. However, it is good practice to at least map the visit date.

How are mandatory visit fields populated when visit fields are not mapped?

Ans: The mandatory visit fields are populated as follows:

1. When the visit date field is not mapped, its value is set to the date/time of the corresponding REDCap event data entry.

2. When the visit site field is not mapped, its value is picked either from the mapped event's default site or the project site.

What sites are available for selection in the project site dropdown?

...

Ans: The project site dropdown is restricted to the sites linked with the collection protocol.

How map the picklist values from REDCap to corresponding PVs of OpenSpecimen?

Ans: It is a very common problem faced between the integration of any two disparate systems. Let's take an example and understand how this problem is solved in OpenSpecimen. Let's say you want to map the gender value "M" received from the REDCap project to the corresponding OpenSpecimen PV "Male". You can follow the given steps:

1. Go to Extras from the homepage and select Dropdown Manager.

2. Click on the "Gender" attribute and click on 'Male' PV.

3. Click on 'Edit', and under properties, enter the name of PV Mapping Key and its corresponding value for your REDCap study.

Mapping Specific CP

...

Mapping for all CPs under a single site

...

Mapping all REDCap projects irrespective of sites or CP

...

Why is the events mapping table not displayed in the REDCap project setup wizard?

Ans: The events mapping table is not displayed when the project is either a "classical project" or has only one event. In the latter case, REDcap considers the project with a single event as "classical projects".

...

Ans: The project site dropdown is restricted to the sites linked with the collection protocol.

REDCap Data

For the value "Save PHI data" please explain the ramifications of clicking "Enabled" or "Disabled".

Ans: When you use the ''Save PHI data" option as Disabled, OS will not pull any RC field marked as 'Identifier' in the RC instrument/form set-up. Usually, these fields are 'First Name', 'Last Name', 'MRN', 'DOB', etc.

PHI such as Name, MRN, DOB must always be saved to identify a record. So what is the use of having a Disabled option for "Save PHI data" ?

Ans: This is a user based preference. We have a few clients who don't want PHI fields pulled in OS. They use these setting as Disabled.

If we remove forms or fields in REDCap after the go-live date, would OS remove them for us automatically during the overnight job?

Ans: Yes, it will. However, in the regular runs, it will only pull the records created from the previous run. If you want ALL the data for the new form, you have to re-run the full synchronization.

If we create a new form in REDCap after the go-live date, would OS sync it for us automatically during the overnight job?

Ans: It depends on the type of configuration you set during the REDCap configuration:

1. The new instrument would be configured automatically if the forms were not defined in the 'Form inclusion/exclusion list'.

2. If any forms are included in the list, you will have to add the new form in the include list.

Why does OpenSpecimen does not delete participants that are deleted in REDCap?

Ans: Since specimen information such as collection, storage, distribution, etc., might be still crucial to the researcher, we do not delete the participant forcefully. We expect that the end-user to login to the OpenSpecimen server and manually delete the participant.

Why does OpenSpecimen pull deleted participants from REDCap?

Ans: OpenSpecimen process the REDCap audit logs in chronological order. Since it first encounters the insert participant audit log, it creates the participant in OpenSpecimen. Later, when it encounters the delete participant audit log, it ignores it since (as per the previous question) OpenSpecimen does not delete participants forcefully. These participants have to be deleted manually.

How to handle data deletes in OpenSpecimen?

Ans: OpenSpecimen maintains a few mapping tables to maintain the mapping between OS-RC objects. To do a total cleanup and resync, you must either truncate or delete rows from these tables.

From v6.3 onwards, the mapping tables can be cleaned from the user interface itself. To do a full resync of the data, delete the data from OpenSpecimen using bulk import. Then navigate to the REDCap settings and click on ‘Delete Data’ under ‘More’. It will clear all the mapping tables, and now the data can be resynced into OpenSpecimen.

Before version v6.3, the cleanup needs to be done at the database level by truncating or deleting rows from these tables.

1. os_redcap_projects - Table that stores all the RC project details.

2. os_redcap_records - Table that stores all the RC participant mapping.

3. os_redcap_instrument_instances - Table that stores all the RC visit form mapping

4. os_redcap_record_visits - Table that stores all the RC visit mapping (only for longitudinal studies)

If you delete a participant or visit in OpenSpecimen and resync ALL data from RC, it will not pull the data for deleted OS objects from RC. However, in the odd case, where you want the resync to fetch ALL data, you will need to delete the entries from the below table:

1. os_redcap_records: delete participant mapping

2. os_redcap_record_visits: delete visit mapping

3. os_redcap_instrument_instances: join with #1 (and/or #2) above and delete the instrument entries

4. If you want to resync all: delete all the entries for a project by joining with os_redcap_projects.

Errors

“java.lang.ClassCastException: java.lang.Integer cannot be cast to java.lang.String"

Solution: OpenSpecimen versions older than v6.1 are not compatible with REDCap 8.10. Due to which when setting up the project, this error is displayed on the user interface. Hence, you need to update to the higher version of OpenSpecimen.

Error Connecting to REDCap API

MysqlDataTruncation:Data truncation: Incorrect datetime

You do not have permission to use the API or Invalid token.

...

Expand
title Click her to view details
This error is raised when the JAVA_HOME is pointing to an unstable or incorrect installation. Unless the JAVA_HOME variable is fixed, the user is not navigated to the next page. Image Removed an unstable or incorrect installation. Unless the JAVA_HOME variable is fixed, the user is not navigated to the next page.

Row size too large