...

Ans: This is by design. The REDCap’s export records API gives a snapshot of ALL records of the project. This is undesirable for the following reasons - 

1.  Fetching all project records for a medium to a large project can be quite taxing on REDCap and OpenSpecimen server resources.

2.  The plugin (frequently) synchronizes records between REDCap and OpenSpecimen. The plugin requires only those changes that have occurred in the REDCap project since the last synchronization run. There is no need to fetch all records every time the plugin run synchronization.

3. The plugin has been designed to fetch changes in the REDCap project in manageable chunks of 100 changes. It ensures that there is no undue memory pressure either on the REDCap server or OpenSpecimen server.

The redcap_log_events table will have no changes recorded for copied REDCap projects. How is this scenario handled?

...

Ans: Refer above question for the preamble. The custom REDCap PHP plugin is required for the following reasons:

1. There is no REDCap API to fetch changes in data that have occurred in a project since a given time point. The API always responds with ‘ALL changes’. The custom PHP plugin solves the problem by implementing an API that provides options like time point from when the changes need to be picked, how many changes, etc.

2. For longitudinal projects, the REDCap events API does not provide event IDs. It poses a problem in enforcing mapping between REDCap and OpenSpecimen events when there is a change in the REDCap event name (including unique_event_name). The custom PHP plugin implements an API to export project events along with their IDs. The OpenSpecimen plugin then uses these event IDs to create events mapping between REDCap and OpenSpecimen. The event IDs remain the same even when there is a change of event name. Significantly, data change records fetched from REDCap contain references to event ID.

Synchronization Run

How are visits synchronized between REDCap and OpenSpecimen?

>>If the user has multiple visit dates in redcap, and we mapped them into OS, but

>>there is no visit names (required field in OS) in redcap.  After we map the visit date into

>>OpenSpecimen, can OS automatically generate visit names and required visit fields in

>>OpenSpecimen to be able to let the user start to enter the specimen information?

...

If PHI is marked as "Disabled", can record matching occur on MRN or DOB or Name ?

Ans: If "Save PHI data" is set to "Disabled," record matching can still occur on fields like MRN, eMPI, External Subj ID, etc if the mapping is added to participant identifier fields. OpenSpecimen will prioritize the participant mapping over the "Save PHI data" setting, ignoring the "Disabled" option for the mapped fields and performing record matching based on the identifiers.

How are visits synchronized between REDCap and OpenSpecimen?

>>If the user has multiple visit dates in redcap, and we mapped them into OS, but

>>there is no visit names (required field in OS) in redcap.  After we map the visit date into

>>OpenSpecimen, can OS automatically generate visit names and required visit fields in

>>OpenSpecimen to be able to let the user start to enter the specimen information?

Ans: Yes, visits will be created automatically using the default visit naming scheme. You can then collect specimens under that visit.

...

select
  cp.short_title as "CP", p.name as "Project", a.protocol_participant_id as "PPID"
from 
  os_redcap_records r
  inner join os_redcap_projects p 
    on p.identifier = r.project_id
  inner join catissue_coll_prot_reg_aud a 
    on a.identifier = r.cpr_id
  inner join catissue_collection_protocol cp 
    on cp.identifier = a.collection_protocol_id
where
  -- 0 for insert, 1 for update 
  a.revtype = 0
order by
  r.identifier;

select
  cp.short_title as "CP", p.name as "Project", a.identifier as "Visit ID", a.name as "Visit Name"
from 
  os_redcap_records r
  inner join os_redcap_record_visits vr
    on vr.record_id = r.identifier
  inner join os_redcap_projects p 
    on p.identifier = r.project_id
  inner join cat_specimen_coll_group_aud a 
    on a.identifier = vr.visit_id
  inner join catissue_coll_prot_reg reg 
    on reg.identifier = a.collection_protocol_reg_id
  inner join catissue_collection_protocol cp 
    on cp.identifier = reg.collection_protocol_id
where 
  -- 0 for insert, 1 for update
  a.revtype = 0 

order by
  r.identifier;

Is it possible to get a subset of participants from REDCap?

Ans: No. Example case: There are 1000 participants in REDCap, but only 100 are to be registered to OpenSpecimen. It is not possible to fetch a subset of participants from REDCap. But if you want to register the only specific types of participants such as participants who are female and above 18 years old, then you can use the 'Record Inclusion Rule' as shown in the section here.

Is it possible to prefix values fetched from REDCap?

Ans: No. Example case: Append RC project ID to the PPID/MPI in OpenSpecimen.

Forms and Mapping

What if the fields are not available in OpenSpecimen's default setting? Does it mean that we first need to create the custom form to store these fields?

Ans: No, the missing fields are automatically created as custom forms/fields by the plugin.

Can the plugin map the REDCap fields into existing custom form fields? 

Ans: No, the plugin cannot map the REDCap fields into existing custom forms. The plugin will create new forms for each REDCap instrument being synced.

Why are events mapping required?

Ans: Events mapping is exclusively for longitudinal projects. It specifies the anticipated or existing visits to which the instrument records from the REDCap are linked.

Is it mandatory to specify events mapping? What happens when events mapping is not specified?

Ans: No, it is not required to specify events mapping. Unplanned visits are created in the absence of such mapping, and the records are linked to those visits.

Is it mandatory to map all REDCap events?

Ans: No, it is not required to map all REDCap events to the CP visits. However, unplanned visits are created for each unmapped REDCap event.

Is it mandatory to specify the visit fields mapping?

Ans: No, it is not mandatory. However, it is good practice to at least map the visit date.

How are mandatory visit fields populated when visit fields are not mapped?

Ans: The mandatory visit fields are populated as follows:

1. When the visit date field is not mapped, its value is set to the date/time of the corresponding REDCap event data entry.

2. When the visit site field is not mapped, its value is picked either from the mapped event's default site or the project site.

What sites are available for selection in the project site dropdown?

Ans: The project site dropdown is restricted to the sites linked with the collection protocol.

...

order by
  r.identifier;

Is it possible to get a subset of participants from REDCap?

Ans: No. Example case: There are 1000 participants in REDCap, but only 100 are to be registered to OpenSpecimen. It is not possible to fetch a subset of participants from REDCap. But if you want to register the only specific types of participants such as participants who are female and above 18 years old, then you can use the 'Record Inclusion Rule' as shown in the section here.

Is it possible to prefix values fetched from REDCap?

Ans: No. Example case: Append RC project ID to the PPID/MPI in OpenSpecimen.

Forms and Mapping

What is the difference between “Record_id=participant.extensionDetail.attrsMap.redcap_record_id” and “record_id=ppid” mapping?

Ans: The first one is to map the "RC record ID" to a custom field in the OS. Here you will use the option to auto-generate PPID in OS. The next is to directly map "RC record ID" to a PPID field in OS and hence PPID are not auto-generated.

That being said, the purpose of using one option over another depends upon your specific requirements.

What if the fields are not available in OpenSpecimen's default setting? Does it mean that we first need to create the custom form to store these fields?

Ans: No, the missing fields are automatically created as custom forms/fields by the plugin.

Can the plugin map the REDCap fields into existing custom form fields? 

Ans: No, the plugin cannot map the REDCap fields into existing custom forms. The plugin will create new forms for each REDCap instrument being synced.

Why are events mapping required?

Ans: Events mapping is exclusively for longitudinal projects. It specifies the anticipated or existing visits to which the instrument records from the REDCap are linked.

Is it mandatory to specify events mapping? What happens when events mapping is not specified?

Ans: No, it is not required to specify events mapping. Unplanned visits are created in the absence of such mapping, and the records are linked to those visits.

Is it mandatory to map all REDCap events?

Ans: No, it is not required to map all REDCap events to the CP visits. However, unplanned visits are created for each unmapped REDCap event.

Is it mandatory to specify the visit fields mapping?

Ans: No, it is not mandatory. However, it is good practice to at least map the visit date.

How are mandatory visit fields populated when visit fields are not mapped?

Ans: The mandatory visit fields are populated as follows:

1. When the visit date field is not mapped, its value is set to the date/time of the corresponding REDCap event data entry.

2. When the visit site field is not mapped, its value is picked either from the mapped event's default site or the project site.

What sites are available for selection in the project site dropdown?

Ans: The project site dropdown is restricted to the sites linked with the collection protocol.

REDCap Data

For the value "Save PHI data" please explain the ramifications of clicking "Enabled" or "Disabled".

Ans: When you use the ''Save PHI data" option as Disabled, OS will not pull any RC field marked as 'Identifier' in the RC instrument/form set-up. Usually, these fields are 'First Name', 'Last Name', 'MRN', 'DOB', etc.

PHI such as Name, MRN, DOB must always be saved to identify a record. So what is the use of having a Disabled option for "Save PHI data" ?

Ans: This is a user based preference. We have a few clients who don't want PHI fields pulled in OS. They use these setting as Disabled.

If we remove forms or fields in REDCap after the go-live date, would OS remove them for us automatically during the overnight job?

...

Ans: It depends on the type of configuration you set during the REDCap configuration:

1. The new instrument would be configured automatically if the forms were not defined in the 'Form inclusion/exclusion list'.

2. If any forms are included in the list, you will have to add the new form in the include list.

Why does OpenSpecimen does not delete participants that are deleted in REDCap?

...

Before version v6.3, the cleanup needs to be done at the database level by truncating or deleting rows from these tables.

1. os_redcap_projects - Table that stores all the RC project details.

2. os_redcap_records - Table that stores all the RC participant mapping.

3. os_redcap_instrument_instances - Table that stores all the RC visit form mapping

4. os_redcap_record_visits - Table that stores all the RC visit mapping (only for longitudinal studies)

If you delete a participant or visit in OpenSpecimen and resync ALL data from RC, it will not pull the data for deleted OS objects from RC. However, in the odd case, where you want the resync to fetch ALL data, you will need to delete the entries from the below table:

1. os_redcap_records: delete participant mapping

2. os_redcap_record_visits: delete visit mapping

3. os_redcap_instrument_instances: join with #1 (and/or #2) above and delete the instrument entries

4. If you want to resync all: delete all the entries for a project by joining with os_redcap_projects.

Errors

“java.lang.ClassCastException: java.lang.Integer cannot be cast to java.lang.String"

...

MysqlDataTruncation:Data truncation: Incorrect datetime

You do not have permission to use the API or Invalid token.

...

Row size too large