-
Notifications
You must be signed in to change notification settings - Fork 659
Flexporter
Warning
🐉 Here be dragons. This feature is a very rough work in progress. Features that you expect may be missing, and features that do exist may be broken. Proceed with caution.
The Synthea Flexible Exporter a.k.a. "Flexporter" is a utility designed to make it easy to add new fields, values, and resource types to Synthea-generated FHIR, without having to modify the Synthea engine. Primary users of this feature are expected to be Implementation Guide authors and users, electronic clinical quality measure developers and testers, and anyone who needs just a slight tweak to the data exported by Synthea.
The basic idea of the Flexporter is that users will define a series of transformations to apply to Synthea data. A predefined set of transformations, such as "set field X to value Y" or "create a new resource" is intended to cover the majority of use cases while still remaining easy to use.
./run_synthea -fm mapping_file [-ig ig_folder]
The Flexporter can also run standalone to post-process FHIR Bundles.
./run_flexporter -fm mapping_file -s source_fhir [-ig ig_folder]
https://github.com/synthetichealth/synthea/blob/flexporter/src/main/java/RunFlexporter.java
The format of the mapping file is as follows:
---
# name is just a friendly name for this mapping
name: Flexporter Mapping
# applicability determines whether this mapping applies to a given file.
# for now the assumption is 1 file = 1 synthea patient bundle.
# this should be a FHIRPath which should return a "truthy" value if the mapping applies
applicability: true
# actions is a list of Action objects. see below for details
actions:
- name: Apply Profiles
...Actions are driven by a key property name on the object. (technical note: this is because the YML library doesn't support discriminators like GSON does for JSON, think like how state types get mapped into a different class by type)
| Field Name | Type | Required | Description | |
|---|---|---|---|---|
| 🗝️ | profiles | List | Yes | List of profile/applicability pairs |
| profile | URL | Yes | Profile URL to be added to resources | |
| applicability | FHIRPath | Yes | FHIRPath to select resources that the profile should be added to |
Example:
This example will apply the qicore-patient profile to all Patient resources, and qicore-encounter to all Encounters.
- name: Apply Profiles
profiles:
- profile: http://hl7.org/fhir/us/qicore/StructureDefinition/qicore-patient
applicability: Patient
- profile: http://hl7.org/fhir/us/qicore/StructureDefinition/qicore-encounter
applicability: Encounter| Field Name | Type | Required | Description | |
|---|---|---|---|---|
| 🗝️ | set_values | List | Yes | List of profile/applicability pairs |
| applicability | FHIRPath | Yes | FHIRPath to select resources to set fields on | |
| fields | List | Yes | List of location/value pairs to set | |
| location | FHIRPath | Yes | FHIRPath to the field to set the value on | |
| value | raw value, Value Function, object | Yes | The value to set on the target, or the function to fetch a value to set on the target | |
| transform | Value Transformation | No | Transformation to apply to the value before setting on the target |
Example 1:
This example will read the value from Immunization.occurence[x] and set it on the Immunization.recorded field, for all Immunization resources.
- name: testSetValues_getField
set_values:
- applicability: Immunization
fields:
- location: Immunization.recorded
value: $getField([Immunization.occurrence])Example 2:
This example will set a full coding, with code/system/display, to Patient.maritalStatus.coding.
- name: testSetValues_object
set_values:
- applicability: Patient
fields:
- location: Patient.maritalStatus.coding
value:
system: http://snomedct.io
code: "36629006"
display: "Legally married (finding)"| Field Name | Type | Required | Description | |
|---|---|---|---|---|
| 🗝️ | create_resource | List | Yes | List of resource definitions to create |
| resourceType | FHIR resource type | Yes | Resource type to create (Patient, Encounter, etc) | |
| based_on | object | No | ||
| based_on.resource | FHIRPath | No | FHIRPath to select resources to base the new created resource off of. (Example use case, create an Appointment based on every Encounter) | |
| based_on.module | GMF Module Name | No | The name of a Synthea Module, where an instance of the current resource will be created for each instance of the patient passing through the state | |
| based_on.state | GMF Module State | No | The state to use as the basis for creating the current resource | |
| fields | List | Yes | List of location/value pairs to set on the created resource | |
| writeback | List (same options as fields) | No | List of location/value pairs to set on the selected based_on resource | |
| fields.location | FHIRPath | Yes | FHIRPath to the field to set the value on | |
| fields.value | raw value, Value Function, object | Yes | The value to set on the target, or the function to fetch a value to set on the target | |
| fields.transform | Value Transformation | No | Transformation to apply to the value before setting on the target |
Example:
This example will create an instance of ServiceRequest for every instance of Procedure, with the following:
- ServiceRequest.intent set to "plan"
- ServiceRequest.encounter.reference set to the same as the source Procedure.encounter.reference
- ServiceRequest.subject.reference set to a reference to the Patient resource
- Procedure.basedOn.reference set to a reference to the newly created ServiceRequest
- name: testCreateResources_createBasedOn
create_resource:
- resourceType: ServiceRequest
based_on:
resource: Procedure
fields:
- location: ServiceRequest.intent
value: plan
- location: ServiceRequest.encounter.reference
value: $getField([Procedure.encounter.reference])
- location: ServiceRequest.subject.reference
value: $findRef([Patient])
writeback:
- location: Procedure.basedOn.reference
value: $setRef([ServiceRequest])| Field Name | Type | Required | Description | |
|---|---|---|---|---|
| 🗝️ | keep_resources | List<FHIRPath> | Yes | List of FHIRPath to select resources that should be retained in the bundle, anything not selected by a rule will be removed |
Example:
This example will keep only the Patient, Encounter, and Condition resources. All other resources will be removed.
- name: testKeepResources
keep_resources:
- Patient
- Encounter
- Condition| Field Name | Type | Required | Description | |
|---|---|---|---|---|
| 🗝️ | delete_resources | List<FHIRPath> | Yes | List of FHIRPath to select resources that should be deleted in the bundle, anything selected by a rule will be removed |
Example:
This example deletes the Provenance resource:
- name: testDeleteResources
delete_resources:
- ProvenanceIf the above options aren't sufficient, the Flexporter allows for executing arbitrary JavaScript code against the Bundle. (The ability to load arbitrary libraries is not [yet] supported)
| Field Name | Type | Required | Description | |
|---|---|---|---|---|
| 🗝️ | execute_script | List | Yes | List of script definitions to execute (in order) |
| function | JavaScript code (string) | Yes | Javascript code including the function to be executed. More than one function may be defined in this field at a time | |
| function_name | string | Yes | Name of the function to be invoked | |
| apply_to |
"resources" or "bundle"
|
Yes | Whether the function should be invoked on the "bundle" as a whole, or on the "resources" each individually |
Example:
This example executes two scripts. The first adds a dummy profile to the first resource in the Bundle, the second sets the birthDate on any Patient resources in the Bundle to '2022-02-22'.
- name: testExecuteScript
execute_script:
- apply_to: bundle
function_name: apply
function: |
function apply(bundle) {
bundle['entry'][0]['resource']['meta'] = {profile: ['http://example.com/dummy-profile']}
}
- apply_to: resource
function_name: apply2
function: |
function apply2(resource, bundle) {
if (resource.resourceType == 'Patient') {
resource.birthDate = '2022-02-22';
}
}| Function | Arguments | Description |
|---|---|---|
getField |
FHIRPath pointing to field | Get value from a field from current or based-on resource |
findValue |
FHIRPath pointing to resource/field | Find a value from another resource within the entire Bundle |
setRef |
Create a reference to the current resource | |
findRef |
FHIRPath pointing to resource | Find a reference to another resource in the bundle |
getAttribute |
Attribute key | Read value from a Synthea attribute. Note this can only be used when running in Synthea, not standalone. |
randomCode |
ValueSet URL | Pick a random code from the target ValueSet |
Basic naming scheme here is "set" and "get" refer to a particular resource, (ie, the current one, or the resource a new one is being based on) and "find" refers to searching the entire bundle. The syntax for value functions is function([argument1,argument2,...]) where arguments are included without quotes.
- create_resource:
- resourceType: ServiceRequest
based_on:
resource: Procedure
fields:
- location: ServiceRequest.intent
value: plan
- location: ServiceRequest.encounter.reference
value: $getField([Procedure.encounter.reference])
- location: ServiceRequest.subject.reference
value: $findRef([Patient])
- location: ServiceRequest.code.coding
value: $randomCode([http://hl7.org/fhir/us/mcode/ValueSet/mcode-laterality-vs])| Function | Description |
|---|---|
toInstant |
Convert the given date/time into an instant: YYYY-MM-DDThh:mm:ss.sss+zz:zz. If the given item doesn't have that level of precision, random values will populate the missing pieces. |
toDate |
Convert the given date/time into a date: YYYY-MM-DD. If the given item doesn't have that level of precision, random values will populate the missing pieces. |
toDateTime |
Convert the given date/time into a dateTime: YYYY-MM-DDThh:mm:ss+zz:zz. If the given item doesn't have that level of precision, random values will populate the missing pieces. |
toTime |
Convert the given date/time into a time: hh:mm:ss. If the given item doesn't have that level of precision, random values will populate the missing pieces. |
actions:
- name: Apply Profiles
profiles:
- profile: http://hl7.org/fhir/us/mcode/StructureDefinition/mcode-cancer-patient
applicability: Patient
- profile: http://example.com/bloodpressure
applicability: Observation.code.coding.where($this.code = '85354-9')
- profile: http://hl7.org/fhir/us/mcode/StructureDefinition/mcode-tnm-clinical-stage-group
applicability: Observation.code.coding.where($this.code = '21908-9')
- profile: http://hl7.org/fhir/us/mcode/StructureDefinition/mcode-tnm-clinical-primary-tumor-category
applicability: Observation.code.coding.where($this.code = '21905-5')
- profile: http://hl7.org/fhir/us/mcode/StructureDefinition/mcode-tnm-clinical-regional-nodes-category
applicability: Observation.code.coding.where($this.code = '21906-3')
- profile: http://hl7.org/fhir/us/mcode/StructureDefinition/mcode-tnm-clinical-distant-metastases-category
applicability: Observation.code.coding.where($this.code = '21907-1')
- name: Create TNM References
set_values:
- applicability: Observation.code.coding.where($this.code = '21908-9')
fields:
- location: Observation.hasMember.where(display='Tumor Category').reference
value: $findRef([Observation.code.coding.where($this.code = '21905-5')])
- location: Observation.hasMember.where(display='Nodes Category').reference
value: $findRef([Observation.code.coding.where($this.code = '21906-3')])
- location: Observation.hasMember.where(display='Metastases Category').reference
value: $findRef([Observation.code.coding.where($this.code = '21907-1')])- ValueSets can only be loaded from the IG when they are fully defined and expanded within the IG. The existing Synthea terminology support works, but you have to provide your own terminology server.
- Resources removed by keep_resources or delete_resources will still be referenced by other resources. Need to define the best approach for what to do here. (Maybe options? "Cascade" = delete resources that reference deleted resources, "dangle" = leave dangling references, "prune field" = remove the reference field but leave the rest of the resource)
- Copying objects from one field to another doesn't currently work, only primitives. The workaround for now is to list out all the individual sub-fields, but this doesn't work well when the fields that are populated might change. One hack I tried to implement was to deconstruct the provided object into its component fields, but that didn't work because HAPI doesn't provide a way to serialize generic FHIR objects, only Resources. (Ex, you can't serialize a CodeableConcept) See Actions.createFhirPathMapping https://github.com/synthetichealth/synthea/blob/d08c545d168610df7ba42326477a14e17daaa78c/src/main/java/org/mitre/synthea/export/flexporter/Actions.java#L176
- Sometimes YAML needs to be told what an object is. For example things that look like dates will become java Date objects unless explicitly marked as strings:
- name: testSetValues
set_values:
- applicability: Patient
fields:
- location: Patient.birthDate
value: !!str "1987-06-05"At the core of the flexporter is the CustomFHIRPathResourceGeneratorR4 class - this is based on a community contribution within HAPI which we've added some additional functionality to. It does some magic using HAPI's internal reflection (on top of Java reflection), so it can be a bear to wrap your head around. Debugging and stepping through examples makes it easier. https://github.com/synthetichealth/synthea/blob/flexporter/src/main/java/org/mitre/synthea/export/flexporter/CustomFHIRPathResourceGeneratorR4.java