[Importer] Add the ability to import flags from a CSV file generated from OpenSRP 1 data.

[dubdabasoduba]

Context

• EUSM has been using OpenSRP 1 to collect data for a while now. As they prepare to migrate to OpenSRP 2, they want to migrate the flags data collected via OpenSRP 1.
• This data does not need to be shown on the app but is required for the dashboard. The idea is just to migrate the Flags and give them all the relevant information that allows them to be displayed on the Dashboard and probably closed via OpenSRP web.
• The import will follow the current workflow where we use a CSV file to define the data needed for upload/import then we update the tooling to allow that import.
• More information can be found here https://github.com/onaio/fhir-resources/issues/3605

Implementation

• Create a visit Encounter for imported data. Reuse the existing one if available. A sample of the Encounter is linked below.
• For each row on the CSV create a Form encounter. These can be found on the different form scoping tickets.
  • Service Point Check Form encounter https://github.com/onaio/fhir-resources/issues/3020
  • Consult Beneficiaries Check Form encounter https://github.com/onaio/fhir-resources/issues/3031
  • Product check Form encounter https://github.com/onaio/fhir-resources/issues/3035
• The above Form encounters will be determined by the entityType column on the CSV.
  • flag_problem - Means it's a product check flag
  • beneficiary_consultation - Means it's a consult beneficiaries flag
  • service_point_check - This means it's a service point or warehouse check flag.
    • This Row in the CSV is only a valid Flag if the service_point_good_order == No. Ignore the row if the value is Yes
    • A Row in the CSV will only be interpreted as a Warehouse check in warehouse_required_action is not null.
      • This Row is a valid Flag if the warehouse_required_action == Yes. Ignore the Row if the value is No
• The locationId column is used to create the Location reference on the Flag or Observation.
• The Form Encounters created above are used are the references on the encounter property of the Flag.
• The Visit Encounter created above is used as the partOf reference of the Form encounters.
• The EUSM dashboard displays Flag reasons. These reasons are stored on Observations. The Observations for each of the flags will be linked below

Acceptance Criteria

• All the flags from OpenSRP 1 should be uploaded correctly to OpenSRP 2.
• All the Observations linked to the above flags should be created correctly.
• All the Encounters should be created correctly.
• The relevant references should be added correctly. These include the following
  • The Encounter
  • The Product
  • The Location

[dubdabasoduba]

Product Flag sample

{
  "resourceType": "Flag",
  "id": "a065c211-cf3e-4b5b-972f-fdac0e45fef7",
  "identifier": [
    {"use": "usual", "value": "a065c211-cf3e-4b5b-972f-fdac0e45fef7"}
  ],
  "status": "active",
  "category": [
    {
      "coding": [
        {
          "system" : "http://smartregister.org/",
          "code"   : "PRODCHECK"                ,
          "display": "Product Check"
        }
      ],
      "text": "Product Check"
    }
  ],
  "code": {
    "coding": [
      {
        "system" : "http://smartregister.org/",
        "code"   : "65347579"                 ,
        "display": "Vist Flag"
      }
    ],
    "text": "Vist Flag"
  },
  "subject": {"reference": "Group/6f3980e0-d1d6-4a7a-a950-939f3ca7b301"},
  "period": {"start": "2024-02-01T00:00:00.00Z"},
  "encounter": {"reference": "Encounter/6f3980e0-d1d6-4a7a-a950-939f3ca7b301"}
}

Product Observation Sample

{
  "resourceType": "Observation",
  "id": "a065c211-cf3e-4b5b-972f-fdac0e45fef7",
  "identifier": [
    {"use": "usual", "value": "a065c211-cf3e-4b5b-972f-fdac0e45fef7"}
  ],
  "status": "final",
  "category": [
    {
      "coding": [
        {
          "system" : "http://smartregister.org/",
          "code"   : "PRODCHECK"                ,
          "display": "Product Check"
        }
      ],
      "text": "Product Check"
    }
  ],
  "code": {
    "coding": [
      {
        "system" : "http://snomed.info/sct",
        "code"   : "issue_details"         ,
        "display": "Issue Details"
      }
    ],
    "text": "Issue Details"
  },
  "subject": {"reference": "Group/6f3980e0-d1d6-4a7a-a950-939f3ca7b301"},
  "focus": [
    {"reference": "Location/6f3980e0-d1d6-4a7a-a950-939f3ca7b301"}
  ],
  "encounter": {"reference": "Encounter/6f3980e0-d1d6-4a7a-a950-939f3ca7b301"},
  "effectivePeriod": {
    "start": "2024-02-01T00:00:00.00Z",
    "end"  : "2024-02-01T00:00:00.00Z"
  },
  "performer": [
    {"reference": "Practitioner/6f3980e0-d1d6-4a7a-a950-939f3ca7b301"}
  ],
  "valueString": "Issue Details"
}

• subject - This is the productId in the CSV.
• focus - This is the locationId in the CSV
• encounter - This is the Form encounter created for this ROW
• effectivePeriod.startand period.start - This is the event_date on the CSV
• effectivePeriod.end - This is the event_date on the CSV
• performer - We can fetch the Practitioner ID for the practitioner used to run the tooling.
• valueString - this is the value of product_flag_problem, (product_not_there and product_not_there_specify_other | product_not_good and product_not_good_specify_other | product_misuse and product_issue_details

[dubdabasoduba]

Service Point check Flag sample

{
  "resourceType": "Flag",
  "id": "a065c211-cf3e-4b5b-972f-fdac0e45fef7",
  "identifier": [
    {"use": "usual", "value": "a065c211-cf3e-4b5b-972f-fdac0e45fef7"}
  ],
  "status": "active",
  "category": [
    {
      "coding": [
        {
          "system" : "http://smartregister.org/",
          "code"   : "SPCHECK"                  ,
          "display": "Service Point Check"
        }
      ],
      "text": "Service Point Check"
    }
  ],
  "code": {
    "coding": [
      {
        "system" : "http://smartregister.org/",
        "code"   : "65347579"                 ,
        "display": "Vist Flag"
      }
    ],
    "text": "Vist Flag"
  },
  "subject": {"reference": "Location/6f3980e0-d1d6-4a7a-a950-939f3ca7b301"},
  "period": {"start": "2024-02-01T00:00:00.00Z"},
  "encounter": {"reference": "Encounter/6f3980e0-d1d6-4a7a-a950-939f3ca7b301"}
}

Service Point Good Order (NO) Reason

{
  "resourceType": "Observation",
  "id": "a065c211-cf3e-4b5b-972f-fdac0e45fef7",
  "identifier": [
    {"use": "usual", "value": "a065c211-cf3e-4b5b-972f-fdac0e45fef7"}
  ],
  "status": "final",
  "category": [
    {
      "coding": [
        {
          "system" : "http://smartregister.org/",
          "code"   : "SPCHECK"                  ,
          "display": "Service Point Check"
        }
      ],
      "text": "Service Point Check"
    }
  ],
  "code": {
    "coding": [
      {
        "system" : "http://smartregister.org/"     ,
        "code"   : "34657579"                      ,
        "display": "Service Point Good Order Check"
      }
    ],
    "text": "Service Point Good Order Check"
  },
  "subject": {"reference": "Location/6f3980e0-d1d6-4a7a-a950-939f3ca7b301"},
  "encounter": {"reference": "Encounter/6f3980e0-d1d6-4a7a-a950-939f3ca7b301"},
  "effectivePeriod": {
    "start": "2024-02-01T00:00:00.00Z",
    "end"  : "2024-02-01T00:00:00.00Z"
  },
  "performer": [
    {"reference": "Practitioner/6f3980e0-d1d6-4a7a-a950-939f3ca7b301"}
  ],
  "valueCodeableConcept": {
    "coding": [
      {
        "system" : "http://snomed.info/sct",
        "code"   : "373067005"             ,
        "display": "No (qualifier value)"
      }
    ],
    "text": "No (qualifier value)"
  }
"note": [
    {"time": "2024-02-01T00:00:00.00Z", "text": "Please describe:"}
  ]
}

• subject - This is the locationId in the CSV.
• encounter - This is the Form encounter created for this ROW
• effectivePeriod.startand period.start - This is the event_date on the CSV
• effectivePeriod.end - This is the event_date on the CSV
• performer - We can fetch the Practitioner ID for the practitioner used to run the tooling.
• note - this is the value of service_point_not_good_order_reason

[dubdabasoduba]

Consult beneficiaries flag sample

{
  "resourceType": "Flag",
  "id": "a065c211-cf3e-4b5b-972f-fdac0e45fef7",
  "identifier": [
    {"use": "usual", "value": "a065c211-cf3e-4b5b-972f-fdac0e45fef7"}
  ],
  "status": "inactive",
  "category": [
    {
      "coding": [
        {
          "system" : "http://smartregister.org/",
          "code"   : "CNBEN"                      ,
          "display": "Consult Beneficiaries Visit"
        }
      ],
      "text": "Consult Beneficiaries Visit"
    }
  ],
  "code": {
    "coding": [
      {
        "system" : "http://smartregister.org/",
        "code"   : "65347579"                 ,
        "display": "Vist Flag"
      }
    ],
    "text": "Vist Flag"
  },
  "subject": {"reference": "Location/6f3980e0-d1d6-4a7a-a950-939f3ca7b301"},
  "period": {
    "start": "2024-02-01T00:00:00.00Z",
    "end"  : "2024-02-01T00:00:00.00Z"
  },
  "encounter": {"reference": "Encounter/6f3980e0-d1d6-4a7a-a950-939f3ca7b301"}
}

Consult Beneficiaries Issue raised

{
  "resourceType": "Observation",
  "id": "a065c211-cf3e-4b5b-972f-fdac0e45fef7",
  "identifier": [
    {"use": "usual", "value": "a065c211-cf3e-4b5b-972f-fdac0e45fef7"}
  ],
  "status": "final",
  "category": [
    {
      "coding": [
        {
          "system" : "http://smartregister.org/",
          "code"   : "CNBEN"                      ,
          "display": "Consult Beneficiaries Visit"
        }
      ],
      "text": "Consult Beneficiaries Visit"
    }
  ],
  "code": {
    "coding": [
      {
        "system" : "http://smartregister.org/",
        "code"   : "77223346"                 ,
        "display": "Consult Beneficiaries"
      }
    ],
    "text": "Consult Beneficiaries"
  },
  "subject": {"reference": "Location/6f3980e0-d1d6-4a7a-a950-939f3ca7b301"},
  "encounter": {"reference": "Encounter/6f3980e0-d1d6-4a7a-a950-939f3ca7b301"},
  "effectivePeriod": {
    "start": "2024-02-01T00:00:00.00Z",
    "end"  : "2024-02-01T00:00:00.00Z"
  },
  "performer": [
    {"reference": "Practitioner/6f3980e0-d1d6-4a7a-a950-939f3ca7b301"}
  ],
  "valueCodeableConcept": {
    "coding": [
      {
        "system" : "http://snomed.info/sct",
        "code"   : "373066001"             ,
        "display": "Yes (qualifier value)"
      }
    ],
    "text": "Yes (qualifier value)"
  },
  "note": [
    {"time": "2024-02-01T00:00:00.00Z", "text": "Issues raised"    }
  ]
}

• subject - This is the locationId in the CSV.
• encounter - This is the Form encounter created for this ROW
• effectivePeriod.startand period.start - This is the event_date on the CSV
• effectivePeriod.end - This is the event_date on the CSV
• performer - We can fetch the Practitioner ID for the practitioner used to run the tooling.
• note - this is the value of consult_beneficiaries_issues_raised

[dubdabasoduba]

Warehouse Flag

{
  "resourceType": "Flag",
  "id": "a065c211-cf3e-4b5b-972f-fdac0e45fef7",
  "identifier": [
    {"use": "usual", "value": "a065c211-cf3e-4b5b-972f-fdac0e45fef7"}
  ],
  "status": "active",
  "category": [
    {
      "coding": [
        {
          "system" : "http://smartregister.org/",
          "code"   : "WHCHECK"                      ,
          "display": "WareHouse check Visit"
        }
      ],
      "text": "Consult Beneficiaries Visit"
    }
  ],
  "code": {
    "coding": [
      {
        "system" : "http://smartregister.org/",
        "code"   : "65347579"                 ,
        "display": "Vist Flag"
      }
    ],
    "text": "Vist Flag"
  },
  "subject": {"reference": "Location/6f3980e0-d1d6-4a7a-a950-939f3ca7b301"},
  "period": {"start": "2024-02-01T00:00:00.00Z"},
  "encounter": {"reference": "Encounter/6f3980e0-d1d6-4a7a-a950-939f3ca7b301"}
}

Warehouse Flag Obsersavation

{
  "resource": {
    "resourceType": "Observation",
    "id": "observation_id",
    "identifier": {
      "value": "obs_identifier_value",
      "use": "usual"
    },
    "status": "final",
    "category": {
      "coding": [
        {
          "system": "https://smartregister.org/",
          "code": "WHCHECK",
          "display": "Warehouse check Visit"
        }
      ],
      "text": "Warehouse check Visit"
    },
    "code": {
      "coding": [
        {
          "system": "https://smartregister.org/",
          "code": "68561322",
          "display": "Required action"
        }
      ],
      "text": "Required action"
    },
    "subject": {
      "reference": "LocationReference"
    },
    "encounter": {
      "reference": "Encounter"
    },
    "effective": {
      "start": "2024-08-02T00:00:00Z",
      "end": "2024-08-02T00:00:00Z"
    },
    "valueCodeableConcept": {
      "coding": [
        {
          "system": "http://snomed.info/sct",
          "code": "373066001",
          "display": "Yes (enter action required)"
        }
      ],
      "text": "Yes (enter action required)"
    }
  }
}

• subject - This is the locationId in the CSV.
• encounter - This is the Form encounter created for this ROW
• effectivePeriod.startand period.start - This is the event_date on the CSV
• effectivePeriod.end - This is the event_date on the CSV
• performer - We can fetch the Practitioner ID for the practitioner used to run the tooling.
• note - this is the value of warehouse_required_action_yes

[Wambere]

@dubdabasoduba thanks for the detailed samples above
Quick clarification, for the visit encounter observation, we said that we want to have the start and end periods set to when the import process starts and ends. Are we assuming that this import is only run once? e.g is there a case where some flags are imported today and others next week? if yes, how does that affect the period times? do we update them both to the latest import times? just update the end?

[dubdabasoduba]

Sample Import Visit Encounter

{
  "resourceType": "Encounter",
  "id": "2492dd29-81bf-4628-96cd-f747f48d7e16",
  "status": "in-progress",
  "class": {
    "system" : "http://terminology.hl7.org/CodeSystem/v3-ActCode",
    "code"   : "OBSENC"                                          ,
    "display": "Observation Encounter"
  },
  "type": [
    {
      "coding": [
        {
          "system" : "http://smartregister.org/",
          "code"   : "SVISIT"                   ,
          "display": "Service Point Visit"
        }
      ],
      "text": "Service Point Visit"
    },
    {
      "coding": [
        {
          "system" : "http://smartregister.org/CodeSystem/visit" ,
          "code"   : "SVIST_IMPORT"              ,
          "display": "Service Point Visit Import"
        }
      ],
      "text": "Service Point Visit Import"
    }
  ],
  "priority": {
    "coding": [
      {
        "system" : "http://terminology.hl7.org/ValueSet/v3-ActPriority",
        "code"   : "EL"                                                ,
        "display": "elective"
      }
    ],
    "text": "elective"
  },
  "participant": [
    {
      "individual": {
        "reference": "Practitioner/631f2c90-43cc-4ae3-a3a5-c539dff7bc3c"
      }
    }
  ],
  "period": {
    "start": "2023-09-13T03:56:00.000+00:00",
    "end"  : "2023-09-13T23:59:00.000+00:00"
  },
  "reasonCode": [
    {
      "coding": [
        {
          "system" : "http://smartregister.org/",
          "code"   : "SVISIT"                   ,
          "display": "Service Point Visit"
        }
      ],
      "text": "Service Point Visit"
    },
    {
      "coding": [
        {
          "system" : "http://smartregister.org/CodeSystem/visit" ,
          "code"   : "SVIST_IMPORT"              ,
          "display": "Service Point Visit Import"
        }
      ],
      "text": "Service Point Visit Import"
    }
  ],
  "location": [
    {
      "location": {
        "reference": "Location/493f46d8-6dfe-4505-ab63-9d78c789400e"
      },
      "status": "active"
    }
  ]
}

[dubdabasoduba]

Quick clarification, for the visit encounter observation, we said that we want to have the start and end periods set to when the import process starts and ends. Are we assuming that this import is only run once? e.g is there a case where some flags are imported today and others next week? if yes, how does that affect the period of time? do we update them both to the latest import times? just update the end?

1. Clarification - We are not creating a Visit encounter Observation. We just need to create the Visit Encounter. A sample of it is added above

We said that we want to have the start and end periods set to when the import process starts and ends. Are we assuming that this import is only run once?

Yes, we are assuming that this would only be run once but, there is a potential of running it multiple times in case the clients use OpenSRP 1 to collect data. Based on this I was thinking we can create a new Visit encounter every time the CSV import is run. What are your thoughts on this?

[Wambere]

1. Clarification - We are not creating a Visit encounter Observation. We just need to create the Visit Encounter. A sample of it is added above

Oh my bad, typo. Yes I mean Visit encounter, not visit encounter observation

Yes, we are assuming that this would only be run once but, there is a potential of running it multiple times in case the clients use OpenSRP 1 to collect data. Based on this I was thinking we can create a new Visit encounter every time the CSV import is run. What are your thoughts on this?

That is definitely possible, it would however be the responsibility of whoever is running the import to put in a new encounter_id (since this is at the moment passed in as a variable when running the command), then the creation of the new encounter will be automatically done.
I also think that this will add a whole layer of complexity, especially if the main purpose of this visit encounter is to track when the records were imported. For example in a case where the same csv is imported twice on different days, or maybe a couple of rows added and then re-run... keeping track of the history of the different visit encounters might be tricky. But let's try it out and see

[Wambere]

@dubdabasoduba I have imported the records to the preview instance, except for a few with issues

• 100 rows of products that had missing ids
• Many rows that were linked to 18 non-existing locations

Please review when you have some time