[Performance] POST /api/subject

[mahalakshme]

Need:

POST and PATCH /api/subject appear in the top 5 in Newrelic consistently

Analysis:

2 things causing slowness:

• select from title_lineage_locations_view
  - doing lowest on titleLineage causing additional slowness
  - querying on the view affects us from being able to add indices on titleLineage
• select from virtual_catchment_address_mapping table based on titleLineage

AC:

• Any solution shouldn't cause further slowness in other APIs
• on POST and PATCH apis of /api/subject only when includeCatchments optional parameter is true, return catchments in the response else by default don't return. The change can be done in existing version itself since that was the understanding that was communicated and other APIs like GET do work like this.
• Add a version=3 parameter, for POST, PUT, PATCH api of subjects. In this version, POST Address as map instead of comma separated values. Lets see the below eg, where location types in an org are Country, State and District.

What does it accept as of version 2:

{
  "External ID": "string",
  "Subject type": "Individual",
  "Address": "India, Uttarakhand",
  "Location ID": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
  "Date of birth": "2024-10-11",
  "Gender": "Male",
  "Registration date": "2024-10-11",
  "First name": "string",
  "Middle name": "string",
  "Last name": "string",
  "Registration location": {
    "X": 19.1253108,
    "Y": 74.7364501
  },
  "observations": {},
  "Profile picture": "string",
  "Voided": true
}

What it should accept as of version 3:

{
   "External ID": "string",
  "Subject type": "Individual",
  "Address map": {
     "Country": "India",
     "State": "Uttarakhand"
   },
  "Location ID": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
  "Date of birth": "2024-10-11",
  "Gender": "Male",
  "Registration date": "2024-10-11",
  "First name": "string",
  "Middle name": "string",
  "Last name": "string",
  "Registration location": {
    "X": 19.1253108,
    "Y": 74.7364501
  },
  "observations": {},
  "Profile picture": "string",
  "Voided": true
}

• So instead of comma separated values, we need to change Address to map. Keys need to be name of location type, and values need to be the location value.
• Error message is generic if the passed map does not resolve to a valid address - Address corresponding to 'Address map' not found
• Should not have the need to mention all location types.
• Should work in orgs with multiple location hierarchies.
• Searching for a location presence to be done in case-insensitive manner. ie., if input via API is 'karnataka' and the location created in Avni is 'Karnataka' they need to still match.

Tech inputs:

• To match the API Address map with locations in Avni use address_level and address_level_type table and not title_lineage_locations_view

Out of scope:

• Modifying the integration service interface
• Making sure the individual is registered only in the allowed location types configured on subject type. Will be handled as part of this card.

Analysis notes:

How integration services uses Avni APIs:

• Bahmni - both ways - karnataka TW pilot - forked
• Goonj - we have State and District separately, so we can update
• Amrit - Avni to Amrit only
• Ashwini - POST, PUT only to encounter and enrolment, no usage of PATCH, not subject
• LAHI - doesn't 've any
• PoWER - not relevant
• Shelter calls PUT /api/subject

Old: ignore

• Since majority of slowness is because of finding address id based on lineage in the request
  - Suggestion: Moving the title_lineage from view to addressLevel table might help, but this will need updating of this column(whenever updating lineage in address_level table - or the lineage column can be removed based on usages) when parent of a location is updated via CSV or UI

Questions:

Can we do this? Say the titleLineage is A,B,C
Find id of C - if only one present then set individual to that id, else - mostly will end here itself
Then find id of B, if only one present then set individual to addresses with title C and parent_id of B, else - Lets make child and parent combination unique and just do this - but there are 34 entries which have them same and 7k entries have , followed by space in the title
Then find id of A, if only one present then id of B = title with B and ....

Inputs:

• disallow , in location

• just inputting address

• 2nd approach - A then B and then C

• reaching to the users is no

• changing the API endpoints better

• can be done via APIs

• change structure - array, map(better: with different location hierarchy)

• what do callers ve - type info, etc.,

• usercatchment csv

• find by titleLineage

• so patch also needs change

• ashwini - bahmni

Inputs:

• subject type -> location type
• case insensitive -

[1t5j0y]

Changes from AC:

• new element named as 'Address map' to be able to support 'Address' in pre-version 3 and 'Address map' in version 3 onwards with type safety.
• Error message is generic if the passed map does not resolve to a valid address - Address corresponding to 'Address map' not found

[mahalakshme]

@1t5j0y Also suggest updating the swagger documentation.

[1t5j0y]

@mahalakshme already done.. anything missing?

[himeshr]

See code-review comment: c587590#r148343774

[himeshr]

Need to also handle the scenarion where there are more than one address level with same title for a given addressLevelType in different lineages. Due to which the locationRepository.findLocation(String title, String type) call would return more than 1 entry.

[AchalaBelokar]

• [ ]

• I put the body and authtoken still it is not showing the result......

[AchalaBelokar]

kskks

[himeshr]

During PATCH "/api/subject?version=3" testing, for updating address, where subject type has the "Location type where this subject can be registered" is set to an intermediate "Address Level Type", then the server responds with error "Address corresponding to 'Address map' not found".

[himeshr]

KNOWN ISSUES - NOT TO BE FIXED IN THIS CARD

• We do not have validations for Address and AddressMap fields, to ensure their values adhere to SubjectType "Location type where this subject can be registered" restriction
• There are no data validations performed for Observation values, so we are able to add non-related fields, invalid values during PUT / POST / PATCH operations

[himeshr]

Validated following test-cases as well after latest fix (Across both versions)

PATCH

• Multiple addressLevelType hierarchies
• Partial addressLevelType hierarchies

DifferentAddressTypeHierarchiesIndividualPatch.txt

PUT / POST

• Create / update subject

PostRequestBodySubjectCreatePerfTesting.txt