Update BioThings x-bte annotation to use filter

[colleenXu]

@newgene says that once the code freeze is over, the BioThings APIs will be updated to allow POST queries to use post_filter.

The initial issue is here: biothings/biothings.api#208

post_filter can be set in the parameters or the request body. @newgene, could you give an example of each in a comment?

---

Once it is live on BioThings APIs prod instances, we'll want to use it. It'll involve looking for BioThings x-bte annotation that uses more complicated "q" querying (maybe uses wrap, replPrefix, _exists_...some way of setting >1 field value) and seeing if post_query will work with it.

[colleenXu]

Making a quick note: this related BioThings feature filter may be coming downstream eventually and is related but distinct from post_filter biothings/biothings.api#296

[colleenXu]

@newgene

This isn't urgent, but can you give an update?

• an example of using post_filter in POST /query, in the parameters and in the request body
• are the BioThings APIs updated for post_filter in POST /query ? If not, do you have a timeline for when this will be available?

[newgene]

A optimized example with post_filter might look like this:

      requestBody:
        body: >-
          {q: "{{ queryInputs }}"  
          scopes: subject.umls}
      parameters:
        fields: object.umls,predication.pmid,predication.sentence,subject.name,object.name
	post_filter: >-
          predicate:LOCATION_OF AND object.semantic_type_abbreviation:neop 
          AND _exists_:object.umls AND pmid_count:>3
        size: 1000

from https://raw.githubusercontent.com/NCATS-Tangerine/translator-api-registry/master/semmeddb/smartapi.yaml

[colleenXu]

From discussion during lab meeting today:

• post_filter is already live on all BioThings APIs (core and pending)
• we discussed how it should work (see Chunlei's comment above)
• Chunlei suggested moving forward on this issue:
  • try building the parameters/request-body using this feature, querying the BioThings API directly, testing changes to x-bte annotation...
  • if that looks good, then change all x-bte annotation
• filter is an upcoming BioThings API feature, and should be an improvement on post_filter (it also has differences for facets / aggregation but we haven't been using those for BTE). So do find-replace once that goes live

EDIT: adding notes from more thinking

• this involves editing the BioThings semmeddb x-bte-annotation-generating notebook to update that resource's x-bte annotation...

[colleenXu]

This BioThings feature may be useful (look at multiple fields for a value, while setting other fields as well), or it may be obsolete now when we adopt the post_filter/filter feature...

[colleenXu]

Update:

• right now filter is deployed to pending BioThings APIs only. Update x-bte annotation to use this, so it can be tested
  • core BioThings APIs won't be updated to use filter until this testing is done
• should be helpful for performance, so it's related to concurrency/performance issues.

[colleenXu]

I prioritized working on this issue before my trip, because it should improve performance.

Already done

in NCATS-Tangerine/translator-api-registry#135 and registrations are refreshed

• TTD
• semmeddb
• text-mining targeted
• dgidb
• suppkg

Not tested

EDIT 12/5: now tested because bug was fixed and these operations are all working as-intended

• Some operations from TTD and Text-Mining Targeted that are affected by Multiple x-bte operations not working (records dropped during edge-management) #756 (see collapsed section below)
• commented-out operations (adjusted some to use filter)

click to expand

TTD:

• pubchem_treats_mondo-rev
• ttd_drug_id_treats_mondo-rev
• uniprotkb_target_for_mondo-rev
• ttd_target_id_target_for_mondo-rev

Text-Mining Targeted:

• drugbank-treats-mondo-rev
• drugbank-treats-hp-rev
• uniprotkb-contributes_to-mondo-rev
• uniprotkb-contributes_to-hp-rev
• drugbank-contributes_to-hp-rev
• drugbank-contributes_to-mondo-rev

Not done yet

• core:
  • MyChem: 10 operations, but potentially complicated because list_filter/jmespath is also involved
• pending Multiomics:
  • drug response: relatively easy (20 operations)
  • wellness: kinda hard/tedious to do (108 operations)
  • ehr risk: kinda hard/tedious to do (148 operations)

---

Notes:

• I included all APIs that could use this new feature in the lists above. I found them by searching for requestBodyType, which indicates more complex requestBody that could be changed to use filter
• I've only worked on pending BioThings APIs, because post_filter and filter seemed to only be deployed on those instances (lab Slack thread)
• I've been using filter because Chunlei recommended it over post_filter
• I noticed that:
  • post_filter and filter seem to work even when the fields in their statements aren't in the parameters.fields (this differs from list_filter/jmespath behavior)
  • sometimes it's easier to make the same filter statement for the forward + reverse operation, even if that adds more than is strictly needed for the statement…

[colleenXu]

Update: the bug blocking testing was addressed, and the "Not tested" operations above were tested and working as-intended.

[colleenXu]

[IN-PROGRESS, WILL CONTINUE TO EDIT]

Done updating to use "filter" parameter:

• Multiomics BigGIM drug-response
• MyChem: only drugcentral bioactivity operations were using the old format, so they were updated (1 operation, rest of operations)
• (commented out text-mining targeted and ttd stuff?)

Not done yet:

• RaMP
• Multiomics wellness: kinda hard/tedious to do (108 operations)
• Multiomics ehr risk: kinda hard/tedious to do (148 operations)

Note!

• Start w/ direct queries to BioThings API
• note when yaml comments are old, but don't need to do all the work to update them