Improve subquery timing

[tokebe]

Currently, BTE queues subqueries by organizing them into buckets. Each bucket may only carry 3 queries per API, with no limit on the number of APIs per bucket. Buckets are then executed sequentially, with each subsequent bucket waiting for the previous to complete all contained queries.

This setup has advantages -- namely, it prevents BTE from making too many queries to the same API at once. However, it also limits BTE's subquery throughput substantially. In most cases, a bucket will contain only 3 queries to the same API, and then only 3 queries are made at a time, waiting for the longest of each, until all buckets are complete. Additionally, there's no theoretical limit to the number of outbound queries, provided they're all to different APIs (though, in practice, this isn't an issue).

We should instead use a different model that allows for higher throughput. Some APIs provide rate limits, which BTE already attempts to respect (albeit, naively). These should be better taken into account, and for those which are unknown, BTE should better handle 429 responses to automatically determine a limit -- one which could be saved both locally to some global (for local, non-redis installations) and in redis (for live instances). Additionally, BTE should use a number of 'slots' and start new subqueries as slots open up, rather than as the slowest of a bucket completes.

Further consideration for the model to implement can be determined in further discussion after some tinkering has been done.

[colleenXu]

I'm wondering if scrolling BioThings queries need special consideration here (I don't know how they relate to "buckets". I'm also not sure that they're included in the logs of how many sub-queries are executed / successful).

[tokebe]

Note: we also want to add a (probably configurable) retry system, so if a query errors out for most non-400, non-timeout reasons, it should be recycled back into the queue (probabably only once or twice).

[colleenXu]

@tokebe It looks like the related PRs aren't linked. Is that okay? (For example: maybe this one?)

[tokebe]

Apologies, bit of an oversight.

Related PRs:

• Concurrency improvements bte_trapi_query_graph_handler#174
• Concurrency improvements call-apis.js#74
• Concurrency improvements bte-server#5
• Concurrency improvements #765

[tokebe]

Relevant changes deployed to Prod.