[RFC] [gatsby-source-drupal]: fetching individual entities instead of lists for higher cache hit rate

[KyleAMathews]

The source plugin currently fetches entities in pages of 50. Which is very fast when the requests are cached. But Drupal's cache eviction policy unfortunately evicts all API pages whenever any entity is modified. Which means that most sources are hitting Drupal. Which is dramatically slower — 10-20x slower.

@Auspicus, @rcaracaus, and I have been discussing that we change the source plugin to fetch individual entities. This seems slower as it'd mean ~50x more API requests. But given that Drupal only evicts the entity's cache when the actual entity changes, the source plugin's cache rate should go from ~10% to perhaps 90%+ So could end up being quite a bit faster on average.

We'd need a new endpoint in the Gatsby module which gives gatsby-source-drupal a list of all entities that need fetched.

A downside to this approach (perhaps can be worked around) is that JSON:API's filters wouldn't work.

[larowlan]

We'd need to see comparable numbers for the cost of making 50x more requests.

[KyleAMathews]

Yeah next step is to build a POC and see how fast this is for different scenarios.

[KyleAMathews]

We did a POC and Gatsby can make 2-3k requests / second for the individual entities so it's plenty fast enough to be competitive.