feat(explorers): allow referencing indicators by catalog path

[marcelgerber]

Fixes #3711.

So!

I worked on using ETL paths in indicator-based explorers for the last few days.
This means that something like this is now valid:

How this works

The way this works is that, at baking time, we resolve all catalog paths contained in the graphers and columns sections of the explorer config.
The explorer config is then transformed in such a way that any catalog paths are replaced by their indicator IDs, and this transformed explorer config is then baked and shipped to the browser. That is, the browser never really needs to know about the existence of catalog paths.

The fields that are transformed are:

• graphers:
  • yVariableIds
  • xVariableId
  • sizeVariableId
  • colorVariableId
• columns
  • catalogPath -> variableId

As you can see above, most fields have not been renamed. That is, any of yVariableIds etc. can now contain either an integer variable ID or a string catalog path (or, in the case of yVariableIds, in theory also a mix of both).

Error handling

One caveat here is error handling, i.e. when an indicator can't be found for a catalog path.
In that case:

• An error is sent to Bugsnag at bake time, mentioning the count and names of the corresponding catalog paths
• The "baked explorer" still contains some catalog paths in fields like yIndicatorIds, xIndicatorId, etc., and just chooses to ignore them

This means: If any of the ETL paths are invalid, a Bugsnag error will be sent at bake time. The explorer will still get baked, but will be partially-broken.

PR tips

On an explorer page, you can get to the baked explorer config using explorer.explorerProgram.prettify() in the console. You can then copy this as plaintext (Chrome: "Copy string contents") and inspect it somewhere.

---

Todos

• Do we also send Bugsnag errors when previewing?
• Check that embedding these explorers works fine
• Think about having a explorer-global catalogPathPrefix or so?
  • Easy to add later on, if we so desire :)
• Currently, we don't do any error handling for unknown catalog paths in the columns.catalogPath section. Should we?
• It's not great that we have string types for some columns now, which are pretty much only present before baking.
• Check logic determining whether it's a variable-based or slug-based explorer (explorerProgram.chartCreationMode)
• Oh god, editing in admin is broken because it's using localStorage
• Button / or toggle to clear localStorage, or switch between localStorage and authored
• Check that various existing explorers still work as intended
• After merge: Update explorer mirroring automation https://github.com/owid/automation/blob/main/automation/mirror_explorers.py#L318

[owidbot]

Quick links (staging server):

Site	Admin	Wizard

Login: ssh owid@staging-site-explorer-by-etl-path

• Site-screenshots: https://github.com/owid/site-screenshots/compare/explorer-by-etl-path
• SVG tester: https://github.com/owid/owid-grapher-svgs/compare/explorer-by-etl-path

SVG tester:

Number of differences (default views): 94 (322819) ❌
Number of differences (all views): 250 (6f38b9) ❌

Edited: 2024-07-02 15:59:43 UTC
Execution time: 1.19 seconds

[marcelgerber]

For review purposes: https://github.com/owid/owid-content/tree/explorer-etl-path-review contains two catalog-path-powered explorers: population-and-demography and population-source.

The staging server for this branch also has a catalog-path-based population-and-demography, but using the 2024 data already because lucas is currently working on that.

[marcelgerber]

I now added an admin-only checkbox that lets one easily switch between viewing the saved or changed explorer config (preview will only work properly using the saved version).

CleanShot.2024-07-02.at.16.20.08.mp4

[danyx23]

Nice, this is a neat feature! Have you adapted the script in automation that imports explorers into the db? If not, can you create an issue and assign it to me?