🔨 change CF thumbnail function to use chart configs from R2

[danyx23]

This PR changes how the CF thumnail worker gets the config for a chart. It used to be the case that it would fetch the HTML file of the grapher page at the given slug and extract the config from that HTML. Now it looks up the grapher config json file in an R2 bucket.

This PR only changes this for published charts accessed by slug. A later PR will enable this also by UUID.

Cloudflare is a bit weird with the intersection of support of various features between CF workers/pages functions, R2 and local/remote dev support:

• Initially I wanted to use bindings in CF functions. For local development, wrangler dev does not support using a remote R2 bucket which is annoying
• I then thought I'd switch to fetching data from R2 buckets via HTTPS instead of bindings, similar to what we do with our variable data/metadata.json files. I realized then though that this only works if the bucket itself is accessible publicly via HTTPS. I'd like to keep the R2 bucket internal and not expose it directly.
• My next approach was to try to use the S3 API in the CF pages function. This didn't work because the default AWS library doesn't work in CF functions and the simpler replacement libraries that I could find were very barebones and made you deal with XML manually
• I then decided to go back to bindings and accepting the somewhat annoying dev story
• But then I realized that once we get MDims, our data managers will want to make changes to charts and also to make changes to MDims and see their changes. If we fetch chart configs for MDims from CF functions in the future then on staging servers this would have been a painful story
• So in the end I decided to make the R2 buckets accessible via the public internet and use fetch instead of R2 bindings.

To test this PR, the main change to your local setup is to add an entry to your .dev.vars example to specify the path within the staging R2 bucket to use for thumbnail rendering:

GRAPHER_CONFIG_R2_BUCKET_PATH=devs/YOURNAME

Before you can test the thumnail rendering locally for a particular chart, you will either have to edit one chart in the admin to make sure it gets uploaded to R2, or run the sync script in devTolls/syncGraphersToR2

This PR adds a few new env vars for CF functions:

• GRAPHER_CONFIG_R2_BUCKET_URL - the primary bucket to read from
• GRAPHER_CONFIG_R2_BUCKET_FALLBACK_URL - the fallback bucket to read from (this should be the prod bucket; on prod it should be empty)
• GRAPHER_CONFIG_R2_BUCKET_FALLBACK_PATH - the path in the fallback bucket to read from (the path inside the prod bucket; on prod it should be empty)
• GRAPHER_CONFIG_R2_BUCKET_PATH - in wrangler.toml we set this only for prod as it's constant there. For local dev setups it should be used from the .dev.vars as per above. Staging servers fall back to the branch name.

[danyx23]

• 🔨 change CF thumbnail function to use chart configs from R2 #3867 : 2 dependent PRs (#3865 , #3870 ) 👈
• 🔨 unify r2 config dirs #3945
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @danyx23 and the rest of your teammates on Graphite

[owidbot]

Quick links (staging server):

Site	Admin	Wizard

Login: ssh owid@staging-site-use-r2-in-cf-functions

• Site-screenshots: https://github.com/owid/site-screenshots/compare/use-r2-in-cf-functions
• SVG tester: https://github.com/owid/owid-grapher-svgs/compare/use-r2-in-cf-functions

SVG tester:

Number of differences (default views): 0 ✅
Number of differences (all views): 0 ✅

Edited: 2024-09-09 14:51:24 UTC
Execution time: 1.37 seconds

[danyx23]

Merge activity

• Sep 10, 6:33 AM EDT: @danyx23 started a stack merge that includes this pull request via Graphite.
• Sep 10, 6:34 AM EDT: @danyx23 merged this pull request with Graphite.