Fix ETL staging server MDim caching issues #4296
Conversation
|
Quick links (staging server):
Login:
SVG tester:Number of differences (default views): 0 ✅ Edited: 2024-12-12 18:00:41 UTC |
| @@ -242,7 +248,7 @@ export const MultiDimDataPageContent = ({ | |||
|
|
|||
| const currentView = useView(currentSettings, config) | |||
| const { varDatapageData, varGrapherConfig, grapherConfigIsReady } = | |||
| useVarDatapageData(config, currentView) | |||
| useVarDatapageData(config, currentView, isPreviewing ?? false) | |||
There was a problem hiding this comment.
could also just set a default in the parameter destructuring 🙂
There was a problem hiding this comment.
Thanks for the fix!
Since we are already introducing a change in behavior between prod and staging (for a good reason) would it make sense to use the environment we run in to decide on what to do? So instead of passing the flag around, we could disable caching in staging for all requests to this endpoint.
There was a problem hiding this comment.
@rakyi yes I thought of this as well - but it can also be useful to have a test environment behave very similar to the prod setup including caching - so that when you make non-admin requests you can test if the caching is what you expect. Let's chat about it in the dev call next week - for now I'd leave it as is.
This PR fixes an issue in the MDim workflow when working in the ETL with staging servers. This is the setup that a lot of data managers use.
Lucas reported that when he makes changes in the ETL on his machine and runs the ETL update targeting a staging server, MDim views would not get updated.
It turns out that the reason this happens (and is different from the local workflow) is that on staging servers, /grapher/by-uuid/UUID.config.json is not handled by the mock site router but instead by CF functions (atm wrangler if I understand correctly, soon probably the CF preview deploy).
These CF functions have s-MaxAge set and so cache the results for a good while.
This PR enables the IsPreviewing flag that we use in the runSiteFooterScripts script and if it is set then reqeusts to the above url append ?nocache which then leads to the CF function (either wrangler or preview deploy) to request the freshest version from R2.
While I was at it I also reduced the cache times. These small json objects don't have to be cached for a full hour, 5 min should be plenty for prod.