Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automatically archive grapher datasets after a data update #3554

Open
pabloarosado opened this issue Nov 18, 2024 · 0 comments
Open

Automatically archive grapher datasets after a data update #3554

pabloarosado opened this issue Nov 18, 2024 · 0 comments
Labels

Comments

@pabloarosado
Copy link
Contributor

pabloarosado commented Nov 18, 2024

Currently, as part of our data update workflow, we need to manually archive a grapher dataset after we have created a new version for it.

But (in principle) this could be done automatically after chart-sync is completed: If we have moved all indicators from one grapher dataset to its corresponding new version (and we have also checked that the indicators of the old grapher dataset are not used in any indicator-based explorer) we could also automatically set isArchived=1 on the old grapher dataset.

NOTE: After discussion, we thought it could be better to do it as part of the archiving CLI tool in ETL.

  • @pabloarosado It could have a flag --grapher to also archive the corresponding grapher datasets.
  • @Marigold Once we know this works well, and takes care of all dependencies and possible issues, it could be used by chart-sync process.

Technical notes

  • This issue is mainly about archiving grapher datasets after an update (i.e. setting isArchive=1). We'd need to check that this dataset does not have charts (and, if possible, that it's not used by indicator-based explorers).
  • We could consider if it should also archive etl steps automatically (we should have a working CLI). However, it's not clear what to do if it fails.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant