Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate monkeypox from ops to ETL #3034

Closed
Marigold opened this issue Jul 29, 2024 · 3 comments
Closed

Migrate monkeypox from ops to ETL #3034

Marigold opened this issue Jul 29, 2024 · 3 comments

Comments

@Marigold
Copy link
Collaborator

We're updating monkeypox data from owid-admin-prod and schedule it with CRON. It's hidden and shouldn't run there, so the best would be migrating it to ETL.

@larsyencken
Copy link
Collaborator

larsyencken commented Aug 1, 2024

From triage discussion:

  • How should it get run?

    • The data in Github hasn't updated in 2 months -- is it still running?
    • This is super related to Add cron-like scheduling into the ETL for regular updates #3016, and if we want it in the ETL we should decide the approach there first
    • It's a reasonable candidate for the ETL since it doesn't do scraping
    • It would also be ok to self-update using Github actions, then use the repo as input to the ETL (a github step)
  • How should we keep the repo CSV up to date

    • Pablo suggested an external channel, which exists already; the code there publishes to Github as a side-effect
    • We could put these into the DAG as an explicit target

Suggest to look at again after #3016

@larsyencken
Copy link
Collaborator

Update from @Marigold :

  • It's going well, and it's not blocked on the regular updates discussion
  • The initial approach is to have a common pipeline for all frequent updates, run daily in Buildkite

@Marigold
Copy link
Collaborator Author

Mpox explorer is now powered by dataset from ETL. We still have to merge changes that would be committing data to github repo.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants