Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support changing file types in scmd import #4805

Open
wants to merge 7 commits into
base: main
Choose a base branch
from

Conversation

LFISHER7
Copy link

@LFISHER7 LFISHER7 commented May 24, 2024

This updates the SCMD import script to support updating of monthly data subsets when they transition from provisional-->wip or wip-->finalised. Resolves https://github.com/ebmdatalab/op-hospitals/issues/20

To do so:

  1. First, gets dataset urls from the bsa open data API instead of scraping them.
  2. Inserts the file type into the scmd data
  3. Partitions on the date to make updating the data easier.
  4. Updates monthly partitions where the file type on BQ does not match the file type returned by the api

@LFISHER7 LFISHER7 force-pushed the update-scmd-import branch from c122849 to 9420e46 Compare May 24, 2024 14:46
@LFISHER7 LFISHER7 changed the title Use API for scmd import Support changing file types in scmd import May 31, 2024
@LFISHER7 LFISHER7 force-pushed the update-scmd-import branch from 8f48704 to 67b1db6 Compare May 31, 2024 14:06
LFISHER7 added 3 commits June 4, 2024 13:53
Out of date months are months for which the existing file type attached to a month in the scmd table does not match that taken from the url. File type changes only go in one direction so we don't have to worry about what the file type changes to.
This way we can easily overwrite data for a given month when the file type has changed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants