Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Scheduler for Downloaders #183

Open
1 of 3 tasks
wjbmattingly opened this issue Dec 10, 2024 · 0 comments
Open
1 of 3 tasks

Scheduler for Downloaders #183

wjbmattingly opened this issue Dec 10, 2024 · 0 comments
Assignees
@kkdavis14 @wjbmattingly
Labels
development enhancement New feature to add to the code Low Low priority task v0.0.2 Change being developed for v.0.0.2

Comments

@wjbmattingly
Copy link

wjbmattingly commented Dec 10, 2024 •
edited
Loading

Description

While developing our downloaders, we noted that it would be useful to be able to automate them in some fashion via specified time intervals. While this works for some external sources that update regularly, not all external sources provide such consistency. Some, such as VIAF, update irregularly and need to be flagged for update.

Task

  • Identify how external data dumps are updated and at what intervals
  • Develop config schemas that capture this variance
  • Write a scheduler.py file in scripts/download/ that uses the configs to then call scripts/download/downloader.py for the specific source.
@wjbmattingly wjbmattingly added enhancement New feature to add to the code development v0.0.2 Change being developed for v.0.0.2 labels Dec 10, 2024
@wjbmattingly wjbmattingly self-assigned this Dec 10, 2024
@wjbmattingly wjbmattingly added the Low Low priority task label Dec 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kkdavis14 kkdavis14

@wjbmattingly wjbmattingly

Labels
development enhancement New feature to add to the code Low Low priority task v0.0.2 Change being developed for v.0.0.2
Projects
None yet
Milestone
No milestone
Development

When branches are created from issues, their pull requests are automatically linked.

refactor/v0.0.2 project-lux/data-pipeline
2 participants
@kkdavis14 @wjbmattingly