Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Harvesting fixes #633

Closed
1 task done
drjova opened this issue Dec 12, 2024 · 1 comment
Closed
1 task done

Harvesting fixes #633

drjova opened this issue Dec 12, 2024 · 1 comment
Assignees
@DonHaul
Labels
project: backoffice type: bug Something isn't working
Milestone
data MVP

Comments

@drjova
Copy link
Contributor

drjova commented Dec 12, 2024 •
edited by DonHaul
Loading

There are some issues with the metadata from the harvesting

  • observables keywords are messed up, they should be split in multiple keywords, one per element
@drjova drjova added project: backoffice type: bug Something isn't working labels Dec 12, 2024
@drjova drjova added this to the data MVP milestone Dec 12, 2024
@DonHaul
Copy link
Collaborator

DonHaul commented Dec 12, 2024 •
edited
Loading

Additional things to check:

  • add parameter for setting since when we want to harvest.
  • make sure that collected_ids always fetches the data of the day before the first execution. i.e. if it was first run on 14/12 but it failed, if its rerun on 17/12 is should still fetch the records of 13/12 not of 16/12

@DonHaul DonHaul self-assigned this Dec 20, 2024
DonHaul added a commit to DonHaul/inspirehep that referenced this issue Dec 20, 2024
@DonHaul
workflows: fix data harvesting
7ff5a74 
- correctly format observables
- add date parameter to dag
- fetch collections ids relative to first run

* ref: cern-sis/issues-inspire/issues/633
@DonHaul DonHaul mentioned this issue Dec 20, 2024
Merged
DonHaul added a commit to DonHaul/inspirehep that referenced this issue Dec 20, 2024
@DonHaul
workflows: fix data harvesting
2e952a4 
- correctly format observables
- add date parameter to dag
- fetch collections ids relative to first run

* ref: cern-sis/issues-inspire/issues/633
DonHaul added a commit to DonHaul/inspirehep that referenced this issue Dec 20, 2024
@DonHaul
workflows: fix data harvesting
33a1957 
- correctly format observables
- add date parameter to dag
- fetch collections ids relative to the logical date

* ref: cern-sis/issues-inspire/issues/633
DonHaul added a commit to DonHaul/inspirehep that referenced this issue Dec 20, 2024
@DonHaul
workflows: fix data harvesting
4cacd48 
- correctly format observables
- add date parameter to dag
- fetch collections ids relative to the logical date

* ref: cern-sis/issues-inspire/issues/633
drjova pushed a commit to inspirehep/inspirehep that referenced this issue Jan 6, 2025
@DonHaul @drjova
workflows: fix data harvesting
6746be5 
- correctly format observables
- add date parameter to dag
- fetch collections ids relative to the logical date

* ref: cern-sis/issues-inspire/issues/633
@DonHaul DonHaul closed this as completed Jan 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@DonHaul DonHaul

Labels
project: backoffice type: bug Something isn't working
Projects
None yet
Milestone
data MVP
Development

When branches are created from issues, their pull requests are automatically linked.

refs/heads/633-harvesting-fixes inspirehep/inspirehep
2 participants
@drjova @DonHaul