BI-Library: CDS. New items in the institutional repository

[ErnestaP]

KPIs descriptions can be found here

• The queries are not full. Ask Salome, Paulina, and Lydia for clarification. We might use the output from the CDS script. More information in the ticket link below.
  Queries: Records harvested by INSPIRE through arXiv:
  037:'arXiv' and year:YYYY not 980:hidden
  Year 2021: 037:'arXiv' and year:2021 not 980:hidden
  https://cds.cern.ch/search?ln=en&p=037%3A%27arXiv%27+and+year%3A2021+not+980%3Ahidden&action_search=Search&op1=a&m1=a&p1=&f1=&c=CERN+Document+Server&sf=&so=d&rm=&rg=10&sc=1&of=hb&wl=0
  Year 2022: 037:'arXiv' and year:2022 not 980:hidden
  https://cds.cern.ch/search?ln=en&p=037%3A%27arXiv%27+and+year%3A2022+not+980%3Ahidden&action_search=Search&op1=a&m1=a&p1=&f1=&c=CERN+Document+Server&sf=&so=d&rm=&rg=10&sc=1&of=hb&wl=0
  Year 2023: 037:'arXiv' and year:2023 not 980:hidden
  https://cds.cern.ch/search?ln=en&p=037%3A%27arXiv%27+and+year%3A2023+not+980%3Ahidden&action_search=Search&op1=a&m1=a&p1=&f1=&c=CERN+Document+Server&sf=&so=d&rm=&rg=10&sc=1&of=hb&wl=0

  Records harvested by INSPIRE through curators:
  035:'oai:inspirehep.net' not 037:'arXiv' and year:YYYY

  Year 2021: 035:'oai:inspirehep.net' not 037:'arXiv' and year:2021
  https://cds.cern.ch/search?ln=en&p=035%3A%27oai%3Ainspirehep.net%27+not+037%3A%27arXiv%27+and+year%3A2021&action_search=Search&op1=a&m1=a&p1=&f1=&c=CERN+Document+Server&sf=&so=d&rm=&rg=10&sc=1&of=hb&wl=0
  Year 2022: 035:'oai:inspirehep.net' not 037:'arXiv' and year:2022
  https://cds.cern.ch/search?ln=en&p=035%3A%27oai%3Ainspirehep.net%27+not+037%3A%27arXiv%27+and+year%3A2022&action_search=Search&op1=a&m1=a&p1=&f1=&c=CERN+Document+Server&sf=&so=d&rm=&rg=10&sc=1&of=hb&wl=0
  Year 2023: 035:'oai:inspirehep.net' not 037:'arXiv' and year:2023
  https://cds.cern.ch/search?ln=en&p=035%3A%27oai%3Ainspirehep.net%27+not+037%3A%27arXiv%27+and+year%3A2023&action_search=Search&op1=a&m1=a&p1=&f1=&c=CERN+Document+Server&sf=&so=d&rm=&rg=10&sc=1&of=hb&wl=0

• Ask Paulina, how often we want to retrieve this data Follow up: Every year

---

Setup Airflow workflow:

• Add tables migration
• Write a DAG

---

DB

• Connect Airflow workflow to DB tables
• Connect Supserset BI to DB tables

---

Feedback regarding results

• Ask Salome, Paulina, Lydia for feedback regarding the results we are getting from CDS. Adjust if needed

---

Deploy

• Deploy to QA
• Deploy to PROD

[ErnestaP]

Ticket: https://cern.service-now.com/service-portal?id=ticket&table=u_request_fulfillment&n=RQF2659173
Considering: solution, that CDS will run a script on their side and we will upload the result in db