Codecov Report

Attention: Patch coverage is 21.00000% with 79 lines in your changes missing coverage. Please review.

Project coverage is 61.72%. Comparing base (a45dccd) to head (0d31fb9).

@@            Coverage Diff             @@
##           master     #696      +/-   ##
==========================================
- Coverage   62.09%   61.72%   -0.38%     
==========================================
  Files          50       51       +1     
  Lines        9167     9266      +99     
==========================================
+ Hits         5692     5719      +27     
- Misses       3475     3547      +72     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

I did quite some progress on this PR in the last two days.

Few important first points:

• an endpoint in the dispatcher takes as arg the query in ADQL format, converts it to a postgresql query and runs it, the list of rows (query result) are returned
  • queries in ADQL format are translated in postgresql using the queryparser library
  • a special extension pg_spehere has be installed on the postgresql (strict version requirements apply here) in order to execute certain functions
• the idea is to use a view defined within the gallery DB which is on a MySQL DB
  • thanks to the pgloader the data can be easily transferred from one DB to the other

Why postgresql?

There is an extension, called mysql_sphere, but this is no longer developed and now it's quite outdated. However I gave it a try but I was not able to compile it. I had a brief interaction with the development team of queryparser and they advised against it.

On the contrary, I am testing pg_sphere (running postgresql version 16) and I currently am successful in running ADQL queries using the gallery DB.

I cant really understand this error : https://github.com/oda-hub/dispatcher-app/actions/runs/11054770284/job/30712623502?pr=696

Any ideas? It should not be related to this PR

I cant really understand this error : https://github.com/oda-hub/dispatcher-app/actions/runs/11054770284/job/30712623502?pr=696

Any ideas? It should not be related to this PR

caused by a conflict between the pytest and pytest-xdist libraries, introduced by the queryparser-python3 library (only for python 3.8)

Having tried to directly import the whole product gallery production DB, into the PostgreSQL (trying to create a PostgreSQL table out of the MySQL view, ie materializing) instance, resulted in the whole process to take an unsustainable amount of time, probably causing disruptions.

So I decided to try and skip the materialization step. With that, the import was very quick (less than a minute), and after some initial testing, I realized that querying the view was still an option as the time taken was acceptable.