Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding functionality to read from local database, unit tests #90

Merged
merged 18 commits into from
Mar 28, 2024

Conversation

astronomerritt
Copy link
Collaborator

@astronomerritt astronomerritt commented Mar 19, 2024

Fixes #75.

  • Constructor added to AdlerPlanetoid object, AdlerPlanetoid.construct_from_SQL(), which allows population from a SQL database.
  • Updated dataclass_utilities.get_data_table() to create the data table from a local SQL database. Currently it does this via Pandas because this is the easiest way I know, and Pandas is already a dependency via lsst-rsp. Probably not optimal behaviour.
  • The code in AdlerPlanetoid now allows specifying what database/schema to pull the tables from. The default is "dp03_catalogs_10yr" when constructing from the RSP and None when pulling from a local database. This is futureproofing, as obviously we won't be pulling data from "dp03_catalogs_10yr" forever... but it's also the only way I could get the testing database to work :D
  • Several small tweaks mostly to do with the fact that Pandas requires case consistency and SQL doesn't give a crap.

Fixes #31.

  • Testing data can be found in ./tests/data/testing_database.db. This is simply all of the data in DP0.3 for the ssObjectId "8268570668335894776", which we've been using for testing anyway.
  • Changed the .gitignore so it doesn't ignore .db files, which caused me some confusion earlier.

Fixes #76.

  • Every class/method/function in the dataclasses folder is now covered by unit tests.
  • There's now a utilities folder which currently contains some utility functions for tests which can be used within unit tests to locate the testing data using relative paths. This means the unit tests will always run no matter what system they're being run on, including GitHub CI.
  • Edited the GitHub workflow to pip install into an editable environment so relative paths remain as expected and the tests can locate the testing data correctly.
  • Added a .coveragerc file so functions commented with # pragma: no cover are ignored by code coverage tools.
  • Used warnings to suppress an entirely useless FutureWarning on a Pandas function instead of using Pandas' own in-built suppression, which doesn't work in whatever version of Pandas runs in Python 3.8.

Also a few minor fixes - some missing lambdas, tweaks to error messages.

Review Checklist for Source Code Changes

  • Does pip install still work?
  • Have you written a unit test for any new functions?
  • Do all the units tests run successfully?
  • Does adler run successfully on a test set of input files/databases?
  • Have you used black on the files you have updated to confirm python programming style guide enforcement?

@codecov-commenter
Copy link

codecov-commenter commented Mar 19, 2024

Codecov Report

Attention: Patch coverage is 94.02985% with 4 lines in your changes are missing coverage. Please review.

Project coverage is 84.71%. Comparing base (b005e3e) to head (cd1b10c).

Files Patch % Lines
src/adler/dataclasses/AdlerPlanetoid.py 80.95% 4 Missing ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@             Coverage Diff             @@
##             main      #90       +/-   ##
===========================================
+ Coverage   71.34%   84.71%   +13.37%     
===========================================
  Files           4       14       +10     
  Lines         164      373      +209     
===========================================
+ Hits          117      316      +199     
- Misses         47       57       +10     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@astronomerritt
Copy link
Collaborator Author

Python 3.8 tests are failing because we require a later Pandas version than the latest 3.8-compatible version. I might be able to code my way around this, but if not I'll just set a requirement for Python >= 3.9 and change the tests to test 3.9, 3.10 and 3.11.

@astronomerritt astronomerritt marked this pull request as ready for review March 26, 2024 15:57
@astronomerritt astronomerritt requested a review from jrob93 March 26, 2024 15:57
@jrob93
Copy link
Collaborator

jrob93 commented Mar 27, 2024

Hey, very awesome! I have just added an example notebook to the PR to demo querying the db file. Hope that's okay. Otherwise it all looks good to me

@astronomerritt astronomerritt merged commit 7f0094b into main Mar 28, 2024
6 checks passed
@astronomerritt astronomerritt deleted the local_tests branch March 28, 2024 10:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants