Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DM-43722 EFD Transform Implementation #18

Open
wants to merge 94 commits into
base: main
Choose a base branch
from
Open

DM-43722 EFD Transform Implementation #18

wants to merge 94 commits into from

Conversation

rcboufleur
Copy link

This pull request implements the basic EFD transformation, including:

  • EFT topic transformation: class and input generalization.
  • Summarization of EFD topics over exposures.
  • Summarization of EFD topics over visits.
  • Ingestion of results into a test database (SQLite).
  • Schema validation of the configuration file using Pydantic.
  • Documentation at the docstring level.

@rcboufleur rcboufleur requested a review from ktlim June 4, 2024 17:30
@glaubervila
Copy link

The following updates were added to this pull request:

  • The lsst_efd_client library has been replaced with InfluxDB API queries

glaubervila and others added 27 commits October 15, 2024 12:52
@glaubervila
Implement first version of EFD transformations
57e1b52
@rcboufleur @glaubervila
Update misssing values in the config file
83eadd3
@rcboufleur @glaubervila
Fix exception message for unimplemented dialect
07ea890
@rcboufleur @glaubervila
Refactor Aggregate class to handle column-wise operations
b7c30cc
@rcboufleur @glaubervila
Refactor Aggregate class to Summary for better clarity and consistency
5e587ab
@rcboufleur @glaubervila
Refactor ATAOS_correctionOffsets_w function to use mean
7134953
@glaubervila
the ExposureEfd table schema was changed to be compatible with config…
5cc2231 
…_LATISS.yml
@glaubervila
Added VisitEFD table and fixed empty values
8732968
@glaubervila
Upsert transactions now commit every 100 rows. Docstrings were added …
ead6f10 
…to dao classes.
@glaubervila
Refactor configuration file loading and validation
3816d47
@glaubervila
Changed lsst_efd_client -> InfluxDB API for topic queries
e1a94ee
@glaubervila
fixed lint
d65a16f
@rcboufleur @glaubervila
Lint fixes
f512a97
@rcboufleur @glaubervila
Additional parameters inserted in config yml
7225480
@rcboufleur @glaubervila
Lint fixes
11eba2d
@glaubervila
Added Dockerfile for Efd Transform
6fd11b4
@glaubervila
The main command was added to the dockerfile
70be00e
@rcboufleur @glaubervila
Implement retrieval of packed time series from InfluxDB using API que…
f663262 
…ries
@rcboufleur @glaubervila
Lint fixes
4366644
@glaubervila
Fixed lint
92462c8
@glaubervila
Fixed lint
45559a5
@rcboufleur @glaubervila
Implements new config file formats and validation
21dc88e
@rcboufleur @glaubervila
Implements unique topic querying to avoid duplicate queries
fb66a51
@glaubervila
Changed base image to w_2024_33
7c73a43
@glaubervila
Connection to usdf_efd api now uses environment variables
880e782
@glaubervila
Added copy of sqlite test database
4c7c5ad
@rcboufleur @glaubervila
Config changes refactored into transformations
0643536
glaubervila and others added 30 commits November 6, 2024 10:50
@glaubervila
Added permision
7a7f99d
@glaubervila
fixed dockerfile
df31088
@glaubervila
diferent schemas by instruments
9c38464
@rcboufleur
Update to properly handle schemas by instrument
e114cce
@rcboufleur
yaml extensions fixed
f4b416d
@glaubervila
Fixed hardcoded schema for Queue manager
626f148
@glaubervila
Added timewindow to queue manager table
927b852
@rcboufleur
Timewindow column added to the yaml files
1d7556a
@rcboufleur
Schema generator yaml files updated
160d265
@rcboufleur
Test files updated
38a8589
@rcboufleur
Update Dockerfile.efdtransform
0e53e5c
@rcboufleur
Update Dockerfile.efdtransform
4fd17e3
@rcboufleur
Merge remote-tracking branch 'origin/main' into tickets/DM-43722
b43bc79
@rcboufleur
Updates configuration files
92a2a4b
@rcboufleur
Updates schemas yaml files
0cd3d38
@rcboufleur
Implements unpivoted array support for array like results
8ff1413
@rcboufleur
Updates test files
6bc51b8
@rcboufleur
Implements new unpivoted tables
8ecd421
@rcboufleur
Updates column name in unpivoted tables
7cc9885
@rcboufleur
Rerun pre-commit
c889ec8
@rcboufleur
Fixes error on github actions
9de2cc6
@rcboufleur
Update summary.py
28a900c
@rcboufleur
Fix column name bug unpivoted data
59d2e2a
@rcboufleur
Realocates testing files
36fae9c
@rcboufleur
Fixes pytest for config_model
38c12fd
@rcboufleur
Fixes bugs in processing unpivoted results
8acfdd0
@rcboufleur
Updates temporary files
a5c1e66
@rcboufleur
Fix trailing spaces
84f9295
@rcboufleur
Fixes configuration files and schemas
cbf7755
@rcboufleur
Update Dockerfile.efdtransform
c15a890
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ktlim ktlim Awaiting requested review from ktlim

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@rcboufleur @glaubervila