Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support configuration for multiple projects #309

Merged
merged 126 commits into from
Mar 5, 2024

Conversation

milanmlft
Copy link
Member

@milanmlft milanmlft commented Feb 19, 2024

Current status

  • Created an example project config file in config-template/example-config.yml
  • Implemented pydantic model of project configuration
  • Added some tests to test the pydantic model
  • Refactored core.upload so it now has an Uploader interface for uploading DICOM and Parquet files to the right destination depending on the project's config
  • Fetch project secrets, such as credentials for the uploader, from Azure Key Vault

To do


Fixes #232

@peshence
Copy link
Contributor

Looks great! I've not used pydantic before, pretty cool!

I would consider having the tag operations be an array to allow for multilevel configuration sharing, as we could have e.g.

  • base profile for all modalities
  • extension profile for X-Rays
  • extension profile specific to this project

or have groups of tags, or something else. Just an idea though, not sure if it actually reflects the project needs, this should be enough for the time being

@milanmlft milanmlft mentioned this pull request Feb 20, 2024
3 tasks
peshence and others added 13 commits February 22, 2024 13:25
## Project configs
## Common
- add env variable for project configs location
- mount project configs location
- make tag operations a list in config (limited to length 1 for now)
- make config functions return `PixlConfig | Any` to get type hints
- replace deprecated pydantic features
- add utility function to load project config by slug only
### CLI
- install cli as editable to find env files correctly
- add function to check env for all env.sample keys
### orthanc-anon project config
- orthanc anon now cofigurable (tag ops, modalities, destionation)
- add query for project slug by non-hashed values
- move anonymisation logic to dcmd package
### EHR project config
- mark only processing tests with run_containers fixture
- use project config to determine destination

---------

Co-authored-by: Milan Malfait <[email protected]>
* Add documentation for Parquet files and export process

* Formatting

* Move `TODO` to issue #306

* Remove PR references

* Formatting

* Move specific details to `pixl_core` docs and add links

* Update directory structure on the FTPS server

* Formatting

* Rename docs/data -> docs/file_types

* Link to `file_types` documentation

* Add directory structures to docstrings

* Update upload.py

Co-authored-by: Stef Piatek <[email protected]>

* Fix docs link

Co-authored-by: Jeremy Stein <[email protected]>

* Clarify that the radiology reports go through Cogstack

Co-authored-by: Jeremy Stein <[email protected]>

* Add note about test files

Co-authored-by: Jeremy Stein <[email protected]>

---------

Co-authored-by: Stef Piatek <[email protected]>
Co-authored-by: Jeremy Stein <[email protected]>
* docs: better editable install instructions

* docs: editable install of pytest-pixl
`field_validator` is only available in `pydantic>-2.0`
Copy link
Contributor

@stefpiatek stefpiatek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really nice, thanks both 🎉

.github/workflows/main.yml Outdated Show resolved Hide resolved
README.md Show resolved Hide resolved
docker-compose.yml Show resolved Hide resolved
docs/design/diagrams/pixl-multi-project-config.png Outdated Show resolved Hide resolved
pixl_core/src/core/project_config.py Outdated Show resolved Hide resolved
@milanmlft milanmlft enabled auto-merge (squash) March 5, 2024 14:33
@milanmlft milanmlft merged commit 7e5a28b into main Mar 5, 2024
8 checks passed
@milanmlft milanmlft deleted the milanmlft/232-multi-project-config branch March 5, 2024 18:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Configuration for multiple projects
3 participants