Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support configuration for multiple projects #309

Merged
merged 126 commits into from
Mar 5, 2024
Merged

Support configuration for multiple projects #309

Changes from 1 commit
Commits
Show all changes
126 commits
Select commit Hold shift + click to select a range
0b5b694
Update config design diagram
milanmlft Feb 19, 2024
0b863de
Add example config yaml
milanmlft Feb 19, 2024
028be09
Add pydantic model for project configuration
milanmlft Feb 19, 2024
1cbedbc
Add tests for pydantic config model
milanmlft Feb 19, 2024
5a74734
Add config loading helper
milanmlft Feb 20, 2024
4fa71ef
fix: Remove `classmethod` decorators and tell ruff that pydantic's va…
milanmlft Feb 20, 2024
07188ff
Refactor config tests: split up invalid fields tests and create commo…
milanmlft Feb 20, 2024
1741888
Add some more asserts to config test and use `load_config()` helper
milanmlft Feb 20, 2024
96980e7
Add PyYAML as core dependency
milanmlft Feb 20, 2024
d656679
Add copyright headers
milanmlft Feb 20, 2024
2703bba
Move `load_config` to top of file
milanmlft Feb 20, 2024
158b663
feat: orthanc-anon project config (#312)
peshence Feb 22, 2024
582c995
Add project config template
milanmlft Feb 22, 2024
c24e86c
Add documentation for Parquet files and export process (#280)
milanmlft Feb 20, 2024
948414d
Correct instructions for editable installs (#314)
peshence Feb 22, 2024
f0fb25d
Add copyright header to config template
milanmlft Feb 22, 2024
40d7c6b
Add secret fetching from Azure keyvault
milanmlft Feb 22, 2024
60e95a9
Ignore local secrets
milanmlft Feb 22, 2024
e1aadb3
Add Azure dependenices
milanmlft Feb 22, 2024
1442354
Refactor: move FTPS setup to separet module
milanmlft Feb 22, 2024
6dba81e
Create AzureKeyVault class to handle project secrets fetching
milanmlft Feb 23, 2024
c68eaf9
Refactor `_connect_to_ftp` to take FTP settings as parameters
milanmlft Feb 23, 2024
5dad1be
Add abstract `Uploader` and `FTPSUploader` classes to define uploadin…
milanmlft Feb 23, 2024
58afdbb
Update upload tests to use `Uploader` interface
milanmlft Feb 23, 2024
b01fd16
Revert back to `pydantic.validator`
milanmlft Feb 23, 2024
1bc7871
Add `MockFTPSUploader` for tests
milanmlft Feb 23, 2024
f59a11b
Update tests with `MockFTPSUploader` class
milanmlft Feb 23, 2024
331d600
Partially added docs to describe project configuration
milanmlft Feb 23, 2024
697840f
Small code reordering
milanmlft Feb 26, 2024
6564127
Add `upload` method to `ParquetExports` class using the uploader stra…
milanmlft Feb 26, 2024
5a8a540
Update EHR API to use the new uploader strategy
milanmlft Feb 26, 2024
633ad54
Load project config only when necessary
milanmlft Feb 26, 2024
d1208db
Move `parquet_export` fixture out of conftest
milanmlft Feb 26, 2024
e8930d2
Update test instruction in EHR API
milanmlft Feb 26, 2024
5302276
Use uploader strategy in orthanc-anonn for DICOM uploads
milanmlft Feb 26, 2024
e155979
Update required environment variables
milanmlft Feb 26, 2024
0596205
Format docker-compose
milanmlft Feb 26, 2024
51ab9d9
Load `.secrets.env` in system tests
milanmlft Feb 26, 2024
da46f95
Add instructions for setting up project config
milanmlft Feb 26, 2024
325f344
Merge branch 'main' into milanmlft/232-multi-project-config
milanmlft Feb 26, 2024
2f20eb6
Update core docs
milanmlft Feb 26, 2024
06841ca
Set PROJECT_CONFIGS_DIR envvar for cli tests
milanmlft Feb 26, 2024
ea0aa8b
Use same config for `MockFTPSUploader` as for the `ftps_server` fixture
milanmlft Feb 26, 2024
df89d77
Fix core tests
milanmlft Feb 26, 2024
54d3cee
Format docker-compose
milanmlft Feb 26, 2024
15f6d5e
Merge branch 'main' into milanmlft/232-multi-project-config
milanmlft Feb 26, 2024
3244b57
Update cli docs
milanmlft Feb 26, 2024
52388cb
Fix: keyvault name envvar
milanmlft Feb 26, 2024
155d60a
Load local `.env` if it exists before setting `PROJECT_CONFIGS_DIR`
milanmlft Feb 26, 2024
ad758b6
Improve logging
milanmlft Feb 26, 2024
3a8e3c9
Merge branch 'main' into milanmlft/232-multi-project-config
milanmlft Feb 26, 2024
cb0c2f3
Move project configs in top-level `projects/` directory
milanmlft Feb 26, 2024
0de653a
Move exports dir in top-level projects
milanmlft Feb 26, 2024
1f58028
Allow setting an alias for Azure KV secret fetching
milanmlft Feb 26, 2024
93a13dc
Remove debugging message
milanmlft Feb 26, 2024
20617f7
Set Azure Keyvault credentials as secrets in CI
milanmlft Feb 26, 2024
1dc640c
Add Azure keyvault setup instructions
milanmlft Feb 26, 2024
ecaf450
Add sample `.secrets.env`
milanmlft Feb 27, 2024
72c21ea
Report which secret is missing
milanmlft Feb 27, 2024
7c973b0
Set default value for `PROJECT_CONFIGS_DIR` in sample `.env`
milanmlft Feb 27, 2024
e99d59c
Update destination options in README
milanmlft Feb 27, 2024
e1fd394
Update Azure KV setup instructions
milanmlft Feb 27, 2024
eee4caa
Raise wanring when destination not supported instead of error
milanmlft Feb 27, 2024
878e22d
Don't mention unsupported alternatives
milanmlft Feb 27, 2024
7815259
No need for line continuation
milanmlft Feb 27, 2024
3350bcc
Better function naming
milanmlft Feb 27, 2024
2865a91
Get FTP port from Azure Keyvault as well
milanmlft Feb 27, 2024
9e25175
Remove more references to `FTP_` environment variables
milanmlft Feb 27, 2024
60314f0
Fix formatting
milanmlft Feb 27, 2024
8edbe55
Forgot another renaming instance
milanmlft Feb 27, 2024
6a555e2
Remove more instances of `FTP_*` env variables
milanmlft Feb 27, 2024
56fad9d
No more FTP dummy service
milanmlft Feb 28, 2024
313cab9
Refer to pytest-pixl plugin in test docs
milanmlft Feb 28, 2024
5ccd23a
Get FTP port from Keyvault as well
milanmlft Feb 28, 2024
72553f4
Allow 'none' destination for uploading
milanmlft Feb 28, 2024
8fbb330
This string expansion isn't working
milanmlft Feb 28, 2024
795060d
Exports are now under `projects/exports`
milanmlft Feb 28, 2024
52a44ef
Docker mounts need absolute paths
milanmlft Feb 28, 2024
19537bf
Seems that we still need the `FTP_*` environment variables
milanmlft Feb 28, 2024
9410466
Update exports dir in gitignore
milanmlft Feb 28, 2024
48d1963
Mostly for debugging but probably a useful check to avoid surprises
milanmlft Feb 28, 2024
893ec56
Think I finally found the bug
milanmlft Feb 28, 2024
de3dbe0
Update docs
milanmlft Feb 28, 2024
26f77a3
Make sure exports dir exists in unit tests
milanmlft Feb 28, 2024
0e19bfe
Add check for public parquet exports
milanmlft Feb 28, 2024
fbd7b5b
Rename system test checks for exports
milanmlft Feb 28, 2024
b8b4f2a
refactor[config]: remove tag-operations from filenames
peshence Feb 29, 2024
bc13dab
Update README.md typo
peshence Feb 29, 2024
bcf0a5f
Update template_config.yaml
peshence Feb 29, 2024
dc84e0c
Update cli/README.md
peshence Feb 29, 2024
dcd5b8b
Switch back to `pydantic.field_validator`
milanmlft Feb 29, 2024
10f85db
Record pydantic version
milanmlft Feb 29, 2024
d0f2ac2
Revert "Format docker-compose"
milanmlft Feb 29, 2024
5779623
Cache secret fetching from AZ keyvault
milanmlft Feb 29, 2024
438144f
Set secret ENV variables for the whole system-test job
milanmlft Feb 29, 2024
83a1b5b
Fix Keyvault secret names in diagram
milanmlft Feb 29, 2024
ba7d180
Add more abstract methods into the base Uploader class
milanmlft Feb 29, 2024
f1e60ce
add slugify reference
peshence Feb 29, 2024
2a557b2
fix[dcmd]: type hints
peshence Feb 29, 2024
682a882
refactor[core]: define uploader subpackage
peshence Feb 29, 2024
c277423
fix[cli]: convert to_posix to str()
peshence Feb 29, 2024
e92819a
fix[imports]
peshence Feb 29, 2024
56c8394
fix docker compose
peshence Feb 29, 2024
c7a20a4
fix[core]: add back ftps uploader __init__
peshence Mar 1, 2024
b0725c0
Add static `create` method to `Uploader` base class to handle upload …
milanmlft Mar 1, 2024
b137f31
Make `uploader._base` non-private
milanmlft Mar 1, 2024
87e6064
Make `uploader.ftps` private
milanmlft Mar 1, 2024
c6f39e0
Update client code to use new `Uploader.create` method
milanmlft Mar 1, 2024
6ce589e
Move `FTPSUploader` to `core.uploader.base` and make private
milanmlft Mar 1, 2024
19ca997
Fix test
milanmlft Mar 1, 2024
6e62cb7
remove mention of non-existing options in enum and in diagram
peshence Mar 1, 2024
bc64636
clarify lastpass secrets note
peshence Mar 1, 2024
3fa5411
Limit scope of secret envvars
milanmlft Mar 1, 2024
cd0d134
Revert "Move `FTPSUploader` to `core.uploader.base` and make private"
milanmlft Mar 4, 2024
adee916
Use package-level `get_uploader` factory instead of static method
milanmlft Mar 4, 2024
991b17e
Update client code to use `get_uploader`
milanmlft Mar 4, 2024
f969d50
Merge branch 'main' into milanmlft/232-multi-project-config
milanmlft Mar 4, 2024
d908c56
Remove duplicate template
milanmlft Mar 4, 2024
2f97327
Merge branch 'main' into milanmlft/232-multi-project-config
milanmlft Mar 5, 2024
0baef01
Doc fix
milanmlft Mar 5, 2024
56044c8
Fix import
milanmlft Mar 5, 2024
c8a7ebb
Fix export dir for cli tests
milanmlft Mar 5, 2024
80e66fc
Fix: Delay checking for `export_dir` existence until it's needed
milanmlft Mar 5, 2024
18673ef
Switch back to editable installs
milanmlft Mar 5, 2024
c48e1be
fix orthanc raw?
peshence Mar 5, 2024
a85533e
Merge branch 'main' into milanmlft/232-multi-project-config
milanmlft Mar 5, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading