Add documentation for the PIXL database #306
Labels
documentation
Improvements or additions to documentation
Comments
This was referenced Feb 16, 2024
milanmlft
added a commit
that referenced
this issue
Feb 20, 2024
* Add documentation for Parquet files and export process * Formatting * Move `TODO` to issue #306 * Remove PR references * Formatting * Move specific details to `pixl_core` docs and add links * Update directory structure on the FTPS server * Formatting * Rename docs/data -> docs/file_types * Link to `file_types` documentation * Add directory structures to docstrings * Update upload.py Co-authored-by: Stef Piatek <[email protected]> * Fix docs link Co-authored-by: Jeremy Stein <[email protected]> * Clarify that the radiology reports go through Cogstack Co-authored-by: Jeremy Stein <[email protected]> * Add note about test files Co-authored-by: Jeremy Stein <[email protected]> --------- Co-authored-by: Stef Piatek <[email protected]> Co-authored-by: Jeremy Stein <[email protected]>
milanmlft
added a commit
that referenced
this issue
Feb 22, 2024
* Add documentation for Parquet files and export process * Formatting * Move `TODO` to issue #306 * Remove PR references * Formatting * Move specific details to `pixl_core` docs and add links * Update directory structure on the FTPS server * Formatting * Rename docs/data -> docs/file_types * Link to `file_types` documentation * Add directory structures to docstrings * Update upload.py Co-authored-by: Stef Piatek <[email protected]> * Fix docs link Co-authored-by: Jeremy Stein <[email protected]> * Clarify that the radiology reports go through Cogstack Co-authored-by: Jeremy Stein <[email protected]> * Add note about test files Co-authored-by: Jeremy Stein <[email protected]> --------- Co-authored-by: Stef Piatek <[email protected]> Co-authored-by: Jeremy Stein <[email protected]>
milanmlft
added a commit
that referenced
this issue
Mar 5, 2024
* Update config design diagram * Add example config yaml * Add pydantic model for project configuration * Add tests for pydantic config model * Add config loading helper * fix: Remove `classmethod` decorators and tell ruff that pydantic's validator is also a classmethod Setting the validators as `classmethod` will fail for some validations * Refactor config tests: split up invalid fields tests and create common base data object * Add some more asserts to config test and use `load_config()` helper * Add PyYAML as core dependency `config.py` needs it * Add copyright headers * Move `load_config` to top of file * feat: orthanc-anon project config (#312) ## Project configs ## Common - add env variable for project configs location - mount project configs location - make tag operations a list in config (limited to length 1 for now) - make config functions return `PixlConfig | Any` to get type hints - replace deprecated pydantic features - add utility function to load project config by slug only ### CLI - install cli as editable to find env files correctly - add function to check env for all env.sample keys ### orthanc-anon project config - orthanc anon now cofigurable (tag ops, modalities, destionation) - add query for project slug by non-hashed values - move anonymisation logic to dcmd package ### EHR project config - mark only processing tests with run_containers fixture - use project config to determine destination --------- Co-authored-by: Milan Malfait <[email protected]> * Add project config template * Add documentation for Parquet files and export process (#280) * Add documentation for Parquet files and export process * Formatting * Move `TODO` to issue #306 * Remove PR references * Formatting * Move specific details to `pixl_core` docs and add links * Update directory structure on the FTPS server * Formatting * Rename docs/data -> docs/file_types * Link to `file_types` documentation * Add directory structures to docstrings * Update upload.py Co-authored-by: Stef Piatek <[email protected]> * Fix docs link Co-authored-by: Jeremy Stein <[email protected]> * Clarify that the radiology reports go through Cogstack Co-authored-by: Jeremy Stein <[email protected]> * Add note about test files Co-authored-by: Jeremy Stein <[email protected]> --------- Co-authored-by: Stef Piatek <[email protected]> Co-authored-by: Jeremy Stein <[email protected]> * Correct instructions for editable installs (#314) * docs: better editable install instructions * docs: editable install of pytest-pixl * Add copyright header to config template * Add secret fetching from Azure keyvault * Ignore local secrets * Add Azure dependenices * Refactor: move FTPS setup to separet module * Create AzureKeyVault class to handle project secrets fetching * Refactor `_connect_to_ftp` to take FTP settings as parameters * Add abstract `Uploader` and `FTPSUploader` classes to define uploading interface * Update upload tests to use `Uploader` interface * Revert back to `pydantic.validator` `field_validator` is only available in `pydantic>-2.0` * Add `MockFTPSUploader` for tests Opted to mock out the `FTPSUploader` for tests instead of the `AzureKeyVault` to set the configuration fields directly. * Update tests with `MockFTPSUploader` class * Partially added docs to describe project configuration * Small code reordering * Add `upload` method to `ParquetExports` class using the uploader strategy * Update EHR API to use the new uploader strategy * Load project config only when necessary * Move `parquet_export` fixture out of conftest Fixture is only used in `test_upload` anyway, and it avoids an issue where importing the `ParquetEport` class requires the `PROJECT_CONFIGS_DIR` envvar to be set, but is only set after the imports in the conftest. * Update test instruction in EHR API * Use uploader strategy in orthanc-anonn for DICOM uploads * Update required environment variables - Remove the `FTP_` variables (except `FTP_PORT`), as these are now retrievd from the Azure Keyvault - Add Azure Keyvault variables to docker-compose * Format docker-compose * Load `.secrets.env` in system tests Contains the Azure Keyvault credentials * Add instructions for setting up project config * Update core docs * Set PROJECT_CONFIGS_DIR envvar for cli tests * Use same config for `MockFTPSUploader` as for the `ftps_server` fixture * Fix core tests * Format docker-compose * Update cli docs Running tests no longer requires the extra script. * Fix: keyvault name envvar * Load local `.env` if it exists before setting `PROJECT_CONFIGS_DIR` * Improve logging * Move project configs in top-level `projects/` directory * Move exports dir in top-level projects * Allow setting an alias for Azure KV secret fetching * Remove debugging message * Set Azure Keyvault credentials as secrets in CI * Add Azure keyvault setup instructions Copied from the Hasher API docs * Add sample `.secrets.env` * Report which secret is missing * Set default value for `PROJECT_CONFIGS_DIR` in sample `.env` * Update destination options in README * Update Azure KV setup instructions * Raise wanring when destination not supported instead of error * Don't mention unsupported alternatives * No need for line continuation * Better function naming * Get FTP port from Azure Keyvault as well * Remove more references to `FTP_` environment variables * Fix formatting * Forgot another renaming instance * Remove more instances of `FTP_*` env variables * No more FTP dummy service * Refer to pytest-pixl plugin in test docs * Get FTP port from Keyvault as well * Allow 'none' destination for uploading * This string expansion isn't working * Exports are now under `projects/exports` * Docker mounts need absolute paths * Seems that we still need the `FTP_*` environment variables I suspect because they are still used by the pytest plugin ftpserver fixture. * Update exports dir in gitignore * Mostly for debugging but probably a useful check to avoid surprises * Think I finally found the bug * Update docs * Make sure exports dir exists in unit tests * Add check for public parquet exports * Rename system test checks for exports * refactor[config]: remove tag-operations from filenames * Update README.md typo * Update template_config.yaml * Update cli/README.md Co-authored-by: Stef Piatek <[email protected]> * Switch back to `pydantic.field_validator` `pydantic.validator` is deprecated * Record pydantic version * Revert "Format docker-compose" This reverts commit 54d3cee. * Cache secret fetching from AZ keyvault * Set secret ENV variables for the whole system-test job This should omit the need to set them explicitly in `.secrets.env` * Fix Keyvault secret names in diagram * Add more abstract methods into the base Uploader class * add slugify reference * fix[dcmd]: type hints * refactor[core]: define uploader subpackage * fix[cli]: convert to_posix to str() * fix[imports] * fix docker compose * fix[core]: add back ftps uploader __init__ * Add static `create` method to `Uploader` base class to handle upload strategies * Make `uploader._base` non-private * Make `uploader.ftps` private * Update client code to use new `Uploader.create` method * Move `FTPSUploader` to `core.uploader.base` and make private Avoids circular imports * Fix test * remove mention of non-existing options in enum and in diagram * clarify lastpass secrets note * Limit scope of secret envvars * Revert "Move `FTPSUploader` to `core.uploader.base` and make private" This reverts commit 6ce589e. * Use package-level `get_uploader` factory instead of static method * Update client code to use `get_uploader` * Remove duplicate template --------- Co-authored-by: Peter Tsrunchev <[email protected]> Co-authored-by: Stef Piatek <[email protected]> Co-authored-by: Jeremy Stein <[email protected]>
Merged
milanmlft
added a commit
that referenced
this issue
Mar 27, 2024
Mostly as an encouragement to further flesh this out Fixes #306
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Definition of Done / Acceptance Criteria
docs/services/pixl_database.mdfile describing the PIXL databaseTesting
No response
Documentation
No response
Dependencies
No response
Details and Comments
Part of #236.
The text was updated successfully, but these errors were encountered: