[SCHEMATIC-210] Add attribute to nones data model #1555
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
thomasyu888
changed the title
thomasyu888
reviewed
Dec 9, 2024
There was a problem hiding this comment.
🔥 LGTM - thanks for the quick work! I'll defer to @andrewelamb to final review and wait for tests to pass
|
Just a note that |
andrewelamb
approved these changes
Dec 9, 2024
|
@GiaJordan feel free to merge this into the feature branch, and I can finish it up with some tests I was working on. |
@andrewelamb / @GiaJordan since Bryan is OOO, could I count on the two of you to self organize and decide who adds this test? Thanks for raising that! Edit - I see Andrew is on it! I missed a comment ... |
@thomasyu888 The two tests that use this data model, |
|
thomasyu888
added a commit
that referenced
this pull request
Dec 13, 2024
… with the NA values argument (#1553) * Wrap pandas functions to support not including `None` with the NA values argument * Ignore types * pylint issues * ordering of ignore * Add to integration test to cover none in a manifest * Add additional test for manifest * [SCHEMATIC-210] Add attribute to nones data model (#1555) Update example_test_nones.model.csv component and add new invalid manifest with nones * first commit * ran black * add test for validateModelManifest * [SCHEMATIC-214] change data model and component (#1556) * add valid values to Patient attributes * update data model * add test manifests * update test for new model * update test for new valid value * change test to use new manifests * remove uneeded test file * revert file * revert file * change tests to use new manifests * remove uneeded manifests * ran black * add tests back in * ran black * revert manifest * Split up valid and errored test as separate testing functions * Remove unused import --------- Co-authored-by: Gianna Jordan <[email protected]> Co-authored-by: Andrew Lamb <[email protected]> Co-authored-by: Thomas Yu <[email protected]>
andrewelamb
added a commit
that referenced
this pull request
Dec 16, 2024
* add new tests * add unit tests * ran black * Update schematic/models/validate_attribute.py Co-authored-by: BryanFauble <[email protected]> * added tests * Update README.md * Update README.md * add unit tests * run black * Update README.md * temp commit * remove old tests * [FDS-2386] Synapse entity tracking and code concurrency updates (#1505) * [FDS-2386] Synapse entity tracking and code concurrency updates * ran black * Update CODEOWNERS * updated data model type rules to include error param * fix validate type attribute to use msg level param * added error handling * run black * create Node class * sat up Node class so that nodes with no displayName fields cause an error on creation * ran black * ran mypy * added new configs for CLI tests * added new manifests for testing CLI commands * automate manual CLI tests * ran black * Update CODEOWNERS * Update scan_repo.yml * Update .github/CODEOWNERS * Update .github/workflows/scan_repo.yml * Attach additional telemetry data to OTEL traces (#1519) * Attach additional telemetry data to OTEL traces * feat: added tracing for cross manifest validation and file name validation (#1509) * add tracing for GX validation * temp commit * Updating contribution doc to expect squash and merge (#1534) * [FDS-2491] Integration tests for Schematic API Test plan (#1512) Integration tests for Schematic API Test plan * [FDS-2500] Add Integration Tests for: Manifest Validation (#1516) * Add Integration Tests for: Manifest Validation * [FDS-2449] Lock `sphinx` version and update `poetry.lock` (#1530) Also install `typing-extensions` in the build * manual test files now being saved in manifests folder * manual test files now being saved in manifests folder * remove lines to delete json files that were under git control * ran black * add try finally blocks to remove created files * ran black * add lines to remove created json files * Update file annotation store process to require filename be present in order to annotate file * add lines to remove created json files * Revert "Update file annotation store process to require filename be present in order to annotate file" This reverts commit f57c718. * Don't attempt to annotate the table * add code in finally blocks to reset config to default values, when tests change them * complete submit manifest command test * ran black * add test for bug case * update test for table tidyness * remove unused import * remove etag column if already present when building temp file view * catch all exceptions to switch to sequential mode * update test for updated data * Revert "update test for updated data" This reverts commit 255e3c0. * Revert "catch all exceptions to switch to sequential mode" This reverts commit 68b0b24. * catch ValueErrors as well * Updates for integration test failures (#1537) * Updates for integration test failures, Config file reset and scope changes * add todos for removing config resets * [FDS-2525] Authenticated export of telemetry data (#1527) * Authenticated export of telemetry data, updating to HTTP otel library * temp reduce tests * restore tests * uncomment tests * redid how files are deleted, manual tests values are set * ran black * [SCHEMATIC-157] Make some dependencies required to avoid `schematic CLI` commands from potentially erroring when doing a pip install (#1540) * Make otel flash non-optional * Add dependencies as non-optional * Include schematic_api for now (#1547) * update toml version to 24.11.1 (#1548) * [SCHEMATIC-193] Support exporting telemetry data from GH integration test runs (#1550) * Support exporting telemetry data from GH run via access token retrieved via oauth2 * [SCHEMATIC-30, SCHEMATIC-200] Add version to click cli / use pathlib.Path module for checking cache size (#1542) * Add version to click cli * Add version * Run black * Reformat * Fix * Update schematic/schemas/data_model_parser.py * Add test for check_synapse_cache_size * Reformat * Fix tests * Remove unused parameter * Install all-extras for now * Make otel flash non-optional * Update dockerfile * Add dependencies as non-optional * Update pyproject toml * Fix trivy issue * Add service version * Run black * Move all utils.general tests into separate folder * Use pre-commit * Add updates to contribution doc * Fix * Add service version to log provider --------- Co-authored-by: BryanFauble <[email protected]> * [SCHEMATIC-212] Prevent traces from being combined (#1552) * Set instance id in github CI run, uninstrument flask auto during integration test run * [SCHEMATIC-163] Catch error when manifest is generated and existing one doesn't have `entityId` (#1551) * adds error handling * adds unit tests for _get_file_entityIds * updates error message * adds entityid check to parent func * updates docstring * [SCHEMATIC-183] Use paths from file view for manifest generation (#1529) source manifest file paths from synapse fileviews at generation * [SCHEMATIC-214] Wrap pandas functions to support not including `None` with the NA values argument (#1553) * Wrap pandas functions to support not including `None` with the NA values argument * Ignore types * pylint issues * ordering of ignore * Add to integration test to cover none in a manifest * Add additional test for manifest * [SCHEMATIC-210] Add attribute to nones data model (#1555) Update example_test_nones.model.csv component and add new invalid manifest with nones * first commit * ran black * add test for validateModelManifest * [SCHEMATIC-214] change data model and component (#1556) * add valid values to Patient attributes * update data model * add test manifests * update test for new model * update test for new valid value * change test to use new manifests * remove uneeded test file * revert file * revert file * change tests to use new manifests * remove uneeded manifests * ran black * add tests back in * ran black * revert manifest * Split up valid and errored test as separate testing functions * Remove unused import --------- Co-authored-by: Gianna Jordan <[email protected]> Co-authored-by: Andrew Lamb <[email protected]> Co-authored-by: Thomas Yu <[email protected]> * incremented packge version number * Update publish.yml * Update test.yml * Update api_test.yml * Update pdoc.yml * Update version.py * updates publish.yml (#1558) (#1561) Co-authored-by: Brad Macdonald <[email protected]> --------- Co-authored-by: BryanFauble <[email protected]> Co-authored-by: Jenny V Medina <[email protected]> Co-authored-by: Thomas Yu <[email protected]> Co-authored-by: Lingling <[email protected]> Co-authored-by: GiaJordan <[email protected]> Co-authored-by: Brad Macdonald <[email protected]> Co-authored-by: Gianna Jordan <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
From #1553
For an attribute to be part of a component, it needs to be listed in the component's
Depends Oncolumn. Without the addition and using the model with the addition the column won't be validated with the jsonschema validation(which would validate theValid Valuesaccording to the data model). The validation rules should still be performed.In the case of the model in #1553 while the manifest can have
Nonestrings included, the validation that these are actuallyNonestrings is not happening as part of the validation tests because the attribute has not been added to the component. This doesn't change the behavior of the valid manifest but when there's a manifest with an invalid value, if the attribute has not been added to the component then no error would be raised.