Validate test cases against provided schema #1
Conversation
There was a problem hiding this comment.
This can and should be submitted as a separate, follow-on PR to unicode-org#767, rather than getting the former blocked by new concerns, such as whether a custom Python script is the right tool for validating JSON test schemas.
| - name: Validate tests for schema v0.0.1 | ||
| run: | | ||
| cd test | ||
| python3 validate.py --schema-dir ./schemas/v0-0-1/ |
There was a problem hiding this comment.
With something like this, we don't need to define a custom validator script:
| - name: Validate tests for schema v0.0.1 | |
| run: | | |
| cd test | |
| python3 validate.py --schema-dir ./schemas/v0-0-1/ | |
| - run: npm install --global ajv-cli | |
| - run: > | |
| ajv validate --spec=draft2020 | |
| -s schemas/v0-0-1/tests.schema.json | |
| -d 'tests/**/*.json' | |
| working-directory: ./test |
There was a problem hiding this comment.
I like the idea of reusing a tool and not having to write a script. But having to enumerate each nested directory is annoying. Also, do we want a command to automatically figure out our latest schema version? The ajv tool only takes a hard coded schema version.
There was a problem hiding this comment.
Oh, the ** segment in the glob pattern takes care of nested directories, so that's not an issue. The only issue is whether we want to have this CI job automatically take the latest schema version, or if we want to manually maintain the CI job to reflect the latest schema every time we change it.
|
Test validation in CI sounds good, @echeran - thanks. Update: The new branch is https://github.com/mradbourne/message-format-wg/tree/validate-tests |
|
Since I moved this PR to upstream at unicode-org#778, I will close this PR now. |
This PR attempts to update unicode-org#767 with automation to ensure that for now & going forward, our test data is stored in well-formed JSON that adheres to the test schema.
The test schema number is currently hard-coded, but if this is good, we can merge this now and look at a followup PR to discern the latest semver automatically from the directory names.