-
-
Notifications
You must be signed in to change notification settings - Fork 34
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Validate test cases against a schema #778
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We do not need a custom script for this. My suggestion in mradbourne#1 (comment) works to replace the current behaviour, but without a custom Python script.
If we want to automatically use the latest schema version, here's how to do that:
ajv validate --spec=draft2020 \
-s $(ls -1v schemas/*/*schema.json | tail -1) \
-d 'tests/**/*.json'
This updated command handles the question of getting the latest schema version. I'm all for not having to write & maintain something if we don't need to, so let's go with that. Just to make sure that it correctly sorts numerically and doesn't fallback to lexicographic sorting, I did a mini test: $ cd /tmp
$ mkdir -p schemas/v{1,2,3,4,5,6,7,8,9,10,11}.0.1
$ touch schemas/v{1,2,3,4,5,6,7,8,9,10,11}.0.1/tests.schema.json
$ mkdir -p schemas/v11.1.0
$ touch schemas/v11.1.0/tests.schema.json
$ ls -1v schemas/*/*schema.json
schemas/v1.0.1/tests.schema.json
schemas/v2.0.1/tests.schema.json
schemas/v3.0.1/tests.schema.json
schemas/v4.0.1/tests.schema.json
schemas/v5.0.1/tests.schema.json
schemas/v6.0.1/tests.schema.json
schemas/v7.0.1/tests.schema.json
schemas/v8.0.1/tests.schema.json
schemas/v9.0.1/tests.schema.json
schemas/v10.0.1/tests.schema.json
schemas/v11.0.1/tests.schema.json
schemas/v11.1.0/tests.schema.json I used semantic versions instead of the autoincrement integer versions that we are currently using just to cover all bases. |
Co-authored-by: Eemeli Aro <[email protected]>
This PR is to add automation so that we guarantee that our test data remains in a consistent state. This prevents problems we had previous of ill-formed JSON and/or invalid contents according to the schema.
This PR moves the work from mradbourne#1 to the upstream, given that #767 was merged already without this addition in order to keep things simple for #767.
From the discussion there, we have a couple of options for how we go about automating the test validation:
ajv-cli
) to validate test cases' JSON filesAside: A decision point from #767 was the format of the version number for the schemas - should it be semantic versioning or an auto-increment integer or something else. #767 changed to auto-increment integer, which I have also have a slight preference for. We're all in agreement there, AFAICT.