Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Validate test cases against a schema #778

Merged
merged 5 commits into from
May 6, 2024

Conversation

echeran
Copy link
Collaborator

@echeran echeran commented Apr 30, 2024

This PR is to add automation so that we guarantee that our test data remains in a consistent state. This prevents problems we had previous of ill-formed JSON and/or invalid contents according to the schema.

This PR moves the work from mradbourne#1 to the upstream, given that #767 was merged already without this addition in order to keep things simple for #767.

From the discussion there, we have a couple of options for how we go about automating the test validation:

  • Option 1: write a custom script to validate test cases' JSON files
    • Pros: Can add logic to pick up the latest schema version
    • Cons: We might have to maintain code in the future
  • Option 2: use an existing tool (ex: ajv-cli) to validate test cases' JSON files
    • Pros: Use an off the shelf tool in JS for JSON
    • Cons: We have to remember to maintain the CI workflow to when there is a new schema version

Aside: A decision point from #767 was the format of the version number for the schemas - should it be semantic versioning or an auto-increment integer or something else. #767 changed to auto-increment integer, which I have also have a slight preference for. We're all in agreement there, AFAICT.

Copy link
Collaborator

@eemeli eemeli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We do not need a custom script for this. My suggestion in mradbourne#1 (comment) works to replace the current behaviour, but without a custom Python script.

If we want to automatically use the latest schema version, here's how to do that:

ajv validate --spec=draft2020 \
-s $(ls -1v schemas/*/*schema.json | tail -1) \
-d 'tests/**/*.json'

@aphillips aphillips added test-suite LDML46 LDML46 Release (Tech Preview - October 2024) labels May 1, 2024
@echeran
Copy link
Collaborator Author

echeran commented May 1, 2024

If we want to automatically use the latest schema version, here's how to do that:

ajv validate --spec=draft2020 \
-s $(ls -1v schemas/*/*schema.json | tail -1) \
-d 'tests/**/*.json'

This updated command handles the question of getting the latest schema version. I'm all for not having to write & maintain something if we don't need to, so let's go with that.

Just to make sure that it correctly sorts numerically and doesn't fallback to lexicographic sorting, I did a mini test:

$ cd /tmp
$ mkdir -p schemas/v{1,2,3,4,5,6,7,8,9,10,11}.0.1
$ touch schemas/v{1,2,3,4,5,6,7,8,9,10,11}.0.1/tests.schema.json
$ mkdir -p schemas/v11.1.0
$ touch schemas/v11.1.0/tests.schema.json

$ ls -1v schemas/*/*schema.json 
schemas/v1.0.1/tests.schema.json
schemas/v2.0.1/tests.schema.json
schemas/v3.0.1/tests.schema.json
schemas/v4.0.1/tests.schema.json
schemas/v5.0.1/tests.schema.json
schemas/v6.0.1/tests.schema.json
schemas/v7.0.1/tests.schema.json
schemas/v8.0.1/tests.schema.json
schemas/v9.0.1/tests.schema.json
schemas/v10.0.1/tests.schema.json
schemas/v11.0.1/tests.schema.json
schemas/v11.1.0/tests.schema.json

I used semantic versions instead of the autoincrement integer versions that we are currently using just to cover all bases.

@echeran echeran requested a review from eemeli May 1, 2024 15:20
.github/workflows/validate_tests.yml Show resolved Hide resolved
.github/workflows/validate_tests.yml Outdated Show resolved Hide resolved
@aphillips aphillips merged commit d3a59da into unicode-org:main May 6, 2024
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
LDML46 LDML46 Release (Tech Preview - October 2024) test-suite
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants