-
-
Notifications
You must be signed in to change notification settings - Fork 23
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Speed up submission result construction #4582
Conversation
In production environments, we no longer run JSON schema validation on all the output of the judge. This saves about 0.5-1s of time on each submission. Instead, we run a validation after everything has been constructed, since we still don't want invalid data in our systems.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am also not a fan of different behavior between development and production here.
As I understood this output is provided for judge developers. I am not sure if running dodona in development is a requirement to develop a judge (might as well test it on naos for example). And it might become rather unintuitive for someone developing a judge that their error messages in production are different from those in their development environment
My proposed solution would be:
- only do final full validation (As production mode is written right now)
- if errors are caught in this process, run the partial validations in an effort to provide a more useful error message
If we go this route, since we don't actually show validations anywhere:
|
The work is storing a reference to Then you need to run something like this in case of an error: split_jsons(judge_output).each do |json|
unless PART_SCHEMER.valid?(json.deep_stringify_keys) || FULL_SCHEMER.valid?(json.deep_stringify_keys)
raise ResultConstructorError.new(
'Judge output is not a valid json',
json.to_s
)
end
end |
For now, we'll begin with using the gem (#4583) and then think about not validating as much later (since there are various factors to consider). |
This pull request modifies how we construct results from the judge output for a submission.
First, it replaces the JSON Schema validator library with another one, which already speeds things up a lot (locally, it went from 5.5s to 0.5s).
Secondly, it modifies the logic to (in production) only validate once at the end, instead of for each command.
I am not 100% convinced if the second part is necessary, since the speedup is fairly small, but the code is uglier in my opinion (and there is then a difference between development/production in an important part of the code).
Closes #4579 .