Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speed up submission result construction #4582

Closed
wants to merge 2 commits into from

Conversation

niknetniko
Copy link
Member

This pull request modifies how we construct results from the judge output for a submission.

First, it replaces the JSON Schema validator library with another one, which already speeds things up a lot (locally, it went from 5.5s to 0.5s).

Secondly, it modifies the logic to (in production) only validate once at the end, instead of for each command.

I am not 100% convinced if the second part is necessary, since the speedup is fairly small, but the code is uglier in my opinion (and there is then a difference between development/production in an important part of the code).

  • Tests were added

Closes #4579 .

In production environments, we no longer run JSON schema validation on
all the output of the judge. This saves about 0.5-1s of time on each
submission.

Instead, we run a validation after everything has been constructed,
since we still don't want invalid data in our systems.
@niknetniko niknetniko requested review from jorg-vr and a team as code owners April 19, 2023 11:24
@niknetniko niknetniko requested review from bmesuere and removed request for a team April 19, 2023 11:24
@niknetniko niknetniko changed the title Speed-up submission result construction Speed up submission result construction Apr 19, 2023
@niknetniko niknetniko requested a review from chvp April 19, 2023 11:25
Copy link
Contributor

@jorg-vr jorg-vr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am also not a fan of different behavior between development and production here.

As I understood this output is provided for judge developers. I am not sure if running dodona in development is a requirement to develop a judge (might as well test it on naos for example). And it might become rather unintuitive for someone developing a judge that their error messages in production are different from those in their development environment

My proposed solution would be:

  • only do final full validation (As production mode is written right now)
  • if errors are caught in this process, run the partial validations in an effort to provide a more useful error message

@niknetniko
Copy link
Member Author

  • if errors are caught in this process, run the partial validations in an effort to provide a more useful error message

If we go this route, since we don't actually show validations anywhere:

  1. If the invalid json is caught as an error (as production mode is written in this PR), we already show the same error as when we currently do, so there is no need to validate again.
  2. If the invalid json is only caught at the end, there no way to know which of the updates from the partial format caused the issue without saving all partial updates and the re-checking all of them, which seems like a lot of work to me.

@jorg-vr
Copy link
Contributor

jorg-vr commented Apr 19, 2023

2. If the invalid json is only caught at the end, there no way to know which of the updates from the partial format caused the issue without saving all partial updates and the re-checking all of them, which seems like a lot of work to me.

The work is storing a reference to judge_output. As all updates happen with cloned json, this will not be changed.

Then you need to run something like this in case of an error:

     split_jsons(judge_output).each do |json|
       unless PART_SCHEMER.valid?(json.deep_stringify_keys) || FULL_SCHEMER.valid?(json.deep_stringify_keys)
          raise ResultConstructorError.new(
            'Judge output is not a valid json',
            json.to_s
          )
        end
      end

@niknetniko niknetniko marked this pull request as draft April 19, 2023 12:40
@niknetniko
Copy link
Member Author

niknetniko commented Apr 19, 2023

For now, we'll begin with using the gem (#4583) and then think about not validating as much later (since there are various factors to consider).

@niknetniko niknetniko closed this Apr 19, 2023
@chvp chvp deleted the enhancement/faster-result-handling branch June 29, 2023 13:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Handling of partial output format is slow
2 participants