Handling of partial output format is slow

[niknetniko]

From executing https://dodona.ugent.be/nl/courses/2263/activities/649388223/ on my local machine:

Looking at the code, some obvious things seem the validation of each JSON object individually. The same exercise without validating each command gives:

[bmesuere]

Validation of non-partial output can also take a while. This is from the legacy python judge from an exercise with 150 tests:

For exercises with smaller json output, the construction time is minimal (a few ms). It thus seems that the overhead for calling the validator is low, but runtimes quickly increase with the size of the json to check.

[niknetniko]

Using the json_schemer gem (for the partial format):

Modifying the code to only validate the result at the end gives:

I would thus propose using the second approach: validate once at the end to ensure we don't get invalid data in the database, and put a rescue block around the partial updates to catch and handle those errors.

This does mean we might be less strict in checking, since each command is not checked individually in the partial format.

[bmesuere]

sounds good 👍

[chvp]

If the checking of the full format fails, could we still check each individual partial command after? For judge development that seems quite important.

I'm also not sure how robust the result constructor is when the partial commands are invalid.

[bmesuere]

Can't we keep the current behaviour in development, or do we also want these granular checks in production?

[chvp]

Keeping the current behaviour in development also seems fine.

[bmesuere]

Switching to the new validator has improved things, but it can still be an issue for certain exercises: https://dodona.ugent.be/nl/submissions/13892016/

I propose to still implement the second part of the original PR.

Do note that there are 1000+ tests in this exercise, so this is also a case of users "consuming" all performance gains by creating more/bigger tests.

[jorg-vr]

The fixing pull request was reverted. see #4647

[bmesuere]

Version 2 of the json schemer gem (merged in #4913) had additional performance improvements.
The legacy exercise with a validation time of 4.75 seconds in https://dodona.be/nl/submissions/13892016/ dropped to 0.58 seconds in https://dodona.be/nl/courses/2263/activities/1565418465/

There is thus no real need to implement additional strategies to speed this up. If others agree, this issue can be closed.

(do note that the prepare time is quite high again)