Bug: Regrade Error with "Each attempt builds on the last" = "Yes" and Updated CodeRunner Question Version

[FelixDiLenarda]

Description of the Issue

We have encountered an issue in Moodle 4.0.5 with a CodeRunner question when "Question behaviour -> Each attempt builds on the last" is set to "Yes". After updating the CodeRunner question (to make a previously incorrect answer become correct), regrading leads to inconsistent results across attempts. It appears that if no change was made to the CodeRunner answer in a subsequent attempt, that attempt still uses the old question version for grading—even though the quiz review screen incorrectly displays it as if the new version is being used.

Below are the detailed steps to reproduce this problem:

Steps to Reproduce

1. Create a Quiz in Moodle 4.0.5

• In Settings under Question behaviour, set Each attempt builds on the last to Yes.

2. Add Two Questions

------- 1. A CodeRunner question (e.g., Python3) with the following:

def v1(parameter):
   if parameter:
       print("TRUE")
   else:
       print("FALSE")

Test cases:
Test 1: v1(True), expected output: TRUE
Test 2: v1(True), expected output: FALSE

------- 2. Another question (type is irrelevant, but must be a separate question).

3. Attempt 1 (as a student)

• For the CodeRunner question, provide a deliberately incorrect answer, e.g.: (Functionname v2 instead of v1)

def v2(parameter):
    if parameter:
        print("TRUE")
    else:
        print("FALSE")

• For the second question, provide a wrong answer as well.
• Save the attempt.

4. Attempt 2 (same student, Re-attempt Quiz)

• Because "Each attempt builds on the last" is Yes, the previous answers are automatically loaded.
• Leave the CodeRunner answer as is (the same def v2(parameter): ...).
• Change the second question’s answer to the correct one.
• Save the second attempt.

5. Edit the CodeRunner Question as Admin or Trainer

• Modify the question so that the previously incorrect def v2(parameter): ... answer is now considered correct.
• In the Quiz settings (under the "Questions" tab for the quiz), ensure the updated version of this CodeRunner question is being used.
• Rationale: This simulates fixing a mistake in the original question such that the student’s original CodeRunner code (v2) should now be correct.

6. View Attempts and Regrade

• As Admin, go to 'Results' to see the 'Attempts: 2' Initially, both attempts still show the old grading:

Attempt 1: CodeRunner question = 0 points, Second question = 0 points
Attempt 2: CodeRunner question = 0 points, Second question = full points

• Use Regrade attempts... to regrade both attempts.
• Observed Outcome after regrading:

Attempt 1: CodeRunner question is now marked correct (full points), Second question remains 0 points.
Attempt 2: CodeRunner question still incorrectly marked wrong (0 points), Second question is correct (full points).

• Reviewing Each Attempt: The Quiz UI shows that version 2 of the CodeRunner question (the updated one) is used for both attempts. However, Attempt 2 still yields 0 points for the CodeRunner answer even though the displayed question version is updated (and v2 should be marked correct now).

Expected Behavior

Both Attempt 1 and Attempt 2 should be regraded against the new version of the CodeRunner question.
Since the student’s def v2(parameter): ... answer is now correct in the updated version, both attempts should show full points for the CodeRunner question.

Actual Behavior

After regrading, Attempt 1 is corrected to full points, but Attempt 2 stays at 0 points.
The review screen for Attempt 2 misleadingly shows that version 2 of the question is used, yet the grading result aligns with the old version (0 points).

Additional Notes

The only difference between Attempt 1 and Attempt 2 is that in Attempt 2, no change was made to the CodeRunner answer before saving.
This behavior only appears when "Question behaviour → Each attempt builds on the last" = Yes is enabled. If that setting is disabled, it seems to behave as expected.

Conclusion

It appears that during the regrade process, attempts that had no change to the CodeRunner response in subsequent attempts might still use the old grading logic, causing inconsistent grading outcomes. This is exacerbated by the UI incorrectly indicating the new question version is in effect, even though the old version’s grading is actually used.

I added an quiz mbz file with the needed questions and the 2 attempts of a user. You can see the results there.
If you use this file to reproduce the bug,:
First delete the attempts and use v1 of the coderunner question, do 2 Attempts like described above and than regrade with v2...

QuizCR.zip

[FelixDiLenarda]

I tested this with a clean moodle 405 install and the Canterburry Jobe server aswell as our own 404 moodle instance and our own Jobe setup.

[trampgeek]

Thank you for the very thorough bug report - a great help.

The issue turns out to be with the companion plugin qbehaviour_adaptive_adapted_for_coderunner.. When a new quiz attempt building on the last was started, cached grading information, which CodeRunner uses to prevent expensive regrading, was being pulled in as well from the last graded attempt.

We've pushed a bug fix to github and updated the Moodle plugins repo as well.

We note as an aside that if you edit the question and the student re-attempts the quiz before you do the regrade, they will still be seeing the same version of the question that they saw before. But this is a Moodle feature, not CodeRunner's.