Development: Fix issues in the user data export

[tobias-lippert]

Checklist

General

• I tested all changes and their related features with all corresponding user types on a test server.
• Language: I followed the guidelines for inclusive, diversity-sensitive, and appreciative language.
• I chose a title conforming to the naming conventions for pull requests.

Server

• Important: I implemented the changes with a very good performance and prevented too many (unnecessary) database calls.
• I followed the coding and design guidelines.
• I added multiple integration tests (Spring) related to the features (with a high test coverage).
• I documented the Java code using JavaDoc style.

Client

• Important: I implemented the changes with a very good performance, prevented too many (unnecessary) REST calls and made sure the UI is responsive, even with large data.
• I followed the coding and design guidelines.
• Following the theming guidelines, I specified colors only in the theming variable files and checked that the changes look consistent in both the light and the dark theme.
• I added multiple integration tests (Jest) related to the features (with a high test coverage), while following the test guidelines.
• I added authorities to all new routes and checked the course groups for displaying navigation elements (links, buttons).
• I documented the TypeScript code using JSDoc style.
• I added multiple screenshots/screencasts of my UI changes (not really applicable)
• I translated all newly inserted strings into English and German.

Motivation and Context

I still had a couple of small things to improve for the data export on my list and I wanted to address them.
Additionally, while testing this feature on production, it turned out that the current download solution is not viable because the export can become quite large and the download fails because it runs into timeouts.
Moreover, this PR fixes a small issue that we no longer include exam information whether an exam was passed and with which grade if no gradingscale for this exam is defined.

Description

• Added additional subdirectories to separate exams and exercises in a course
• Added an exercise prefix to exercise directories to be consistent with exams and courses
• Do no longer download the file from the server using the client and then hand it over to the browser but instead hand it over directly to the browser.
• Checks if an grading scale for an exam exists and only if it does, add information about grades and passed/not passed
• Add general exam information only if it is available.
• Use an executor service to create exports in parallel. We allow 60 minutes to create the data exports. Afterward, all creation processes in progress are cancelled and will be started again the next time the cron job is run.
• Only include automatic results for programming exercises before the assessment due date if user is not at least a tutor in the course
• consider due date instead of assessment due for quiz exercises

Steps for Testing

For testing purposes, data exports are created every 5 min, after ~ 5 minutes just reload the page again. You also receive a webapp notification and also an email (but most likely you do not have access to the email address of the user)

You can only request a data export if there has not been requested one for your user in the last 14 days.

Prerequisites:

• 1 User
• Make sure the user has at least participated in one exam without a grading scale and in one exam with a grading scale and has at least participated in one programming exercise with manual assessment with an assessment due date in the future and participated in a quiz that has not yet ended.

1. Log in to Artemis
2. Navigate to Privacy Statement
3. Navigate to Data Export
4. Request a data export
5. Once created, download it
6. Make sure that the browser shows the progress
7. Open the data export and verify that the new subdirectories exist
8. Verify that exercise directories have an exercise_ prefix
9. Verify that for an exam without grading scale the exam_results.csv file has no information/columns about passed and grade
10. Verfiy that for an exam with a grading scale the columns are included.
11. Verify that no columns in the general_exam_Information.csv file exist without content
12. Verify that for the programming exercise no manual correction feedback or a new score is leaked
13. Verify that for the quiz exercise no results are included.

Review Progress

Performance Review

• I (as a reviewer) confirm that the client changes (in particular related to REST calls and UI responsiveness) are implemented with a very good performance
• I (as a reviewer) confirm that the server changes (in particular related to database calls) are implemented with a very good performance

Code Review

• Code Review 1
• Code Review 2

Manual Tests

• Test 1
• Test 2

Test Coverage

Server

Class/File	Line Coverage	Confirmation (assert/expect)
DataExportExerciseCreationService	93%	✅
DataExportExamCreationService	95%	✅
DataExportResource	97%	✅
DataExportQuizExerciseCreationService	100%	✅

Client

Class/File	Line Coverage	Confirmation (assert/expect)
data-export.component.ts	100%	✅
data-export.service.ts	100%	✅

[lennart-keller]

Code looks very good to me. Just two very minor things.

[MaximilianAnzinger]

In general code looks good to me.

I left one comment and I've one suggestion: I would appreciate JavaDocs for Services. Although many of your services don't do this consistently, it would make sense to help other developers by including documentation.

[Strohgelaender]

lgtm

[MaximilianAnzinger]

Overall code looks good. Found some inconsistencies in the doc comments.

[pal03377]

Good improvements! Especially well done with all the new comments

[pal03377]

code

[MaximilianAnzinger]

Code

[Strohgelaender]

new changes lgtm

[dearjasmina]

good changes

[pal03377]

Reapprove code after cronjob was changed to value for production