Programming exercises: Add blackbox tests as another Java project type

[chrisknedl]

WARNING: DELETE EXERCISES CREATED WITH THIS PR AFTER TESTING! OTHERWISE THE TEST SERVER IS BROKEN FOR OTHER PRs

Checklist

General

• I tested all changes and their related features with all corresponding user types on a test server.
• This is a small issue that I tested locally and was confirmed by another developer on a test server.
• Language: I followed the guidelines for inclusive, diversity-sensitive, and appreciative language.
• I chose a title conforming to the naming conventions for pull requests.

Server

• Important: I implemented the changes with a very good performance and prevented too many (unnecessary) database calls.
• I followed the coding and design guidelines.
• I added multiple integration tests (Spring) related to the features (with a high test coverage).
• I documented the Java code using JavaDoc style.

Client

• Important: I implemented the changes with a very good performance, prevented too many (unnecessary) REST calls and made sure the UI is responsive, even with large data.
• I followed the coding and design guidelines.
• Following the theming guidelines, I specified colors only in the theming variable files and checked that the changes look consistent in both the light and the dark theme.
• I added multiple integration tests (Jest) related to the features (with a high test coverage), while following the test guidelines.
• I documented the TypeScript code using JSDoc style.
• I added multiple screenshots/screencasts of my UI changes.
• I translated all newly inserted strings into English and German.

Changes affecting Programming Exercises

• I tested all changes and their related features with all corresponding user types on Test Server 2 (Jenkins and Gitlab).

Motivation and Context

In some cases, it is preferable to leave it up to the student how their program is structured. In such cases, methods and classes of the solution are not known beforehand. In order to test such solutions, blackbox tests are required. This PR adds that functionality for Java programming exercises.

Description

The testing framework that is used is called DejaGnu (https://www.gnu.org/software/dejagnu/). A DejaGnu button is added to the project types for Java exercises in the client. When selected, the exemplary dependency, testwise coverage and sequential test runs are disabled.

The template files for this exercise are stored in templates/java/maven_blackbox. As implied by the folder name, the DejaGnu image is based on Maven. The DejaGnu template exercise adopts the sorting exercise from the other two project types and turns it into a user input based exercise.

The pipeline.groovy file that defines the Jenkins pipeline steps is located in the templates/java/test/blackbox.projectTemplate directory. This folder also includes all the files required for testing. The actual docker image is defined in the pipeline.groovy file (ghcr.io/uni-passau-artemis/artemis-dejagnu:20).

Steps for Testing

Prerequisites:

• 1 Instructor
• 1 Course
• TS2 (Jenkins Setup)

1. Log in to Artemis
2. Navigate to Course Management
3. Create a Java programming exercise with online editor enabled, and choose the new DejaGnu project type.
4. Make sure the BASE build fails, and the SOLUTION build works
5. Look inside the test-repository via the online editor, it should contain the files that are shown in the two screenshots at the bottom.

Review Progress

Performance Review

• I confirm that the client changes (in particular related to REST calls and UI responsiveness) are implemented with a very good performance
• I confirm that the server changes (in particular related to database calls) are implemented with a very good performance

Code Review

• Code Review 1
• Code Review 2

Manual Tests

• Test 1
• Test 2

Test Coverage

Class/File	Line Coverage	Confirmation (assert/expect)
programming-exercise-update.component.ts	80%	✅

Screenshots

UI change:

Test Repo folder structure:


bb will be replaced by your package name.

[Strohgelaender]

I don't see in the changes where the build plan for this new type gets created. Since this is also a maven project, I assume that the normal maven build plan just gets used here. However, isMavenProject() returns false for this type (which in itself is confusing for a type called maven_blackbox). Does this just rely on a if (isGradle()) {} else {} code in the repository?

At a minimum, I would add a comment at the respective place explaining that this brach must cover 3 project types.

In general, I'd like to have isMavenProject() return true for this new type, since it really is a maven project. For places where not the same logic can be applied, an explicit check like if (type == MAVEN_BLACKBOX) { } else if (isMavenProject(type)) { } would be better.

[Strohgelaender]

Does this type need the special command-pattern-exercise template? In a future PR I'd like to remove the duplicated template files between the PLAIN_MAVEN and MAVEN_MAVEN types (which use the same template / solution code but just slightly different configuration files). If we could also use the same code template here, this would make this change a bit easier. And for most exercises, the template code does not survive anyways.

[b-fein]

Does this type need the special command-pattern-exercise template?

Do you mean the different Client.java and its dependencies in the input package? If so, yes they are needed. The idea of this template is to be able to test programs as blackboxes via their CLI rather than unit-tests. This requires a different Client implementation, unless the Client is changed in a future separate PR to accept a command line parameter that switches from non-interactive to interactive mode.

[Strohgelaender]

thanks for answering the questions, lgtm now.

[laadvo]

reapprove code

[b-fein]

Re-tested exercise creation and project import.

[MarkusPaulsen]

Code looks good.