diff --git a/docs/dev/cypress/cypress-open-screenshot.png b/docs/dev/cypress/cypress-open-screenshot.png deleted file mode 100644 index 7c32c512b6e5..000000000000 Binary files a/docs/dev/cypress/cypress-open-screenshot.png and /dev/null differ diff --git a/docs/dev/cypress/cypress_bamboo_deployment_diagram.svg b/docs/dev/cypress/cypress_bamboo_deployment_diagram.svg deleted file mode 100644 index 8ff7e84792a7..000000000000 --- a/docs/dev/cypress/cypress_bamboo_deployment_diagram.svg +++ /dev/null @@ -1,4 +0,0 @@ - - - -
<<virtual machine>>
Bamboo Build Agent
<<virtual machine>>...
<<execution environment>>
Docker container
<<execution environment>>...
<<infrastructure>>
University Data Center
<<infrastructure>>...
<<component>>
Bamboo
<<component>>...
<<component>>
Jira
<<component>>...
<<component>>
Bitbucket
<<component>>...
<<execution environment>>
Docker container
<<execution environment>>...
<<execution environment>>
Docker container
<<execution environment>>...
<<component>>
Artemis Server
<<component>>...
<<component>>
MySQL Database
<<component>>...
<<component>>
Artemis Client
<<component>>...
<<component>>
Cypress Test Runner
<<component>>...
Text is not SVG - cannot display
\ No newline at end of file diff --git a/docs/dev/cypress/cypress_test_environment_deployment_diagram.svg b/docs/dev/cypress/cypress_test_environment_deployment_diagram.svg deleted file mode 100644 index d7f6ccc7eac7..000000000000 --- a/docs/dev/cypress/cypress_test_environment_deployment_diagram.svg +++ /dev/null @@ -1,4 +0,0 @@ - - - -
<<virtual machine>>
Bamboo Build Agent
<<virtual machine>>...
<<virtual machine>>
Test Server 3
<<virtual machine>>...
<<component>>
Artemis Server
<<component>>...
<<infrastructure>>
University Data Center
<<infrastructure>>...
<<component>>
Bamboo
<<component>>...
<<component>>
Jira
<<component>>...
<<component>>
Bitbucket
<<component>>...
<<component>>
MySQL Database
<<component>>...
<<execution environment>>
Docker container
<<execution environment>>...
<<component>>
Artemis Client
<<component>>...
<<component>>
Cypress Test Runner
<<component>>...
Text is not SVG - cannot display
\ No newline at end of file diff --git a/docs/dev/cypress/sorry-cypress-dashboard.png b/docs/dev/cypress/sorry-cypress-dashboard.png deleted file mode 100644 index bf2d38932af2..000000000000 Binary files a/docs/dev/cypress/sorry-cypress-dashboard.png and /dev/null differ diff --git a/docs/dev/cypress/sorry-cypress-run.png b/docs/dev/cypress/sorry-cypress-run.png deleted file mode 100644 index ca2e93daea70..000000000000 Binary files a/docs/dev/cypress/sorry-cypress-run.png and /dev/null differ diff --git a/docs/dev/cypress/sorry-cypress-runs.png b/docs/dev/cypress/sorry-cypress-runs.png deleted file mode 100644 index 60a47a352dbb..000000000000 Binary files a/docs/dev/cypress/sorry-cypress-runs.png and /dev/null differ diff --git a/docs/dev/cypress/sorry-cypress-test.png b/docs/dev/cypress/sorry-cypress-test.png deleted file mode 100644 index 83148bb2fed3..000000000000 Binary files a/docs/dev/cypress/sorry-cypress-test.png and /dev/null differ diff --git a/docs/dev/playwright.rst b/docs/dev/playwright.rst index c4edd85d5399..ce6536c96e54 100644 --- a/docs/dev/playwright.rst +++ b/docs/dev/playwright.rst @@ -1,6 +1,23 @@ E2E Testing with Playwright =========================== +**Background** + +The Playwright test suite contains system tests verifying the most important features of Artemis. +System tests test the whole system and therefore require a complete deployment of Artemis first. +In order to prevent as many faults (bugs) as possible from being introduced into the develop branch, +we want to execute the Playwright test suite whenever new commits are pushed to a Git branch +(just like the unit and integration test suites). + +To accomplish this we need to be able to dynamically deploy multiple different instances of Artemis at the same time. +An ideal setup would be to deploy the whole Artemis system using Kubernetes. +However, this setup is too complex at the moment. +The main reason for the complexity is that it is very hard to automatically setup Docker containers for +the external services (e.g. Gitlab, Jenkins) and connect them directly with Artemis. + +Therefore, the current setup only dynamically deploys the Artemis server and configures it to connect to +the prelive system, which is already properly setup in the university data center. + Set up Playwright locally ------------------------- @@ -137,8 +154,72 @@ To run tests sequentially (one after another), set the ``workers`` option to ``1 sequentially, while running test files in parallel, set the ``fullyParallel`` option to ``false``. -Best practices for writing tests in Playwright ----------------------------------------------- +Best practices when writing new E2E tests +----------------------------------------- + +**Understanding the System and Requirements** + +Before writing tests, a deep understanding of the system and its requirements is crucial. +This understanding guides determining what needs testing and what defines a successful test. +The best way to understand is to consolidate the original system`s developer or a person actively working on this +component. + +**Identify Main Test Scenarios** + +Identify what are the main ways the component is supposed to be used. Try +the action with all involved user roles and test as many different inputs as +feasible. + +**Identify Edge Test Scenarios** + +Next to the main test scenarios, there are also edge case scenarios. These +tests include inputs/actions that are not supposed to be performed (e.g. enter +a too-long input into a field) and test the error-handling capabilities of the +platform. + +**Write Tests as Development Progresses** + +Rather than leaving testing until the end, write tests alongside each piece of +functionality. This approach ensures the code remains testable and makes +identifying and fixing issues as they arise easier. + +**Keep Tests Focused** + +Keep each test focused on one specific aspect of the code. If a test fails, it is +easier to identify the issue when it does not check multiple functionalities at +the same time. + +**Make Tests Independent** + +Tests should operate independently from each other and external factors like +the current date or time. Each test should be isolated. Use API calls for unrelated tasks, such as creating a +course, and UI interaction for the appropriate testing steps. This also involves +setting up a clean environment for every test suite. + +**Use Descriptive Test Names** + +Ensure each test name clearly describes what the test does. This strategy +makes the test suite easier to understand and quickly identifies which test +has failed. + +**Use Similar Test Setups** + +Avoid using different setups for each test suit. For example, always check +for the same HTTP response when deleting a course. + +**Do Not Ignore Failing Tests** + +If a test consistently fails, pay attention to it. Investigate as soon as possible +and fx the issue, or update the test if the requirements have changed. + +**Regularly Review and Refactor Your Tests** + +Tests, like code, can accumulate technical debt. Regular reviews for duplication, +unnecessary complexity, and other issues help maintain tests and enhance reliability. + + +Playwright testing best practices +--------------------------------- 1. **Use page objects for common interactions**: @@ -230,3 +311,102 @@ Best practices for writing tests in Playwright Waiting for the page load state is not recommended if we are only interested in specific elements appearing on the page - use ``waitFor()`` function of a locator instead. + +Artemis Deployment on Bamboo Build Agent +---------------------------------------- +Every execution of the Playwright test suite requires its own deployment of Artemis. +The easiest way to accomplish this is to deploy Artemis locally on the build agent, which executes the Playwright tests. +Using ``docker compose`` we can start a MySQL database and the Artemis server locally on the build agent and +connect it to the prelive system in the university data center. + +.. figure:: playwright/playwright_bamboo_deployment_diagram.svg + :align: center + :alt: Artemis Deployment on Bamboo Build Agent for Playwright + + Artemis Deployment on Bamboo Build Agent for Playwright + +In total there are three Docker containers started in the Bamboo build agent: + +1. MySQL + + This container starts a MySQL database and exposes it on port 3306. + The container automatically creates a new database 'Artemis' and configures it + with the recommended settings for Artemis. + The Playwright setup reuses the already existing + `MySQL docker image `__ + from the standard Artemis Docker setup. + +2. Artemis Application + + The Docker image for the Artemis container is created from the already existing + `Dockerfile `__. + When the Bamboo build of the Playwright test suite starts, it retrieves the Artemis executable (.war file) + from the `Artemis build plan `_. + Upon creation of the Artemis Docker image the executable is copied into the image together with configuration files + for the Artemis server. + + The main configuration of the Artemis server are contained in the + `Playwright environment configuration files `__. + However, those files do not contain any security relevant information. + Security relevant settings are instead passed to the Docker container via environment variables. This information is + accessible to the Bamboo build agent via + `Bamboo plan variables `__. + + The Artemis container is also configured to + `depend on `__ + the MySQL container and uses + `health checks `__ + to wait until the MySQL container is up and running. + +3. Playwright + + Playwright offers a test environment `docker image `__ + to execute Playwright tests. + The image contains Playwright browsers and browser system dependencies. + However, Playwright itself is not included in the image. + This is convenient for us because the image is smaller and the Artemis Playwright project requires + additional dependencies to fully function. + Therefore, the Artemis Playwright Docker container is configured to install all dependencies + (using :code:`npm ci`) upon start. This will also install Playwright itself. + Afterwards the Artemis Playwright test suite is executed. + + The necessary configuration for the Playwright test suite is also passed in via environment variables. + Furthermore, the Playwright container depends on the Artemis container and is only started + once Artemis has been fully booted. + +**Bamboo webhook** + +The Artemis instance deployed on the build agent is not publicly available to improve the security of this setup. +However, in order to get the build results for programming exercise submissions Artemis relies on a webhook from Bamboo +to send POST requests to Artemis. +To allow this, an extra rule has been added to the firewall allowing only the Bamboo instance in the prelive system +to connect to the Artemis instance in the build agent. + +**Timing** + +As mentioned above, we want the Playwright test suite to be executed whenever new commits are pushed to a Git branch. +This has been achieved by adding the +`Playwright build plan `__ +as a `child dependency `__ +to the `Artemis Build build plan `__. +The *Artemis Build* build plan is triggered whenever a new commit has been pushed to a branch. + +The Playwright build plan is only triggered after a successful build of the Artemis executable. +This does imply a delay (about 10 minutes on average) between the push of new commits and the execution +of the Playwright test suite, since the new Artemis executable first has to be built. + +**NOTE:** The Playwright test suite is only automatically executed for internal branches and pull requests +(requires access to this GitHub repository), **not** for external ones. +In case you need access rights, please contact the maintainer `Stephan Krusche `__. + +Maintenance +----------- +The Artemis Dockerfile as well as the MySQL image are already maintained because they are used in +other Artemis Docker setups. +Therefore, only Playwright and the Playwright Docker image require active maintenance. +Since the Playwright test suite simulates a real user, it makes sense to execute the test suite with +the latest browser versions. +The Playwright Docker image we use always has browsers with specific versions installed. +Therefore, the +`docker-compose file `__ +should be updated every month to make sure that the latest Playwright image is used. diff --git a/docs/dev/playwright/playwright_bamboo_deployment_diagram.svg b/docs/dev/playwright/playwright_bamboo_deployment_diagram.svg new file mode 100644 index 000000000000..14bdb96ef47e --- /dev/null +++ b/docs/dev/playwright/playwright_bamboo_deployment_diagram.svg @@ -0,0 +1,4 @@ + + + +
<<virtual machine>>
Bamboo Build Agent
<<virtual machine>>...
<<docker container>>
artemis-app
<<docker container>>...
<<docker container>>
mysql
<<docker container>>...
<<component>>
MySQL Database
<<component>>...
<<component>>
Artemis Server
<<component>>...
<<component>>
Artemis Client
<<component>>...
<<docker container>>
artemis-playwright
<<docker container>>...
<<component>>
Playwright Test Environment
<<component>>...
Text is not SVG - cannot display
\ No newline at end of file