Experiment: API tests using cypress (#1647)

* Add cypress to project * Refactor * Remove example tests * chore: renaming, refactoring, moving files around * chore: add newline to .gitignore * undo pytest.ini change * Undo pytest.ini take 2 * Code cleanup * Refactor code and files * feat(cypress): update local config * feat(cypress): add tests for sms parts * Refactoring * fix: make tests more resilient * Refactor credential handling * fix: update login test to use proper credential * Use desktop viewport by default * chore: Update AXE plugin * Update GCA tests * feat: automatically scan different view port sizes * fix: add `testIsolation: false` to cypress config until we can get `cy.session()` working correctly * chore: upgrade cypress from v10 -> v12 * fix(gca tests): no longer need `preserveOnce()` call with `testIsolation` set to `false` * chore: refactor login code to all be in the LoginPage * chore: remove calls to `.preserveOnce()` * chore: remove unused code * fix: add option to ignore certs as gmail IMAP was not working * chore: remove defunct user * feat: add new bounce rate pages to GCA test * chore: update deps * chore: add smoke_test template id to config * feat: add new `fetchEmail` and `createEmailAccount` tasks * feat: add smoke test that sends email via API and verifies receipt * chore: formatting * feat: add command line for cypress smoke tests: `npm run smoke` * feat: add support to devcontainer to run headless ui tests * chore: update dependencies * feat: fix up email checking/account creation * chore: add new templates to config * fix: increase viewport height in cypress * feat(smoke test): add smoke test suite; add new page models * fix: override etherreal_email env var to ensure new account for every test * fix: add config name to config * fix: refactor how API keys are stored/accessed * chore: add info to config * feat(a11y): add tests to scan for a11y and HTML validation on app pages * chore: move gca tests to a11y folder * chore: remove `.only()` from smoke tests * task: refactor login a bit - move cookie deletion into login code - run deleteAllEmails task inside of login code - Add assertion inside login code * chore: fix getting API keys after refactor * fix: remove verbose logging * chore: add amazon simulated address to config * chore: fix up tests * fix(smoke): use [email protected] instead of internal notify simulated emails because they have a bug * fix(smoke): remove redundant commands * fix(smoke): remove `.only()` command * testing cypress on CI * fix: put new workflow in the correct directory * troubleshooting failing github action * More github action troubleshooting * Remove experimental workflow * chore: add cypress open script * chore(devcontainer): install cypress in notify-dev-entrypoint * fix: update local tempalte ids * fix: add newrelic.com to reduce noise in tests * chore(deps): update cypress and axe-core * fix: simplify login test * fix: skip send/api key combinations that dont work by design * fix(file_attach): use the same template as other tests * fix(smoke tests): if testing locally, dont wait for admin to update notification to delivered because it wont * feat(script): add a new script to run the a11y tests from the command line * chore(refactor): refactor config and update tests accordingly * chore: add README to tests_cypress folder * chore: remove admin related code * fix(scripts): remov admin scripts; add new script for API tests * chore: remove admin code * chore: add cypress videos to gitignore * chore: change config.js to cjs; automatically set baseurl in cypress.config.js * fix: dont set baseurl in each test anymore * feat: add test for scheduled bulk sends
cds-snc · Aug 2, 2023 · 2c3e421 · 2c3e421
1 parent 77f337d
commit 2c3e421
Show file tree

Hide file tree

Showing 21 changed files with 5,672 additions and 1 deletion.
diff --git a/.devcontainer/Dockerfile b/.devcontainer/Dockerfile
@@ -24,12 +24,25 @@ RUN apt-get update \
     man-db \
     manpages \
     net-tools \
+    nodejs \
+    npm \
     openssh-client \
     procps \
     sudo \
     tldr \
     unzip \
     vim \
+    libgtk2.0-0 \
+    libgtk-3-0 \
+    libgbm-dev \
+    libnotify-dev \
+    libgconf-2-4 \
+    libnss3 \
+    libxss1 \
+    libasound2 \
+    libxtst6 \
+    xauth \
+    xvfb \
     && apt-get autoremove -y \
     && apt-get clean -y \
     && rm -rf /var/lib/apt/lists/*

diff --git a/.devcontainer/devcontainer.json b/.devcontainer/devcontainer.json
@@ -52,8 +52,12 @@
 			"version": "latest",
 			"helm": "latest",
 			"minikube": "none"
+		},
+		"ghcr.io/devcontainers/features/node:1": {
+			"version": "14.17.4"
 		}
 	},
 	"postCreateCommand": "notify-dev-entrypoint.sh",
-	"remoteUser": "vscode"
+	"remoteUser": "vscode",
+
 }
diff --git a/.devcontainer/scripts/notify-dev-entrypoint.sh b/.devcontainer/scripts/notify-dev-entrypoint.sh
@@ -49,3 +49,6 @@ poetry install
 
 # Upgrade schema of the notification_api database.
 poetry run flask db upgrade
+
+# install npm deps (i.e. cypress)
+cd tests_cypress && npm install && npx cypress install && cd ..
diff --git a/.gitignore b/.gitignore
@@ -87,3 +87,7 @@ celerybeat.pid
 # Misc
 /scripts/run_my_tests.sh
 jinja_templates/
+
+cypress.env.json
+node_modules/
+tests_cypress/cypress/videos/
diff --git a/tests_cypress/README b/tests_cypress/README
@@ -0,0 +1,43 @@
+# Notify + Cypress 🎉
+
+## Setup
+This folder contains Cypress tests suites.  In order to run them, you'll need to install cypress and its dependencies. If you're running inside the dev container, rebuild your dev container to get the necessary packages.  
+
+## Running the tests
+### In your devcontainer
+There are some issues getting the cypress UI to launch within the devcontainer.  For now, you can run the headless tests inside the dev container but if you want to launch the cypress UI you will need to do that outside of the dev container.  
+
+There are 2 helper scripts in `package.json` to run 2 of the test suites.  Run these from the `tests_cypress/` folder:
+- `npm run smoke`:  this will run the smoke tests in headless mode using the electron browser
+- `npm run a11y`: this will run the accessibility tests in headless mode using the electron browser
+
+### Outside of your devcontainer
+To launch the cypress UI, where you can choose your test suite and visually debug and inspect tests, run (from the `tests_cypress/` folder):
+- `npm run cypress`: this will open the cypress UI where you can choose which tests to run and in which browser
+
+### Local installation
+To install cypress locally, use the following command, from the `tests_cypress/` folder:
+```bash
+npm install
+npx cypress install
+```
+
+## Configuration 
+- `cypress.env.json`: this file contains sensitive items like api keys and passphrases that you'll need to run the tests. You'll need to add the file `cypress.env.json` into the `tests_cypress/` folder and its contents can be found in lastpass.
+- `config.js`: this file contains non-sensitive items like template ids and hostnames that you'll need to run the tests
+
+### `cypress.env.json` contents
+| key             | description                                     |
+| --------------- | ----------------------------------------------- |
+| ADMIN_SECRET    | Secret admin uses to authenticate against API   |
+| ADMIN_USERNAME  | Username admin uses to authenticate against API |
+| NOTIFY_USER     | Notify user used by the tests                   |
+| NOTIFY_PASSWORD | Password of NOTIFY_USER                         |
+| IMAP_PASSWORD   | IMAP password of gmail account for NOTIFY_USER  |
+
+### Target environment 🎯
+The tests are configured to run against the staging environment by default.  To run the tests against your local environment, you'll need to create a local service and API keys and store these values in your config.  You will also need to update the `ConfigToUse` variable in `config.js` file:
+```js
+const ConfigToUse = config.LOCAL;
+```
+
diff --git a/tests_cypress/config.js b/tests_cypress/config.js
@@ -0,0 +1,77 @@
+let STAGING = {
+    CONFIG_NAME: "STAGING",
+    Hostnames: {
+        API: 'https://api.staging.notification.cdssandbox.xyz',
+        Admin: 'https://staging.notification.cdssandbox.xyz',
+        DDAPI: 'https://api.document.staging.notification.cdssandbox.xyz',
+    },
+    Services: {
+        Notify: 'd6aa2c68-a2d9-4437-ab19-3ae8eb202553',
+        Cypress: '5c8a0501-2aa8-433a-ba51-cefb8063ab93'
+    },
+    Templates: {
+        'FILE_ATTACH_TEMPLATE_ID': '7246c71e-3d60-458b-96af-af17a5b07659',
+        'SIMPLE_EMAIL_TEMPLATE_ID': '939dafde-1b60-47f0-a6d5-c9080d92a4a8',
+        'VARIABLES_EMAIL_TEMPLATE_ID': '1101a00a-11b7-4036-865c-add43fcff7c9',
+        'SMOKE_TEST_EMAIL': '5e26fae6-3565-44d5-bfed-b18680b6bd39',
+        'SMOKE_TEST_EMAIL_BULK': '04145882-0f21-4d57-940d-69883fc23e77',
+        'SMOKE_TEST_EMAIL_ATTACH': 'bf85def8-01b4-4c72-98a8-86f2bc10f2a4',
+        'SMOKE_TEST_EMAIL_LINK': '37924e87-038d-48b8-b122-f6dddefd56d5',
+        'SMOKE_TEST_SMS': '16cae0b3-1d44-47ad-a537-fd12cc0646b6'
+    },
+    Users: {
+        Team: ['[email protected]'],
+        NonTeam: ['[email protected]'],
+        Simulated: ['[email protected]', '[email protected]', '[email protected]'],
+        SimulatedPhone: ['+16132532222', '+16132532223', '+16132532224']
+    },
+    ReplyTos: {
+        Default: '24e5288d-8bfa-4ad4-93aa-592c11a694cd',
+        Second: '797865c4-788b-4184-91ae-8e45eb07e40b'
+    },
+    viewports: [320,375,640,768]
+};
+
+let LOCAL = {
+    CONFIG_NAME: "LOCAL",
+    Hostnames: {
+        API: 'http://localhost:6011',
+        Admin: 'http://localhost:6012',
+        DDAPI: 'http://localhost:7000',
+    },
+    Services: {
+        Notify: 'd6aa2c68-a2d9-4437-ab19-3ae8eb202553',
+        Cypress: '4049c2d0-0cab-455c-8f4c-f356dff51810'
+    },
+    Templates: {
+        'FILE_ATTACH_TEMPLATE_ID': '7246c71e-3d60-458b-96af-af17a5b07659',
+        'SIMPLE_EMAIL_TEMPLATE_ID': 'b4692883-4182-4a23-b1b9-7b9df66a66e8',
+        'VARIABLES_EMAIL_TEMPLATE_ID': '258d8617-da88-4faa-ad28-46cc69f5a458',
+        'SMOKE_TEST_EMAIL': '136e951e-05c8-4db4-bc50-fe122d72fcaa',
+        'SMOKE_TEST_EMAIL_BULK': '48207d93-144d-4ebb-92c5-99ff1f1baead',
+        'SMOKE_TEST_EMAIL_ATTACH': '58db03d6-a9d8-4482-8621-26f473f3980a',
+        'SMOKE_TEST_EMAIL_LINK': '2d52d997-42d3-4ac0-a597-7afc94d4339a',
+        'SMOKE_TEST_SMS': '5945e2f0-3e37-4813-9a60-e0665e02e9c8'
+    },
+    Users: {
+        Team: ['[email protected]'],
+        NonTeam: ['[email protected]'],
+        Simulated: ['[email protected]', '[email protected]', '[email protected]'],
+        SimulatedPhone: ['+16132532222', '+16132532223', '+16132532224']
+    },
+    ReplyTos: {
+        Default: '1bc45a34-f4de-4635-b36f-7da2e2d248ed',
+        Second: 'aaa58593-fc0a-46b0-82b8-b303ae662a41'
+    },
+    viewports: [320,375,640,768]
+};
+
+const config = {
+    STAGING,
+    LOCAL,
+};
+
+// choose which config to use here
+const ConfigToUse = config.STAGING;
+
+module.exports = ConfigToUse;
diff --git a/tests_cypress/cypress.config.js b/tests_cypress/cypress.config.js
@@ -0,0 +1,43 @@
+var config = require('./config');
+
+const { defineConfig } = require("cypress");
+const EmailAccount = require('./cypress/plugins/email-account')
+const htmlvalidate = require("cypress-html-validate/plugin");
+
+module.exports = defineConfig({
+  e2e: {
+    baseUrl: config.Hostnames.API,
+    setupNodeEvents: async (on, config) => {
+      htmlvalidate.install(on);
+
+      const emailAccount = await EmailAccount()
+      on('task', {
+        getLastEmail() {
+          return emailAccount.getLastEmail()
+        },
+        deleteAllEmails() {
+          return emailAccount.deleteAllEmails()
+        },
+        fetchEmail(acct) {
+          return emailAccount.fetchEmail(acct)
+        },
+        createEmailAccount() {
+          return emailAccount.createEmailAccount();
+        }
+      });
+
+      on('before:browser:launch', (browser = {}, launchOptions) => {
+        if (browser.family === 'chromium' && browser.name !== 'electron') {
+          launchOptions.extensions = [];
+        }
+        return launchOptions;
+      });
+    },
+    specPattern: '**/e2e/**/*.cy.js',
+    watchForFileChanges: false,
+    blockHosts: ['*google-analytics.com', 'stats.g.doubleclick.net', 'bam.nr-data.net', '*newrelic.com'],
+    viewportWidth: 1280,
+    viewportHeight: 850,
+    testIsolation: false
+  },
+});
diff --git a/tests_cypress/cypress/Notify/NotifyAPI.js b/tests_cypress/cypress/Notify/NotifyAPI.js
@@ -0,0 +1,111 @@
+import jwt from "jsonwebtoken";
+import config from "../../config";
+
+const Utilities = {
+    CreateJWT: () => {
+        const claims = {
+            'iss': Cypress.env('ADMIN_USERNAME'),
+            'iat': Math.round(Date.now() / 1000)
+        }
+
+        var token = jwt.sign(claims, Cypress.env('ADMIN_SECRET'));
+
+        return token;
+    },
+};
+const Admin = {
+    SendOneOff: ({to, template_id}) => {
+
+        var token = Utilities.CreateJWT();
+        return cy.request({
+            url: `/service/${config.Services.Cypress}/send-notification`,
+            method: 'POST',
+            headers: {
+                Authorization: `Bearer ${token}`,
+            },
+            body: {
+                'to': to,
+                'template_id': template_id,
+                'created_by': Cypress.env('NOTIFY_USER_ID'),
+            }
+        });
+    }
+}
+
+const API = {
+    SendEmail: ({ api_key, to, template_id, personalisation, failOnStatusCode = true, email_reply_to_id }) => {
+        return cy.request({
+            failOnStatusCode: failOnStatusCode,
+            url: '/v2/notifications/email',
+            method: 'POST',
+            headers: {
+                'Content-Type': 'application/json',
+                Authorization: 'ApiKey-v1 ' + api_key,
+            },
+            body: {
+                "email_address": to,
+                "template_id": template_id,
+                "personalisation": personalisation,
+                ...(email_reply_to_id) && { email_reply_to_id: email_reply_to_id } // only add email_reply_to_id if it's defined
+            }
+        });
+    },
+    SendBulkEmail: ({ api_key, to, bulk_name, template_id, personalisation, failOnStatusCode = true, scheduled_for}) => {
+        return cy.request({
+            failOnStatusCode: failOnStatusCode,
+            url: '/v2/notifications/bulk',
+            method: 'POST',
+            headers: {
+                'Content-Type': 'application/json',
+                Authorization: 'ApiKey-v1 ' + api_key,
+            },
+            body: {
+                "name": bulk_name,
+                "template_id": template_id,
+                "rows": [
+                    ["email address"],
+                    ...to
+                ],
+                ...(scheduled_for) && { scheduled_for: scheduled_for } // only add scheduled_for if it's defined
+            }
+        });
+    },
+    SendSMS: ({ api_key, to, template_id, personalisation, failOnStatusCode = true }) => {
+        return cy.request({
+            failOnStatusCode: failOnStatusCode,
+            url: '/v2/notifications/sms',
+            method: 'POST',
+            headers: {
+                'Content-Type': 'application/json',
+                Authorization: 'ApiKey-v1 ' + api_key,
+            },
+            body: {
+                "phone_number": to,
+                "template_id": template_id,
+                "personalisation": personalisation,
+            }
+        });
+    },
+    SendBulkSMS: ({ api_key, to, bulk_name, template_id, personalisation, failOnStatusCode = true }) => {
+        return cy.request({
+            failOnStatusCode: failOnStatusCode,
+            url: '/v2/notifications/bulk',
+            method: 'POST',
+            headers: {
+                'Content-Type': 'application/json',
+                Authorization: 'ApiKey-v1 ' + api_key,
+            },
+            body: {
+                "name": bulk_name,
+                "template_id": template_id,
+                "rows": [
+                    ["phone number"],
+                    ...to
+                ],
+            }
+        });
+    },
+
+}
+
+export default { API, Utilities, Admin };
diff --git a/tests_cypress/cypress/e2e/api/all.cy.js b/tests_cypress/cypress/e2e/api/all.cy.js
@@ -0,0 +1,2 @@
+import './email_notifications.cy';
+import './file_attach.cy';
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1,2 @@
		import './email_notifications.cy';
		import './file_attach.cy';