Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SMS soak test #2011

Merged
merged 15 commits into from
Dec 11, 2023
1 change: 1 addition & 0 deletions scripts/soak_test/.env.example
Original file line number Diff line number Diff line change
@@ -1,2 +1,3 @@
API_KEY=
EMAIL_TEMPLATE_ID=
SMS_TEMPLATE_ID=
20 changes: 11 additions & 9 deletions scripts/soak_test/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
The goal of this code is to do a soak test of api while we make significant application or infrastructure changes.

There are two soak tests here:
- `soak_test_send_email.py` will POST an email to api every second.
- `soak_test_send_notification.py` will POST an email or SMS to api every second.
- `soak_test_all_servers.py` will do a GET to all our servers (admin, api, dd-api, api-k8s, documentation), on average hitting each server once a second

## How to configure
Expand All @@ -14,19 +14,21 @@ Run the setup.sh to install the python pre-requisites or run in the repo devcont

Default configuration is in the `locust.conf` file. Note that the `host` is the base address of the system you are testing, for example `https://staging.notification.cdssandbox.xyz` **not** `https://api.staging.notification.cdssandbox.xyz`. The "api" prefix will be added in the code.

The python file `soak_test_send_email.py` requires environment variables `API_KEY` and `EMAIL_TEMPLATE_ID`. The template should have no variables.
The python file `soak_test_send_notification.py` requires environment variables `API_KEY`, `EMAIL_TEMPLATE_ID`, and `SMS_TEMPLATE_ID` . The template should have no personalisation variables.

```
API_KEY=gcntfy-notAKey-f6c7cc49-b5b7-4e67-a8ff-24f34be34523-f6c7cc49-b5b7-4e67-a8ff-24f34be34523
EMAIL_TEMPLATE_ID=f6c7cc49-b5b7-4e67-a8ff-24f34be34523
SMS_TEMPLATE_ID=f6c7cc49-b5b7-4e67-aeef-24f34be34523
```
These can be in a `.env` file in the soak_test directory.

__See Last Pass note "Soak Test Staging API Key and Template" in Shared-New-Notify-Staging folder__

Note that the default configuration in `locust.conf` is to send one email per second.

You can supply a `--ref` option to `soak_test_send_email.py` that will set the notification's `client_reference`. This is useful in testing that all POSTs were processed successfully.
Notes:
- The default configuration in `locust.conf` is to send one email per second.
- You can supply a `--ref` option to `soak_test_send_notification.py` that will set the notification's `client_reference`. This is useful in testing that all POSTs were processed successfully.
- You can also supply a `--sms` option that will sens sms instead of email.

## How to run

Expand All @@ -37,7 +39,7 @@ There are two ways to run Locust, with the UI or headless.
Locally you can run the email soak test with:

```shell
locust -f ./soak_test_send_email.py --ref=soak-2023-05-30-A
locust -f ./soak_test_send_notification.py [--ref=soak-2023-05-30-A] [--sms]
```

Follow the localhost address that the console will display to get to the UI. It will ask you how many total users and spawned users you want configured. Once setup, you can manually start the tests via the UI and follow the summary data and charts visually.
Expand All @@ -53,7 +55,7 @@ locust -f ./soak_test_all_servers.py
You can pass the necessary parameters to the command line to run in the headless mode. For example:

```shell
locust -f ./soak_test_send_email.py --headless --ref=soak-2023-05-30-A
locust -f ./soak_test_send_notification.py --headless [--ref=soak-2023-05-30-A] [--sms]
```

The defaults in `locust.conf` may be overridden by command line options
Expand All @@ -64,9 +66,9 @@ The server soak test can be run with
locust -f ./soak_test_all_servers.py --headless
```

## Checking if all emails were sent
## Checking if all notifications were sent

To check whether all the POSTs from `soak_test_send_email.py` made it into the database, run the "Soak test" query on blazer. The query is already in staging, or you can run:
To check whether all the POSTs from `soak_test_send_notification.py` made it into the database, run the "Soak test" query on blazer. The query is already in staging, or you can run:

```sql
WITH
Expand Down
30 changes: 0 additions & 30 deletions scripts/soak_test/soak_test_send_email.py

This file was deleted.

38 changes: 38 additions & 0 deletions scripts/soak_test/soak_test_send_notification.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
import os
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We didn't want this to sit in a subfolder of tests-stress?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤔 We could think of this as a very low level stress test... generally I've been using high priority templates for these soak tests and low priority ones for the larger stress tests so it would require either rewriting the code a bit or changing the .env file when switching between them. 🤷


from dotenv import load_dotenv
from locust import HttpUser, constant_pacing, events, task
from soak_utils import url_with_prefix

load_dotenv()


@events.init_command_line_parser.add_listener
def _(parser):
parser.add_argument("--ref", type=str, default="test", help="reference")
parser.add_argument("--sms", action='store_true', help="send sms")


class NotifyApiUser(HttpUser):
wait_time = constant_pacing(1) # each user makes one post per second

def __init__(self, *args, **kwargs):
self.host = url_with_prefix(self.host, "api")

super(NotifyApiUser, self).__init__(*args, **kwargs)
self.headers = {"Authorization": f"apikey-v1 {os.getenv('API_KEY')}"}
self.email_template = os.getenv("EMAIL_TEMPLATE_ID")
self.sms_template = os.getenv("SMS_TEMPLATE_ID")
self.email_address = "[email protected]"
self.phone_number = "+16135550123" # INTERNAL_TEST_NUMBER
self.reference_id = self.environment.parsed_options.ref
self.send_sms = self.environment.parsed_options.sms

@task(1)
def send_notification(self):
if self.send_sms:
json = {"phone_number": self.phone_number, "template_id": self.sms_template, "reference": self.reference_id}
self.client.post("/v2/notifications/sms", json=json, headers=self.headers)
else:
json = {"email_address": self.email_address, "template_id": self.email_template, "reference": self.reference_id}
self.client.post("/v2/notifications/email", json=json, headers=self.headers)
Loading