Merge remote-tracking branch 'origin/main' into feat/decrease-retry-f…

…or-email-high

Makefile

@@ -56,11 +56,7 @@ run-celery-local: ## Run the celery workers with all the queues
 
 .PHONY: run-celery-local-filtered
 run-celery-local-filtered: ## Run the celery workers with all queues but filter out common scheduled tasks
-	./scripts/run_celery_local.sh 2>&1 >/dev/null | grep -Ev 'beat|in-flight-to-inbox|run-scheduled-jobs|check-job-status'
-
-.PHONY: run-celery-beat-local
-run-celery-beat-local: ## Run the celery beat
-	./scripts/run_celery_beat_local.sh
+	./scripts/run_celery_local.sh 2>&1 >/dev/null | grep -iEv 'beat|in-flight-to-inbox|run-scheduled-jobs|check-job-status'
 
 .PHONY: run-celery-purge
 run-celery-purge: ## Purge the celery queues

README.md

@@ -109,13 +109,8 @@ file. Copy that file to `.env` and customize it to your needs.
 
 
 ##  To run the queues
-```
-scripts/run_celery_local.sh
-```
 
-```
-scripts/run_celery_beat.sh
-```
+Run `make run-celery-local` or `make run-celery-local-filtered`. Note that the "filtered" option does not show the beat worker logs nor most scheduled tasks (this makes it easier to trace notification sending).
 
 ### Python version
 

scripts/load_test/README.md

@@ -1,4 +1,4 @@
-# Soak test
+# Load test
 
 ## Goals
 
@@ -12,7 +12,7 @@ Default configuration is in the `locust.conf` file.
 
 The python file `load_test.py` requires environment variables as listed in `.env.example`. The templates should have no variables.
 
-__See Last Pass note "Load Test Variables" in Shared-New-Notify-Staging folder__
+__See One Password note "Load Test Variables" in Shared-New-Notify-Staging folder__
 
 
 ## How to run

scripts/run_celery_local.sh

@@ -7,4 +7,4 @@ set -e
 
 echo "Start celery, concurrency: ${CELERY_CONCURRENCY-4}"
 
-celery -A run_celery.notify_celery worker --pidfile="/tmp/celery.pid" --loglevel=INFO --concurrency="${CELERY_CONCURRENCY-4}" -Q database-tasks,-priority-database-tasks.fifo,-normal-database-tasks,-bulk-database-tasks,job-tasks,notify-internal-tasks,periodic-tasks,priority-tasks,normal-tasks,bulk-tasks,reporting-tasks,research-mode-tasks,retry-tasks,send-sms-tasks,send-sms-high,send-sms-medium,send-sms-low,send-throttled-sms-tasks,send-email-high,send-email-medium,send-email-low,send-email-tasks,service-callbacks,delivery-receipts
+celery -A run_celery.notify_celery worker --beat --pidfile="/tmp/celery.pid" --loglevel=INFO --concurrency="${CELERY_CONCURRENCY-4}" -Q database-tasks,-priority-database-tasks.fifo,-normal-database-tasks,-bulk-database-tasks,job-tasks,notify-internal-tasks,periodic-tasks,priority-tasks,normal-tasks,bulk-tasks,reporting-tasks,research-mode-tasks,retry-tasks,send-sms-tasks,send-sms-high,send-sms-medium,send-sms-low,send-throttled-sms-tasks,send-email-high,send-email-medium,send-email-low,send-email-tasks,service-callbacks,delivery-receipts

scripts/soak_test/.env.example

@@ -1,2 +1,3 @@
 API_KEY=
 EMAIL_TEMPLATE_ID=
+SMS_TEMPLATE_ID=

scripts/soak_test/README.md

@@ -5,7 +5,7 @@
 The goal of this code is to do a soak test of api while we make significant application or infrastructure changes.
 
 There are two soak tests here:
-- `soak_test_send_email.py` will POST an email to api every second.
+- `soak_test_send_notification.py` will POST an email or SMS to api every second.
 - `soak_test_all_servers.py` will do a GET to all our servers (admin, api, dd-api, api-k8s, documentation), on average hitting each server once a second
 
 ## How to configure
@@ -14,19 +14,21 @@ Run the setup.sh to install the python pre-requisites or run in the repo devcont
 
 Default configuration is in the `locust.conf` file. Note that the `host` is the base address of the system you are testing, for example `https://staging.notification.cdssandbox.xyz` **not** `https://api.staging.notification.cdssandbox.xyz`. The "api" prefix will be added in the code.
 
-The python file `soak_test_send_email.py` requires environment variables `API_KEY` and `EMAIL_TEMPLATE_ID`. The template should have no variables.
+The python file `soak_test_send_notification.py` requires environment variables `API_KEY`, `EMAIL_TEMPLATE_ID`, and `SMS_TEMPLATE_ID` . The template should have no personalisation variables.
 
 ```
 API_KEY=gcntfy-notAKey-f6c7cc49-b5b7-4e67-a8ff-24f34be34523-f6c7cc49-b5b7-4e67-a8ff-24f34be34523
 EMAIL_TEMPLATE_ID=f6c7cc49-b5b7-4e67-a8ff-24f34be34523
+SMS_TEMPLATE_ID=f6c7cc49-b5b7-4e67-aeef-24f34be34523
 ```
 These can be in a `.env` file in the soak_test directory.
 
 __See Last Pass note "Soak Test Staging API Key and Template" in Shared-New-Notify-Staging folder__
 
-Note that the default configuration in `locust.conf` is to send one email per second.
-
-You can supply a `--ref` option to `soak_test_send_email.py` that will set the notification's `client_reference`. This is useful in testing that all POSTs were processed successfully.
+Notes:
+- The default configuration in `locust.conf` is to send one email per second.
+- You can supply a `--ref` option to `soak_test_send_notification.py` that will set the notification's `client_reference`. This is useful in testing that all POSTs were processed successfully.
+- You can also supply a `--sms` option that will sens sms instead of email.
 
 ## How to run
 
@@ -37,7 +39,7 @@ There are two ways to run Locust, with the UI or headless.
 Locally you can run the email soak test with:
 
 ```shell
-locust -f ./soak_test_send_email.py --ref=soak-2023-05-30-A
+locust -f ./soak_test_send_notification.py [--ref=soak-2023-05-30-A] [--sms]
 ```
 
 Follow the localhost address that the console will display to get to the UI. It will ask you how many total users and spawned users you want configured. Once setup, you can manually start the tests via the UI and follow the summary data and charts visually.
@@ -53,7 +55,7 @@ locust -f ./soak_test_all_servers.py
 You can pass the necessary parameters to the command line to run in the headless mode. For example:
 
 ```shell
-locust -f ./soak_test_send_email.py --headless --ref=soak-2023-05-30-A
+locust -f ./soak_test_send_notification.py --headless [--ref=soak-2023-05-30-A] [--sms]
 ```
 
 The defaults in `locust.conf` may be overridden by command line options
@@ -64,9 +66,9 @@ The server soak test can be run with
 locust -f ./soak_test_all_servers.py --headless
 ```
 
-## Checking if all emails were sent
+## Checking if all notifications were sent
 
-To check whether all the POSTs from `soak_test_send_email.py` made it into the database, run the "Soak test" query on blazer. The query is already in staging, or you can run:
+To check whether all the POSTs from `soak_test_send_notification.py` made it into the database, run the "Soak test" query on blazer. The query is already in staging, or you can run:
 
 ```sql
 WITH

scripts/soak_test/soak_test_send_notification.py

@@ -0,0 +1,38 @@
+import os
+
+from dotenv import load_dotenv
+from locust import HttpUser, constant_pacing, events, task
+from soak_utils import url_with_prefix
+
+load_dotenv()
+
+
+@events.init_command_line_parser.add_listener
+def _(parser):
+    parser.add_argument("--ref", type=str, default="test", help="reference")
+    parser.add_argument("--sms", action='store_true', help="send sms")
+
+
+class NotifyApiUser(HttpUser):
+    wait_time = constant_pacing(1)  # each user makes one post per second
+
+    def __init__(self, *args, **kwargs):
+        self.host = url_with_prefix(self.host, "api")
+
+        super(NotifyApiUser, self).__init__(*args, **kwargs)
+        self.headers = {"Authorization": f"apikey-v1 {os.getenv('API_KEY')}"}
+        self.email_template = os.getenv("EMAIL_TEMPLATE_ID")
+        self.sms_template = os.getenv("SMS_TEMPLATE_ID")
+        self.email_address = "[email protected]"
+        self.phone_number = "+16135550123"  # INTERNAL_TEST_NUMBER
+        self.reference_id = self.environment.parsed_options.ref
+        self.send_sms = self.environment.parsed_options.sms
+
+    @task(1)
+    def send_notification(self):
+        if self.send_sms:
+            json = {"phone_number": self.phone_number, "template_id": self.sms_template, "reference": self.reference_id}
+            self.client.post("/v2/notifications/sms", json=json, headers=self.headers)
+        else:
+            json = {"email_address": self.email_address, "template_id": self.email_template, "reference": self.reference_id}
+            self.client.post("/v2/notifications/email", json=json, headers=self.headers)