Slack webhook used in workflows no longer working

[andrewleith]

Describe the bug

The slack webhook used in a number of workflows is no longer working due to invalid JSON. Nothing has changed here, but it looks as though we are using single quotes in our JSON instead of double quotes - which is not valid. It is possible that the slack code has been updated recently to be more strict.

Potentially affected workflows:

notification-admin

• cypress-staging.yaml
• docker.yaml
• s3-backup.yml
• test_prod_config.yaml
• test.yaml

notification-adr

• s3-backup.yml

notification-api

• build_and_push_performance_test.yml
• docker.yaml
• lambda_production.yml
• lambda_staging.yml
• performance.yml
• s3-backup.yml
• test.yaml

notification-document-download-api

• docker.yaml
• s3-backup.yml

notification-documentation

• docker.yml
• master.yml
• s3-backup.yml

notification-lambdas

• s3-backup.yml

notification-manifests

• helmfile_production_apply.yaml
• helmfile_staging_apply_specific_app.yaml
• helmfile_staging_apply.yaml
• merge_to_main_production.yaml
• merge_to_main_staging.yaml
• s3-backup.yml
• smoke_test.yaml

notification-system-status-frontend

• prod_upload_to_s3.yml
• upload_to_s3.yml

notification-terraform

• check_eks_ami_update.yml
• check_eks_cluster_update.yml
• check_rds_cluster_update.yml
• s3-backup.yml
• sanitize_production_sms_usage.yml
• sanitize_staging_sms_usage.yml
• workflow_failure.yml

notification-utils

• s3-backup.yml

SEV-2 Major

Expected behavior

When a workflow fails, a notification should be sent to slack to alert us.

Impact

Describe overall user/system impact to consider when prioritizing this issue.

If applicable

Impact on Notify users:

Impact on Recipients:

Impact on Notify team:

• Unaware of certain issues happening in the system

[andrewleith]

Admin has been fixed in cds-snc/notification-admin#2041

[andrewleith]

API fix: cds-snc/notification-api#2413

[andrewleith]

document-download-api: cds-snc/notification-document-download-api#209

[andrewleith]

notification-documentation: cds-snc/notification-documentation#171

[andrewleith]

manifests:
cds-snc/notification-manifests#3295
TF:
cds-snc/notification-terraform#1715