Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cleanup unused queues #1230

Merged
merged 7 commits into from
Apr 2, 2024
Merged

Cleanup unused queues #1230

merged 7 commits into from
Apr 2, 2024

Conversation

sastels
Copy link
Contributor

@sastels sastels commented Mar 27, 2024

Summary | Résumé

Delete infrastructure related to the send-email-tasks and send-sms-tasks queues. We stopped using these queues a few months ago.

Related Issues | Cartes liées

Test instructions | Instructions pour tester la modification

review dashboards

Release Instructions | Instructions pour le déploiement

None.

Reviewer checklist | Liste de vérification du réviseur

  • This PR does not break existing functionality.
  • This PR does not violate GCNotify's privacy policies.
  • This PR does not raise new security concerns. Refer to our GC Notify Risk Register document on our Google drive.
  • This PR does not significantly alter performance.
  • Additional required documentation resulting of these changes is covered (such as the README, setup instructions, a related ADR or the technical documentation).

⚠ If boxes cannot be checked off before merging the PR, they should be moved to the "Release Instructions" section with appropriate steps required to verify before release. For example, changes to celery code may require tests on staging to verify that performance has not been affected.

@sastels sastels force-pushed the cleanup-unused-queues branch from 38c21a3 to 7c918c0 Compare March 27, 2024 17:54
@sastels sastels marked this pull request as ready for review March 27, 2024 18:15
@sastels sastels requested a review from jimleroyer as a code owner March 27, 2024 18:15
@sastels sastels requested review from ben851 and P0NDER0SA March 27, 2024 18:15
Copy link

Updating alarms ⏰? Great! Please update the Google Sheet and add a 👍 to this message after 🙏

2 similar comments
Copy link

Updating alarms ⏰? Great! Please update the Google Sheet and add a 👍 to this message after 🙏

Copy link

Updating alarms ⏰? Great! Please update the Google Sheet and add a 👍 to this message after 🙏

Copy link
Contributor

@ben851 ben851 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link

github-actions bot commented Apr 2, 2024

Staging: common

✅   Terraform Init: success
✅   Terraform Validate: success
✅   Terraform Format: success
✅   Terraform Plan: success
✅   Conftest: success

⚠️   Warning: resources will be destroyed by this change!

Plan: 0 to add, 2 to change, 4 to destroy
Show summary
CHANGE NAME
update aws_cloudwatch_dashboard.emails[0]
aws_cloudwatch_dashboard.sms[0]
delete aws_cloudwatch_metric_alarm.sqs-email-queue-delay-critical[0]
aws_cloudwatch_metric_alarm.sqs-email-queue-delay-warning[0]
aws_cloudwatch_metric_alarm.sqs-sms-stuck-in-queue-critical[0]
aws_cloudwatch_metric_alarm.sqs-sms-stuck-in-queue-warning[0]

✂   Warning: plan has been truncated! See the full plan in the logs.

Show plan
Resource actions are indicated with the following symbols:
  ~ update in-place
  - destroy

Terraform will perform the following actions:

  # aws_cloudwatch_dashboard.emails[0] will be updated in-place
  ~ resource "aws_cloudwatch_dashboard" "emails" {
      ~ dashboard_body = jsonencode(
          ~ {
              ~ widgets = [
                    {
                        height     = 6
                        properties = {
                            liveData = true
                            metrics  = [
                                [
                                    "AWS/SES",
                                    "Send",
                                ],
                            ]
                            period   = 300
                            region   = "ca-central-1"
                            stacked  = true
                            stat     = "Sum"
                            title    = "Sent emails per 5mn"
                            view     = "timeSeries"
                        }
                        type       = "metric"
                        width      = 9
                        x          = 0
                        y          = 4
                    },
                  ~ {
                      ~ properties = {
                          ~ alarms = [
                                # (9 unchanged elements hidden)
                                "arn:aws:cloudwatch:ca-central-1:239043911459:alarm:ses-complaint-rate-critical",
                              - "arn:aws:cloudwatch:ca-central-1:239043911459:alarm:sqs-email-queue-delay-critical",
                                "arn:aws:cloudwatch:ca-central-1:239043911459:alarm:no-emails-sent-5-minutes-critical",
                                # (1 unchanged element hidden)
                            ]
                            # (1 unchanged attribute hidden)
                        }
                        # (5 unchanged attributes hidden)
                    },
                    {
                        height     = 6
                        properties = {
                            metrics = [
                                [
                                    "AWS/SES",
                                    "Bounce",
                                    {
                                        color = "#ff7f0e"
                                    },
                                ],
                            ]
                            period  = 900
                            region  = "ca-central-1"
                            stacked = true
                            stat    = "Sum"
                            title   = "Bounces per 15mn"
                            view    = "timeSeries"
                        }
                        type       = "metric"
                        width      = 9
                        x          = 9
                        y          = 4
                    },
                    # (1 unchanged element hidden)
                    {
                        height     = 6
                        properties = {
                            annotations = {
                                horizontal = [
                                    {
                                        label = "60 sec"
                                        value = 60
                                    },
                                    {
                                        fill  = "above"
                                        label = "20 sec"
                                        value = 20
                                    },
                                ]
                            }
                            metrics     = [
                                [
                                    "AWS/SQS",
                                    "ApproximateAgeOfOldestMessage",
                                    "QueueName",
                                    "eks-notification-canada-casend-email-high",
                                    {
                                        color  = "#1f77b4"
                                        label  = "Age of oldest message"
                                        region = "ca-central-1"
                                    },
                                ],
                            ]
                            period      = 60
                            region      = "ca-central-1"
                            stacked     = true
                            stat        = "Average"
                            title       = "Average approximate age of oldest message in send-email-high"
                            view        = "timeSeries"
                        }
                        type       = "metric"
                        width      = 8
                        x          = 0
                        y          = 24
                    },
                  ~ {
                      ~ properties = {
                          ~ markdown = <<-EOT
                                # Sending emails
                                
                                Emails are sent with [SES](https://ca-central-1.console.aws.amazon.com/ses/home?region=ca-central-1#dashboard:).
                                
                                Our limits are:
                                - 1,000,000 emails per 24 hour period
                                - 100 emails/second
                                
                              - Emails are sent by Celery through the `deliver_email` task through the [send-email-tasks](https://ca-central-1.console.aws.amazon.com/sqs/v2/home?region=ca-central-1#/queues/https%3A%2F%2Fsqs.ca-central-1.amazonaws.com%2F239043911459%2Feks-notification-canada-casend-email-tasks) queue.
                              + Emails are sent by Celery through the `deliver_email` task through the [send-email-low](https://ca-central-1.console.aws.amazon.com/sqs/v2/home?region=ca-central-1#/queues/https%3A%2F%2Fsqs.ca-central-1.amazonaws.com%2F239043911459%2Feks-notification-canada-casend-email-low), [send-email-medium](https://ca-central-1.console.aws.amazon.com/sqs/v2/home?region=ca-central-1#/queues/https%3A%2F%2Fsqs.ca-central-1.amazonaws.com%2F239043911459%2Feks-notification-canada-casend-email-medium), or [send-email-high](https://ca-central-1.console.aws.amazon.com/sqs/v2/home?region=ca-central-1#/queues/https%3A%2F%2Fsqs.ca-central-1.amazonaws.com%2F239043911459%2Feks-notification-canada-casend-email-high) queues.
                                
                                ## Message flow
                                
                              - After a notification has been created in the database, Celery sends the email to the provider using the deliver_email Celery task. This Celery task is assigned to the SQS queue eks-notification-canada-casend-email-tasks, unless a specific queue has been assigned to the queue (for example: eks-notification-canada-capriority-tasks for priority notifications or eks-notification-canada-cabulk-tasks through the API REST service). This task calls the AWS SES API to send a text message.
                              + After a notification has been created in the database, Celery sends the email to the provider using the deliver_email Celery task. This Celery task is assigned to the send-email-low, send-email-medium, or send-email-high SQS queue depending on the email's priority. This task calls the AWS SES API to send a text message.
                                
                                ## Delivery receipts
                                
                                Receipts from SES are dispatched to SNS -> [Lambda](https://ca-central-1.console.aws.amazon.com/lambda/home?region=ca-central-1#/functions/ses-to-sqs-email-callbacks) -> [SQS](https://ca-central-1.console.aws.amazon.com/sqs/v2/home?region=ca-central-1#/queues/https%3A%2F%2Fsqs.ca-central-1.amazonaws.com%2F239043911459%2Feks-notification-canada-cadelivery-receipts) in the `delivery-receipts` queue.
                                
                                A delay in this queue means that we are slow to process delivery receipts (delivered, bounce, complaints).
                            EOT
                        }
                        # (5 unchanged attributes hidden)
                    },
                    {
                        height     = 6
                        properties = {
                            metrics = [
                                [
                                    "AWS/SQS",
                                    "ApproximateAgeOfOldestMessage",
                                    "QueueName",
                                    "eks-notification-canada-cadelivery-receipts",
                                ],
                            ]
                            period  = 60
                            region  = "ca-central-1"
                            stacked = true
                            stat    = "Average"
                            title   = "Approximate age of oldest message in delivery-receipts"
                            view    = "timeSeries"
                        }
                        type       = "metric"
                        width      = 9
                        x          = 9
                        y          = 16
                    },
                    # (2 unchanged elements hidden)
                    {
                        height     = 6
                        properties = {
                            metrics = [
                                [
                                    "NotificationCanadaCa",
                                    "staging_notifications_celery_tasks_process_ses_results",
                                    "metric_type",
                                    "counter",
                                ],
                            ]
                            period  = 300
                            region  = "ca-central-1"
                            stacked = true
                            stat    = "Sum"
                            title   = "Celery: Number of SES delivery receipts processed"
                            view    = "timeSeries"
                        }
                        type       = "metric"
                        width      = 9
                        x          = 9
                        y          = 10
                    },
                  ~ {
                      ~ width      = 9 -> 8
                      ~ x          = 9 -> 8
                      ~ y          = 42 -> 36
                        # (3 unchanged attributes hidden)
                    },
                  ~ {
                      ~ width      = 9 -> 8
                      ~ y          = 48 -> 36
                        # (4 unchanged attributes hidden)
                    },
                  ~ {
                      ~ width      = 9 -> 8
                      ~ x          = 9 -> 16
                      ~ y          = 48 -> 36
                        # (3 unchanged attributes hidden)
                    },
                    {
                        height     = 2
                        properties = {
                            markdown = <<-EOT
                                # Delivery Queues
                            EOT
                        }
                        type       = "text"
                        width      = 24
                        x          = 0
                        y          = 22
                    },
                  - {
                      - height     = 6
                      - properties = {
                          - metrics = [
                              - [
                                  - "AWS/SQS",
                                  - "ApproximateAgeOfOldestMessage",
                                  - "QueueName",
                                  - "eks-notification-canada-casend-email-tasks",
                                  - {
                                      - color = "#1f77b4"
                                    },
                                ],
                            ]
                          - period  = 60
                          - region  = "ca-central-1"
                          - stacked = true
                          - stat    = "Average"
                          - title   = "Average approximate age of oldest message in send-email-tasks"
                          - view    = "timeSeries"
                        }
                      - type       = "metric"
                      - width      = 9
                      - x          = 0
                      - y          = 42
                    },
                  - {
                      - height     = 6
                      - properties = {
                          - metrics = [
                              - [
                                  - "AWS/SQS",
                                  - "ApproximateNumberOfMessagesVisible",
                                  - "QueueName",
                                  - "eks-notification-canada-casend-email-tasks",
                                ],
                            ]
                          - period  = 60
                          - region  = "ca-central-1"
                          - stacked = false
                          - stat    = "Average"
                          - title   = "Approximate number of messages in send-email-tasks"
                          - view    = "timeSeries"
                        }
                      - type       = "metric"
                      - width      = 9
                      - x          = 0
                      - y          = 36
                    },
                  - {
                      - height     = 6
                      - properties = {
                          - metrics = [
                              - [
                                  - "AWS/SQS",
                                  - "ApproximateAgeOfOldestMessage",
                                  - "QueueName",
                                  - "eks-notification-canada-casend-email-tasks",
                                  - {
                                      - color = "#1f77b4"
                                    },
                                ],
                            ]
                          - period  = 60
                          - region  = "ca-central-1"
                          - stacked = true
                          - stat    = "Average"
                          - title   = "Average approximate age of oldest message in send-email-tasks"
                          - view    = "timeSeries"
                        }
                      - type       = "metric"
                      - width      = 9
                      - x          = 9
                      - y          = 36
                    },
                    {
                        height     = 6
                        properties = {
                            annotations = {
                                horizontal = [
                                    {
                                        label = "45 min"
                                        value = 2700
                                    },
                                    {
                                        fill  = "above"
                                        label = "30 min"
                                        value = 1800
                                    },
                                ]
                            }
                            metrics     = [
                                [
                                    "AWS/SQS",
                                    "ApproximateAgeOfOldestMessage",
                                    "QueueName",
                                    "eks-notification-canada-casend-email-medium",
                                    {
                                        color  = "#1f77b4"
                                        label  = "Age of oldest message"
                                        region = "ca-central-1"
                                    },
                                ],
                            ]
                            period      = 60
                            region      = "ca-central-1"
                            stacked     = true
                            stat        = "Average"
                            title       = "Average approximate age of oldest message in send-email-medium"
                            view        = "timeSeries"
                        }
                        type       = "metric"
                        width      = 8
                        x          = 8
                        y          = 24
                    },
                    # (3 unchanged elements hidden)
                ]
            }
        )
        id             = "Emails"
        # (2 unchanged attributes hidden)
    }

  # aws_cloudwatch_dashboard.sms[0] will be updated in-place
  ~ resource "aws_cloudwatch_dashboard" "sms" {
      ~ dashboard_body = jsonencode(
          ~ {
              ~ widgets = [
                  ~ {
                      ~ properties = {
                          ~ alarms = [
                                # (1 unchanged element hidden)
                                "arn:aws:cloudwatch:ca-central-1:239043911459:alarm:sns-sms-success-rate-canadian-numbers-warning",
                              - "arn:aws:cloudwatch:ca-central-1:239043911459:alarm:sqs-sms-stuck-in-queue-warning",
                              - "arn:aws:cloudwatch:ca-central-1:239043911459:alarm:sqs-sms-stuck-in-queue-critical",
                                "arn:aws:cloudwatch:ca-central-1:239043911459:alarm:sqs-send-sms-high-queue-delay-warning",
                                # (7 unchanged elements hidden)
                            ]
                            # (1 unchanged attribute hidden)
                        }
                        # (5 unchanged attributes hidden)
                    },
                    {
                        height     = 6
                        properties = {
                            metrics = [
                                [
                                    "AWS/SNS",
                                    "NumberOfNotificationsDelivered",
                                    "PhoneNumber",
                                    "PhoneNumberDirect",
                                ],
                            ]
                            period  = 300
                            region  = "ca-central-1"
                            stacked = true
                            stat    = "Sum"
                            title   = "SMS delivered per 5m"
                            view    = "timeSeries"
                        }
                        type       = "metric"
                        width      = 9
                        x          = 0
                        y          = 5
                    },
                    # (7 unchanged elements hidden)
                    {
                        height     = 6
                        properties = {
                            metrics = [
                                [
                                    "AWS/Lambda",
                                    "Invocations",
                                    "FunctionName",
                                    "sns-to-sqs-sms-callbacks",
                                ],
                                [
                                    ".",
                                    "Errors",
                                    ".",
                                    ".",
                                    {
                                        color = "#d62728"
                                        yAxis = "right"
                                    },
                                ],
                            ]
                            period  = 300
                            region  = "ca-central-1"
                            stacked = true
                            stat    = "Sum"
                            title   = "Lambda invocations per 5m"
                            view    = "timeSeries"
                        }
                        type       = "metric"
                        width      = 9
                        x          = 0
                        y          = 39
                    },
                  ~ {
                      ~ properties = {
                          ~ markdown = <<-EOT
                                ## Limits
                                - SNS [maximum sending rate](https://docs.aws.amazon.com/general/latest/gr/sns.html#limits_sns): 20 SMS/second
                                - [Spending limit](https://ca-central-1.console.aws.amazon.com/sns/v3/home?region=ca-central-1#/mobile/text-messaging) of 30,000 USD/month
                                
                                ## Message flow
                              - After a notification has been created in the database, Celery sends the SMS to the provider using the `deliver_sms` Celery task. This Celery task is assigned to the SQS queue [eks-notification-canada-casend-sms-tasks](#/queues/https%3A%2F%2Fsqs.ca-central-1.amazonaws.com%2F239043911459%2Feks-notification-canada-casend-sms-tasks), unless a specific queue has been assigned to the queue (for example priority templates, SMS sent by the Notify service etc.). This task calls the SNS API to send a text message.
                              + After a notification has been created in the database, Celery sends the SMS to the provider using the `deliver_sms` Celery task. This Celery task is assigned to the SQS queue [eks-notification-canada-casend-sms-low](#/queues/https%3A%2F%2Fsqs.ca-central-1.amazonaws.com%2F239043911459%2Feks-notification-canada-casend-sms-low), [eks-notification-canada-casend-sms-medium](#/queues/https%3A%2F%2Fsqs.ca-central-1.amazonaws.com%2F239043911459%2Feks-notification-canada-casend-sms-medium), or [eks-notification-canada-casend-sms-high](#/queues/https%3A%2F%2Fsqs.ca-central-1.amazonaws.com%2F239043911459%2Feks-notification-canada-casend-sms-high) depending on the SMS priority. This task calls the SNS API to send a text message.
                                
                                ## SNS IDs
                                SNS keeps track of SMS with a `messageId`, the value of SNS' `messageId` is stored in the `Notification` object in the `reference` column.
                                
                                ## Logging
                                Celery tasks output multiple messages when processing tasks/calling the SNS API, take a look at the relevant Celery code to know more.
                                
                                After an SMS has been sent by SNS, the delivery details are stored in CloudWatch Log groups:
                                
                                - [sns/ca-central-1/239043911459/DirectPublishToPhoneNumber](#logsV2:log-groups/log-group/sns$252Fca-central-1$252F239043911459$252FDirectPublishToPhoneNumber) for successful deliveries
                                - [sns/ca-central-1/239043911459/DirectPublishToPhoneNumber/Failure](#logsV2:log-groups/log-group/sns$252Fca-central-1$252F239043911459$252FDirectPublishToPhoneNumber$252FFailure) for failures
                                
                                ## Phone numbers
                                
                                SMS sent in `ca-central-1` use random phone numbers managed by AWS.
                                
                                ### ⚠️  SNS in `us-west-2`
                                If a Notify service has an inbound number attached, SMS will be sent with SNS using a long code phone number ordered on Pinpoint in the `us-west-2` region. Statistics for this region and alarms are **not visible on this dashboard**.
                            EOT
                        }
                        # (5 unchanged attributes hidden)
                    },
                    {
                        height     = 6
                        properties = {
                            metrics = [
                                [
                                    "NotificationCanadaCa",
                                    "staging_notifications_celery_tasks_deliver_sms",
                                    "metric_type",
                                    "counter",
                                ],
                            ]
                            period  = 300
                            region  = "ca-central-1"
                            stacked = true
                            stat    = "Sum"
                            title   = "Number of deliver_sms Celery tasks per 5m"
                            view    = "timeSeries"
                        }
                        type       = "metric"
                        width      = 9
                        x          = 9
                        y          = 17
                    },
                    # (13 unchanged elements hidden)
                ]
            }
        )
        id             = "SMS"
        # (2 unchanged attributes hidden)
    }

  # aws_cloudwatch_metric_alarm.sqs-email-queue-delay-critical[0] will be destroyed
  # (because aws_cloudwatch_metric_alarm.sqs-email-queue-delay-critical is not in configuration)
  - resource "aws_cloudwatch_metric_alarm" "sqs-email-queue-delay-critical" {
      - actions_enabled           = true -> null
      - alarm_actions             = [
          - "arn:aws:sns:ca-central-1:239043911459:alert-critical",
        ] -> null
      - alarm_description         = "ApproximateAgeOfOldestMessage in email queue >= 45 minutes for 5 minutes" -> null
      - alarm_name                = "sqs-email-queue-delay-critical" -> null
      - arn                       = "arn:aws:cloudwatch:ca-central-1:239043911459:alarm:sqs-email-queue-delay-critical" -> null
      - comparison_operator       = "GreaterThanOrEqualToThreshold" -> null
      - datapoints_to_alarm       = 0 -> null
      - dimensions                = {
          - "QueueName" = "eks-notification-canada-casend-email-tasks"
        } -> null
      - evaluation_periods        = 5 -> null
      - id                        = "sqs-email-queue-delay-critical" -> null
      - insufficient_data_actions = [
          - "arn:aws:sns:ca-central-1:239043911459:alert-warning",
        ] -> null
      - metric_name               = "ApproximateAgeOfOldestMessage" -> null
      - namespace                 = "AWS/SQS" -> null
      - ok_actions                = [
          - "arn:aws:sns:ca-central-1:239043911459:alert-ok",
        ] -> null
      - period                    = 60 -> null
      - statistic                 = "Maximum" -> null
      - tags                      = {} -> null
      - tags_all                  = {} -> null
      - threshold                 = 2700 -> null
      - treat_missing_data        = "missing" -> null
    }

  # aws_cloudwatch_metric_alarm.sqs-email-queue-delay-warning[0] will be destroyed
  # (because aws_cloudwatch_metric_alarm.sqs-email-queue-delay-warning is not in configuration)
  - resource "aws_cloudwatch_metric_alarm" "sqs-email-queue-delay-warning" {
      - actions_enabled           = true -> null
      - alarm_actions             = [
          - "arn:aws:sns:ca-central-1:239043911459:alert-warning",
        ] -> null
      - alarm_description         = "ApproximateAgeOfOldestMessage in email queue >= 30 minutes for 5 minutes" -> null
      - alarm_name                = "sqs-email-queue-delay-warning" -> null
      - arn                       = "arn:aws:cloudwatch:ca-central-1:239043911459:alarm:sqs-email-queue-delay-warning" -> null
      - comparison_operator       = "GreaterThanOrEqualToThreshold" -> null
      - datapoints_to_alarm       = 0 -> null
      - dimensions                = {
          - "QueueName" = "eks-notification-canada-casend-email-tasks"
        } -> null
      - evaluation_periods        = 5 -> null
      - id                        = "sqs-email-queue-delay-warning" -> null
      - insufficient_data_actions = [] -> null
      - metric_name               = "ApproximateAgeOfOldestMessage" -> null
      - namespace                 = "AWS/SQS" -> null
      - ok_actions                = [] -> null
      - period                    = 60 -> null
      - statistic                 = "Maximum" -> null
      - tags                      = {} -> null
      - tags_all                  = {} -> null
      - threshold                 = 1800 -> null
      - treat_missing_data        = "missing" -> null
    }

  # aws_cloudwatch_metric_alarm.sqs-sms-stuck-in-queue-critical[0] will be destroyed
  # (because aws_cloudwatch_metric_alarm.sqs-sms-stuck-in-queue-critical is not in configuration)
  - resource "aws_cloudwatch_metric_alarm" "sqs-sms-stuck-in-queue-critical" {
      - actions_enabled           = true -> null
      - alarm_actions             = [
          - "arn:aws:sns:ca-central-1:239043911459:alert-critical",
        ] -> null
      - alarm_description         = "ApproximateAgeOfOldestMessage in SMS queue is older than 15 minutes for 5 minutes" -> null
      - alarm_name                = "sqs-sms-stuck-in-queue-critical" -> null
      - arn                       = "arn:aws:cloudwatch:ca-central-1:239043911459:alarm:sqs-sms-stuck-in-queue-critical" -> null
      - comparison_operator       = "GreaterThanOrEqualToThreshold" -> null
      - datapoints_to_alarm       = 0 -> null
      - dimensions                = {
          - "QueueName" = "eks-notification-canada-casend-sms-tasks"
        } -> null
      - evaluation_periods        =...
Show Conftest results
WARN - plan.json - main - Missing Common Tags: ["aws_acm_certificate.client_vpn"]
WARN - plan.json - main - Missing Common Tags: ["aws_athena_workgroup.ad_hoc"]
WARN - plan.json - main - Missing Common Tags: ["aws_athena_workgroup.build_tables"]
WARN - plan.json - main - Missing Common Tags: ["aws_athena_workgroup.primary"]
WARN - plan.json - main - Missing Common Tags: ["aws_athena_workgroup.support"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_event_rule.aws_health[0]"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_log_group.route53_resolver_query_log[0]"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_log_group.sns_deliveries[0]"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_log_group.sns_deliveries_failures[0]"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_log_group.sns_deliveries_failures_us_west_2[0]"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_log_group.sns_deliveries_us_west_2[0]"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.bulk-bulk-not-being-processed-critical[0]"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.bulk-bulk-not-being-processed-warning[0]"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.bulk-inflights-not-being-processed-critical[0]"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.bulk-inflights-not-being-processed-warning[0]"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.bulk-not-being-processed-critical[0]"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.bulk-not-being-processed-warning[0]"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.contact-3-500-error-15-minutes-critical[0]"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.document-download-bucket-size-warning[0]"]
WARN - plan.json -...

Copy link

github-actions bot commented Apr 2, 2024

Staging: eks

✅   Terraform Init: success
✅   Terraform Validate: success
✅   Terraform Format: success
✅   Terraform Plan: success
✅   Conftest: success

Plan: 0 to add, 1 to change, 0 to destroy
Show summary
CHANGE NAME
update aws_cloudwatch_dashboard.notify_system[0]
Show plan
Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # aws_cloudwatch_dashboard.notify_system[0] will be updated in-place
  ~ resource "aws_cloudwatch_dashboard" "notify_system" {
      ~ dashboard_body = jsonencode(
          ~ {
              ~ widgets = [
                    # (5 unchanged elements hidden)
                    {
                        height     = 4
                        properties = {
                            metrics   = [
                                [
                                    "ContainerInsights/Prometheus",
                                    "kube_deployment_status_replicas_available",
                                    "namespace",
                                    "notification-canada-ca",
                                    "ClusterName",
                                    "notification-canada-ca-staging-eks-cluster",
                                    "deployment",
                                    "admin",
                                    {
                                        color  = "#69ae34"
                                        label  = "admin"
                                        region = "ca-central-1"
                                    },
                                ],
                                [
                                    "...",
                                    "document-download-api",
                                    {
                                        label  = "document-download-api"
                                        region = "ca-central-1"
                                    },
                                ],
                                [
                                    "...",
                                    "documentation",
                                    {
                                        label  = "documentation"
                                        region = "ca-central-1"
                                    },
                                ],
                                [
                                    "...",
                                    "api",
                                    {
                                        label  = "api-k8s"
                                        region = "ca-central-1"
                                    },
                                ],
                            ]
                            period    = 60
                            region    = "ca-central-1"
                            sparkline = true
                            stat      = "Maximum"
                            title     = "Application pods"
                            view      = "singleValue"
                        }
                        type       = "metric"
                        width      = 9
                        x          = 15
                        y          = 17
                    },
                  ~ {
                      ~ properties = {
                          ~ alarms = [
                                # (25 unchanged elements hidden)
                                "arn:aws:cloudwatch:ca-central-1:239043911459:alarm:sqs-sms-stuck-in-queue-critical",
                              - "arn:aws:cloudwatch:ca-central-1:239043911459:alarm:sqs-email-queue-delay-critical",
                                "arn:aws:cloudwatch:ca-central-1:239043911459:alarm:sns-sms-success-rate-canadian-numbers-critical",
                                # (18 unchanged elements hidden)
                            ]
                            # (2 unchanged attributes hidden)
                        }
                        # (5 unchanged attributes hidden)
                    },
                    {
                        height     = 4
                        properties = {
                            metrics   = [
                                [
                                    "ContainerInsights",
                                    "cluster_node_count",
                                    "ClusterName",
                                    "notification-canada-ca-staging-eks-cluster",
                                    {
                                        color  = "#dfb52c"
                                        label  = "Node Count"
                                        region = "ca-central-1"
                                    },
                                ],
                            ]
                            period    = 60
                            region    = "ca-central-1"
                            sparkline = true
                            stat      = "Maximum"
                            title     = "Nodes"
                            view      = "singleValue"
                        }
                        type       = "metric"
                        width      = 3
                        x          = 12
                        y          = 17
                    },
                    # (16 unchanged elements hidden)
                ]
            }
        )
        id             = "Notify-System-Overview"
        # (2 unchanged attributes hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

─────────────────────────────────────────────────────────────────────────────

Saved the plan to: plan.tfplan

To perform exactly these actions, run the following command to apply:
    terraform apply "plan.tfplan"
Show Conftest results
WARN - plan.json - main - Cloudwatch log metric pattern is invalid: ["aws_cloudwatch_log_metric_filter.celery-error[0]"]
WARN - plan.json - main - Cloudwatch log metric pattern is invalid: ["aws_cloudwatch_log_metric_filter.scanfiles-timeout[0]"]
WARN - plan.json - main - Missing Common Tags: ["aws_acm_certificate.notification-canada-ca"]
WARN - plan.json - main - Missing Common Tags: ["aws_acm_certificate.notification-canada-ca-alt[0]"]
WARN - plan.json - main - Missing Common Tags: ["aws_alb.notification-canada-ca"]
WARN - plan.json - main - Missing Common Tags: ["aws_alb_listener.internal_alb_tls"]
WARN - plan.json - main - Missing Common Tags: ["aws_alb_listener.notification-canada-ca"]
WARN - plan.json - main - Missing Common Tags: ["aws_alb_target_group.internal_nginx_http"]
WARN - plan.json - main - Missing Common Tags: ["aws_alb_target_group.notification-canada-ca-admin"]
WARN - plan.json - main - Missing Common Tags: ["aws_alb_target_group.notification-canada-ca-api"]
WARN - plan.json - main - Missing Common Tags: ["aws_alb_target_group.notification-canada-ca-document"]
WARN - plan.json - main - Missing Common Tags: ["aws_alb_target_group.notification-canada-ca-document-api"]
WARN - plan.json - main - Missing Common Tags: ["aws_alb_target_group.notification-canada-ca-documentation"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_log_group.notification-canada-ca-eks-application-logs[0]"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_log_group.notification-canada-ca-eks-cluster-logs[0]"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_log_group.notification-canada-ca-eks-prometheus-logs[0]"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.admin-evicted-pods[0]"]
WARN - plan.json - main - Missing Common Tags: ["aws_cloudwatch_metric_alarm.admin-pods-high-cpu-warning[0]"]
WARN - plan.json - main - Missing Common Tags:...

@sastels sastels merged commit a7f6194 into main Apr 2, 2024
3 checks passed
@sastels sastels deleted the cleanup-unused-queues branch April 2, 2024 17:27
@whabanks whabanks mentioned this pull request Apr 3, 2024
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants