Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add warnings for throttling exceptions #1555

Merged
merged 5 commits into from
Sep 26, 2024
Merged

add warnings for throttling exceptions #1555

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
e1761af
add warnings for throttling exceptions
sastels Sep 25, 2024
491baa1
ai [review] plz
sastels Sep 25, 2024
e0c40b7
add newline
sastels Sep 25, 2024
03ea79e
tweak service-callback-too-many-failures-critical
sastels Sep 25, 2024
ea056bb
Merge branch 'main' into throttling-exception-warning
sastels Sep 26, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 31 additions & 1 deletion aws/eks/cloudwatch_alarms.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -972,7 +972,7 @@ resource "aws_cloudwatch_metric_alarm" "service-callback-too-many-failures-warni

resource "aws_cloudwatch_metric_alarm" "service-callback-too-many-failures-critical" {
count = var.cloudwatch_enabled ? 1 : 0
alarm_name = "service-callback-too-many-failures-warning"
alarm_name = "service-callback-too-many-failures-critical"
alarm_description = "Service reached the max number of callback retries 100 times in 10 minutes"
comparison_operator = "GreaterThanOrEqualToThreshold"
evaluation_periods = "1"
Expand All @@ -984,3 +984,33 @@ resource "aws_cloudwatch_metric_alarm" "service-callback-too-many-failures-criti
treat_missing_data = "notBreaching"
alarm_actions = [var.sns_alert_critical_arn]
}

resource "aws_cloudwatch_metric_alarm" "throttling-exception-warning" {
count = var.cloudwatch_enabled ? 1 : 0
alarm_name = "throttling-exception-warning"
alarm_description = "Have received a throttling exception in the last minute"
comparison_operator = "GreaterThanOrEqualToThreshold"
evaluation_periods = "1"
metric_name = aws_cloudwatch_log_metric_filter.throttling-exceptions[0].metric_transformation[0].name
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The metric_name should be a string value, but it seems to be referencing an array element. Ensure that aws_cloudwatch_log_metric_filter.throttling-exceptions[0].metric_transformation[0].name is correctly defined and accessible.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True that we'd have a problem if those [0] array elements didn't exist, but that doesn't have anything to do with the type of metric_name, and it's the pattern we use everywhere. (and we only declare all these things if cloudwatch_enabled is set so they all work together).

namespace = aws_cloudwatch_log_metric_filter.throttling-exceptions[0].metric_transformation[0].namespace
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The namespace should be a string value, but it seems to be referencing an array element. Ensure that aws_cloudwatch_log_metric_filter.throttling-exceptions[0].metric_transformation[0].namespace is correctly defined and accessible.

period = 60
statistic = "Sum"
threshold = 1
treat_missing_data = "notBreaching"
alarm_actions = [var.sns_alert_warning_arn]
}

resource "aws_cloudwatch_metric_alarm" "many-throttling-exceptions-warning" {
count = var.cloudwatch_enabled ? 1 : 0
alarm_name = "many-throttling-exceptions-warning"
alarm_description = "Have received 100 throttling exception in the last minute"
comparison_operator = "GreaterThanOrEqualToThreshold"
evaluation_periods = "1"
metric_name = aws_cloudwatch_log_metric_filter.throttling-exceptions[0].metric_transformation[0].name
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The metric_name should be a string value, but it seems to be referencing an array element. Ensure that aws_cloudwatch_log_metric_filter.throttling-exceptions[0].metric_transformation[0].name is correctly defined and accessible.

namespace = aws_cloudwatch_log_metric_filter.throttling-exceptions[0].metric_transformation[0].namespace
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The namespace should be a string value, but it seems to be referencing an array element. Ensure that aws_cloudwatch_log_metric_filter.throttling-exceptions[0].metric_transformation[0].namespace is correctly defined and accessible.

period = 60
statistic = "Sum"
threshold = 100
treat_missing_data = "notBreaching"
alarm_actions = [var.sns_alert_warning_arn]
}
13 changes: 13 additions & 0 deletions aws/eks/cloudwatch_log.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -192,3 +192,16 @@ resource "aws_cloudwatch_log_metric_filter" "callback-request-failures" {
value = "1"
}
}

resource "aws_cloudwatch_log_metric_filter" "throttling-exceptions" {
count = var.cloudwatch_enabled ? 1 : 0
name = "throttling-exceptions"
pattern = "ThrottlingException"
log_group_name = aws_cloudwatch_log_group.notification-canada-ca-eks-application-logs[0].name

metric_transformation {
name = "throttling-exceptions"
namespace = "LogMetrics"
value = "1"
}
}
Loading