Skip to content

Latest commit

 

History

History
173 lines (142 loc) · 4.39 KB

slides.md

File metadata and controls

173 lines (142 loc) · 4.39 KB
Raw
theme background title info class drawings transition mdc overviewSnapshots
seriph
Introduction to Grafana alerting
## Slidev Starter Template Presentation slides for developers. Learn more at [Sli.dev](https://sli.dev)
text-center
persist
slide-left
true
true

Grafana alerting

A Short Introduction

Press Space for next page

Grafana alerting in a nutshell

  • Automatically run queries on an interval
  • Define rule to determine whether alerts should be generated
  • Use policies to route alerts to appropriate contact point
  • Use templates to format the alert message

transition: slide-up

Anatomy of an Alert Rule

Query(/ies)

  • Can be created from existing Panel
    Edit -> Alert -> New alert rule
  • Have a time range and min interval
    • Adjust these based on how you want your alert to behave
    • Set range only as big as needed
  • Click Preview to show query results

Each time series generated by queries becomes an alert instance

  • One alert rule can monitor conditions for multiple time series independently!
  • Choose appropriate GROUP BYs to make alert more or less selective

transition: slide-up

Expressions

  • Apply some processing to time series/previous expression result
  • Reduce: Apply function to time series, e.g. max, mean, last
    • Might need to choose different Mode
  • Threshold: Compare result to value/range
  • Math: Perform arbitrary calculation on expression(s)
    • $B > 70
    • $A - $B
    • again: applied to each time series
    • extremely powerful!
    • not super well documented

transition: slide-up

Evaluation Behavior

  • Folder (for tidiness)
  • Evaluation group
    • Decides evaluation interval
  • Pending period
    • Delays firing of alert rule
    • Only makes sense if larger than evaluation interval
  • No Data handling
    • If query returns no data at all, what should happen?

transition: slide-up

Annotations

  • Summary and Description
  • Can use templating:
    • Evaluated per alert instance
    • Access labels:
      {{ index $labels "labelName" }}
      • Determined by query groupBy
    • Access values:
      {{ index $values "A" }}
  • Can generate more understandable and actionable message
CPU usage for {{ index $labels "instance" }} has exceeded 80% ({{ index $values "A" }}) for the last 5 minutes.

Labels

  • Manually add labels
  • Severity of alert, (sub)team, …
  • Used to route alerts

Policies

  • Match on labels
  • Specify contact point
  • Configure mute timings
  • Other timing options

Contact points

  • e.g. Slack channel
  • Also configure which message and title templates should be used!
    • Different teams can use different message template

Notification templates

  • Use go template syntax

  • Generate message based on list of alerts

  • Can access labels, annotations and values

    • Including the summary definitely makes sense

  • Can include links to the linked panel, dashboard, alert rule or for generating a silence

  • Slack flavoured markup is really annoying…

layout: image image: https://grafana.com/media/docs/alerting/how-notification-templates-works.png backgroundSize: contain

https://grafana.com/docs/grafana/latest/alerting/