tutorial: Writing Prometheus Alert Rules with Nickel

[mipmip]

I just published a new blog about Nickel: https://technative.eu/blog/writing_prometheus_alert_rules_with_nickel/

Feedback is highly appreciated.

[yannham]

Hi @mipmip, very cool to see a blogpost about trying Nickel out!

A couple of quick, focused feedback first:

• the normalize_string function can be written more pleasantly using the reverse application operator |> (or sometimes called pipe in other languages, not very far from shell's piping!). x |> f is just f x, but this becomes useful when chaining applications, because x |> f |> g |> h is h (g (f x)), the latter being a bit uglier. So, back to normalize, you can write it like:

  let normalize_string = fun instring => 
    instring
    |> std.string.lowercase
    |> std.string.replace " " "_"
    |> std.string.replace_regex "[%,.]+" ""
  in
  

• it's not mandatory, especially for a first getting started blog post like this, but in general we tend to statically type helper functions when it's easy to do. Here, that would mean writing e.g. let normalize_string : String -> String = ..., and something like
  alert_rule : String -> String -> String -> String -> Number -> String -> _. That would prevent some errors when calling to this functions, for example (and also that the body of the function themselves is ok!)

More general feedback about idiomatic Nickel. Once again, it might depart from your original idea of showing off a very simple example, so feel free to ignore 🙂 However, please at least look at the very last section (middle ground) as I think it would improve readability of your example without changing much of the rest.

In the grand scheme of things, I think I would actually write alert_rule like this:

let alert_rule = {
  config | not_exported = {
    service_name,
    problem_txt_short,
    problem_txt_long,
    duration,
    sev,
    expression,
  },
  alert = "%{normalize_string config.service_name}_%{normalize_string config.problem_txt_short}",
  expr = config.expression,
  for = config.duration,
  labels = {
    team = "devops",
    platform = "aws"
  },
  annotations = {
    summary = "%{config.service_name} is unhealty. %{config.problem_txt_short}",
    description = "%{config.service_name} is unhealty. %{config.problem_txt_long} For at least %{config.duration}.",
    severity = std.string.from_number config.sev,
    }
}

This is a bare record, with a new field config that will hold what were before the parameters of the function. Instead of using the arguments directly as before, I now substituted them for config.xxx.

The metadata not_exported means that config will be ignored during nickel export or any kind of serialization: somehow, this value is only meaningful during runtime, but is not exported. As defined, alert_rule is, so to speak, a partial configuration: you can try to inspect it and it might work (e.g. evaluating alert_rule.labels succeed), but if you try to export it, Nickel will error out with "missing field definition for xxx". So, how do you "apply" this configuration snippet? You can use Nickel's merging operator &! let's see the rest of your example with this new alert_rule:

{
  groups = [
    {
      name = "mycompany",
      rules = [
        alert_rule
        & {
          config = {
            service_name = "DocumentDB",
            problem_txt_short = "freeable memory low",
            problem_txt_long = "The instance has less then 25% freeable memory.",
            duration = "10m",
            sev = 2,
            expression = m%"
              16 /
              (aws_docdb_freeable_memory_average{dbinstance_identifier="tf-created-AABBCCDD"} / (1024^3))
              * 100 < 25
            "%,
          }
        },
        alert_rule
        & {
          config = {
            service_name = "DocumentDB",
            problem_txt_long = "cpu usage too high",
            problem_txt_short = "The instance are using more than 90% of the cpu assigned.",
            duration = "10m",
            sev = 2,
            expression = m%"
              aws_docdb_cpuutilization_average > 90
            "%
          }
        },
      ]
    }
  ]
}

In the end, what's the difference? The goal achieved is similar to functions. You use a combo of undefined fields, Nickel's record ability to self-reference and Nickel's merging instead of application to encode functions in a strange way. But, as mentioned before, alert_rule is not a function, so it can be partially evaluated - as long as you don't need the inputs -. It can be queried by the LSP (you can hover on it, and it will probably show interesting information. Right now, merging-based completion isn't yet implemented, but we can imagine soon that the LSP would autocomplete your arguments inside config as well.

As a second step, you can add contracts, documentation and default value to alert_rule:

config | not_exported = {
    service_name
      | String
      | doc "The name of the service",
    problem_txt_short
      | String
      | doc "The problem description, in less than XX characters",
    problem_txt_long
      | String
      | doc "The full problem description without length limit",
    duration
      | String
      | doc "The duration, as a string containing a decimal number followed by a time unit (s, m, etc.)"
      | default
      = "10m",
    sev
      | Number
      | doc "...",
    expression
      | String
      | doc "The expression of the condition triggering this alert",
  },

You could even define custom contracts to check that problem_txt_short is actually shorter than XX characters, or that the duration is a valid duration. Since it looks like you're a Nix user, it's basically close the NixOS module system, but baked in the Nickel language as a native feature 🙂

Middle ground

However, the former approach might be a bit harder (or is it?) to expose in one short blog post. Could maybe make for a follow-up post 🙃 ? If you want to stick to functions, you can also - as in Nix - use records and destructuring to emulate named arguments, which I think would improve readability. The alert_rule application is a bit hard to follow currently. You could rewrite the definition of alert_rule as:

let alert_rule = fun {
  service_name,
  problem_txt_short,
  problem_txt_long,
  duration,
  sev,
  expression
  }  => {
  # ...
}

This is a function which expects its argument to be a record, and destructure it on-the-fly. To apply it, you now have to pass a record, but which has the advantage of naming things:

alert_rule {
  service_name = "DocumentDB",
  problem_txt_long = "cpu usage too high",
  problem_txt_short = "The instance are using more than 90% of the cpu assigned.",
  duration = "10m",
  sev = 2,
  expression = m%"
    aws_docdb_cpuutilization_average > 90
  "%
}

[mipmip]

Wow this is great feedback! I will definitely improve the blog with suggestions. I feel honored to get this private training 😀 thanks

[yannham]

You're welcome! It makes think that the whole explanation about "modules" should really have a dedicated section in the manual.