Export job run times via Prometheus

[csweichel]

As operator for our dev-environment I want to have an overview of the job run times for some jobs. For example, if our build job suddenly takes much longer that may be something I want to react to.

A job-metrics plugin could provide this information on a Prometheus endpoint. This plugin would get a list of filter expressions and names. For each such entry we would produce a histogram of build times and a counter of successful builds, failed builds and total builds.

The configuration for this plugin could look like:

metrics:
# name will become part of the metric names which in this case would be
#   werft_job_metric_werft_build_duration_sec
#   werft_job_metric_werft_build_total
#   werft_job_metric_werft_build_success
#   werft_job_metric_werft_build_failure
- name: werft_build
  # filter are "AND'ed" filter expressions. A job has to match all of them to become part of the metric
  filter:
  - or: ["name ~== -build"]
  - or: ["repo.owner == csweichel"]
  - or: ["repo.repo == werft"]
  # histogram lists the time steps of duration the histogram
  histogram:
  - 0.01
  - 1
  - 10
  - 30
  - 60