Production and use of logs, metrics

[iawells]

This is a catch-all discussion on logging, metrics and how the system should produce and consume them.

Top-down view:

Operators require logs and metrics to

• proactively identify trends that indicate a change will at some point be required (e.g. 'our load is increasing and we will run out on day X'
• set up alerts that will page them when a crisis starts to happen (e.g. 'our subscribers can no longer access the internet')
• root-cause problems ('our subscribers can no longer access the internet because thing X has a problem, and I now understand what I must do to fix that')
• set up alert-like triggers to begin an automated workflow (this can be LCM: 'app scales out when...', 'we murder this pod when...'; this can be automated repair: 'if this alert happens, reroute traffic away')

Logs and metrics are produced in multiple levels of the stack:

• the network fabric
• the server hardware: fan speeds, temperatures, etc.
• the platform software: the OS, Kubernetes, and other components
• the application: low-level internal component information and service-level KPIs

A problem may be located (q.v.) by using one source of information or by using multiple sources of information. Location typically involves identifying a broader area and 'drilling down' into that area to verify a hypothesis on the type of fault during root-cause.

On the k8s/app boundary, there are some questions:

• app stdout is recorded by k8s and can be converted to logs. Applications can, alternatively, implement their own logging. How should each of these be used?
• apps sometimes require metrics from the platform to calculate their own KPIs or work on their orchestration. How should they get hold of these metrics?

Logging and metric export styles:

• logs can be unstructured (lines of text, typically exported via the syslog protocol, stdout, or a stream and processed using regexps specific to different producers)
• logs can be structured (fluentd/logstash; a dictionary of items, structured at the source, so that regexp decomposition is not required).

Should we pick one for a best practice? Should we detail how it should be used?

• metrics can be exported using traditional network protocols (netconf, SNMP) or more modern app-design techniques

What could we recommend as a best practice?

• alerts (based particularly on thresholds) can be created on metrics at the producer or on a central system where the metrics go. App-local alerts can be processed locally, making a tight and robust feedback loop. Central alerts can consult data from multiple locations (see above).

What could we recommend as a best practice?

[electrocucaracha]

Regarding roles and responsibilities, log entries are implemented by the CNF developers but consumed by the CNF operators, so the reasons described before(proactively identify trends, set up alerts, root-cause problems, etc.) establish the criteria for adding log entries right?