Initial stab at facilitating /proc/stat sampling

[oschaaf]

This contains a first stab at a service which can be deployed to containers to sample proc stat, and expose results in Prometheus format over http. Creating an early PR for discussion.

This also has a shell script that will build a standalone binary for the python service, and deploy/start that in the containers that run the istio-injected side car proxies. On top of that there's a few bits and pieces that might come around useful later, if we want to stream the raw time-series data to disk for later inspection.

Please see README.md for more PR details.

[istio-testing]

Hi @oschaaf. Thanks for your PR.

I'm waiting for a istio member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[howardjohn]

Sorry I am not too familiar with this, just wondering how it compares/relates to https://github.com/prometheus/node_exporter?

[mandarjog]

Node exporter looks very promising. It has great coverage. . @oschaaf if it supports higher sampling rate, ability to run on vms and baremetal, then we have full coverage.

[oschaaf]

Looking at node exporter, this is similar, yet different. Summarizing findings after a short exploration:

• stats collection is independent. Node exporter ties stats collection to its networking code, no polling of /metrics means no sampling.
• for when trying to correlate latency and metrics in time, this optionally streams the raw
  samples to storage. (independent of the metrics we're exposing over http).
• This opens /proc/stat once and subsequently calls read()once per second.
  Node exporter repeatedly opens/reads/closes /sys/devices/system/cpu/cpuXXX/[frequency|throttling|etc].
  It seems that the overhead to get to the crown jewels (irq metrics are the goal) are determined by two factors: the rate at which the metrics are polled plus the cpu core count.

I might be overly paranoid, but on the one hand, to me it seems that doing less means less (chance of) introducing noise into the actual measurements we're doing. And also directly streaming collected samples to storage seems like a good way to make sure all captured information is retained for future reference (or at least know about it when something is lost).
OTOH, I can also see that less code is more, and not having a DIY to maintain is worth a lot . Plus having easy (opt-in) access to the full set of metrics exposed by node collector could be handy.

[mandarjog]

@oschaaf apologies for delayed response.

I think that using an existing tool is preferred until it is proven to be an issue, like excessive cost of sampling. It would be better to send PRs to nodeexporter if excessive re-opening of files becomes an issue.

I think we should document this work, and re-open it in the future if nodeexporter is unworkable.

[oschaaf]

Closing this one, as instead #890 will bring up node_exporter when the profileMode which is introduced there gets enabled in `values.yaml'