runtime/metrics: add GOMAXPROCS, NumCgoCall, GC CPU time

[hub-adda]

Hi
I was testing and using the runtime/metrics package introduced in 1.16
I'm using go version 1.16.5.
I didn't find the following metrics that exists in runtime package in the runtime/metrics package :

• func NumCgoCall() int64
• func (r *MemProfileRecord) InUseBytes() int64
• func (r *MemProfileRecord) InUseObjects() int64

Does it make sense to add them in runtime/metrics pacake?

Does it makes sense to add those values (which are static) as well?

• func Version() string
• func GOMAXPROCS(n int) int
• func NumCPU() int
  Thanks a lot, G.

[seankhliao]

cc @mknyszek

[mknyszek]

Hi
I was testing and using the runtime/metrics package introduced in 1.16
I'm using go version 1.16.5.
I didn't find the following metrics that exists in runtime package in the runtime/metrics package :

• func NumCgoCall() int64
• func (r *MemProfileRecord) InUseBytes() int64
• func (r *MemProfileRecord) InUseObjects() int64

NumCgoCall could be a good candidate. The other two should already be accounted for as /gc/heap/objects:objects and /memory/classes/heap/objects:bytes respectively.

Does it make sense to add them in runtime/metrics pacake?

Does it makes sense to add those values (which are static) as well?

• func Version() string
• func GOMAXPROCS(n int) int
• func NumCPU() int

Version and NumCPU aren't quite what the package was made for. It doesn't really make sense to track these values over time, since they're static. Though, simultaneously, I understand how analyzing something like this across a fleet is nice, and it's also nice to have one API to export everything. I'll think about it.

GOMAXPROCS on the other hand is probably a good candidate. It can change over time (usually it doesn't, but it can) and it seems like it might be useful to correlate against other metrics. It would cost us very little to support this. Same goes for runtime/debug.SetGCPercent.

Thanks a lot, G.

Thanks for the suggestions!

[rsc]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

[rsc]

ping @mknyszek. any new thoughts?

[mpx]

Please consider exporting the total time spent perfoming GC as well (work.totaltime or similar). It would be more useful than the processed GCCPUFraction currently provided by runtime.Memstats

[mknyszek]

@mpx I'm already ahead of you on that one. GCCPUFraction is intentionally missing from runtime/metrics because of issues with it. Instead, I'm hoping to export a distribution of GC CPU utilizations by GC cycle. It's still preprocessed, but given that the API is sample-based rather than event-based, I think it's the best we can do for the moment.

Total work time also seems fine to me, though it doesn't seem quite as useful without a "total CPU time" counter also (sample them together, take a diff to determine GC CPU utilization since the last sample). We could expose something like that, but measuring exact CPU time is unfortunately kind of tough. In the runtime we just make some assumptions that luckily hold true in 99% (maybe more) of cases. It might be useful to export those assumptions anyway, but I think the aforementioned distribution of GC CPU time is more directly useful for monitoring (and includes those assumptions, necessarily, anyway).

Do you have another use-case for total time that I'm not considering?

[hub-adda]

Hi all, thanks for the efforts you invest in that issue. As a telemetry consumer telemetry, we use external libraries to measure the total cpu usage of the process. I'm not sure that this is the right place to discuss it. What are the limitations of the solution you present?
Thanks Gil

[mknyszek]

Alrighty, from this issue, I think we should add the following metrics:

1. GOMAXPROCS
2. Total GC CPU time.
3. NumCgoCall.

I don't think there's any point to doing Version or NumCPU because they're static. Those should just be exported yourself (to whatever system) if you want them. It seems unnecessary to "resample" them regularly via the runtime/metrics API, though if you have a good use-case, I'm open to hearing it.

Any objections?

[rsc]

Based on the discussion above, this proposal seems like a likely accept.
— rsc for the proposal review group

[mpx]

@mknyszek For total GC time I'd like to compare it against rusage to understand whether a task is GC heavy without needing all the profiling machinery (which is inaccurate/more involved in other ways). I understand we can't sample both simultaneously, but any error should be small, and will become relatively smaller over the lifetime of the process.

Recording a histogram of GC times would be interesting, but I don't have a specific use for it atm.

[rsc]

No change in consensus, so accepted. 🎉
This issue now tracks the work of implementing the proposal.
— rsc for the proposal review group

[gopherbot]

Change https://go.dev/cl/404307 mentions this issue: runtime/metrics: add CPU stats

[gopherbot]

Change https://go.dev/cl/404306 mentions this issue: runtime/metrics: add the number of cgo calls

[gopherbot]

Change https://go.dev/cl/404305 mentions this issue: runtime/metrics: add gomaxprocs metric

[mknyszek]

I got to everything except the GC CPU stats. To get that right it needs a little more thought and a little more implementation work that I just couldn't get to before the freeze. :( My apologies. It's about 80% done, so my hope is that I can land it early in the 1.20 cycle.

[jmacd]

OpenTelemetry-Go would like runtime/metrics access to GCCPUFraction -- however it would be better expressed as a total number of CPU seconds spent in GC for reasons explained here: open-telemetry/opentelemetry-go-contrib#316 and above

I.e., it should have Cumulative=True, ValueKind=Float64, name=/gc/cpu/time:seconds

However, to make that useful, we should also have the rusage number we're comparing to in the same snapshot, so metrics like /cpu/user/time:seconds and /cpu/sys/time:seconds containing the cumulative count of CPU seconds used. Then, the former GCCPUFraction would equal to the metric derived from (/gc/cpu:seconds / (/cpu/user:seconds + /cpu/sys:seconds)) or something like that.

[mknyszek]

OpenTelemetry-Go would like runtime/metrics access to GCCPUFraction

We intentionally omitted GCCPUFraction from runtime/metrics initially because it's a somewhat misleading metric. It's averaged over the entire lifetime application, so you'll never be able to identify the GC overheads over a critical window you actually care about. Idle periods, or even just startup time can significantly skew the value, and its changes over time are less useful. Occasionally it still is useful, but as you suggest, it's probably better to just export what it's derived from in the first place.

I.e., it should have Cumulative=True, ValueKind=Float64, name=/gc/cpu/time:seconds

I'm already ahead of you on that. :) Please see https://go/dev/cl/404307 which I hope to land for Go 1.20.

However, to make that useful, we should also have the rusage number we're comparing to in the same snapshot

Comparing against rusage isn't going to work because measuring true CPU time efficiently across platforms is not something the Go runtime currently supports internally (and would likely require a substantial amount of work to do so). But, as you suggest, the aforementioned CL does export a metric for the Go runtime's estimate of total user CPU time that has elapsed (the estimate will be accurate provided that GOMAXPROCS is set such that the environment's resources are not oversubscribed). The other metrics can be compared against that total. The metric descriptions will also explain this caveat.

The metrics may be updated in the future to reflect true CPU time, and I suspect it's better to have an estimate than to not have anything at all.