Skip to content

Commit

Permalink
docs: address additional comments
Browse files Browse the repository at this point in the history
Signed-off-by: mbshields <[email protected]>
  • Loading branch information
mbshields committed Oct 9, 2023
1 parent a6168ed commit 7d36d18
Show file tree
Hide file tree
Showing 8 changed files with 54 additions and 60 deletions.
1 change: 0 additions & 1 deletion .wordlist.txt
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,6 @@ CRI
crt
CSS
cURL
curl
cve
CVE
CVEs
Expand Down
10 changes: 8 additions & 2 deletions docs/admin-guide/admin-configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -314,18 +314,24 @@ zot supports a range of monitoring tools including the following:

* Metrics

Metrics data is available in a Prometheus format. A full zot image with extensions includes a node exporter. A minimal zot image can use an external node exporter such as `zxp`.
Metrics data is available in a Prometheus format. A full zot image with extensions includes a node exporter. A minimal zot image can use an external node exporter such as `zxp`.

* Benchmarking

The zot project includes the `zb` tool, which allows you to benchmark a zot registry or any other container image registry that conforms to the [OCI Distribution Specification](https://github.com/opencontainers/distribution-spec).
The zot project includes the `zb` tool, which allows you to benchmark a zot registry or any other container image registry that conforms to the [OCI Distribution Specification](https://github.com/opencontainers/distribution-spec).

* Performance profiling

Performance profiling capabilities within zot allow a zot [administrator](../articles/authn-authz.md) to collect and export a range of diagnostic performance data such as CPU intensive function calls, memory allocations, and execution traces. The collected data can then be analyzed using Go tools and a variety of available visualization tools.

When zot is deployed in a Kubernetes setup, a site reliability engineering (SRE) operator can monitor service level indicators (SLI) such as metrics and logs. Metrics will appear in [Prometheus](https://prometheus.io/docs/guides/query-log/) using the zot `metrics` extension, while logs will appear in the Elasticsearch stack ([ELK stack](https://www.elastic.co/what-is/elk-stack)) using [Filebeat](https://www.elastic.co/beats/filebeat).

For detailed information about the monitoring tools, see [Monitoring the registry](../articles/monitoring.md).

For detailed information about benchmarking, see [Benchmarking zot with zb](../articles/benchmarking-with-zb.md).

For detailed information about performance profiling, see [Performance Profiling in zot](../articles/pprofiling.md).

<a name="cluster_config"></a>

## Clustering zot
Expand Down
95 changes: 39 additions & 56 deletions docs/developer-guide/pprofiling.md → docs/articles/pprofiling.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,15 @@

> :point_right: Use zot's built-in profiling tools to collect and analyze runtime performance.
The profiling capabilities within zot allow a zot [administrator](../articles/authn-authz.md) to collect and export a range of diagnostic performance data such as CPU intensive function calls, memory allocations, and execution traces. The collected data can then be analyzed using go tools and a variety of available visualization tools.
The profiling capabilities within zot allow a zot [administrator](../articles/authn-authz.md) to collect and export a range of diagnostic performance data such as CPU intensive function calls, memory allocations, and execution traces. The collected data can then be analyzed using Go tools and a variety of available visualization tools.

> :pencil2: If authentication is enabled, only a zot admin user can access the APIs for profiling.
>
> :pencil2: All examples in this article assume that the zot registry is running at `localhost:8080`.
## What data is available?

The zot source code incorporates runtime analysis tools to collect data for the following performance-related profiles:
The zot source code incorporates [golang's pprof package](https://pkg.go.dev/runtime/pprof) of runtime analysis tools to collect data for the following performance-related profiles:

| Profile | Description |
| ------- | --------- |
Expand All @@ -20,7 +24,7 @@ The zot source code incorporates runtime analysis tools to collect data for the
| threadcreate | Stack traces that led to the creation of new OS threads. |
| trace | A trace of execution of the current program. You can specify the duration in the `seconds` URL query parameter. After you get the trace file, use the `go tool trace` command to investigate the trace. |

To return an HTML-format profile list along with a count of currently available records for each profile, use the following API command:
To return a current HTML-format profile list along with a count of currently available records for each profile, use the following API command:

/v2/_zot/pprof/

Expand All @@ -30,80 +34,60 @@ To return an HTML-format profile list along with a count of currently available

To collect and export any available profile, use the following API command format:

/v2/_zot/pprof/<profile-name>[?<query-parameters>]

The following example shows an API request for the CPU usage profile named `profile` using a collection window of 10 seconds:

$ curl -s -v http://localhost:8080/v2/_zot/pprof/profile?seconds=10 > cpu10.prof
* Trying 127.0.0.1:8080...
* Connected to localhost (127.0.0.1) port 8080 (#0)
> GET /v2/_zot/pprof/profile?seconds=10 HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/8.1.2
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Disposition: attachment; filename="profile"
< Content-Type: application/octet-stream
< X-Content-Type-Options: nosniff
< Date: Tue, 26 Sep 2023 17:01:28 GMT
< Content-Length: 107
<
{ [107 bytes data]
* Connection #0 to host localhost left intact

This command creates an output data file named "cpu10.prof".

- Using the query parameter `?seconds=<number>`, we specify the number of seconds to gather the profile data. If this parameter is not specified, the default is 30 seconds.
- In this example, the raw output data is redirected to a file named "cpu10.prof". Alternatively, you can use `curl -O` to create a file with the default profile name (in this case, "profile"). If no output file is specified by either a cURL flag or an output redirection, the cURL command fails with "Failure writing output to destination".
/v2/_zot/pprof/<profile-type>[?<query-parameters>]

The following example shows an API request for the CPU usage profile named `profile` using a collection window of 30 seconds:

$ curl -s http://localhost:8080/v2/_zot/pprof/profile?seconds=30 > cpu.prof

This command example creates an output data file named "cpu.prof".

- The query parameter `?seconds=<number>` specifies the number of seconds to gather the profile data. If this parameter is not specified, the default is 30 seconds.
- In this example, the raw output data is redirected to a file named "cpu.prof". Alternatively, you can use `curl -O` to create a file with the default profile name (in this case, "profile"). If no output file is specified by either a cURL flag or an output redirection, the cURL command fails with "Failure writing output to destination".
- The command output file is in a machine-readable format that can be interpreted by performance analyzers.


## Analyzing the CPU usage profile using `go tool pprof`

The pprof package provides a variety of presentation formats for analyzing runtime performance.
Go's pprof package provides a variety of presentation formats for analyzing runtime performance.

For detailed information, see the [pprof documentation](https://pkg.go.dev/runtime/pprof).


### Generating a pprof web presentation

When an HTTP port is specified as a command flag, the `go tool pprof` command installs and opens a local web server that provides a web interface for viewing and analyzing the profile data. This example opens a localhost page at port 9090 for viewing the CPU usage data captured in the profile file named "cpu10.prof".
When an HTTP port is specified as a command flag, the `go tool pprof` command installs and opens a local web server that provides a web interface for viewing and analyzing the profile data. This example opens a localhost page at port 9090 for viewing the CPU usage data captured in the profile file named "cpu.prof".

$ go tool pprof -http=:9090 cpu10.prof
$ go tool pprof -http=:9090 cpu.prof
Serving web UI on http://localhost:9090

The pprof web view offers several options for viewing and interpreting the collected performance data. Select VIEW to see the following options:
The pprof web view offers several options for viewing and interpreting the collected performance data. Select VIEW to see the available options:

![pprof-view.jpg](../assets/images/pprof-view.jpg){width="300"}

A Flame Graph can be very useful for analyzing CPU usage:

![flame.svg](../assets/images/flame.svg){width="300"}
![profiling-flame.svg](../assets/images/profiling-flame.svg){width="300"}


### Generating a graphic image

The pprof package can generate graphic representations of profile data in many formats. This example generates a GIF file representing the CPU usage in the "cpu10.prof" file.

$ go tool pprof -gif cpu10.prof
Generating report in profile001.gif

![profile001.gif](../assets/images/profile001.gif){width="300"}
The pprof package can generate graphic representations of profile data in many formats. This example generates a PNG file representing the CPU usage in the "cpu.prof" file.

!!! note "Documentation comments"
$ go tool pprof -png cpu.prof
Generating report in profile001.png

Is this section helpful? Need a more interesting illustration (maybe Flame?)
![profile001.png](../assets/images/profile001.png){width="300"}


### Opening a pprof interactive session

This example opens an interactive session with pprof and executes the pprof `top` command, which displays the top ten modules by CPU usage during the profiling capture window.

$ go tool pprof cpu10.prof
$ go tool pprof cpu.prof
Type: cpu
Time: Sep 26, 2023 at 10:01am (PDT)
Duration: 10s, Total samples = 10ms ( 0.1%)
Duration: 30s, Total samples = 10ms ( 0.1%)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top
Showing nodes accounting for 10ms, 100% of 10ms total
Expand All @@ -118,29 +102,28 @@ This example opens an interactive session with pprof and executes the pprof `top
0 0% 100% 10ms 100% runtime.semawakeup
(pprof)

!!! note "Documentation comments"

Is this section helpful? If so, it would be better to have more realistic data.

## Analyzing the trace profile using `go tool trace`

!!! note "Documentation comments"

Is this section helpful?

You can collect trace data with the `trace` profile, as in this example:

$ curl -s -v http://localhost:8080/v2/_zot/pprof/trace?seconds=10 > trace10.prof
$ curl -s -v http://localhost:8080/v2/_zot/pprof/trace?seconds=30 > trace.prof

Using the `go tool trace` package, you can analyze the trace data captured in the "trace10.prof" example file:
Using the `go tool trace` package, you can analyze the trace data captured in the "trace.prof" example file:

$ go tool trace trace10.prof
$ go tool trace trace.prof
2023/09/21 16:58:58 Parsing trace...
2023/09/21 16:58:58 Splitting trace...
2023/09/21 16:58:58 Opening browser. Trace viewer is listening on http://127.0.0.1:62606

The `go tool trace` command installs and opens a local web server that provides a web interface for viewing and analyzing the trace data.

> :bulb: As an alternative, you can generate a pprof-like profile from the trace file using the following command:
As an alternative, you can generate a pprof-like profile from the trace file using the following command:

$ go tool trace -pprof=[net|sync|syscall|sched] <filename>

For example:

$ go tool trace -pprof=net trace.prof


go tool trace -pprof=[net|sync|syscall|sched] trace10.prof
Binary file removed docs/assets/images/profile001.gif
Binary file not shown.
Binary file added docs/assets/images/profile001.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
File renamed without changes
6 changes: 6 additions & 0 deletions docs/developer-guide/onboarding.md
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,12 @@ Delve is a powerful open-source debugger for the Go programming
language. Downloads and documentation for Delve are available on GitHub
at <https://github.com/go-delve/delve>.

## Performance profiling

Performance profiling capabilities within zot allow a zot [administrator](../articles/authn-authz.md) to collect and export a range of diagnostic performance data such as CPU intensive function calls, memory allocations, and execution traces. The collected data can then be analyzed using Go tools and a variety of available visualization tools.

For detailed information about performance profiling, see [Performance Profiling in zot](../articles/pprofiling.md).

## Code Organization

The zot project codebase is organized as follows:
Expand Down
2 changes: 1 addition & 1 deletion mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,6 @@ nav:
- Developer Guide:
- Onboarding: developer-guide/onboarding.md
- Extensions: developer-guide/extensions-dev.md
- Performance Profiling: developer-guide/pprofiling.md
- Contributing: developer-guide/contributing.md
- Articles:
- CI/CD Pipeline: articles/building-ci-cd-pipeline.md
Expand All @@ -122,6 +121,7 @@ nav:
- Clustering: articles/clustering.md
- Monitoring: articles/monitoring.md
- Benchmarking with zb: articles/benchmarking-with-zb.md
- Performance Profiling: articles/pprofiling.md
- Using kind for Deployment Testing: articles/kind-deploy.md
extra_css:
- stylesheets/custom.css

0 comments on commit 7d36d18

Please sign in to comment.