diff --git a/.wordlist.txt b/.wordlist.txt index dbeda2b..c687aad 100644 --- a/.wordlist.txt +++ b/.wordlist.txt @@ -41,7 +41,6 @@ CRI crt CSS cURL -curl cve CVE CVEs diff --git a/docs/admin-guide/admin-configuration.md b/docs/admin-guide/admin-configuration.md index 8ee201e..e100243 100644 --- a/docs/admin-guide/admin-configuration.md +++ b/docs/admin-guide/admin-configuration.md @@ -314,11 +314,15 @@ zot supports a range of monitoring tools including the following: * Metrics - Metrics data is available in a Prometheus format. A full zot image with extensions includes a node exporter. A minimal zot image can use an external node exporter such as `zxp`. + Metrics data is available in a Prometheus format. A full zot image with extensions includes a node exporter. A minimal zot image can use an external node exporter such as `zxp`. * Benchmarking - The zot project includes the `zb` tool, which allows you to benchmark a zot registry or any other container image registry that conforms to the [OCI Distribution Specification](https://github.com/opencontainers/distribution-spec). + The zot project includes the `zb` tool, which allows you to benchmark a zot registry or any other container image registry that conforms to the [OCI Distribution Specification](https://github.com/opencontainers/distribution-spec). + +* Performance profiling + + Performance profiling capabilities within zot allow a zot [administrator](../articles/authn-authz.md) to collect and export a range of diagnostic performance data such as CPU intensive function calls, memory allocations, and execution traces. The collected data can then be analyzed using Go tools and a variety of available visualization tools. When zot is deployed in a Kubernetes setup, a site reliability engineering (SRE) operator can monitor service level indicators (SLI) such as metrics and logs. Metrics will appear in [Prometheus](https://prometheus.io/docs/guides/query-log/) using the zot `metrics` extension, while logs will appear in the Elasticsearch stack ([ELK stack](https://www.elastic.co/what-is/elk-stack)) using [Filebeat](https://www.elastic.co/beats/filebeat). @@ -326,6 +330,8 @@ For detailed information about the monitoring tools, see [Monitoring the registr For detailed information about benchmarking, see [Benchmarking zot with zb](../articles/benchmarking-with-zb.md). +For detailed information about performance profiling, see [Performance Profiling in zot](../articles/pprofiling.md). + ## Clustering zot diff --git a/docs/developer-guide/pprofiling.md b/docs/articles/pprofiling.md similarity index 60% rename from docs/developer-guide/pprofiling.md rename to docs/articles/pprofiling.md index de7bb2e..65c72b3 100644 --- a/docs/developer-guide/pprofiling.md +++ b/docs/articles/pprofiling.md @@ -2,11 +2,15 @@ > :point_right: Use zot's built-in profiling tools to collect and analyze runtime performance. -The profiling capabilities within zot allow a zot [administrator](../articles/authn-authz.md) to collect and export a range of diagnostic performance data such as CPU intensive function calls, memory allocations, and execution traces. The collected data can then be analyzed using go tools and a variety of available visualization tools. +The profiling capabilities within zot allow a zot [administrator](../articles/authn-authz.md) to collect and export a range of diagnostic performance data such as CPU intensive function calls, memory allocations, and execution traces. The collected data can then be analyzed using Go tools and a variety of available visualization tools. + +> :pencil2: If authentication is enabled, only a zot admin user can access the APIs for profiling. +> +> :pencil2: All examples in this article assume that the zot registry is running at `localhost:8080`. ## What data is available? -The zot source code incorporates runtime analysis tools to collect data for the following performance-related profiles: +The zot source code incorporates [golang's pprof package](https://pkg.go.dev/runtime/pprof) of runtime analysis tools to collect data for the following performance-related profiles: | Profile | Description | | ------- | --------- | @@ -20,7 +24,7 @@ The zot source code incorporates runtime analysis tools to collect data for the | threadcreate | Stack traces that led to the creation of new OS threads. | | trace | A trace of execution of the current program. You can specify the duration in the `seconds` URL query parameter. After you get the trace file, use the `go tool trace` command to investigate the trace. | -To return an HTML-format profile list along with a count of currently available records for each profile, use the following API command: +To return a current HTML-format profile list along with a count of currently available records for each profile, use the following API command: /v2/_zot/pprof/ @@ -30,80 +34,60 @@ To return an HTML-format profile list along with a count of currently available To collect and export any available profile, use the following API command format: - /v2/_zot/pprof/[?] - -The following example shows an API request for the CPU usage profile named `profile` using a collection window of 10 seconds: - - $ curl -s -v http://localhost:8080/v2/_zot/pprof/profile?seconds=10 > cpu10.prof - * Trying 127.0.0.1:8080... - * Connected to localhost (127.0.0.1) port 8080 (#0) - > GET /v2/_zot/pprof/profile?seconds=10 HTTP/1.1 - > Host: localhost:8080 - > User-Agent: curl/8.1.2 - > Accept: */* - > - < HTTP/1.1 200 OK - < Content-Disposition: attachment; filename="profile" - < Content-Type: application/octet-stream - < X-Content-Type-Options: nosniff - < Date: Tue, 26 Sep 2023 17:01:28 GMT - < Content-Length: 107 - < - { [107 bytes data] - * Connection #0 to host localhost left intact - -This command creates an output data file named "cpu10.prof". - -- Using the query parameter `?seconds=`, we specify the number of seconds to gather the profile data. If this parameter is not specified, the default is 30 seconds. -- In this example, the raw output data is redirected to a file named "cpu10.prof". Alternatively, you can use `curl -O` to create a file with the default profile name (in this case, "profile"). If no output file is specified by either a cURL flag or an output redirection, the cURL command fails with "Failure writing output to destination". + /v2/_zot/pprof/[?] + +The following example shows an API request for the CPU usage profile named `profile` using a collection window of 30 seconds: + + $ curl -s http://localhost:8080/v2/_zot/pprof/profile?seconds=30 > cpu.prof + +This command example creates an output data file named "cpu.prof". + +- The query parameter `?seconds=` specifies the number of seconds to gather the profile data. If this parameter is not specified, the default is 30 seconds. +- In this example, the raw output data is redirected to a file named "cpu.prof". Alternatively, you can use `curl -O` to create a file with the default profile name (in this case, "profile"). If no output file is specified by either a cURL flag or an output redirection, the cURL command fails with "Failure writing output to destination". - The command output file is in a machine-readable format that can be interpreted by performance analyzers. ## Analyzing the CPU usage profile using `go tool pprof` -The pprof package provides a variety of presentation formats for analyzing runtime performance. +Go's pprof package provides a variety of presentation formats for analyzing runtime performance. For detailed information, see the [pprof documentation](https://pkg.go.dev/runtime/pprof). ### Generating a pprof web presentation -When an HTTP port is specified as a command flag, the `go tool pprof` command installs and opens a local web server that provides a web interface for viewing and analyzing the profile data. This example opens a localhost page at port 9090 for viewing the CPU usage data captured in the profile file named "cpu10.prof". +When an HTTP port is specified as a command flag, the `go tool pprof` command installs and opens a local web server that provides a web interface for viewing and analyzing the profile data. This example opens a localhost page at port 9090 for viewing the CPU usage data captured in the profile file named "cpu.prof". - $ go tool pprof -http=:9090 cpu10.prof + $ go tool pprof -http=:9090 cpu.prof Serving web UI on http://localhost:9090 -The pprof web view offers several options for viewing and interpreting the collected performance data. Select VIEW to see the following options: +The pprof web view offers several options for viewing and interpreting the collected performance data. Select VIEW to see the available options: ![pprof-view.jpg](../assets/images/pprof-view.jpg){width="300"} A Flame Graph can be very useful for analyzing CPU usage: -![flame.svg](../assets/images/flame.svg){width="300"} +![profiling-flame.svg](../assets/images/profiling-flame.svg){width="300"} ### Generating a graphic image -The pprof package can generate graphic representations of profile data in many formats. This example generates a GIF file representing the CPU usage in the "cpu10.prof" file. - - $ go tool pprof -gif cpu10.prof - Generating report in profile001.gif - -![profile001.gif](../assets/images/profile001.gif){width="300"} +The pprof package can generate graphic representations of profile data in many formats. This example generates a PNG file representing the CPU usage in the "cpu.prof" file. -!!! note "Documentation comments" + $ go tool pprof -png cpu.prof + Generating report in profile001.png - Is this section helpful? Need a more interesting illustration (maybe Flame?) +![profile001.png](../assets/images/profile001.png){width="300"} ### Opening a pprof interactive session This example opens an interactive session with pprof and executes the pprof `top` command, which displays the top ten modules by CPU usage during the profiling capture window. - $ go tool pprof cpu10.prof + $ go tool pprof cpu.prof Type: cpu Time: Sep 26, 2023 at 10:01am (PDT) - Duration: 10s, Total samples = 10ms ( 0.1%) + Duration: 30s, Total samples = 10ms ( 0.1%) Entering interactive mode (type "help" for commands, "o" for options) (pprof) top Showing nodes accounting for 10ms, 100% of 10ms total @@ -118,29 +102,28 @@ This example opens an interactive session with pprof and executes the pprof `top 0 0% 100% 10ms 100% runtime.semawakeup (pprof) -!!! note "Documentation comments" - - Is this section helpful? If so, it would be better to have more realistic data. ## Analyzing the trace profile using `go tool trace` -!!! note "Documentation comments" - - Is this section helpful? - You can collect trace data with the `trace` profile, as in this example: - $ curl -s -v http://localhost:8080/v2/_zot/pprof/trace?seconds=10 > trace10.prof + $ curl -s -v http://localhost:8080/v2/_zot/pprof/trace?seconds=30 > trace.prof -Using the `go tool trace` package, you can analyze the trace data captured in the "trace10.prof" example file: +Using the `go tool trace` package, you can analyze the trace data captured in the "trace.prof" example file: - $ go tool trace trace10.prof + $ go tool trace trace.prof 2023/09/21 16:58:58 Parsing trace... 2023/09/21 16:58:58 Splitting trace... 2023/09/21 16:58:58 Opening browser. Trace viewer is listening on http://127.0.0.1:62606 The `go tool trace` command installs and opens a local web server that provides a web interface for viewing and analyzing the trace data. -> :bulb: As an alternative, you can generate a pprof-like profile from the trace file using the following command: +As an alternative, you can generate a pprof-like profile from the trace file using the following command: + + $ go tool trace -pprof=[net|sync|syscall|sched] + +For example: + + $ go tool trace -pprof=net trace.prof + - go tool trace -pprof=[net|sync|syscall|sched] trace10.prof diff --git a/docs/assets/images/profile001.gif b/docs/assets/images/profile001.gif deleted file mode 100644 index 54dc969..0000000 Binary files a/docs/assets/images/profile001.gif and /dev/null differ diff --git a/docs/assets/images/profile001.png b/docs/assets/images/profile001.png new file mode 100644 index 0000000..094d72f Binary files /dev/null and b/docs/assets/images/profile001.png differ diff --git a/docs/assets/images/flame.svg b/docs/assets/images/profiling-flame.svg similarity index 100% rename from docs/assets/images/flame.svg rename to docs/assets/images/profiling-flame.svg diff --git a/docs/developer-guide/onboarding.md b/docs/developer-guide/onboarding.md index cee4908..ea7fe58 100644 --- a/docs/developer-guide/onboarding.md +++ b/docs/developer-guide/onboarding.md @@ -102,6 +102,12 @@ Delve is a powerful open-source debugger for the Go programming language. Downloads and documentation for Delve are available on GitHub at . +## Performance profiling + +Performance profiling capabilities within zot allow a zot [administrator](../articles/authn-authz.md) to collect and export a range of diagnostic performance data such as CPU intensive function calls, memory allocations, and execution traces. The collected data can then be analyzed using Go tools and a variety of available visualization tools. + +For detailed information about performance profiling, see [Performance Profiling in zot](../articles/pprofiling.md). + ## Code Organization The zot project codebase is organized as follows: diff --git a/mkdocs.yml b/mkdocs.yml index 1f42eec..0be5142 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -110,7 +110,6 @@ nav: - Developer Guide: - Onboarding: developer-guide/onboarding.md - Extensions: developer-guide/extensions-dev.md - - Performance Profiling: developer-guide/pprofiling.md - Contributing: developer-guide/contributing.md - Articles: - CI/CD Pipeline: articles/building-ci-cd-pipeline.md @@ -122,6 +121,7 @@ nav: - Clustering: articles/clustering.md - Monitoring: articles/monitoring.md - Benchmarking with zb: articles/benchmarking-with-zb.md + - Performance Profiling: articles/pprofiling.md - Using kind for Deployment Testing: articles/kind-deploy.md extra_css: - stylesheets/custom.css