improve readme and update minimal example #198
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -10,52 +10,36 @@ and monitor the results on GitHub Actions workflow. | |
|
|
||
| - This action can store collected benchmark results in [GitHub pages][gh-pages] branch and provide | ||
| a chart view. Benchmark results are visualized on the GitHub pages of your project. | ||
|
|
||
|  | ||
| - This action can detect possible performance regressions by comparing benchmark results. When | ||
| benchmark results get worse than previous exceeding the specified threshold, it can add [an alert comment][alert-comment-example] to the commit. | ||
|
|
||
|  | ||
|
|
||
| [Japanese Blog post](https://rhysd.hatenablog.com/entry/2019/11/11/131505) | ||
|
|
||
| ## Supported languages & examples | ||
|
|
||
| | Language | Supported | Example Workflow | Example Project | | ||
| |--------------|-----------|-----------------------------------------------------------------------------|--------------------------------------------------------| | ||
| | Rust | Yes | [![cargo bench][rust-badge]][rust-workflow-example] | [examples/rust](./examples/rust) | | ||
| | Go | Yes | [![go test -bench][go-badge]][go-workflow-example] | [examples/go](./examples/go) | | ||
| | JavaScript | Yes | [![benchmark.js][benchmarkjs-badge]][benchmarkjs-workflow-example] | [examples/benchmarkjs](./examples/benchmarkjs) | | ||
| | Python | Yes | [![pytest-benchmark][pytest-benchmark-badge]][pytest-workflow-example] | [examples/pytest](./examples/pytest) | | ||
| | C++ | Yes | [![Google Benchmark Framework][cpp-badge]][cpp-workflow-example] | [examples/cpp](./examples/cpp) | | ||
| | C++ (Catch2) | Yes | [![Catch2][catch2-badge]][catch2-workflow-example] | [examples/catch2](./examples/catch2) | | ||
| | Julia | Yes | [![BenchmarkTools.jl][julia-badge]][julia-workflow-example] | [examples/julia](./examples/julia) | | ||
| | .Net | Yes | [![Benchmark.Net][benchmarkdotnet-badge]][benchmarkdotnet-workflow-example] | [examples/benchmarkdotnet](./examples/benchmarkdotnet) | | ||
| | Java | Yes | [![JMH][java-badge]][java-workflow-example] | [examples/java](./examples/java) | | ||
| | Luau | Yes | Coming soon for benchmarkluau | Coming soon | | ||
|
|
||
| All benchmark charts from above workflows are gathered in GitHub pages: | ||
|
|
||
| https://benchmark-action.github.io/github-action-benchmark/dev/bench/ | ||
|
|
||
| ## Custom benchmarks | ||
|
|
||
| Additionally, even though there is no explicit example for them, you can use | ||
| `customBiggerIsBetter` and `customSmallerIsBetter` to use this | ||
| action and create your own graphs from your own benchmark data. The name in | ||
|
|
@@ -82,62 +66,10 @@ context) properties. Like this: | |
| } | ||
| ] | ||
| ``` | ||
| ## How to use | ||
|
|
||
| ### Minimal setup | ||
|
|
||
| ```yaml | ||
| name: Minimal setup | ||
| on: | ||
|
|
@@ -151,40 +83,39 @@ jobs: | |
| runs-on: ubuntu-latest | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| - uses: actions/setup-go@v5 | ||
| with: | ||
| go-version: "stable" | ||
| # Run benchmark with `go test -bench` and stores the output to a file | ||
| - name: Run benchmark | ||
| run: go test -bench 'BenchmarkFib' | tee output.txt | ||
| # Download previous benchmark result from cache | ||
| - name: Download previous benchmark data | ||
| uses: actions/cache/restore@v4 | ||
| with: | ||
| fail-on-cache-miss: true | ||
| path: ./cache/benchmark-data.json | ||
| key: ${{ runner.os }}-benchmark | ||
| - name: Compare results | ||
| uses: benchmark-action/github-action-benchmark@v1 | ||
| with: | ||
| # What benchmark tool the output.txt came from | ||
| tool: 'go' | ||
| # Extract benchmark result from here | ||
| output-file-path: output.txt | ||
| # Where the previous data file is stored | ||
| external-data-json-path: ./cache/benchmark-data.json | ||
| # Workflow will fail when an alert happens | ||
| fail-on-alert: true | ||
| # Upload the updated cache file for the next job by actions/cache | ||
| - name: Save benchmark JSON | ||
| uses: actions/cache/save@v3 | ||
| with: | ||
| path: ./cache/benchmark-data.json | ||
| # Include OS in key so that we don't compare benchmarks across different OSes | ||
| key: ${{ runner.os }}-benchmark | ||
| ``` | ||
|
|
||
| By default, this action marks the result as performance regression when it is worse than the previous | ||
| exceeding 200% threshold. For example, if the previous benchmark result was 100 iter/ns and this time | ||
| it is 230 iter/ns, it means 230% worse than the previous and an alert will happen. The threshold can | ||
|
|
@@ -193,24 +124,23 @@ be changed by `alert-threshold` input. | |
| A live workflow example is [here](.github/workflows/minimal.yml). And the results of the workflow can | ||
| be seen [here][minimal-workflow-example]. | ||
|
|
||
| ### Comment on commit when regression is found | ||
|
|
||
| ```yaml | ||
| - name: Compare benchmarks and comment on alert | ||
| uses: benchmark-action/github-action-benchmark@v1 | ||
| with: | ||
| tool: 'go' | ||
| # Where the output from the current benchmark is stored | ||
| output-file-path: ${{ github.sha }}_bench_output.txt | ||
| # Where the previous benchmark is stored | ||
| external-data-json-path: ./cache/benchmark-data.json | ||
| fail-on-alert: true | ||
| # GitHub API token to make a commit comment | ||
| github-token: ${{ secrets.GITHUB_TOKEN }} | ||
| # Enable alert commit comment | ||
| comment-on-alert: true | ||
| # Mention @rhysd in the commit comment, not mandatory but highly recommended to ensure the comment is seen | ||
| alert-comment-cc-users: '@rhysd' | ||
| ``` | ||
|
|
||
|
|
@@ -220,34 +150,29 @@ performance regression. | |
|
|
||
| Now, in addition to making workflow fail, the step leaves a commit comment when it detects performance | ||
| regression [like this][alert-comment-example]. Though `alert-comment-cc-users` input is not mandatory for | ||
| this, I recommend to set it to make sure you notice the comment via GitHub notification. Please note | ||
| that this value must be quoted like `'@rhysd'` because [`@` is an indicator in YAML syntax](https://yaml.org/spec/1.2/spec.html#id2772075). | ||
|
|
||
| A live workflow example is [here](.github/workflows/commit-comment.yml). And the results of the workflow | ||
| can be seen [here][commit-comment-workflow-example]. | ||
|
|
||
| ### Leave a comment on PR Summary | ||
|
|
||
| Github Actions [Job Summaries](https://github.blog/2022-05-09-supercharging-github-actions-with-job-summaries/) are | ||
| also supported. In order to use Job Summaries, turn on the `summary-always` option. | ||
|
|
||
| ```yaml | ||
| - name: Compare benchmarks and leave a comment on PR summary | ||
| uses: benchmark-action/github-action-benchmark@v1 | ||
| with: | ||
| tool: 'cargo' | ||
| # Where the output from the current benchmark is stored | ||
| output-file-path: ${{ github.sha }}_bench_output.txt | ||
| # Where the previous benchmark is stored | ||
| external-data-json-path: ./cache/benchmark-data.json | ||
| fail-on-alert: true | ||
| # Enable Job Summary for PRs | ||
| summary-always: true | ||
| ``` | ||
|
|
||
| ### Charts on GitHub Pages | ||
|
|
@@ -293,8 +218,8 @@ jobs: | |
| # Run benchmark with `go test -bench` and stores the output to a file | ||
| - name: Run benchmark | ||
| run: go test -bench 'BenchmarkFib' | tee output.txt | ||
|
|
||
| - name: Compare benchmarks and publish to github pages | ||
| uses: benchmark-action/github-action-benchmark@v1 | ||
| with: | ||
| name: My Project Go Benchmark | ||
|
|
@@ -329,10 +254,10 @@ benchmark results identical. | |
|
|
||
| Please see the above ['Examples' section](#examples) to see live workflow examples for each language. | ||
|
|
||
| If you don't want to pass GitHub API token to this action: | ||
|
|
||
| ```yaml | ||
| - name: Compare benchmarks | ||
| uses: benchmark-action/github-action-benchmark@v1 | ||
| with: | ||
| name: My Project Go Benchmark | ||
|
|
@@ -341,31 +266,12 @@ If you don't want to pass GitHub API token to this action, it's still OK. | |
| # Set auto-push to false since GitHub API token is not given | ||
| auto-push: false | ||
| # Push gh-pages branch by yourself | ||
| - name: Publish to github pages without token | ||
| run: git push 'https://you:${{ secrets.GITHUB_TOKEN }}@github.com/you/repo-name.git' gh-pages:gh-pages | ||
| ``` | ||
|
|
||
| Please add a step to push the branch to the remote. | ||
|
|
||
|
There was a problem hiding this comment. Same. These have value to users. There was a problem hiding this comment. This section is repeated thrice.
I put everything in one place (the table). Please checkout the readme locally and see how it looks |
||
| ### Action inputs | ||
|
|
||
| Input definitions are written in [action.yml](./action.yml). | ||
|
|
||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
IMO, this introduction has some value to users to understand the tool. If we don't want to distract in the main doc, we can move it to separated doc and link it here. Completely deleting it may create the gap and user experience regression.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The screenshots are still there, but I moved them to the very top (except this one https://raw.githubusercontent.com/rhysd/ss/master/github-action-benchmark/tooltip.png which I feel adds no value - people using this tool will notice on their own about this, and people that don't yet know this tool will not care about this detail).
Please checkout the readme locally and see how it looks