Performance history canary #1209
Conversation
| @@ -145,7 +167,10 @@ jobs: | |||
| export RESULT_REPO_BRANCH=${{ env.RESULT_REPO_BRANCH }} | |||
| export RESULT_REPO_ACCESS_TOKEN=${{ secrets.CI_ACCESS_TOKEN }} | |||
| export FROM_DATE=2020-07-10 | |||
| ./ci-perf-kit/scripts/openjdk-history-run.sh ./mmtk-openjdk ./reports/${{ steps.branch.outputs.branch_name }} | |||
| ./ci-perf-kit/scripts/openjdk-history-run.sh \ | |||
There was a problem hiding this comment.
It might be easier to implement if we do two separate runs, one for the commit, one for the canary version, and then plot them in two separate graphs. In that case, the code for plotting the original graph does not need big changes, and can be used to plot the canary graph.
There was a problem hiding this comment.
OK I saw mmtk/ci-perf-kit#47. You are doing that in ci-perf-kit. That's fine.
There was a problem hiding this comment.
@qinsoon The main concern is that ./ci-perf-kit/scripts/openjdk-history-run.sh also commits to the mmtk/ci-perf-result repo and generates the HTML. It is safer to use one single bash script to test both the newest version and the canary.
wks
force-pushed
the
feature/history-canary
branch
from
e470cff to
8363b3b
Compare
This PR adds a "canary" build to the performance regression CI of OpenJDK. The "canary" is a chosen revision of mmtk-core and mmtk-openjdk that is tested alongside each merged PR. The performance of the "canary" should not change unless there is an environment change. Spotting a change in the "canary" performance can help us identify environment changes that are unintended or otherwise unnoticed. TODO: Add a script to select the latest release tag as the canary instead of hard-coding a version.
The latest and the canary version may use different toolchain, and will be selected automatically when compiling them.
Extract canary version to an environment variable. Do not use secret when checking out ci-perf-kit because it is a public repo now. Add a name to the "branch" action.
wks
force-pushed
the
feature/history-canary
branch
from
8363b3b to
569e357
Compare
The canary will be run alongside the newest revision in regression test to identify unnoticed environment changes. Related PR: mmtk/mmtk-core#1209
There was a problem hiding this comment.
I merged mmtk/ci-perf-kit#47 and tagged 0.8.1 for ci-perf-kit. Can you update the version for ci-perf-kit to 0.8.1?
|
I approved the PR. You can merge this PR and #1206 when you think they are ready. |
This PR adds a "canary" build to the performance regression CI of OpenJDK. The "canary" is a chosen revision of mmtk-core and mmtk-openjdk that is tested alongside each merged PR. The performance of the "canary" should not change unless there is an environment change or there is a noise. Spotting a change in the "canary" performance can help us identify environment changes that are unintended or otherwise unnoticed, and also identify the noise level.
Currently, we choose a specific release version as the version of the "canary". Using a release version has the advantage of being easy to specify the exact revision of both the mmtk-core and the mmtk-openjdk repository. We may also switch to some methods of automatically select the canary version in the future.
There are other minor changes made.
latestandcanary. In each of the directories, we check outmmtk-coreandmmtk-openjdkof the latest and the canary versions, respectively.ci-replace-mmtk-dep.pyscript to replace the revision of themmtk-coredependency inmmtk-openjdk. As a result, we no longer need to usesed, and no longer need to copy themmtk-coredirectory intommtk-openjdk/repos.RUSTUP_TOOLCHAINenvironment variable becausecargocommand according to therust-toolchainfile in the directory.