From 9bd9c37d4e8ba605b91f7f5f28e44b721c0b367e Mon Sep 17 00:00:00 2001 From: Yi Lin Date: Fri, 20 Sep 2024 05:09:32 +0000 Subject: [PATCH 1/2] Document our policy about performance testing environment and epochs --- docs/team/ci.md | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/docs/team/ci.md b/docs/team/ci.md index d67d475c64..0478824cb0 100644 --- a/docs/team/ci.md +++ b/docs/team/ci.md @@ -1,6 +1,6 @@ # Continuous Integration -## Testing +## Correctness Testing MMTK core runs CI tests *before* a pull request is merged. @@ -11,3 +11,13 @@ MMTk core sets up two sets of tests, the *minimal tests* and the *extended tests * Extended tests only run for a pull request if the pull request is tagged with the label `PR-extended-testing`. This set of tests may take hours, and usually include integration tests with bindings which run the language implementation's standard test suite as much as possible. + +## Performance Testing + +We conduct performance testing for each MMTk core commit after it has been merged. + +### Testing Environment and Epochs + +We track the performance of MMTk over years. Naturally, changes in the testing environment (hardware or software) and methodology are sometimes necessary. Each time we make such a change, it marks the start of a new *epoch* in our performance evaluation. + +Since changes in the testing environment can significantly impact performance, we do not directly compare performance results across different epochs. Within an epoch, we ensure that **MMTk does not experience performance regressions**, and **we only update the testing environment when there is no performance regression in the current epoch**. From c4d2e2fec50eb0d2055a6434aa30a1baf7220268 Mon Sep 17 00:00:00 2001 From: Kunshan Wang Date: Mon, 21 Oct 2024 11:39:22 +0800 Subject: [PATCH 2/2] Mention canary --- docs/team/ci.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/team/ci.md b/docs/team/ci.md index 0478824cb0..b80dbef801 100644 --- a/docs/team/ci.md +++ b/docs/team/ci.md @@ -21,3 +21,9 @@ We conduct performance testing for each MMTk core commit after it has been merge We track the performance of MMTk over years. Naturally, changes in the testing environment (hardware or software) and methodology are sometimes necessary. Each time we make such a change, it marks the start of a new *epoch* in our performance evaluation. Since changes in the testing environment can significantly impact performance, we do not directly compare performance results across different epochs. Within an epoch, we ensure that **MMTk does not experience performance regressions**, and **we only update the testing environment when there is no performance regression in the current epoch**. + +### Regression Test Canary + +To monitor unnoticed performance changes and to measure the level of noise in the testing environment, we use a canary when doing performance regression tests with the OpenJDK binding. A "canary" is a chosen revision that is run along with any merged pull request. Since the same revision is run again and again, its performance should be relatively constant, within the range of noise. If we notice a change in the performance of the canary (especially something that resembles a [step function](https://en.wikipedia.org/wiki/Heaviside_step_function) in the line plot), we should inspect our testing environment for hardware or software changes. + +We keep running the same canary version until it is no longer possible, for reasons such as the toolchain for compiling that version is no longer available. When that happens, we may choose a different canary version or switch to an automatic mechanism for choosing canary.