diff --git a/commands/runbms.html b/commands/runbms.html index 0fb7aad..930c941 100644 --- a/commands/runbms.html +++ b/commands/runbms.html @@ -184,7 +184,7 @@

Usage

-s: only use the specified heap sizes. This is a comma-separated string of integers or floating point numbers. For each slice s in SLICE, we run benchmarks at s * minheap. -N and ns are ignored.

+N and ns are ignored.

-p: add a prefix to the folder names where the results are stored. By default, the folder that stores the result is named using the host name and the timestamp. However, you can add a prefix to the folder name to signify which experiments the results belong to.

@@ -230,7 +230,7 @@

Zulip

type: Zulip request: type: private - to: ["your user id here"] + to: ["your user id here"]

Keys

request: please follow the Zulip API documentation. @@ -250,7 +250,7 @@

CopyFile

Keys

patterns: a list of patterns following the Python 3 pathlib.Path.glob syntax. diff --git a/cookbook/perf_events.html b/cookbook/perf_events.html index 8860f0a..61b83ea 100644 --- a/cookbook/perf_events.html +++ b/cookbook/perf_events.html @@ -185,15 +185,15 @@

JVMTI

modifiers:
   jvmti_env:
     type: EnvVar
-    var: "LD_PRELOAD"
-    val: "/path/to/distillation/libperf_statistics.so"
+    var: "LD_PRELOAD"
+    val: "/path/to/distillation/libperf_statistics.so"

Second, you need to specify a list of events you want to measure.

modifiers:
   perf:
     type: EnvVar
-    var: "PERF_EVENTS"
-    val: "PERF_COUNT_HW_CPU_CYCLES,PERF_COUNT_HW_INSTRUCTIONS,PERF_COUNT_HW_CACHE_LL:MISS,PERF_COUNT_HW_CACHE_L1D:MISS,PERF_COUNT_HW_CACHE_DTLB:MISS"
+    var: "PERF_EVENTS"
+    val: "PERF_COUNT_HW_CPU_CYCLES,PERF_COUNT_HW_INSTRUCTIONS,PERF_COUNT_HW_CACHE_LL:MISS,PERF_COUNT_HW_CACHE_L1D:MISS,PERF_COUNT_HW_CACHE_DTLB:MISS"

If you want to get a full list of events you can use on a particular machine, you can clone and build libpfm4 and run the showevtinfo program.

Third, you need to tell the JVM to load the agent. @@ -201,23 +201,23 @@

JVMTI

modifiers:
   jvmti:
     type: JVMArg
-    val: "-agentpath:/path/to/distillation/libperf_statistics.so"
+    val: "-agentpath:/path/to/distillation/libperf_statistics.so"

Finally, you need to let the DaCapo benchmark inform the start and the end of a benchmark iteration. We will reuse the RustMMTk probe here, as the callback functions in the JVMTI agent are also called harness_begin and harness_end.

modifiers:
   probes_cp:
     type: JVMClasspath
-    val: "/path/to/probes/out /path/to/probes/out/probes.jar"
+    val: "/path/to/probes/out /path/to/probes/out/probes.jar"
   probes:
     type: JVMArg
-    val: "-Djava.library.path=/path/to/probes/out -Dprobes=RustMMTk"
+    val: "-Djava.library.path=/path/to/probes/out -Dprobes=RustMMTk"

Now, putting it all together, you can define a set of modifiers, and use that set in your config strings.

modifiers:
   jvmti_common:
     type: ModifierSet
-    val: "probes|probes_cp|jvmti|jvmti_env|perf"
+    val: "probes|probes_cp|jvmti|jvmti_env|perf"

MMTk

Please clone and build probes. @@ -226,17 +226,17 @@

MMTk

modifiers:
   probes_cp:
     type: JVMClasspath
-    val: "/path/to/probes/out /path/to/probes/out/probes.jar"
+    val: "/path/to/probes/out /path/to/probes/out/probes.jar"
   probes:
     type: JVMArg
-    val: "-Djava.library.path=/path/to/probes/out -Dprobes=RustMMTk"
+    val: "-Djava.library.path=/path/to/probes/out -Dprobes=RustMMTk"

Then, you can specify a list of events you want to measure.

modifiers:
   mmtk_perf:
     type: EnvVar
-    var: "MMTK_PHASE_PERF_EVENTS"
-    val: "PERF_COUNT_HW_CPU_CYCLES,0,-1;PERF_COUNT_HW_INSTRUCTIONS,0,-1;PERF_COUNT_HW_CACHE_LL:MISS,0,-1;PERF_COUNT_HW_CACHE_L1D:MISS,0,-1;PERF_COUNT_HW_CACHE_DTLB:MISS,0,-1"
+    var: "MMTK_PHASE_PERF_EVENTS"
+    val: "PERF_COUNT_HW_CPU_CYCLES,0,-1;PERF_COUNT_HW_INSTRUCTIONS,0,-1;PERF_COUNT_HW_CACHE_LL:MISS,0,-1;PERF_COUNT_HW_CACHE_L1D:MISS,0,-1;PERF_COUNT_HW_CACHE_DTLB:MISS,0,-1"

Note that the list is semicolon-separated. Each entry consists of three parts, separated by commas. diff --git a/index.html b/index.html index 48e868a..52a731f 100644 --- a/index.html +++ b/index.html @@ -178,7 +178,7 @@

IntroductionDisclaimer

At this stage, the focus of this project is driven by the internal use of members from Steve Blackburn's lab. If you are a member of the lab, you know what to do if you encounter any issue, and you can ignore the below.

-

If you are a member of the public, please kindly note that the project is open-source and documented on a "good-faith" basis. +

If you are a member of the public, please kindly note that the project is open-source and documented on a "good-faith" basis. We might not have the time to consider your features requests. Please don't be offended if we ignore these. Having said that, you are very welcomed to use it, and we will be very pleased if this helps anyone. diff --git a/intro.html b/intro.html index 48e868a..52a731f 100644 --- a/intro.html +++ b/intro.html @@ -178,7 +178,7 @@

IntroductionDisclaimer

At this stage, the focus of this project is driven by the internal use of members from Steve Blackburn's lab. If you are a member of the lab, you know what to do if you encounter any issue, and you can ignore the below.

-

If you are a member of the public, please kindly note that the project is open-source and documented on a "good-faith" basis. +

If you are a member of the public, please kindly note that the project is open-source and documented on a "good-faith" basis. We might not have the time to consider your features requests. Please don't be offended if we ignore these. Having said that, you are very welcomed to use it, and we will be very pleased if this helps anyone. diff --git a/print.html b/print.html index 1476da5..9c6cc0e 100644 --- a/print.html +++ b/print.html @@ -179,7 +179,7 @@

IntroductionDisclaimer

At this stage, the focus of this project is driven by the internal use of members from Steve Blackburn's lab. If you are a member of the lab, you know what to do if you encounter any issue, and you can ignore the below.

-

If you are a member of the public, please kindly note that the project is open-source and documented on a "good-faith" basis. +

If you are a member of the public, please kindly note that the project is open-source and documented on a "good-faith" basis. We might not have the time to consider your features requests. Please don't be offended if we ignore these. Having said that, you are very welcomed to use it, and we will be very pleased if this helps anyone. @@ -240,7 +240,7 @@

Installation

Then, create a file two_builds.yml with the following content.

includes:
-  - "$RUNNING_NG_PACKAGE_DATA/base/runbms.yml"
+  - "$RUNNING_NG_PACKAGE_DATA/base/runbms.yml"

The YAML file represents a dictionary (key-value pairs) that defines the experiments you are running. The includes directive here will populate the dictionary with some default values shipped with running-ng.

@@ -274,11 +274,11 @@

Prepare The benchmark suite is defined in $RUNNING_NG_PACKAGE_DATA/base/dacapo.yml. By default, the minimum heap sizes of dacapochopin-29a657f benchmarks are measured with AdoptOpenJDK 15 using G1 GC. If you are using OpenJDK 11 or 17, you can override the value of suites.dacapochopin-29a657f.minheap to temurin-17-G1 or temurin-11-G1. -That is, you can, for example, add "suites.dacapochopin-29a657f.minheap": "temurin-17-G1" to overrides.

+That is, you can, for example, add "suites.dacapochopin-29a657f.minheap": "temurin-17-G1" to overrides.

Then, add the following to two_builds.yml.

overrides:
-  "suites.dacapochopin-29a657f.timing_iteration": 5
-  "suites.dacapochopin-29a657f.callback": "probe.DacapoChopinCallback"
+  "suites.dacapochopin-29a657f.timing_iteration": 5
+  "suites.dacapochopin-29a657f.callback": "probe.DacapoChopinCallback"

That is, we want to run five iterations for each invocation, and use DacapoChopinCallback because it is the appropriate callback for this release of DaCapo.

Prepare Your Builds

@@ -290,11 +290,11 @@

Prepa build1: type: OpenJDK release: 11 - home: "/path/to/build1/jdk" # make sure /path/to/build1/jdk/bin/java exists + home: "/path/to/build1/jdk" # make sure /path/to/build1/jdk/bin/java exists build2: type: OpenJDK release: 11 - home: "/path/to/build2/jdk" # make sure /path/to/build2/jdk/bin/java exists + home: "/path/to/build2/jdk" # make sure /path/to/build2/jdk/bin/java exists

This defines two builds of runtimes.

I recommend that you use absolute paths for the builds, although relative paths will work, and will be relative to where you run running.

@@ -305,18 +305,18 @@

Prepare Probes< 
modifiers:
   probes_cp:
     type: JVMClasspath
-    val: "/path/to/probes/out /path/to/probes/out/probes.jar"
+    val: "/path/to/probes/out /path/to/probes/out/probes.jar"
   probes:
     type: JVMArg
-    val: "-Djava.library.path=/path/to/probes/out -Dprobes=RustMMTk"
+    val: "-Djava.library.path=/path/to/probes/out -Dprobes=RustMMTk"

This defines two modifiers, which will be used later to modify the JVM command line arguments.

Please only use absolute paths for all the above.

Prepare Configs

Finally, add he following to two_builds.yml.

configs:
-  - "build1|ms|s|c2|mmtk_gc-SemiSpace|tph|probes_cp|probes"
-  - "build2|ms|s|c2|mmtk_gc-SemiSpace|tph|probes_cp|probes"
+  - "build1|ms|s|c2|mmtk_gc-SemiSpace|tph|probes_cp|probes"
+  - "build2|ms|s|c2|mmtk_gc-SemiSpace|tph|probes_cp|probes"

The syntax is described here.

Sanity Checks

@@ -337,7 +337,7 @@

Single In

This allows you to see any issue before wasting several days only realizing that something didn't work.

Run It

Once you are happy with everything, run the experiments.

-
running runbms /path/to/log two_builds.yml 8 -p "two_builds"
+
running runbms /path/to/log two_builds.yml 8 -p "two_builds"
 

 
Don't forget to give the results folder a prefix so that you can later tell what the experiment was for.

 
Analysing Results

@@ -399,11 +399,11 @@ 
benchmarks
 
configs

 
A YAML list of configuration strings to be used to run the benchmarks. These are
 specified as a runtime followed by a '|' separated list of
-modifiers, i.e. "<runtime>|<modifier>|...|<modifier>".

+modifiers, i.e. "<runtime>|<modifier>|...|<modifier>".

 
For example:

 
configs:
-  - "openjdk11|ms|s|c2"
-  - "openjdk15|ms|s"
+  - "openjdk11|ms|s|c2"
+  - "openjdk15|ms|s"
 

 
specifies running to use the openjdk11 runtime with ms, s, and c2
 modifiers; and the openjdk15 runtime with the ms, and s modifiers. In
@@ -414,8 +414,8 @@ 
configs

 so that it's easier for multi-line editing.

 
For example:

 
configs:
-  - "openjdk8 |foo-1 |bar|buzz"
-  - "openjdk15|foo-16|   |buzz"
+  - "openjdk8 |foo-1 |bar|buzz"
+  - "openjdk15|foo-16|   |buzz"
 

 
includes

 
A YAML list of paths to YAML files that are to be included into the current
@@ -430,8 +430,8 @@ 
includes$HOME/bar.yml.
 Similarly,

 
includes:
- - "./base/suites.yml"
- - "./base/modifiers.yml"
+ - "./base/suites.yml"
+ - "./base/modifiers.yml"
 

 
includes the suites.yml and modifiers.yml files located at ./base
 respectively.

@@ -454,10 +454,10 @@ 
modifiersmodifiers:
   s:
     type: JVMArg
-    val: "-server"
+    val: "-server"
   c2:
     type: JVMArg
-    val: "-XX:-TieredCompilation -Xcomp"
+    val: "-XX:-TieredCompilation -Xcomp"
     excludes:
       dacapo2006:
         - eclipse
@@ -472,7 +472,7 @@ 
modifiersmodifiers:
   c2:
     type: JVMArg
-    val: "-XX:-TieredCompilation -Xcomp"
+    val: "-XX:-TieredCompilation -Xcomp"
     includes:
       dacapo2006:
         - eclipse
@@ -484,7 +484,7 @@ 
modifiersmodifiers:
   c2:
     type: JVMArg
-    val: "-XX:-TieredCompilation -Xcomp"
+    val: "-XX:-TieredCompilation -Xcomp"
     includes:
       dacapo2006:
         - eclipse
@@ -505,13 +505,13 @@ 
Value Options
 
modifiers:
   env_var:
     type: EnvVar
-    var: "FOO{0}"
-    val: "{1}"
+    var: "FOO{0}"
+    val: "{1}"
 
 [...]
 
 configs:
-  - "openjdk11|env_var-42-43"
+  - "openjdk11|env_var-42-43"
 

 
specifies to run the openjdk11 runtime with the environment
 variable FOO42 set to 43. Note that value options are not limited only to
@@ -535,10 +535,10 @@ 
Keys

 
programs:
   <BM_NAME_1>:
     path: /full/path/to/benchmark/binary_1
-    args: "Any arguments to binary_1"
+    args: "Any arguments to binary_1"
   <BM_NAME_2>:
     path: /full/path/to/benchmark/binary_2
-    args: "Any arguments to binary_2"
+    args: "Any arguments to binary_2"
   [...]
 

 
A possible use-case could use wrapper shell scripts around the benchmark to
@@ -546,7 +546,7 @@ 
Keys

 
DaCapo

 
DaCapo benchmark suite.

 
Keys

-
release: one of the possible values ["2006", "9.12", "evaluation"].
+
release: one of the possible values ["2006", "9.12", "evaluation"].
 The value is required.

 
path: path to the DaCapo jar.
 The value is required.
@@ -588,19 +588,19 @@ 
Keys

 Therefore, companion programs should have appropriate timeouts or detect when main program finishes.
 Here is an example of using companion to launch bpftrace in the background to count the system calls.

 
includes:
-  - "$RUNNING_NG_PACKAGE_DATA/base/runbms.yml"
+  - "$RUNNING_NG_PACKAGE_DATA/base/runbms.yml"
 
 overrides:
-  "suites.dacapo2006.timing_iteration": 1
-  "suites.dacapo2006.companion": "sudo bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @syscall[args->id] = count(); @process[comm] = count();} interval:s:10 { printf(\"Goodbye world!\\n\"); exit(); }'"
-  "invocations": 1
+  "suites.dacapo2006.timing_iteration": 1
+  "suites.dacapo2006.companion": "sudo bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @syscall[args->id] = count(); @process[comm] = count();} interval:s:10 { printf(\"Goodbye world!\\n\"); exit(); }'"
+  "invocations": 1
 
 benchmarks:
   dacapo2006:
     - fop
 
 configs:
-  - "temurin-17"
+  - "temurin-17"
 

 
In the log file, the output from the main program and the output from the companion program is separated by *****.

 
size: specifying the size of input data.
@@ -621,7 +621,7 @@ 
SPECjbb2015 (preview ⚠️)

 
SPECjbb2015.

 
Keys

-
release: one of the possible values ["1.03"].
+
release: one of the possible values ["1.03"].
 The value is required.

 
path: path to the jar.
 The value is required.
@@ -629,28 +629,28 @@ 
Keys

 Environment variables will be expanded.

 
Benchmark Specification

 
Only strings are allowed, which should correspond to the the mode of the SPECjbb2015 controller.
-Right now, only "composite" is supported.

+Right now, only "composite" is supported.

 
SPECjvm98 (preview ⚠️)

 
SPECjvm98.

 
Note that you will need to prepend probes to the classpaths, so that the modified SpecApplication can be used.

 
Here is an example configuration file.

 
includes:
-  - "/home/zixianc/running-ng/src/running/config/base/runbms.yml"
+  - "/home/zixianc/running-ng/src/running/config/base/runbms.yml"
 
 modifiers:
   probes_cp:
     type: JVMClasspathPrepend
-    val: "/home/zixianc/MMTk-Dev/evaluation/probes /home/zixianc/MMTk-Dev/evaluation/probes/probes.jar"
+    val: "/home/zixianc/MMTk-Dev/evaluation/probes /home/zixianc/MMTk-Dev/evaluation/probes/probes.jar"
 
 benchmarks:
   specjvm98:
     - _213_javac
 
 configs:
-  - "adoptopenjdk-8|probes_cp"
+  - "adoptopenjdk-8|probes_cp"
 

 
Keys

-
release: one of the possible values ["1.03_05"].
+
release: one of the possible values ["1.03_05"].
 The value is required.

 
path: path to the SPECjvm98 folder, where you can find SpecApplication.class.
 The value is required.
@@ -848,7 +848,7 @@ 
Usage

 
-s: only use the specified heap sizes.
 This is a comma-separated string of integers or floating point numbers.
 For each slice s in SLICE, we run benchmarks at s * minheap.
-N and ns are ignored. 

+N and ns are ignored.

 
-p: add a prefix to the folder names where the results are stored.
 By default, the folder that stores the result is named using the host name and the timestamp.
 However, you can add a prefix to the folder name to signify which experiments the results belong to.

@@ -894,7 +894,7 @@ 
Zulip

     type: Zulip
     request:
       type: private
-      to: ["your user id here"]
+      to: ["your user id here"]

Keys

request: please follow the Zulip API documentation. @@ -914,7 +914,7 @@

CopyFile

Keys

patterns: a list of patterns following the Python 3 pathlib.Path.glob syntax. @@ -1009,15 +1009,15 @@

JVMTI

modifiers:
   jvmti_env:
     type: EnvVar
-    var: "LD_PRELOAD"
-    val: "/path/to/distillation/libperf_statistics.so"
+    var: "LD_PRELOAD"
+    val: "/path/to/distillation/libperf_statistics.so"

Second, you need to specify a list of events you want to measure.

modifiers:
   perf:
     type: EnvVar
-    var: "PERF_EVENTS"
-    val: "PERF_COUNT_HW_CPU_CYCLES,PERF_COUNT_HW_INSTRUCTIONS,PERF_COUNT_HW_CACHE_LL:MISS,PERF_COUNT_HW_CACHE_L1D:MISS,PERF_COUNT_HW_CACHE_DTLB:MISS"
+    var: "PERF_EVENTS"
+    val: "PERF_COUNT_HW_CPU_CYCLES,PERF_COUNT_HW_INSTRUCTIONS,PERF_COUNT_HW_CACHE_LL:MISS,PERF_COUNT_HW_CACHE_L1D:MISS,PERF_COUNT_HW_CACHE_DTLB:MISS"

If you want to get a full list of events you can use on a particular machine, you can clone and build libpfm4 and run the showevtinfo program.

Third, you need to tell the JVM to load the agent. @@ -1025,23 +1025,23 @@

JVMTI

modifiers:
   jvmti:
     type: JVMArg
-    val: "-agentpath:/path/to/distillation/libperf_statistics.so"
+    val: "-agentpath:/path/to/distillation/libperf_statistics.so"

Finally, you need to let the DaCapo benchmark inform the start and the end of a benchmark iteration. We will reuse the RustMMTk probe here, as the callback functions in the JVMTI agent are also called harness_begin and harness_end.

modifiers:
   probes_cp:
     type: JVMClasspath
-    val: "/path/to/probes/out /path/to/probes/out/probes.jar"
+    val: "/path/to/probes/out /path/to/probes/out/probes.jar"
   probes:
     type: JVMArg
-    val: "-Djava.library.path=/path/to/probes/out -Dprobes=RustMMTk"
+    val: "-Djava.library.path=/path/to/probes/out -Dprobes=RustMMTk"

Now, putting it all together, you can define a set of modifiers, and use that set in your config strings.

modifiers:
   jvmti_common:
     type: ModifierSet
-    val: "probes|probes_cp|jvmti|jvmti_env|perf"
+    val: "probes|probes_cp|jvmti|jvmti_env|perf"

MMTk

Please clone and build probes. @@ -1050,17 +1050,17 @@

MMTk

modifiers:
   probes_cp:
     type: JVMClasspath
-    val: "/path/to/probes/out /path/to/probes/out/probes.jar"
+    val: "/path/to/probes/out /path/to/probes/out/probes.jar"
   probes:
     type: JVMArg
-    val: "-Djava.library.path=/path/to/probes/out -Dprobes=RustMMTk"
+    val: "-Djava.library.path=/path/to/probes/out -Dprobes=RustMMTk"

Then, you can specify a list of events you want to measure.

modifiers:
   mmtk_perf:
     type: EnvVar
-    var: "MMTK_PHASE_PERF_EVENTS"
-    val: "PERF_COUNT_HW_CPU_CYCLES,0,-1;PERF_COUNT_HW_INSTRUCTIONS,0,-1;PERF_COUNT_HW_CACHE_LL:MISS,0,-1;PERF_COUNT_HW_CACHE_L1D:MISS,0,-1;PERF_COUNT_HW_CACHE_DTLB:MISS,0,-1"
+    var: "MMTK_PHASE_PERF_EVENTS"
+    val: "PERF_COUNT_HW_CPU_CYCLES,0,-1;PERF_COUNT_HW_INSTRUCTIONS,0,-1;PERF_COUNT_HW_CACHE_LL:MISS,0,-1;PERF_COUNT_HW_CACHE_L1D:MISS,0,-1;PERF_COUNT_HW_CACHE_DTLB:MISS,0,-1"

Note that the list is semicolon-separated. Each entry consists of three parts, separated by commas. diff --git a/quickstart.html b/quickstart.html index 873f48b..30b0a4c 100644 --- a/quickstart.html +++ b/quickstart.html @@ -181,7 +181,7 @@

Installation

Then, create a file two_builds.yml with the following content.

includes:
-  - "$RUNNING_NG_PACKAGE_DATA/base/runbms.yml"
+  - "$RUNNING_NG_PACKAGE_DATA/base/runbms.yml"

The YAML file represents a dictionary (key-value pairs) that defines the experiments you are running. The includes directive here will populate the dictionary with some default values shipped with running-ng.

@@ -215,11 +215,11 @@

Prepare The benchmark suite is defined in $RUNNING_NG_PACKAGE_DATA/base/dacapo.yml. By default, the minimum heap sizes of dacapochopin-29a657f benchmarks are measured with AdoptOpenJDK 15 using G1 GC. If you are using OpenJDK 11 or 17, you can override the value of suites.dacapochopin-29a657f.minheap to temurin-17-G1 or temurin-11-G1. -That is, you can, for example, add "suites.dacapochopin-29a657f.minheap": "temurin-17-G1" to overrides.

+That is, you can, for example, add "suites.dacapochopin-29a657f.minheap": "temurin-17-G1" to overrides.

Then, add the following to two_builds.yml.

overrides:
-  "suites.dacapochopin-29a657f.timing_iteration": 5
-  "suites.dacapochopin-29a657f.callback": "probe.DacapoChopinCallback"
+  "suites.dacapochopin-29a657f.timing_iteration": 5
+  "suites.dacapochopin-29a657f.callback": "probe.DacapoChopinCallback"

That is, we want to run five iterations for each invocation, and use DacapoChopinCallback because it is the appropriate callback for this release of DaCapo.

Prepare Your Builds

@@ -231,11 +231,11 @@

Prepa build1: type: OpenJDK release: 11 - home: "/path/to/build1/jdk" # make sure /path/to/build1/jdk/bin/java exists + home: "/path/to/build1/jdk" # make sure /path/to/build1/jdk/bin/java exists build2: type: OpenJDK release: 11 - home: "/path/to/build2/jdk" # make sure /path/to/build2/jdk/bin/java exists + home: "/path/to/build2/jdk" # make sure /path/to/build2/jdk/bin/java exists

This defines two builds of runtimes.

I recommend that you use absolute paths for the builds, although relative paths will work, and will be relative to where you run running.

@@ -246,18 +246,18 @@

Prepare Probes< 
modifiers:
   probes_cp:
     type: JVMClasspath
-    val: "/path/to/probes/out /path/to/probes/out/probes.jar"
+    val: "/path/to/probes/out /path/to/probes/out/probes.jar"
   probes:
     type: JVMArg
-    val: "-Djava.library.path=/path/to/probes/out -Dprobes=RustMMTk"
+    val: "-Djava.library.path=/path/to/probes/out -Dprobes=RustMMTk"

This defines two modifiers, which will be used later to modify the JVM command line arguments.

Please only use absolute paths for all the above.

Prepare Configs

Finally, add he following to two_builds.yml.

configs:
-  - "build1|ms|s|c2|mmtk_gc-SemiSpace|tph|probes_cp|probes"
-  - "build2|ms|s|c2|mmtk_gc-SemiSpace|tph|probes_cp|probes"
+  - "build1|ms|s|c2|mmtk_gc-SemiSpace|tph|probes_cp|probes"
+  - "build2|ms|s|c2|mmtk_gc-SemiSpace|tph|probes_cp|probes"

The syntax is described here.

Sanity Checks

@@ -278,7 +278,7 @@

Single In

This allows you to see any issue before wasting several days only realizing that something didn't work.

Run It

Once you are happy with everything, run the experiments.

-
running runbms /path/to/log two_builds.yml 8 -p "two_builds"
+
running runbms /path/to/log two_builds.yml 8 -p "two_builds"
 

 
Don't forget to give the results folder a prefix so that you can later tell what the experiment was for.

 
Analysing Results

diff --git a/references/index.html b/references/index.html
index ae8fadd..dc54975 100644
--- a/references/index.html
+++ b/references/index.html
@@ -206,11 +206,11 @@ 
benchmarks
 
configs

 
A YAML list of configuration strings to be used to run the benchmarks. These are
 specified as a runtime followed by a '|' separated list of
-modifiers, i.e. "<runtime>|<modifier>|...|<modifier>".

+modifiers, i.e. "<runtime>|<modifier>|...|<modifier>".

 
For example:

 
configs:
-  - "openjdk11|ms|s|c2"
-  - "openjdk15|ms|s"
+  - "openjdk11|ms|s|c2"
+  - "openjdk15|ms|s"
 

 
specifies running to use the openjdk11 runtime with ms, s, and c2
 modifiers; and the openjdk15 runtime with the ms, and s modifiers. In
@@ -221,8 +221,8 @@ 
configs

 so that it's easier for multi-line editing.

 
For example:

 
configs:
-  - "openjdk8 |foo-1 |bar|buzz"
-  - "openjdk15|foo-16|   |buzz"
+  - "openjdk8 |foo-1 |bar|buzz"
+  - "openjdk15|foo-16|   |buzz"
 

 
includes

 
A YAML list of paths to YAML files that are to be included into the current
@@ -237,8 +237,8 @@ 
includes$HOME/bar.yml.
 Similarly,

 
includes:
- - "./base/suites.yml"
- - "./base/modifiers.yml"
+ - "./base/suites.yml"
+ - "./base/modifiers.yml"
 

 
includes the suites.yml and modifiers.yml files located at ./base
 respectively.

@@ -261,10 +261,10 @@ 
modifiersmodifiers:
   s:
     type: JVMArg
-    val: "-server"
+    val: "-server"
   c2:
     type: JVMArg
-    val: "-XX:-TieredCompilation -Xcomp"
+    val: "-XX:-TieredCompilation -Xcomp"
     excludes:
       dacapo2006:
         - eclipse
@@ -279,7 +279,7 @@ 
modifiersmodifiers:
   c2:
     type: JVMArg
-    val: "-XX:-TieredCompilation -Xcomp"
+    val: "-XX:-TieredCompilation -Xcomp"
     includes:
       dacapo2006:
         - eclipse
@@ -291,7 +291,7 @@ 
modifiersmodifiers:
   c2:
     type: JVMArg
-    val: "-XX:-TieredCompilation -Xcomp"
+    val: "-XX:-TieredCompilation -Xcomp"
     includes:
       dacapo2006:
         - eclipse
@@ -312,13 +312,13 @@ 
Value Options
 
modifiers:
   env_var:
     type: EnvVar
-    var: "FOO{0}"
-    val: "{1}"
+    var: "FOO{0}"
+    val: "{1}"
 
 [...]
 
 configs:
-  - "openjdk11|env_var-42-43"
+  - "openjdk11|env_var-42-43"
 

 
specifies to run the openjdk11 runtime with the environment
 variable FOO42 set to 43. Note that value options are not limited only to
diff --git a/references/suite.html b/references/suite.html
index dfae6a8..c7220bb 100644
--- a/references/suite.html
+++ b/references/suite.html
@@ -182,10 +182,10 @@ 
Keys

 
programs:
   <BM_NAME_1>:
     path: /full/path/to/benchmark/binary_1
-    args: "Any arguments to binary_1"
+    args: "Any arguments to binary_1"
   <BM_NAME_2>:
     path: /full/path/to/benchmark/binary_2
-    args: "Any arguments to binary_2"
+    args: "Any arguments to binary_2"
   [...]
 

 
A possible use-case could use wrapper shell scripts around the benchmark to
@@ -193,7 +193,7 @@ 
Keys

 
DaCapo

 
DaCapo benchmark suite.

 
Keys

-
release: one of the possible values ["2006", "9.12", "evaluation"].
+
release: one of the possible values ["2006", "9.12", "evaluation"].
 The value is required.

 
path: path to the DaCapo jar.
 The value is required.
@@ -235,19 +235,19 @@ 
Keys

 Therefore, companion programs should have appropriate timeouts or detect when main program finishes.
 Here is an example of using companion to launch bpftrace in the background to count the system calls.

 
includes:
-  - "$RUNNING_NG_PACKAGE_DATA/base/runbms.yml"
+  - "$RUNNING_NG_PACKAGE_DATA/base/runbms.yml"
 
 overrides:
-  "suites.dacapo2006.timing_iteration": 1
-  "suites.dacapo2006.companion": "sudo bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @syscall[args->id] = count(); @process[comm] = count();} interval:s:10 { printf(\"Goodbye world!\\n\"); exit(); }'"
-  "invocations": 1
+  "suites.dacapo2006.timing_iteration": 1
+  "suites.dacapo2006.companion": "sudo bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @syscall[args->id] = count(); @process[comm] = count();} interval:s:10 { printf(\"Goodbye world!\\n\"); exit(); }'"
+  "invocations": 1
 
 benchmarks:
   dacapo2006:
     - fop
 
 configs:
-  - "temurin-17"
+  - "temurin-17"
 

 
In the log file, the output from the main program and the output from the companion program is separated by *****.

 
size: specifying the size of input data.
@@ -268,7 +268,7 @@ 
SPECjbb2015 (preview ⚠️)

 
SPECjbb2015.

 
Keys

-
release: one of the possible values ["1.03"].
+
release: one of the possible values ["1.03"].
 The value is required.

 
path: path to the jar.
 The value is required.
@@ -276,28 +276,28 @@ 
Keys

 Environment variables will be expanded.

 
Benchmark Specification

 
Only strings are allowed, which should correspond to the the mode of the SPECjbb2015 controller.
-Right now, only "composite" is supported.

+Right now, only "composite" is supported.

 
SPECjvm98 (preview ⚠️)

 
SPECjvm98.

 
Note that you will need to prepend probes to the classpaths, so that the modified SpecApplication can be used.

 
Here is an example configuration file.

 
includes:
-  - "/home/zixianc/running-ng/src/running/config/base/runbms.yml"
+  - "/home/zixianc/running-ng/src/running/config/base/runbms.yml"
 
 modifiers:
   probes_cp:
     type: JVMClasspathPrepend
-    val: "/home/zixianc/MMTk-Dev/evaluation/probes /home/zixianc/MMTk-Dev/evaluation/probes/probes.jar"
+    val: "/home/zixianc/MMTk-Dev/evaluation/probes /home/zixianc/MMTk-Dev/evaluation/probes/probes.jar"
 
 benchmarks:
   specjvm98:
     - _213_javac
 
 configs:
-  - "adoptopenjdk-8|probes_cp"
+  - "adoptopenjdk-8|probes_cp"
 

 
Keys

-
release: one of the possible values ["1.03_05"].
+
release: one of the possible values ["1.03_05"].
 The value is required.

 
path: path to the SPECjvm98 folder, where you can find SpecApplication.class.
 The value is required.
diff --git a/searcher.js b/searcher.js
index d2b0aee..dc03e0a 100644
--- a/searcher.js
+++ b/searcher.js
@@ -316,7 +316,7 @@ window.search = window.search || {};
     
     // Eventhandler for keyevents on `document`
     function globalKeyHandler(e) {
-        if (e.altKey || e.ctrlKey || e.metaKey || e.shiftKey || e.target.type === 'textarea' || e.target.type === 'text') { return; }
+        if (e.altKey || e.ctrlKey || e.metaKey || e.shiftKey || e.target.type === 'textarea' || e.target.type === 'text' || !hasFocus() && /^(?:input|select|textarea)$/i.test(e.target.nodeName)) { return; }
 
         if (e.keyCode === ESCAPE_KEYCODE) {
             e.preventDefault();