Add benchmark noise reducing info. (#5924)

* Add benchmark noise reducing info.

Add info about knobs making benchmarks more stable
across different runs.

* Add more general info about setting clock freq.

* Move comments out of the code

benchmarks/README.md

@@ -5,6 +5,27 @@ The two main benchmarking scripts are
   - `result_analyzer.py` to aggregate the benchmark result in CSV form.
 
 
+## Reducing benchmark noise 
+
+It is important to keep the benchmark runs safe from external effects 
+to reduce noise. Do the following:
+
+Sets the CPU statically to the highest tuneable frequency.
+Prevent energy saving features to kick in.
+
+```sudo cpupower frequency-set --governor performance```
+
+Lock GPU clocks to lower frequency to reduce the chance of thermal throttling. Choose
+FREQ based on your GPU info. To find out clock frequency on your device run:
+`nvidia-smi -q -d CLOCK`, and look for Graphics/SM in Max Clocks section.
+Setting the clock a couple hundrend MHz below, or ~80% of max
+will most likely prevent thermal throttling effects.
+
+```FREQ=... nvidia-smi --lock-gpu-clocks=$FREQ,$FREQ```
+
+Disable autoboost selecting clock rate based on thermal, and power budget effects.
+```CUDA_AUTO_BOOST=0```
+
 ## Experiment runner
 
 Run the `experiment_runner.py` from the `pytorch` directory, which should be the