Add benchmark noise reducing info.

[golechwierowicz]

Add info about knobs making benchmarks more stable across different runs.

[frgossen]

Thanks!

[zpcore]

Quick questions:

• Is there a recommendation/consensus GPU freq we can use throughout different model testing so that we can compare the results under the same constraint in the same platform?
• Also want to check if the GPU freq can be set arbitrarily as long as it is below the maximum or there is only limited options we can choose from.

[golechwierowicz]

Quick questions:

• Is there a recommendation/consensus GPU freq we can use throughout different model testing so that we can compare the results under the same constraint in the same platform?
• Also want to check if the GPU freq can be set arbitrarily as long as it is below the maximum or there is only limited options we can choose from.

1. I don't think there is a consensus among the team, it will be good to establish it now. (given everyone runs benchmarks manually, once we have an automated suite it will be unnecessary though)
2. AFAIK it has to below the maximum, above the minimum, and it has to be the multiple of 15. But the entire point is setting it to such a value so that thermal throttling does not kick in, and there is no random slowdown on computation. So setting it to ~80% of the maximum seems like a good idea.