feature request: Per-benchmark thresholds

[ollie-etl]

Currently thresholds are specified per testbed, measure, and branch. I'm requesting that it is also extend to include (maybe optional) benchmark

if I generate report which looks like this

{   "benchmark 1": {
            "foo": 17.0,
            "bar": 42.0
     },
     "benchmark 2": {
            "foo": 450000.0,
            "bar": 10000000.0
     }
}

Lets say I'd like to be able to determine that "benchmark 1". "foo" doesn't exceed 20, and "benchmark 2"."foo"` doesn't exceed 50000. I currently can't, or at least it would require normalising the results. If the units have physical significance, this isn't desirable.

[epompeii]

@ollie-etl you are correct, a Threshold is currently tied to the combination of Branch, Testbed, and Measure and then it is applied to all Benchmarks within that set.

Lets say I'd like to be able to determine that "benchmark 1". "foo" doesn't exceed 20, and "benchmark 2". "foo" doesn't exceed 50000.

This should currently be possible using a Percentage Test (percentage) with an Upper Boundary set to 0.0. Likewise, you can also guarantee it doesn't go below that set value by setting a Lower Boundary of 0.0.

You will also want to set the Max Sample Size to 2 so that it is only ever comparing against the most recent historical result.

Better filtering for which Benchmarks to apply a Threshold to is something that I've been putting some thought towards: #366
However, I think the above solution should be sufficient to handle your example use case.