[ci] More aggressively tune the GPU and benchmark

[xuzhao9]

Through the OSS metrics tracking we found the day-to-day run variance is larger than we expected (~4%) where we are shooting for 2% max variance.

This PR will tune the GPU more aggressively and run gemm and softmax benchmarks with cudagraph to see if metrics can be more stabilized.

Test plan: CI

[facebook-github-bot]

@xuzhao9 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[xuzhao9]

We are seeing weird softmax gbps data on this PR:

Need to validate the numbers

Maybe also add roofline checks for the results.

[xuzhao9]

The signal above has cleared, as we can reproduce the result based on the Triton official test script:
https://triton-lang.org/main/getting-started/tutorials/02-fused-softmax.html

$ python softmax.py
softmax-performance:
        N       Triton        Torch
0   256.0  1752.991381  2324.677671
1   384.0  2654.954942  2592.323434
2   512.0  3404.632698  3034.068027
3   640.0  3939.462997  2772.362045
4   768.0  4352.611933  3071.420366
5   896.0  4450.028124  3269.086357
6  1024.0  4186.171295  3368.800611
7  1152.0  3398.567332  2500.780098

[facebook-github-bot]

@xuzhao9 merged this pull request in 18eaf84.