Bencher succeed without running any benchmarks with rust_criterion adapter on Windows.

[TheVeryDarkness]

Hello! I'm using bencher in Github Actions for one of my repos in Rust, and I'm using adapter rust_criterion, but I found bencher works well on macos-latest and linux-latest, but succeed without running any benchmarks on Windows.

You can find results from a workflow run in https://github.com/TheVeryDarkness/iof/actions/runs/11302133786.

• Adapter: rust_criterion
• Platform: Windows

[epompeii]

@TheVeryDarkness thank you for using Bencher, and I'm sorry you're running into trouble.

I do have a CI job that runs the example Criterion benchmarks on Windows.
This is output from my most recent push: https://github.com/bencherdev/bencher/actions/runs/11301003886/job/31434759390#step:4:2883

Taking a look at your workflow file: https://github.com/TheVeryDarkness/iof/blob/main/.github%2Fworkflows%2Fbenchmark.yml#L32
I would recommend trying to wrap cargo bench in quotes: "cargo bench"
And see if that helps.
This is effectively what the example does:

bencher/tasks/test_api/src/task/test/examples.rs

Lines 156 to 158 in 8a1cc54

	Self::RustCriterion | Self::RustIai | Self::RustIaiCallgrind | Self::RustCustom => {
	"cargo bench"
	},

[TheVeryDarkness]

Adding quotes there doesn't seem to be working :(

Workflow run: https://github.com/TheVeryDarkness/iof/actions/runs/11320056080

There was an error when running on linux-latest because I had some bugs in my benchmarks.

I suspect it has something to do with the spaces in my benchmark names.

[TheVeryDarkness]

Well, it seems it's not caused by the spaces.

Workflow run: https://github.com/TheVeryDarkness/iof/actions/runs/11323643624/job/31486739251

[TheVeryDarkness]

And cargo bench -- --list can list all benchmarks.

Workflow run: https://github.com/TheVeryDarkness/iof/actions/runs/11323828042/job/31487307640.

[epompeii]

@TheVeryDarkness thank you for trying out using quotes. I'm not very knowledgeable on Windows development, so I figured it was worth a shot.

If you run your benchmarks on a Windows box without Bencher, what does the output look like?

[epompeii]

@TheVeryDarkness any update on this issue?

[TheVeryDarkness]

@epompeii Thanks for reminding me! Here is a run on Windows: https://github.com/TheVeryDarkness/iof/actions/runs/12478423660/job/34825991109. The benchmark itself runs well on windows, but bencher still does not run benchmarks.

By the way, CI seems to be failing on Linux and MacOS though I think I didn't change too many codes.

[TheVeryDarkness]

Here is the latest run on Windows: https://github.com/TheVeryDarkness/iof/actions/runs/12478628302/job/34826478391#step:7:1

[epompeii]

@TheVeryDarkness thank you for the recent runs. I checked the logs for both runs, and I only see two POST calls to the API backend for each of them. Since the Linux and MacOS Reports are showing up, these would therefore belong to them.

This makes me think the the Windows CLI is not even sending over the POST request.

I currently can't get access to a Windows VM to debug this:

Due to ongoing technical issues, as of October 23, 2024, downloads are temporarily unavailable.

https://developer.microsoft.com/en-us/windows/downloads/virtual-machines/

Hopefully that changes soon.

[epompeii]

@TheVeryDarkness if you put the Threshold boundary (0.99) in quotes, does that change anything?

https://github.com/TheVeryDarkness/iof/blob/main/.github%2Fworkflows%2Fbenchmark.yml#L39

bencher run --project iof --token '${{ secrets.BENCHER_API_TOKEN }}' --branch main --testbed '${{ matrix.os }}' --threshold-measure latency --threshold-test t_test --threshold-max-sample-size 64 --threshold-upper-boundary "0.99" --thresholds-reset --err --adapter rust_criterion --github-actions '${{ secrets.GITHUB_TOKEN }}' "cargo bench"

[TheVeryDarkness]

@epompeii it seems not. The latest run: https://github.com/TheVeryDarkness/iof/actions/runs/12487806742/job/34849436729#step:8:1