I second this, it would be nice if there was something like a test.nix which was also produced, and just running nix-build test.nix would then run all of that tests which were affected by the package set changes.

I would love to add the feature, however, I am short on time :(

EDIT: or better yet, have another switch nix-review pr 82742 --tests and it will also run tests as well

EDIT: or better yet, have another switch nix-review pr 82742 --tests and it will also run tests as well

since you should run tests when reviewing a pr, they should be opt-out

Yes, but they can be very costly in compute, ram, and disk space (may bring in many more dependencies). And I think it would warrant opting-in to a review potentially taking a significant amount of system resources.

That's true, especially for nixosTests building a whole VM. Maybe we can exclude them und just run package tests automatically?

Vm tests would need to be build serial after packages have been build.

Vm tests would need to be build serial after packages have been build.

Why? Nix already does that by consequence.

BTW I am trying to do the evaluation to get all tests of the package differences but got stuck with a issue where getting the outPath of the test and triggering a error (non catchable) because some dependency is marked as broken.

I think it needs either some kind of ofborg support or some nixpkgs adjustments to deal with this.

How about building all tests except for VM tests by default, and opting-in to VM tests?