Adjust number of test cases #496
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The PR seems to make the “can rollback” test flaky because it discards way too many objects. I don't think we should merge it until that is taken care of. |
Niols
force-pushed
the
niols/ajust-test-cases
branch
from
a1764ec to
d370f82
Compare
Niols
force-pushed
the
niols/ajust-test-cases
branch
2 times, most recently
from
677ae88 to
1c4365a
Compare
Niols
force-pushed
the
niols/ajust-test-cases
branch
from
1c4365a to
2dd7f18
Compare
github-merge-queue bot
pushed a commit
that referenced
this pull request
Nov 17, 2023
_This PR has been initially devised as part of the Genesis project, in #496. It now targets `main` because we believe it might benefit the project as a whole._ ### Description This PR aims at normalising the adjustments of the number of QuickCheck tests in the whole codebase. `ouroboros-consensus:unstable-consensus-testlib` provides utilities in a module `Test.Util.TestEnv` to help describing a difference of behaviour of tests when ran in a `dev` (default), `ci` or `nightly` environment. This is typically used to change the number of test cases that QuickCheck generates. This number will be 100 in `dev`, 10,000 in `ci` and 100,000 in `nightly` by default. Now, for some tests, 100 is too much and so we end up wanting to run ten times less than that (or any other ratio, really). The proper way to do this without hard-coding a value (say 10 tests) which would then be the same independently of the environment is by adjusting the value option `QuickCheckTests`, with something like: ```haskell adjustOption (\(QuickCheckTests n) -> QuickCheckTests (1 `max` (div n 10))) :: TestTree -> TestTree ``` This is what is being done in various places of the code base in slightly different ways. This PR introduces a helper `adjustQuickCheckTests` and uses it consistently across the code base. This helper allows writing “let's run 10⨉ less tests” as ```haskell adjustQuickCheckTests (`div` 10) :: TestTree -> TestTree ``` ### Bonus questions I have two bonus questions which I think could be taken care of in this PR. First, I have seen in various place a helper: ```haskell adjustTestEnv :: TestTree -> TestEnv -> TestTree adjustTestEnv tree = \case Nightly -> tree _ -> adjustQuickCheckTests (`div` 10) tree ``` I have the impression that this could be simplified as simply ``adjustQuickCheckTests (`div` 10)``. What do you think? Second, a lot of places hardcode a number of tests with `localOption (QuickCheckTests 14)` for instance. I would be tempted to replace this by something like ``adjustQuickCheckTests (`div` 7)``, which would give a similar result but would behave better when test environments are indeed used in CI or nightly. What do you think?
|
Has to be adapted to target #819; I will do that today. |
|
See #821. |
facundominguez
pushed a commit
that referenced
this pull request
Dec 14, 2023
* Rename `BlockTree.prettyPrint` into `prettyBlockTree`
* Move `PointSchedule`'s pretty-printing closer to its type
* Move `GenesisTest`'s printing closer to its type
* Adjust number of test cases
Reimplementation of #496 targetting #819, which include the more general
changes of #500.
* Add a pretty-printer for the current state of the node under test
facundominguez
pushed a commit
that referenced
this pull request
Dec 15, 2023
* Rename `BlockTree.prettyPrint` into `prettyBlockTree`
* Move `PointSchedule`'s pretty-printing closer to its type
* Move `GenesisTest`'s printing closer to its type
* Adjust number of test cases
Reimplementation of #496 targetting #819, which include the more general
changes of #500.
* Add a pretty-printer for the current state of the node under test
facundominguez
pushed a commit
that referenced
this pull request
Dec 15, 2023
* Rename `BlockTree.prettyPrint` into `prettyBlockTree`
* Move `PointSchedule`'s pretty-printing closer to its type
* Move `GenesisTest`'s printing closer to its type
* Adjust number of test cases
Reimplementation of #496 targetting #819, which include the more general
changes of #500.
* Add a pretty-printer for the current state of the node under test
Niols
added a commit
that referenced
this pull request
Jan 23, 2024
* Rename `BlockTree.prettyPrint` into `prettyBlockTree`
* Move `PointSchedule`'s pretty-printing closer to its type
* Move `GenesisTest`'s printing closer to its type
* Adjust number of test cases
Reimplementation of #496 targetting #819, which include the more general
changes of #500.
* Add a pretty-printer for the current state of the node under test
nbacquey
pushed a commit
that referenced
this pull request
Feb 26, 2024
* Rename `BlockTree.prettyPrint` into `prettyBlockTree`
* Move `PointSchedule`'s pretty-printing closer to its type
* Move `GenesisTest`'s printing closer to its type
* Adjust number of test cases
Reimplementation of #496 targetting #819, which include the more general
changes of #500.
* Add a pretty-printer for the current state of the node under test
nbacquey
pushed a commit
that referenced
this pull request
Feb 26, 2024
* Rename `BlockTree.prettyPrint` into `prettyBlockTree`
* Move `PointSchedule`'s pretty-printing closer to its type
* Move `GenesisTest`'s printing closer to its type
* Adjust number of test cases
Reimplementation of #496 targetting #819, which include the more general
changes of #500.
* Add a pretty-printer for the current state of the node under test
nbacquey
pushed a commit
that referenced
this pull request
Feb 27, 2024
* Rename `BlockTree.prettyPrint` into `prettyBlockTree`
* Move `PointSchedule`'s pretty-printing closer to its type
* Move `GenesisTest`'s printing closer to its type
* Adjust number of test cases
Reimplementation of #496 targetting #819, which include the more general
changes of #500.
* Add a pretty-printer for the current state of the node under test
nbacquey
pushed a commit
that referenced
this pull request
Feb 27, 2024
* Rename `BlockTree.prettyPrint` into `prettyBlockTree`
* Move `PointSchedule`'s pretty-printing closer to its type
* Move `GenesisTest`'s printing closer to its type
* Adjust number of test cases
Reimplementation of #496 targetting #819, which include the more general
changes of #500.
* Add a pretty-printer for the current state of the node under test
nbacquey
pushed a commit
that referenced
this pull request
Feb 27, 2024
* Rename `BlockTree.prettyPrint` into `prettyBlockTree`
* Move `PointSchedule`'s pretty-printing closer to its type
* Move `GenesisTest`'s printing closer to its type
* Adjust number of test cases
Reimplementation of #496 targetting #819, which include the more general
changes of #500.
* Add a pretty-printer for the current state of the node under test
nbacquey
pushed a commit
that referenced
this pull request
Feb 27, 2024
* Rename `BlockTree.prettyPrint` into `prettyBlockTree`
* Move `PointSchedule`'s pretty-printing closer to its type
* Move `GenesisTest`'s printing closer to its type
* Adjust number of test cases
Reimplementation of #496 targetting #819, which include the more general
changes of #500.
* Add a pretty-printer for the current state of the node under test
Niols
added a commit
that referenced
this pull request
Feb 29, 2024
* Rename `BlockTree.prettyPrint` into `prettyBlockTree`
* Move `PointSchedule`'s pretty-printing closer to its type
* Move `GenesisTest`'s printing closer to its type
* Adjust number of test cases
Reimplementation of #496 targetting #819, which include the more general
changes of #500.
* Add a pretty-printer for the current state of the node under test
nbacquey
pushed a commit
that referenced
this pull request
Mar 11, 2024
* Rename `BlockTree.prettyPrint` into `prettyBlockTree`
* Move `PointSchedule`'s pretty-printing closer to its type
* Move `GenesisTest`'s printing closer to its type
* Adjust number of test cases
Reimplementation of #496 targetting #819, which include the more general
changes of #500.
* Add a pretty-printer for the current state of the node under test
facundominguez
pushed a commit
that referenced
this pull request
Mar 11, 2024
* Rename `BlockTree.prettyPrint` into `prettyBlockTree`
* Move `PointSchedule`'s pretty-printing closer to its type
* Move `GenesisTest`'s printing closer to its type
* Adjust number of test cases
Reimplementation of #496 targetting #819, which include the more general
changes of #500.
* Add a pretty-printer for the current state of the node under test
amesgen
pushed a commit
that referenced
this pull request
Mar 25, 2024
* Rename `BlockTree.prettyPrint` into `prettyBlockTree`
* Move `PointSchedule`'s pretty-printing closer to its type
* Move `GenesisTest`'s printing closer to its type
* Adjust number of test cases
Reimplementation of #496 targetting #819, which include the more general
changes of #500.
* Add a pretty-printer for the current state of the node under test
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
For now, we have relied on
withMaxSuccessto reduce the number of tests. This does not allow for an increase of the number of tests in CI or nightly. This PR uses the proper mechanism.