Cache several shrink results for every chain step, so it does not re-access state model state during shrink phase #436
Conversation
…wrong state object when creating transformations This change caches the transformation if it accesses state. fixes jqwik-team#426
…access state model state during shrink phase This might take slightly more memory and produce suboptimal shrinks, however makes shrink possible for state-accessing transformations. Fixes jqwik-team#428
vlsi
changed the title
| @Property | ||
| void chainActionsAreProperlyDescribedEvenAfterChainExecution(@ForAll("setMutatingChain") ActionChain<SetMutatingChainState> chain) { | ||
| chain = chain.withInvariant( | ||
| state -> { | ||
| if (state.hasPrints) { | ||
| assertThat(state.actualOps).hasSizeLessThan(5); |
There was a problem hiding this comment.
Currently, the property is going to fail, so I probably need to add some kind of post-processing to treat "hasSizeLessThan violations as expected".
| .sorted(Comparator.comparing(Shrinkable::distance)) | ||
| .limit(numSamples) |
There was a problem hiding this comment.
This triggers OOM in case .shrink() stream is big. See #338 (comment)
|
I tried re-enabling shrink for my action chain, and it turns out to spend too much time shrinking (too many variations? too many restrictions in the current shrink approach like "after shrink, all the subsequent state-accessing actions must be discarded"?) I tried this change with my current tests. and the property fails after ~8 minutes with shrink=OFF producing ~15 actions. However, if I enable shrinking and re-run with the same seed, then it can't converge after 40-50 minutes, so I am inclined to just go for shrink=OFF and get the failure faster. I consider implementing toString-like methods that would print test code instead of "list of actions". Then I would be able to "shrink" the test manually by commenting-out irrelevant bits of the generated test code. |
Overview
See "a" in #428 (comment)
a) Sample (and keep) several .shrink() results for each action before the action is executed.
Pro: we would have valid shrink results (the shrink result would strictly correspond to the state)
Cons: it would spend time and memory on keeping .shrink() results even in case the chain succeeds.
Cons: it does not fully fit "try many different shrink types, order them, and start with the most promising ones" since we would be able to keep only a few shrink results for each action.
Details
The number of cached shrinkables could be configured via
jqwik.eagerChainShrinkSamplessystem property. I default it to 2.I hereby agree to the terms of the jqwik Contributor Agreement.