Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Output seed information when checking fails during generation #78

Open
SevereOverfl0w opened this issue Oct 6, 2023 · 7 comments
Open

Output seed information when checking fails during generation #78

SevereOverfl0w opened this issue Oct 6, 2023 · 7 comments

Comments

@SevereOverfl0w
Copy link

If generation fails during the bindings phase of the checking macro any seed information is lost. This makes it difficult to reproduce failures.

(deftest fails
  (checking
    "a"
    [i gen/small-integer
     :let [x (/ 1 i)]]
    (is (pos? x))))

Produces this output:

ERROR in (fails) (Numbers.java:188)
Uncaught exception, not in assertion.
expected: nil
  actual: java.lang.ArithmeticException: Divide by zero
 at clojure.lang.Numbers.divide (Numbers.java:188)
    clojure.lang.Numbers.divide (Numbers.java:3877)
    com.gfredericks.test.chuck.clojure_test_test$fn__4276$fn__4277$fn__4284.invoke (clojure_test_test.cljc:55)
    clojure.test.check.rose_tree$fmap.invokeStatic (rose_tree.cljc:77)
    clojure.test.check.rose_tree$fmap.invoke (rose_tree.cljc:73)
    clojure.test.check.rose_tree$fmap$fn__618.invoke (rose_tree.cljc:77)
    clojure.core$map$fn__5884.invoke (core.clj:2759)
    clojure.lang.LazySeq.sval (LazySeq.java:42)
    clojure.lang.LazySeq.seq (LazySeq.java:51)
    clojure.lang.RT.seq (RT.java:535)
    clojure.core$seq__5419.invokeStatic (core.clj:139)
    clojure.core$map$fn__5884.invoke (core.clj:2750)
    clojure.lang.LazySeq.sval (LazySeq.java:42)
    clojure.lang.LazySeq.seq (LazySeq.java:51)
    clojure.lang.RT.seq (RT.java:535)
    clojure.core$seq__5419.invokeStatic (core.clj:139)
    clojure.core/seq (core.clj:139)
    clojure.test.check.rose_tree$permutations$iter__629__635$fn__636$iter__631__640$fn__641.invoke (rose_tree.cljc:102)
    clojure.lang.LazySeq.sval (LazySeq.java:42)
    clojure.lang.LazySeq.seq (LazySeq.java:51)
    clojure.lang.RT.seq (RT.java:535)
    clojure.core$seq__5419.invokeStatic (core.clj:139)
    clojure.core/seq (core.clj:139)
    clojure.test.check.rose_tree$permutations$iter__629__635$fn__636.invoke (rose_tree.cljc:102)
    clojure.lang.LazySeq.sval (LazySeq.java:42)
    clojure.lang.LazySeq.seq (LazySeq.java:51)
    clojure.lang.RT.seq (RT.java:535)
    clojure.core$seq__5419.invokeStatic (core.clj:139)
    clojure.core$map$fn__5884.invoke (core.clj:2750)
    clojure.lang.LazySeq.sval (LazySeq.java:42)
    clojure.lang.LazySeq.seq (LazySeq.java:51)
    clojure.lang.RT.seq (RT.java:535)
    clojure.core$seq__5419.invokeStatic (core.clj:139)
    clojure.core$map$fn__5884.invoke (core.clj:2750)
    clojure.lang.LazySeq.sval (LazySeq.java:42)
    clojure.lang.LazySeq.seq (LazySeq.java:51)
    clojure.lang.RT.seq (RT.java:535)
    clojure.core$seq__5419.invokeStatic (core.clj:139)
    clojure.core$empty_QMARK_.invokeStatic (core.clj:6195)
    clojure.core$empty_QMARK_.invoke (core.clj:6195)
    clojure.test.check$shrink_loop.invokeStatic (check.cljc:263)
    clojure.test.check$shrink_loop.invoke (check.cljc:242)
    clojure.test.check$failure.invokeStatic (check.cljc:314)
    clojure.test.check$failure.invoke (check.cljc:297)
    clojure.test.check$quick_check.invokeStatic (check.cljc:228)
    clojure.test.check$quick_check.doInvoke (check.cljc:59)
    clojure.lang.RestFn.invoke (RestFn.java:425)
    clojure.lang.AFn.applyToHelper (AFn.java:156)
    clojure.lang.RestFn.applyTo (RestFn.java:132)
    clojure.core$apply.invokeStatic (core.clj:671)
    clojure.core$apply.invoke (core.clj:662)
    com.gfredericks.test.chuck.clojure_test_test$fn__4276$fn__4277.invoke (clojure_test_test.cljc:52)
    com.gfredericks.test.chuck.clojure_test$_testing.invokeStatic (clojure_test.cljc:103)
    com.gfredericks.test.chuck.clojure_test$_testing.invoke (clojure_test.cljc:101)
    com.gfredericks.test.chuck.clojure_test_test$fn__4276.invokeStatic (clojure_test_test.cljc:52)
    com.gfredericks.test.chuck.clojure_test_test/fn (clojure_test_test.cljc:51)
    clojure.test$test_var$fn__9761.invoke (test.clj:717)
    clojure.test$test_var.invokeStatic (test.clj:717)
    clojure.test$test_var.invoke (test.clj:708)
    clojure.test$test_vars$fn__9787$fn__9792.invoke (test.clj:735)
    clojure.test$default_fixture.invokeStatic (test.clj:687)
    clojure.test$default_fixture.invoke (test.clj:683)
    clojure.test$test_vars$fn__9787.invoke (test.clj:735)
    clojure.test$default_fixture.invokeStatic (test.clj:687)
    clojure.test$default_fixture.invoke (test.clj:683)
    clojure.test$test_vars.invokeStatic (test.clj:731)
    clojure.test$test_all_vars.invokeStatic (test.clj:737)
    clojure.test$test_ns.invokeStatic (test.clj:758)
    clojure.test$test_ns.invoke (test.clj:743)
    user$eval227$fn__350.invoke (form-init6834647596943620637.clj:1)
    clojure.lang.AFn.applyToHelper (AFn.java:156)
    clojure.lang.AFn.applyTo (AFn.java:144)
    clojure.core$apply.invokeStatic (core.clj:669)
    clojure.core$apply.invoke (core.clj:662)
    leiningen.core.injected$compose_hooks$fn__154.doInvoke (form-init6834647596943620637.clj:1)
    clojure.lang.RestFn.applyTo (RestFn.java:137)
    clojure.core$apply.invokeStatic (core.clj:667)
    clojure.core$apply.invoke (core.clj:662)
    leiningen.core.injected$run_hooks.invokeStatic (form-init6834647596943620637.clj:1)
    leiningen.core.injected$run_hooks.invoke (form-init6834647596943620637.clj:1)
    leiningen.core.injected$prepare_for_hooks$fn__159$fn__160.doInvoke (form-init6834647596943620637.clj:1)
    clojure.lang.RestFn.applyTo (RestFn.java:137)
    clojure.lang.AFunction$1.doInvoke (AFunction.java:31)
    clojure.lang.RestFn.invoke (RestFn.java:408)
    clojure.core$map$fn__5884.invoke (core.clj:2759)
    clojure.lang.LazySeq.sval (LazySeq.java:42)
    clojure.lang.LazySeq.seq (LazySeq.java:51)
    clojure.lang.Cons.next (Cons.java:39)
    clojure.lang.RT.boundedLength (RT.java:1793)
    clojure.lang.RestFn.applyTo (RestFn.java:130)
    clojure.core$apply.invokeStatic (core.clj:669)
    clojure.test$run_tests.invokeStatic (test.clj:768)
    clojure.test$run_tests.doInvoke (test.clj:768)
    clojure.lang.RestFn.applyTo (RestFn.java:137)
    clojure.core$apply.invokeStatic (core.clj:667)
    clojure.core$apply.invoke (core.clj:662)
    user$eval227$fn__360$fn__413.invoke (form-init6834647596943620637.clj:1)
    user$eval227$fn__360$fn__361.invoke (form-init6834647596943620637.clj:1)
    user$eval227$fn__360.invoke (form-init6834647596943620637.clj:1)
    user$eval227.invokeStatic (form-init6834647596943620637.clj:1)
    user$eval227.invoke (form-init6834647596943620637.clj:1)
    clojure.lang.Compiler.eval (Compiler.java:7181)
    clojure.lang.Compiler.eval (Compiler.java:7171)
    clojure.lang.Compiler.load (Compiler.java:7640)
    clojure.lang.Compiler.loadFile (Compiler.java:7578)
    clojure.main$load_script.invokeStatic (main.clj:475)
    clojure.main$init_opt.invokeStatic (main.clj:477)
    clojure.main$init_opt.invoke (main.clj:477)
    clojure.main$initialize.invokeStatic (main.clj:508)
    clojure.main$null_opt.invokeStatic (main.clj:542)
    clojure.main$null_opt.invoke (main.clj:539)
    clojure.main$main.invokeStatic (main.clj:664)
    clojure.main$main.doInvoke (main.clj:616)
    clojure.lang.RestFn.applyTo (RestFn.java:137)
    clojure.lang.Var.applyTo (Var.java:705)
    clojure.main.main (main.java:40)

Ran 5 tests containing 20 assertions.
0 failures, 1 errors.
Tests failed.
@SevereOverfl0w
Copy link
Author

SevereOverfl0w commented Oct 6, 2023 •
edited
Loading

After poking at this a little bit, it seems that we'd need upstream changes in test.check to make this work. When doing a call-gen in quick-check the exception would need to be caught and some handling put in place. Calling :reporter-fn with any relevant information before rethrowing the exception seems like the minimal set of changes to me.

@gfredericks
Copy link
Owner

Okay; we can leave this open and you or somebody else could pursue those changes in test.check.

@frenchy64 frenchy64 mentioned this issue Feb 23, 2024
Open
frenchy64 added a commit to frenchy64/test.chuck that referenced this issue Feb 23, 2024
@frenchy64
Close gfredericks#78: print seed on generation error
54af899
@frenchy64 frenchy64 mentioned this issue Feb 23, 2024
Closed
@frenchy64
Copy link
Contributor

frenchy64 commented Feb 23, 2024 •
edited
Loading

@gfredericks I took a shot at it. #80

What do you think about the test.check changes? Worth proposing on TCHECK-159?

https://github.com/frenchy64/test.check/pull/1/files

@gfredericks
Copy link
Owner

I'm not maintaining test.check anymore, so my opinion doesn't matter too much.

@frenchy64
Copy link
Contributor

Firstly, thanks @gfredericks for all the years of maintainership. In my book, test.check under your direction has been the most successful Clojure library of them all. All that to say, I really appreciate your work and congratulations on an amazing tenure.

Secondly, my hit rate on getting patches merged from JIRA is probably about 5%. Would you consider a patch to test.chuck with this definition of quick-check, and then using it in checking?

@gfredericks
Copy link
Owner

Thanks for the kind words!

Copypasting the definition of quickcheck into this library sounds like it could gen gnarly in the future if implementation details in test.check change.

Is there no way to do this in test.chuck, perhaps by wrapping generators with a fmap that has a try/catch, and then wrapping the checking expression similarly? That'd amount to turning a generator-exception into a test failure, which is a bit weird, but would at least result in the seed being output.

@frenchy64
Copy link
Contributor

@gfredericks great idea, I will try it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@gfredericks @frenchy64 @SevereOverfl0w