- Fix beholder watch functionality that would cause a NullPointerException earlier.
- Handle assertions using
:actual (not= ,,,)output format, this fixed deep-diff when using the Expectations library.
- Always make the notifier plugin a no-op in CI environments, so when
(= "true" (getenv "CI"))
- We no longer depend on clojure or spec.alpha, as it is assumed your app declares its own Clojure dependency, which will pull in the matching spec.alpha version.
- Track all results when
:oncefixtures callfmultiple times. (thanks @NoahTheDuke)
- Version bumps
- Added
kaocha.stacktrace/*stacktrace-stop-list*binding to stop printing a stacktrace after matching a string
- [watch] Fix watching when paths do not exist.
- [watch] Respect disabling of unload/reload when watching.
- [watch] Respect clojure.tools.namespace.repl load/unload metadata.
- Update deep-diff2 to
2.9.202so Kaocha tests can benefit from faster hash-map diffing.
- Get rid of extra space in output when using the
:print-invocationsplugin - Prevent
ClassCastExceptionwhen using Ctrl-C (SIGINT) after enabling--watch
- Bump tools.cli to 1.0.219
- The watcher now prints the output from plugins, eg.
bin/kaocha --watch --plugin kaocha.plugin/profiling
- Expose
:print-config trueand:print-test-plan trueto Clojure CLI -X invocations.
- Smoke tests for babashka on CI
- The
-loadand-runmultimethods now use Kaocha's own hierarchy, rather than the global one. (Needed to support the new doctests type)
- Fix regression in the notifier plugin
- Kaocha is now compatible with Babashka. Running under Babashka is most useful for validating your Clojure code runs correctly on Babashka, but it may also be faster for some test suites because of reduced start up time.
- Fix issue with
--watchand Neovim by bumping Beholder to1.0.2
- Fix bug causing namepaces to not be loaded if an alias was already created for them using
:as-alias
kaocha.repl/configaccepts a:profilekey when specifying extra options.
- Configuration errors and circular dependencies are reported as warnings, rather than causing the entire watcher to crash. (thanks @frenchy64)
- Fix bug added in #384: assertions in the tail position of an each fixture would return the result of the assertion instead of the testable object with the merged report data. (thanks @NoahTheDuke)
kaocha.plugin.capture-output/bypassmacro, for temporarily bypassing output capturing
- Circular dependencies in watch mode no longer kills the process. (thanks @frenchy64)
- Dependency version bumps
- Ensure reloading errors are printed in watch mode when the first test suite is disabled.
- Using a try-catch (without rethrowing) in an
:eachfixture could swallow thrown exceptions, resulting in a test being treated as passing when it should have been reported as an error. Fixed by changing how:eachfixtures wrap the test function in execution. (thanks @NoahTheDuke) - Fix crash on Windows when using
--watchwith the default Beholder watcher.
- Documentation fixes and improvements
- Configure a timeout for notifications with
--notification-timeoutor:kaocha.plugin.notifier/timeout. Note that this feature doesn't work for terminal-notifier on macOS, with Java's built-in TrayIcon, or with notify-send on certain Linux desktop environments and distributions.
- Fix configuration parsing when using
--watch. Previously, profiles would be respected on the initial load, but not after watch reloaded the configuration. - Notifier now reports errors in your notification command instead of silently failing.
- Fixed
java.lang.IllegalArgumentException: No matching clause: []exception when:kaocha.spec.test.check/symsis a collection.
- Add specs for kaocha.plugin/profiling data (#302).
- Fix issue where diffing
(is (= ...))assertions sometimes fails when comparing records. - Fix the
--no-notificationsflag in the Notifier plugin
- Some error codes were duplicated. This is a breaking change if you rely on error codes.
- When a test suite configuration value is not a collection or symbol, the error code is now 250 instead of 252. The error code 252 is still used when the configuration file fails to load.
- When registering a plugin fails due to being unable to load a namespace, the error code is now 249 instead of 254. When registering a plugin fails for other reasons, the error code is now 248 instead of 254. When resolving a reporter var fails, the error code is still 254. You can test for error codes between 240 and 249 to check for plugin errors regardless of cause.
- Upgraded
lambdaisland/deep-difftolambdaisland/deep-diff2
- Fix a regression where in direct usage of
kaocha.apia reporter is specified as a function
- Fix issue in
gc-profiling, which was caused by a prior fix from 1.67.1055.
- Fix misleading error message when all tests are filtered out. Previously, it
would misleadingly suggest you correct the
test-pathsandns-patternsconfiguration keys. - Fix overflow with the
gc-profilingplugin when there's too many bytes.
- Extend
kaocha.config/load-configto also work on resources
- Fix issue with
gc-profilingplugin when there's a syntax error. - Ensure that modifications that are done by deleting and recreating the file
are picked up by using
--watchwith Beholder.
- Ignore test suites that are already excluded through CLI arguments when running the filter plugin. This gets rid of false warnings regarding missing metadata
- Version bumps of spec, expound, test.check
- Catch an exception in the notifier plugin which can occur in headless (CI) setups
- Added configuration
:kaocha.watch/typewhich takes either:beholderor:hawkas values. Defaulting to:beholderas the new fs watcher. - Add
--no-fail-fastCLI option
- Changed default watcher to Beholder which supports OSX/m1 machines natively. Hawk is now deprecated and will be removed in a future release.
- Add support for
:watch?when usingexec-fn
- Fixed an issue where the combination of a load error and using
--focuswould result in "no tests found", shadowing the actual error
- Apply
:kaocha/bindingshigher in the stack, so they are visible tomainandpost-summaryplugin hooks - Fix an issue when the history track reporter gets invoked outside of the scope where the history tracking atom is bound
- Fixed issue in
gc-profilingthat caused itermittentNullPointerExceptions
kaocha.runner/execfor use with Clojure CLI's -X feature- Added
gc-profilingplugin for measuring the memory usage of tests.
- Breaking! Unqualified plugin names containing dots are no longer
normalized to contain the
kaocha.plugin-namespace in front.
- Added support for code using
:as-alias
- Fix only considering public vars when building up the test plan
- Fix load-error handling in
kaocha.watch - Fix
could not resolve symbol requireerror that occured sporadically when requiring certain kaocha namespaces. - Fix printing of boolean options in the print-invocations plugin
- Fix Java reflection warning in the Notifier plugin
- [BREAKING] Remove the Orchestra dependency, and no longer auto-instrument.
You'll have to list Orchestra in your own
deps.edn/project.cljif you want to use the Orchestra plugin. - Version bumps of Clojure, tools.cli, spec.alpha, expound
- Formatting of failed test results using deep-diff can be disabled with
--diff-style :noneon the command line or:diff-style :noneintests.edn.
- Fix at least some cases of syntax errors being suppressed by the "no tests found" message.
- Kaocha watch can now add ignores from
.gitignoreand.ignore. To enable this feature, set:kaocha.watch/use-ignore-fileto true in your deps.edn. - Kaocha now falls back to the notifications provided by Java's AWT when it can't
find
notify-sendorterminal-notifier.
- Clearly alert the user that Clojure versions before 1.9 aren't supported, rather than failing on whatever 1.9 functionality happens to be invoked first.
- Fixed an issue with the definition of spec
:kaocha.test-plan/load-errorthat caused a ClassCastException whenever a generator was created for it. - Errors when loading plugins are more specific, specifying which namespaces, if any, failed to load.
- Warn when running Kaocha without a configuration file. This is fine for experimenting, but for long-term use, we recommend creating a configuration file to avoid changes in behavior between releases.
- Provide a warning when no tests are found.
- Fix exception when running Kaocha on Windows with the built-in notification plugin enabled.
- Fixed an issue with the optional
clojure.test.checkdependency (follow-up)
If the value of a configuration key is not a collection or symbol, a more helpful error message is output. Fixes #124kaocha.type.spec.test.checknow correctly builds fdef testables with configuration options from their enclosing test suites.kaocha.plugin.alpha.spec-test-checknow honors command line arguments based upon all of the configured STC suites rather than the static:generative-fdef-checksselector.- Fix an issue where
clojure.test.checkwould be required for Kaocha to work, rather than being an optional dependency
- Breaking! Test configuration (
tests.edn) is now validated with spec, meaning existing configs may fail. In most cases you should be able to update your config so it is valid, but please do report any issues. kaocha.plugin.alpha.spec-test-checknow respects a priority of supplied configuration. CLI options always take precedence, followed by options specified in individual test suites, followed by global options.- Improved spec definitions and generative fdef coverage
- Fix documentation table of contents
- Make Ctrl-C (SIGINT) handling more reliable, so you can always short-circuit Kaocha to see your failing tests.
- Make spec-test-check plugin honor commond line arguments, so you can run only the generated test suite.
- Don't run group tests (e.g. namespace) when there are no tests inside it that would run (empty or all tests skipped)
- fdef/spec based tests via plugin: honor
:clojure.spec.test.check/instrument?and:clojure.spec.test.check/check-asserts?fromtests.edn
- Added
:kaocha.plugin/debugfor easy introspection of Kaocha's machinery - Added docstrings and markdown docs for the Orchestra and Preloads plugins
- In the filter plugin's pre-load early filtering of test suites, check flags
provided directly in the config, instead of only checking command line
arguments. This fixes kaocha.repl invocations like
(kaocha.repl/run {:kaocha.filter/skip [:unit]})
- Prevent loading of test suites that are excluded from the run
- Added internal diagnostics
- Fix
post-summarywhen used from hooks plugin
- Add two new hooks,
:kaocha.hooks/pre-load-test,:kaocha.hooks/post-load-test - Extend the hooks plugin to allow for
:kaocha.hooks/pre-load-test,:kaocha.hooks/pre-test/:kaocha.hooks/post-testand:kaocha.hooks/post-load-testhooks to be defined on the testable (i.e. on the test suite) - The
:post-summaryhook can now be used through the hooks plugin (before it was only available to plugins) - Allow test type implementations to add
:kaocha.testable/aliasesto testables, these can be used when focusing/skipping
:kaocha.hooks/before/:kaocha.hooks/afternow get converted to:kaocha.hooks/pre-test/:kaocha.hooks/post-testhooks. The former are considered deprecated but will continue to work.- the
post-summaryhook will also be called when invoked viakaocha.repl kaocha.testable/test-seqonly returns actual testables, not a top level config/test-plan map- Bumped Orchestra and Expound, this pulled in a breaking change in Orchestra where it no longer includes the explained spec error in the exception message. To accomodate for this the Orchestra plugin has been updated so the explanation appears in the reported output.
- Only instrument lambdaisland/kaocha namespaces with Orchestra. For
instrumentation of your own code or third party libraries use the
:orchestraplugin.
- The hooks plugin now understands two new hooks at the test suite level,
:kaocha.hooks/beforeand:kaocha.hooks/after - Make the Hawk filesystem watcher configurable with
:kaocha.watch/hawk-opts
- Added a
:kaocha.testable/skip-add-classpath?flag on the test suite to prevent Kaocha from modifying the classpath
- An Orchestra plugin
:kaocha.plugin/orchestrafor instrumenting functions specs with Orchestra - A Preloads plugin
:kaocha.plugin/preloadsfor requiring namespaces before Kaocha loads test suites. This is useful for requiring spec namespaces or other side-effecting namespaces that are not required by test code.
- Fixed an issue where plugin names where not correctly normalized before deduplication, leading to potentially having a plugin twice in the stack
- Added
:kaocha.report/printed-expressionto thedebugreporter, for debugging reporting issues with kaocha-cljs
- Kaocha's own plugins can now use a simple keyword in
tests.edn, e.g.:notifierinstead of:kaocha.plugin/notifier, similar to what we do on the command line.
- Bumped several dependencies: org.clojure/spec.alpha, org.clojure/tools.cli, and aero
- Namespaces where the ns form can not be read by tools.readers are now reported as a test failure, rather than being quietly ignored.
- Added support in the reporter for the
=?macro as used in Expectations (thanks @dharrigan 🎉)
- Fix support for dynamic bindings with
set!in watch mode. (thanks @frenchy64) - Fixes support for
:confighooks in the hooks plugin.
- Breaking change! Focus/skip options are now applied in two passes, once for
options in
tests.edn, once for command-line/REPL options. The result is that command line options can only narrow the set of tests to be run. (thanks @otwieracz)
- Added
#meta-mergereader literal fortests.edn. (thanks @RickyMoynihan)
- Fix an issue with the Kaocha keyword hierarchy where two keys had an ancestor
via two different paths, causing problems when trying to
underive. - Make the version-filter plugin work with non-numeric version segments, as in
"1.8.0_212-20190523183340.buildslave.jdk8u"
- Fix regression, only show the
--focuswarning when applicable
- Added the
--profilecommand line flag, which gets passed to Aero's#profile {}tagged literal reader. Defaults to:ciwhenCI=true. - Output a warning when
--focus TESTABLE-IDdoes not match any tests.
- The
kaocha.report.progress/progressprogress bar reporter now allows the appropriate exception to be reported when there is a syntax error in Clojure source code. Was formerly throwing NullPointerException.
- Consolidate
kaocha.hierarchy, so it can be used for kaocha-cljs
- Be smarter about loading namespaces and resolving vars for dynamic bindings
set in
tests.edn
- Type hints to eliminate reflection warnings.
- Fix
--watchwhen invoked without a[:kaocha/cli-options :config-file], either becausetests.edndoesn't exist, or the config originated elsewhere. - Make
kaocha.repl/configset[:kaocha/cli-options :config-file]if applicable. - Handle exceptions in
--watchhigher up, to prevent certain errors from being silently ignored.
- When providing dynamic var bindings in
tests.edn, we now try to load the given namespaces before setting the bindings.
- Make sure "no tests found" warning only shows up when it really needs to.
- lambdaisland/deep-diff {:mvn/version "0.0-29"} -> {:mvn/version "0.0-47"}
- nubank/matcher-combinators {:mvn/version "0.8.1"} -> {:mvn/version "0.9.0"}
- Added a check to make sure org.clojure/tools.cli is up to date.
- Built in plugins in the
kaocha.plugincan now be specified as simple (rather than namespaced) keywords. - The binding plugin has been removed, instead its functionality is now built-in, which allowed us to address several issues.
- Load errors now end in an immediate failure of the test run, instead of a warning. They are reported as an error so plugins like the notifier and junit.xml can display them.
- dependency upgrades, this fixes an upstream issue with clj-diff
- lambdaisland/deep-diff {:mvn/version "0.0-25"} -> {:mvn/version "0.0-29"}
- orchestra {:mvn/version "2018.12.06-2"} -> {:mvn/version "2019.02.06-1"}
- Added a
pre-reporthook. This allows plugins to inspect and change test events just before they are passed to the reporter. - Added a
:kaocha.plugin/notifierplugin that pops up desktop notifications when a test run passes or fails. - Add the
wrap-runhook to the hooks plugin. - Watch mode: re-run all tests by pressing "enter"
- Watch mode: watch
tests.ednfor changes - Watch mode: ignore certain files with
:kaocha.watch/ignore [".*" ,,,] - To disable output capturing you can now use
:capture-output? false, instead of:kaocha.plugin.capture-output/capture-output? false. Since this is a built-in plugin that's enabled by default it makes sense to provide a shorthand for this. - Added a
:kaocha.plugin/bindingsplugin that allows setting dynamic var bindings fromtests.edn
- Preserve changes to the config made in a
pre-loadhook - Watch mode: if a namespace fails to load then report it clearly and fail the test run, skipping any remaining tests.
- Ignore
--focus-metawhen none of the tests have this particular metadata. - Print a nicer message when a plugin can't be loaded (Daniel Compton)
- Only print random seed when test run failsg
- Added a "version-filter" plugin, which will skip tests if their metadata indicates they are not compatible with the Clojure or Java version being used.
- Mark tests with
^:kaocha/pendingmetadata to skip over them, and report them as "pending" - Added a "hooks" plugin (
:kaocha.plugin/hooks), that allows hooking into various parts of Kaocha's process using simple functions. This provides a more lightweight alternative to end users for full fledged plugins. - The
pre-testhook now runs earlier, so that:kaocha.testable/skipor:kaocha.testable/pendingcan be set from the hook and still be recognized.
- [internal] Extracted
kaocha.runner/run, to be used by alternative command line runners like boot.
- Added a TAP reporter (
kaocha.report/tap) - Added a new
--print-envflag to the:kaocha.plugin.alpha/infoplugin, which outputs the Clojure and Java version before each run.
- Filter out
jdk.internal.reflectstack frames when detecting source file (Java 9+)
- Prefer a stackframe-based file/line detection over taking the file/line of the test definition, this way the reported location is that of the assertion, rather than that of the test.
- The
print-invocationsplugin no longer prints out the--config-fileflag when it hasn't changed from its default value (tests.edn)
- Removed debug prn calls
- [internal] Test types can signal that any remaining sibling tests should be skipped. This is used by the ClojureScript test type: if a test times out then we can no longer rely on the JavaScript environment being responsive. Instead fail the test (signal a timeout) and skip any remaining tests in the same suite.
- A new work-in-progress information plugin,
:kaocha.plugin.alpha/info, currently only prints the list of all test ids.
- When specifying test suites defaults are always provided, so it's no longer
necessary to provide
:src-paths,:test-paths, etc. if they don't deviate from the defaults. This also means all test suites get the default:kaocha.filter/skip-meta [:kaocha/skip].
- In watch mode: Scan test-dirs/source-dirs, rather than letting tools.namespace derive the list of paths from the classpath
- Honor the
capture-output?flag when provided intests.edn - Print captured output when a test fails because it doesn't contain assertions
- Make file/line in failures more accurate
- Consider
(is (= ))assertions with only a single argument as failures, as these are most likely typos, they always evaluate to true.
- Kaocha now also considers the global hierarchy when determining event types. This makes it possible for third-party clojure.test reporters to be Kaocha-aware without having to depend on Kaocha.
- lambdaisland/kaocha-cloverage is now its own project, make sure to include it in your deps/project files.
- Speeded up startup by avoiding loading kaocha.watch, core.async, fipp, puget #14
- Cucumber support, see lambdaisland/kaocha-cucumber
- Function specs are now checked with orchestra + expound
- Plugins in the
kaocha.pluginnamespace can now be specified on the command line with their short name kaocha.assertionsnamespace with customclojure.testassertions. Currently for internal use but might evolve into its own library.- Added Cloverage integration, currently still included in the main
lambdaisland/kaochaartifact. - Added support for "pending" tests (skipped but still reported). Currently only used by Cucumber.
- This release contains several internal changes to support disparate test suite types.
- The test summary now reads "x tests, y assertions, z failures", rather than "test vars", to be more test type agnostic.
- De-dupe plugins, for cases where a plugin is added to
tests.ednand on the CLI
- Fix matcher-combinator support
- Skip tests marked as
^:kaocha/skipby default - Junit.xml output, see lambdaisland/kaocha-junit-xml
- Fix Java 11 compatiblity
- deep-diff functionality has been extracted into its own library
- Looking up of print-handler fails for nil
- Fix regression in
kaocha.report.progress
kaocha.plugin.alpha/xfail, mark failing tests with^:kaocha/xfailto make them pass, and vice versa. (#2)(is (= ,,,))assertions are now deep diffed and pretty printed.- Plugins now take a "docstring" (added under the
:descriptionkey). (For future use.)
--fail-fastmode is incompatible with the check which fails tests when they don't contain any assertions. (#10)kaocha.repldoes not correctly merge in extra config keys- Reported line number in case of failures points to the failed assertion, not to the failed var.
- Passing extra config to
kaocha.repl/runwill still by default run the current*ns*, rather than all tests. - Honor
*print-length*when set. (defaults to 100) - Make sure
kaocha.testable/*current-testable*is bound when plugin'swrap-runresult executes.
- Config merging (defaults + tests.edn + repl values) now uses
meta-mergefor flexible append/prepend/replace. - Print the testable-id in failure messages rather than just the var name, so it can be passed straight on to
--focus
- Fixed
kaocha.repl/run-all
-
BREAKING:
kaocha.repl/run-testsandkaocha.repl/run-all-testshave been renamed torunandrun-all, so a(use 'kaocha.repl)doesn't clash withclojure.test. -
Skip reloading namespaces during load if they are already defined. In watch mode they still get reloaded through tools.namespace when necessary. This change is done to make REPL usage more intuitive. When running
kaocha.repl/run-testsit will refrain from doing a(require ... :reload), instead accepting whatever state your REPL process is in.
- Documentation updates
- Make Kaocha's reporters more easily extendable through keyword hierarchies
- Load errors in
--watchmode no longer cause the process to exit. Instead you get a warning and the loading is retried on next change.
- Made
kaocha.repla lot more useful, making it easy to do a full or partial test run from a REPL or buffer.
- Reporters in the
kaocha.reportnamespace now can be specified on the command line with just their short name, e.g.--reporter dots
- Improved matcher-combinators support, now failure summary is only shown at the end (for dots or docs reporter), and output is correctly captured and displayed.
- Compatibility with newer matcher-combinators.
- BREAKING: Instead of
#kaochause#kaocha/v1as a reader literal that normalizes configuration. The old version is still supported for now but generates a warning.
- Use tools.namespace.track for tracking/reloading namespaces in watch mode. This should make this a lot more reliable.
- Change the
add-classpathclassloader hack so it doesn't mess up the thread binding stack. - Make dots reporter compatible with newer versions of matcher-combinators.
- Due a limitation of Aero
:ns-patternsmust be strings, and not regex literals. Clarified this in the docs.
- filter keys (skip/skip-meta/focus/focus-meta) can now be used without namespace when using the
#kaochareader literal for configuration. Before::kaocha.filter/focus, after::focus. - Added a
:kaocha.hooks/pre-loadhook to complement:kaocha.hooks/post-load.
- BREAKING:
:kaocha.type/suiteis now called:kaocha.type/clojure.test
--versioncommand line flag (only works when running from a JAR)--helpas alternative to--test-help, for environments where--helpisn't shadowed
- Make code base analysable by cljdoc.xyz
- Make sure clojars.org links to correct
kaocha.repl/run-tests/kaocha.repl/run-all-tests(since renamed torunandrun-all)
- Dynamically adding test directories to the classpath should be more robust now.
:kaocha.suite/ns-patterns,:kaocha.suite/source-pathsand:kaocha.suite/test-pathshave been renamed to just use the:kaochanamespace.
- Capture output, this is enabled by default. Only output of failing tests is
printed. This also introduced a new plugin hook,
wrap-run, which allows you to decoraterun-testablesfor doing things like adding bindings. - Reporters now always get the current testable in their clojure.test reporting event.
- Added a progress bar reporter
- Documentation reporter: show in the documentation output which tests fail.
- The randomize plugin could cause an exception because the sort-by-random-number-generator wasn't stable (it violated the contract of a comparator). Instead assign each testable a sort key first, then sort by those. This does mean seeds from before this change will no longer produce the same result.
- When specifying an invalid reporter var, error before trying to load tests.
- Correctly count matcher-combinator mismatch failures when exiting early (Ctrl-C).
- When running in watch mode, first re-run failed tests. Only when they pass do a full re-run.
- When
fail-fastis true, quit immediately when a load error is detected, instead of only failing when the namespace runs. - Suite names can now be specified on the CLI with keyword syntax, i.e.
bin/kaocha :unit
- The profiling plugin can now be configured on the command line and from
tests.edn, with
--[no-]profiling,--profiling-count,:kaocha.plugin.profiling/profiling?,:kaocha.plugin.profiling/count
--focusand--focus-metaoverride config-level:focus/:focus-meta, rather than append. This is more intuitive, when focusing from the command line you don't want extra tests to show up.- Don't run the
post-summaryhook when using the API, this prevents noise from plugins in the--print-test-plan/--print-resultoutput.
- Don't count filtered tests in profiling results.
- Testable now has an optional
:kaocha.var/wrapkey, which can contain a seq of functions that will be used to "wrap" the actual test function, useful e.g. for providing bindings. clojure.test style :each fixtures are also handled through this key, so plugins can choose to add wrapping functions at the start or the end of this seq to wrap "inside" or "around" the fixtures.
- Make the test-plan available in
pre-testandpost-testplugin hooks, so that they have access to top level configuration items.
- Added the
kaocha.plugin/defpluginmacro, making plugins look more like a deftype.
- The configuration format has changed, you should now start with the
#kaocha {}tagged reader literal intests.ednto provide defaults. If you want more control then overwritetests.ednwith the output of--print-configand tweak.