Retry plugin fails with kotest framework

[NikolayMetchev]

Using the kotest framework (https://kotest.io/) breaks the retry plugin;
Here is the error message:

Execution failed for task ':test'.
> org.gradle.test-retry was unable to retry the following test methods, which is unexpected. Please file a bug report at https://github.com/gradle/test-retry-gradle-plugin/issues
     com.test.KoTest#Test 1

Here is a full project that recreates the failure:
kotest.gradle.tar.gz

[ldaley]

@marcphilipp something seems to go wrong in the filtering/selection between Gradle and platform. Could you take a look at kotest when you get a chance please and see if it is implementing this part of platform as we expect.

[NikolayMetchev]

Any news here?

[marcphilipp]

I'm afraid I haven't had time to investigate, yet. It's still on my list, though.

[NikolayMetchev]

Is it still on the list?

[marcphilipp]

Sorry for the delay. I took a look and the problem seems to be that kotest does not use MethodSource which Gradle filters for so even ./gradlew test --tests "KoTest.Test 1" doesn't work. As a stop gap, I think the test-retry plugin could be changed to detect kotest test classes and rerun the entire test class if one of its tests fails.

@NikolayMetchev Would you be interested in contributing a PR for that?

[NikolayMetchev]

I have asked the kotest channel for help: https://kotlinlang.slack.com/archives/CT0G9SD7Z/p1634897392015500

[NikolayMetchev]

Also there is a kotest plugin for Idea. That can run individual tests but it doesn't go through gradle it runs the following. Not sure if that helps.

/Users/nmetchev/.asdf/installs/java/temurin-11.0.12+7/bin/java -Didea.test.cyclic.buffer.size=1048576 -javaagent:/Users/nmetchev/Library/Application Support/JetBrains/Toolbox/apps/IDEA-U/ch-1/212.5457.46/IntelliJ IDEA 2021.2 EAP.app/Contents/lib/idea_rt.jar=52321:/Users/nmetchev/Library/Application Support/JetBrains/Toolbox/apps/IDEA-U/ch-1/212.5457.46/IntelliJ IDEA 2021.2 EAP.app/Contents/bin -Dfile.encoding=UTF-8 -classpath /Users/nmetchev/Library/Application Support/JetBrains/Toolbox/apps/IDEA-U/ch-1/212.5457.46/IntelliJ IDEA 2021.2 EAP.app/Contents/lib/idea_rt.jar:/Users/nmetchev/Downloads/kotest-gradle/build/classes/kotlin/test:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.kotest/kotest-runner-junit5-jvm/4.6.3/8e48bd1180d7bc8c483b61f9531adacfa9dcf919/kotest-runner-junit5-jvm-4.6.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.kotest/kotest-assertions-core-jvm/4.6.3/685fd4aa66941854e60069429c26245b80e05b87/kotest-assertions-core-jvm-4.6.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.kotest/kotest-property-jvm/4.6.3/12a27152669524c9bcd6bcccd82f33f3bdf16b55/kotest-property-jvm-4.6.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-stdlib-jdk8/1.5.31/ff5d99aecd328872494e8921b72bf6e3af97af3e/kotlin-stdlib-jdk8-1.5.31.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlinx/kotlinx-coroutines-core-jvm/1.5.0/d8cebccdcddd029022aa8646a5a953ff88b13ac8/kotlinx-coroutines-core-jvm-1.5.0.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.junit.platform/junit-platform-suite-api/1.6.2/36796f8e82db47f0a09580be1fb68449b3b8ee6f/junit-platform-suite-api-1.6.2.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.junit.jupiter/junit-jupiter-api/5.6.2/c9ba885abfe975cda123bf6f8f0a69a1b46956d0/junit-jupiter-api-5.6.2.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.junit.platform/junit-platform-launcher/1.6.2/d866de2950859ca1c7996351d7b3d97428083cd0/junit-platform-launcher-1.6.2.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.junit.platform/junit-platform-engine/1.6.2/1752cad2579e20c2b224602fe846fc660fb35805/junit-platform-engine-1.6.2.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-stdlib-jdk7/1.5.31/77e0f2568912e45d26c31fd417a332458508acdf/kotlin-stdlib-jdk7-1.5.31.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-stdlib/1.5.31/6628d61d0f5603568e72d2d5915d2c034b4f1c55/kotlin-stdlib-1.5.31.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-stdlib-common/1.5.31/43331609c7de811fed085e0dfd150874b157c32/kotlin-stdlib-common-1.5.31.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.kotest/kotest-framework-engine-jvm/4.6.3/6d6d4b3a3b41125e852e9b50544a6009c9e3ec69/kotest-framework-engine-jvm-4.6.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.kotest/kotest-framework-api-jvm/4.6.3/1f8dd5cf0ea1525da216c7daa507c3c2a7c3651e/kotest-framework-api-jvm-4.6.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.kotest/kotest-framework-discovery-jvm/4.6.3/e9e797db0a56fdf317afc906ef731823e8dbb0a2/kotest-framework-discovery-jvm-4.6.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.kotest/kotest-common-jvm/4.6.3/180634419642124484a14583fec51bfdbbb44330/kotest-common-jvm-4.6.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.kotest/kotest-extensions-jvm/4.6.3/f813249aa945ae1dd17c053c0a6d24f72409386f/kotest-extensions-jvm-4.6.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.kotest/kotest-framework-concurrency-jvm/4.6.3/3528b540b65aa1dea63f22b23499ebc2dc6cb77d/kotest-framework-concurrency-jvm-4.6.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.apiguardian/apiguardian-api/1.1.0/fc9dff4bb36d627bdc553de77e1f17efd790876c/apiguardian-api-1.1.0.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.junit.platform/junit-platform-commons/1.6.2/7644a14b329e76b5fe487628b50fb5eab6ba7d26/junit-platform-commons-1.6.2.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.opentest4j/opentest4j/1.2.0/28c11eb91f9b6d8e200631d46e20a7f407f2a046/opentest4j-1.2.0.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.kotest/kotest-assertions-shared-jvm/4.6.3/7e075a5a0dec96d624a165c9ac14af7cccfe455d/kotest-assertions-shared-jvm-4.6.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.jetbrains/annotations/13.0/919f0dfe192fb4e063e7dacadee7f8bb9a2672a9/annotations-13.0.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/com.github.ajalt/mordant/1.2.1/6cbab1a74ab6dafbf81b7706733d4c2fbaff2e0b/mordant-1.2.1.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-script-runtime/1.5.0/a2d66828077376fc1851a6fdbf25865935f4101d/kotlin-script-runtime-1.5.0.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.github.classgraph/classgraph/4.8.105/74132d64981decf7c76a506b023e753c335e1983/classgraph-4.8.105.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/com.github.ajalt/colormath/1.2.0/c62f49b31f34588dbbfb477c08fd56bc3026d202/colormath-1.2.0.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.kotest/kotest-assertions-api-jvm/4.6.3/2c4f417ffd5fd8f862ab5250bb68b73f5479f31/kotest-assertions-api-jvm-4.6.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-reflect/1.5.0/47806fe8ed30dbdf9e697eda5e9c9a3905ff3363/kotlin-reflect-1.5.0.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlinx/kotlinx-coroutines-jdk8/1.5.0/ec8255f97a375b6014cad1169c057f98f0ebcf28/kotlinx-coroutines-jdk8-1.5.0.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.github.java-diff-utils/java-diff-utils/4.9/3ec791c5aa74a72fb499ae8d9547abe27b637b0f/java-diff-utils-4.9.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/com.github.mifmif/generex/1.0.2/b378f873b4e8d7616c3d920e2132cb1c87679600/generex-1.0.2.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/dk.brics.automaton/automaton/1.11-8/6ebfa65eb431ff4b715a23be7a750cbc4cc96d0f/automaton-1.11-8.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.mockk/mockk/1.9.3/d26e5a4a7fd4ecb5635b28a0b70960ab0b20556d/mockk-1.9.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/commons-io/commons-io/2.6/815893df5f31da2ece4040fe0a12fd44b577afaf/commons-io-2.6.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-script-util/1.5.0/34f5af4377938350a351eca6b7f9e514388e6aa5/kotlin-script-util-1.5.0.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-scripting-jvm/1.5.0/6f2bbad9181cff63b847ac0526cf3d11e5f02f93/kotlin-scripting-jvm-1.5.0.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.mockk/mockk-dsl-jvm/1.9.3/63de994ae1f30b06395325258d2e5683a8bdd525/mockk-dsl-jvm-1.9.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.mockk/mockk-common/1.9.3/7f2d6bc3390a76edeba34947e0de6d2314645f25/mockk-common-1.9.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.mockk/mockk-agent-jvm/1.9.3/d6f7b4d893caf1cda2bb28ee637b247ced5d5eab/mockk-agent-jvm-1.9.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-daemon-client/1.5.0/373d55b6113e6be520d84bfb738bd40520004bfa/kotlin-daemon-client-1.5.0.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.jetbrains.intellij.deps/trove4j/1.0.20181211/216c2e14b070f334479d800987affe4054cd563f/trove4j-1.0.20181211.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-scripting-common/1.5.0/5b78b9cc03552532e8e689a002f1149ab43e44f0/kotlin-scripting-common-1.5.0.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.mockk/mockk-dsl/1.9.3/4f158b4ed161bc4373356733088e09491ecd13da/mockk-dsl-1.9.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.mockk/mockk-agent-common/1.9.3/50f32c350cb9018a62e6de2bed710acf675f887b/mockk-agent-common-1.9.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.mockk/mockk-agent-api/1.9.3/467c1fd4867fc49b98a7de421a717b619767abfe/mockk-agent-api-1.9.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.objenesis/objenesis/3.0.1/11cfac598df9dc48bb9ed9357ed04212694b7808/objenesis-3.0.1.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/net.bytebuddy/byte-buddy/1.9.10/211a2b4d3df1eeef2a6cacf78d74a1f725e7a840/byte-buddy-1.9.10.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/net.bytebuddy/byte-buddy-agent/1.9.10/9674aba5ee793e54b864952b001166848da0f26b/byte-buddy-agent-1.9.10.jar io.kotest.engine.launcher.MainKt --spec com.test.KoTest --testpath Test 1 --reporter teamcity

[christophsturm]

I'm the author of a different kotlin test runner,(https://github.com/failgood/failgood) that also uses a dsl instead of methods to declare tests, and I think autodetecting test runners and then changing the behavior is not a great way to do this. @marcphilipp are you suggesting to create methodSources for tests even if they are not declared in a method? would that work?

[christophsturm]

Also there is a kotest plugin for Idea. That can run individual tests but it doesn't go through gradle it runs the following. Not sure if that helps.

/Users/nmetchev/.asdf/installs/java/temurin-11.0.12+7/bin/java -Didea.test.cyclic.buffer.size=1048576 -javaagent:/Users/nmetchev/Library/Application Support/JetBrains/Toolbox/apps/IDEA-U/ch-1/212.5457.46/IntelliJ IDEA 2021.2 EAP.app/Contents/lib/idea_rt.jar=52321:/Users/nmetchev/Library/Application Support/JetBrains/Toolbox/apps/IDEA-U/ch-1/212.5457.46/IntelliJ IDEA 2021.2 EAP.app/Contents/bin -Dfile.encoding=UTF-8 -classpath /Users/nmetchev/Library/Application Support/JetBrains/Toolbox/apps/IDEA-U/ch-1/212.5457.46/IntelliJ IDEA 2021.2 EAP.app/Contents/lib/idea_rt.jar:/Users/nmetchev/Downloads/kotest-gradle/build/classes/kotlin/test:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.kotest/kotest-runner-junit5-jvm/4.6.3/8e48bd1180d7bc8c483b61f9531adacfa9dcf919/kotest-runner-junit5-jvm-4.6.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.kotest/kotest-assertions-core-jvm/4.6.3/685fd4aa66941854e60069429c26245b80e05b87/kotest-assertions-core-jvm-4.6.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.kotest/kotest-property-jvm/4.6.3/12a27152669524c9bcd6bcccd82f33f3bdf16b55/kotest-property-jvm-4.6.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-stdlib-jdk8/1.5.31/ff5d99aecd328872494e8921b72bf6e3af97af3e/kotlin-stdlib-jdk8-1.5.31.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlinx/kotlinx-coroutines-core-jvm/1.5.0/d8cebccdcddd029022aa8646a5a953ff88b13ac8/kotlinx-coroutines-core-jvm-1.5.0.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.junit.platform/junit-platform-suite-api/1.6.2/36796f8e82db47f0a09580be1fb68449b3b8ee6f/junit-platform-suite-api-1.6.2.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.junit.jupiter/junit-jupiter-api/5.6.2/c9ba885abfe975cda123bf6f8f0a69a1b46956d0/junit-jupiter-api-5.6.2.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.junit.platform/junit-platform-launcher/1.6.2/d866de2950859ca1c7996351d7b3d97428083cd0/junit-platform-launcher-1.6.2.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.junit.platform/junit-platform-engine/1.6.2/1752cad2579e20c2b224602fe846fc660fb35805/junit-platform-engine-1.6.2.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-stdlib-jdk7/1.5.31/77e0f2568912e45d26c31fd417a332458508acdf/kotlin-stdlib-jdk7-1.5.31.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-stdlib/1.5.31/6628d61d0f5603568e72d2d5915d2c034b4f1c55/kotlin-stdlib-1.5.31.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-stdlib-common/1.5.31/43331609c7de811fed085e0dfd150874b157c32/kotlin-stdlib-common-1.5.31.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.kotest/kotest-framework-engine-jvm/4.6.3/6d6d4b3a3b41125e852e9b50544a6009c9e3ec69/kotest-framework-engine-jvm-4.6.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.kotest/kotest-framework-api-jvm/4.6.3/1f8dd5cf0ea1525da216c7daa507c3c2a7c3651e/kotest-framework-api-jvm-4.6.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.kotest/kotest-framework-discovery-jvm/4.6.3/e9e797db0a56fdf317afc906ef731823e8dbb0a2/kotest-framework-discovery-jvm-4.6.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.kotest/kotest-common-jvm/4.6.3/180634419642124484a14583fec51bfdbbb44330/kotest-common-jvm-4.6.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.kotest/kotest-extensions-jvm/4.6.3/f813249aa945ae1dd17c053c0a6d24f72409386f/kotest-extensions-jvm-4.6.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.kotest/kotest-framework-concurrency-jvm/4.6.3/3528b540b65aa1dea63f22b23499ebc2dc6cb77d/kotest-framework-concurrency-jvm-4.6.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.apiguardian/apiguardian-api/1.1.0/fc9dff4bb36d627bdc553de77e1f17efd790876c/apiguardian-api-1.1.0.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.junit.platform/junit-platform-commons/1.6.2/7644a14b329e76b5fe487628b50fb5eab6ba7d26/junit-platform-commons-1.6.2.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.opentest4j/opentest4j/1.2.0/28c11eb91f9b6d8e200631d46e20a7f407f2a046/opentest4j-1.2.0.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.kotest/kotest-assertions-shared-jvm/4.6.3/7e075a5a0dec96d624a165c9ac14af7cccfe455d/kotest-assertions-shared-jvm-4.6.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.jetbrains/annotations/13.0/919f0dfe192fb4e063e7dacadee7f8bb9a2672a9/annotations-13.0.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/com.github.ajalt/mordant/1.2.1/6cbab1a74ab6dafbf81b7706733d4c2fbaff2e0b/mordant-1.2.1.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-script-runtime/1.5.0/a2d66828077376fc1851a6fdbf25865935f4101d/kotlin-script-runtime-1.5.0.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.github.classgraph/classgraph/4.8.105/74132d64981decf7c76a506b023e753c335e1983/classgraph-4.8.105.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/com.github.ajalt/colormath/1.2.0/c62f49b31f34588dbbfb477c08fd56bc3026d202/colormath-1.2.0.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.kotest/kotest-assertions-api-jvm/4.6.3/2c4f417ffd5fd8f862ab5250bb68b73f5479f31/kotest-assertions-api-jvm-4.6.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-reflect/1.5.0/47806fe8ed30dbdf9e697eda5e9c9a3905ff3363/kotlin-reflect-1.5.0.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlinx/kotlinx-coroutines-jdk8/1.5.0/ec8255f97a375b6014cad1169c057f98f0ebcf28/kotlinx-coroutines-jdk8-1.5.0.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.github.java-diff-utils/java-diff-utils/4.9/3ec791c5aa74a72fb499ae8d9547abe27b637b0f/java-diff-utils-4.9.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/com.github.mifmif/generex/1.0.2/b378f873b4e8d7616c3d920e2132cb1c87679600/generex-1.0.2.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/dk.brics.automaton/automaton/1.11-8/6ebfa65eb431ff4b715a23be7a750cbc4cc96d0f/automaton-1.11-8.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.mockk/mockk/1.9.3/d26e5a4a7fd4ecb5635b28a0b70960ab0b20556d/mockk-1.9.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/commons-io/commons-io/2.6/815893df5f31da2ece4040fe0a12fd44b577afaf/commons-io-2.6.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-script-util/1.5.0/34f5af4377938350a351eca6b7f9e514388e6aa5/kotlin-script-util-1.5.0.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-scripting-jvm/1.5.0/6f2bbad9181cff63b847ac0526cf3d11e5f02f93/kotlin-scripting-jvm-1.5.0.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.mockk/mockk-dsl-jvm/1.9.3/63de994ae1f30b06395325258d2e5683a8bdd525/mockk-dsl-jvm-1.9.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.mockk/mockk-common/1.9.3/7f2d6bc3390a76edeba34947e0de6d2314645f25/mockk-common-1.9.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.mockk/mockk-agent-jvm/1.9.3/d6f7b4d893caf1cda2bb28ee637b247ced5d5eab/mockk-agent-jvm-1.9.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-daemon-client/1.5.0/373d55b6113e6be520d84bfb738bd40520004bfa/kotlin-daemon-client-1.5.0.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.jetbrains.intellij.deps/trove4j/1.0.20181211/216c2e14b070f334479d800987affe4054cd563f/trove4j-1.0.20181211.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-scripting-common/1.5.0/5b78b9cc03552532e8e689a002f1149ab43e44f0/kotlin-scripting-common-1.5.0.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.mockk/mockk-dsl/1.9.3/4f158b4ed161bc4373356733088e09491ecd13da/mockk-dsl-1.9.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.mockk/mockk-agent-common/1.9.3/50f32c350cb9018a62e6de2bed710acf675f887b/mockk-agent-common-1.9.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.mockk/mockk-agent-api/1.9.3/467c1fd4867fc49b98a7de421a717b619767abfe/mockk-agent-api-1.9.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.objenesis/objenesis/3.0.1/11cfac598df9dc48bb9ed9357ed04212694b7808/objenesis-3.0.1.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/net.bytebuddy/byte-buddy/1.9.10/211a2b4d3df1eeef2a6cacf78d74a1f725e7a840/byte-buddy-1.9.10.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/net.bytebuddy/byte-buddy-agent/1.9.10/9674aba5ee793e54b864952b001166848da0f26b/byte-buddy-agent-1.9.10.jar io.kotest.engine.launcher.MainKt --spec com.test.KoTest --testpath Test 1 --reporter teamcity

kotest also has a gradle plugin, so it should probably be supported there.

[marcphilipp]

are you suggesting to create methodSources for tests even if they are not declared in a method? would that work?

No, I wasn't suggesting that and it wouldn't help for kotest. The kotest TestEngine implementation currently creates all descriptors dynamically at execution instead of at discovery time and Gradle currently filters the discovery result using a PostDiscoveryFilter.

I think autodetecting test runners and then changing the behavior is not a great way to do this.

If Gradle would somehow support filtering kotest tests that would certainly be preferable. However, I don't see a clear path forward bar reimplementing the kotest TestEngine or adding a different dynamic filtering concept to the JUnit Platform and using it in Gradle. This plugin already has logic for detecting certain Spock classes (e.g. ones annotated with @Stepwise) and rerunning the entire test class. So, while not super elegant, it would provide an interim solution.

[christophsturm]

adding support for tests that are not tied to methods to junit platform would be really great, and imo much better than adding custom behavior for every test engine. isnt the whole point of junit platform not to do that?

[marcphilipp]

The JUnit Platform supports retries via UniqueIdSelector. This is ultimately an issue with Gradle which is very test class/method centric at the moment.

[NikolayMetchev]

The JUnit Platform supports retries via UniqueIdSelector. This is ultimately an issue with Gradle which is very test class/method centric at the moment.

Is it worth opening an issue with Gradle itself?

[sksamuel]

Sorry for the delay. I took a look and the problem seems to be that kotest does not use MethodSource which Gradle filters for so even ./gradlew test --tests "KoTest.Test 1" doesn't work. As a stop gap, I think the test-retry plugin could be changed to detect kotest test classes and rerun the entire test class if one of its tests fails.

@NikolayMetchev Would you be interested in contributing a PR for that?

Kotest will work for --tests if it is a classname only. So the workaround could be as simple as not including the test/method name in the string.

The reason Kotest doesn't support --tests fully is because of the reasons discussed - gradle turns it into a method selector and then doesn't propagate it fully on non method based engines.

[christophsturm]

The JUnit Platform supports retries via UniqueIdSelector. This is ultimately an issue with Gradle which is very test class/method centric at the moment.

Is it worth opening an issue with Gradle itself?

I think it would be a great idea.

[eriwen]

Related to gradle/gradle#19513