Excessive memory usage with highly parameterized test

[ben-manes]

Steps to reproduce

import java.util.stream.IntStream;

import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.MethodSource;

public final class AllocationTest {

  @ParameterizedTest
  @MethodSource("providesInts")
  public void allocate(int i) {
    if (i < 0) {
      throw new AssertionError();
    }
  }

  public static IntStream providesInts() {
    return IntStream.range(0, 2_000_000);
  }
}

When running on Gradle with a 256mb heap, this test performs ~200k executions before an OutOfMemoryError. The memory grab appears to be heavy-weight test results before retained in-memory. This was run against v5.7.0.

1. Download junit-stress.zip project.
2. Run gradlew cleanTest test until it slows down
3. Create a heap dump, e.g.
   jmap -dump:live,format=b,file=/Users/ben/Downloads/junit.hprof "$(pgrep -f GradleWorkerMain)"
4. Open in your favorite profiler, e.g. JMC, YourKit, JProfiler (all free for OSS projects)

Context

I discovered this out of curiosity when facing a similar problem in TestNG (testng-team/testng#2096). It too retains the test results in-memory, though its stress tester runs faster and gets to ~600k before failing. In both cases the results shouldn't be held in-memory, e.g. perhaps streaming the results to disk if retained for a report.

The real-world case is Caffeine, which has a test suite that executes 4M+ scenarios. This is because a cache has many configuration options that interact, so a simple change could break a subsets of configurations for a given test case. Therefore the easiest solution is brute force testing with a custom @CacheSpec applied to the method to declare the specification constraints. The provider inspects the test method to generates all possible combinations.

Because of the memory grab, Caffeine's test suite is broken into ~30 tasks each running ~140k test executions. Over time I've whittled down the source problems by hacking the testing framework's internal structures to drop allocations. Prior tricks were modifying the test results in a listener, e.g. to replace parameters with their stringified version and dedupe test names. As more tests are added it keeps becoming a problem, requiring more effort to keep the build healthy. Recently I hacked the internals to drop the test result objects themselves (as not needed for Gradle's report). This shrank the heap from 315MB (8.6M objects) down to 22MB (0.5M objects), but is extremely hacky by reflectively clearing various collections. It functions enough for my build to shave the CI time by 20 minutes (22% speedup) which only offers a 512mb heap.

In both frameworks resolving this stress case might be an invasive change. While most test suites are small, better GC behavior can improve performance and lets the framework scale to more extreme scenarios.

[marcphilipp]

Thanks for the detailed report! This is effectively a duplicate of #1445. Let's continue the discussion there!

[marcphilipp]

Duplicate of #1445