Parameterization for spec-style tests?

[greghaskins]

#58 focuses specifically on parameterization for Gherkin-style tests (scenarioOutline). It would be nice to provide a cleaner parameterized test API for Spec-style tests too.

There is no obvious consensus among spec-style test runners as to the "canonical" way to do this. RSpec, Jasmine, and Mocha, for example, do not have any special support for parameterization out of the box. There are a few data points from the community, though:

• rspec-parameterized exists, and seems to be reasonably popular (129 stars)
• There was a long discussion about adding parameterized tests to Mocha. They decided not to add it to their core, but lots of ideas were thrown around.
• Ginkgo does have built-in table driven tests support.

We can use this thread to discuss the best approach for Spectrum, either taking ideas from above or going in a different direction.

[ashleyfrieze]

rspec-parameterized has a describe, where, with_them syntax.
ginkgo has a describeTable and entry syntax.

We have prototyped describeParameterized, withExamples

[greghaskins]

For what it’s worth, the Ginkgo DescribeTable feature doesn’t appear that widely used on GitHub, at least based on this search. 822 compared to about 57k usages of Ginkgo overall (~1.5%).

And the rspec-parameterized constructs seem to be used even less relative to the number of public codebases using rspec. 119 of about 816k (below 0.015%), 100x fewer than Ginkgo’s DescribeTable).

There is a ton of churn on this topic in the Jasmine and Mocha communities. It seems to be a desired feature, but adding it has been resisted so there hasn't been a clear "winning" implementation.

These data points lead me to conclude there isn't a particularly strong prior-art way to do this for the Specification DSL. That should give us some license to steal the best ideas from all of the above, and combine that with our own flavor. I personally prefer some variant of the where constructs (as opposed to using the table metaphor). Still trying to figure out a nice syntax, though.

[ashleyfrieze]

How about:

describe("some fixture",
    where(
        example(1,2,3),
        example(2,3,5),
        example(3,4,7)
    ), (a,b,c) -> {
        it("adds up the numbers correctly", () -> {
            assertThat(a+b, is(c));
        });
});

Perhaps we add a synonym of example as testCase... though perhaps example is better.

Or perhaps we replace where with withExamples as we already do for scenarioOutline? In fact, perhaps we keep things very simple and just do what scenarioOutline does, but using describe:

describe("some fixture", (a,b,c) -> {
        it("adds up the numbers correctly", () -> {
            assertThat(a+b, is(c));
        });
    }, withExamples(
        example(1,2,3),
        example(2,3,5),
        example(3,4,7))
);

If the second of these, then the implementation is ~0 effort ;)

[basdijkstra]

Just throwing in my $0.02:

The first example is quite similar to the closest I've seen in data driven testing support in Jasmine, which I found (and stole from) here:

http://blog.jphpsf.com/2012/08/30/drying-up-your-javascript-jasmine-tests

I like the second one a little better, though, if only because it's much more similar to how data driven testing is supported in the Cucumber style specs.

Not sure if my vote counts for anything, but hey :)