Table-driven testing in JUnit with TableTest

[nchaugen]

First off all, thank you @junit-team for your great work in developing and maintaining JUnit.

I am a long-time fan of using tables to drive tests, ever since being introduced to Ward Cunningham’s Fit Framework 20 years ago.

Having used Spock Framework for table-driven testing in recent projects, I found that not all Java and Kotlin developers got comfortable with the Groovy-ness of it.

Based on this experience, plus some ideas for how to further improve support for table-driven testing, I set out to create an alternative based on JUnit.

The result is TableTest. I also wrote a couple of short blog posts that introduce table-driven testing and TableTest, and I created a plugin for IntelliJ IDEA to help with auto formatting, colouring and syntax.

I have received good feedback from people who have looked into it and started using it in their projects.

One question I get is if it would be possible to make TableTest built-in to JUnit. Personally I would love that as it would make table-driven testing available to a lot of programmers.

So I wanted to ask your thoughts on this. If you think it is worth looking into, I would be happy to discuss how we could approach it, and also to do the necessary adaptations if we decide to move forward.

Thanks!

[marcphilipp]

First of all, thanks for reaching out! I think TableTest looks very useful. There is quite a bit of overlap with regular @ParameterizedTests with @CsvSource, isn't there? It looks like you were able to reuse quite a bit of the framework, including JUnit's converter support (at least as a fallback). We're currently thinking about making that extensible (see ). If you have a chance, it would be great to get your feedback on that work so it will ideally be useful to you as well.

One question I get is if it would be possible to make TableTest built-in to JUnit. Personally I would love that as it would make table-driven testing available to a lot of programmers.

So I wanted to ask your thoughts on this. If you think it is worth looking into, I would be happy to discuss how we could approach it, and also to do the necessary adaptations if we decide to move forward.

We can certainly discuss it! Please note that we're currently very busy with the upcoming 6.0 release so it will take even longer than usual. I'll add it to our discussion backlog.

[nchaugen]

Thank you for the swift response!

You are right that @TableTest is overlapping with @ParamerizedTest with @CsvSource. That is what I would be using previously for table-driven tests in JUnit.

I would say the main improvements @TableTest brings are readability, support for collection types, and the ability use factory methods for custom, automatic conversion to method parameter types.

I also find that the less verbose annotation setup and the scenario column adds to the developer comfort.

A more extensible converter service sounds helpful to this kind of string-based value providers for parameterized tests. I will have a look. Is this work happening in PR #4219?

Happy to contribute in a discussion when you have the time.

Looking forward to the 6.0 release!

[marcphilipp]

Is this work happening in PR #4219?

Yes, that's the one. 👍

[scordio]

support for collection types

Hi @nchaugen, just FYI, on top of the work at #4219, I'm also creating a junit-converters library which will eventually leverage the new SPI in the PR. I planned to have a converter for collection-like strings, and I see that your library already has work in this direction.

I'd be happy to get your feedback there, too, in case you find it useful 🙂

[nchaugen]

Sorry this took some time, but I have finally had a chance to look into the new ConversionSupport to see how it would work with TableTest.

This post turned rather long, so TLDR:

• In general I think ConversionSupport is a nice improvement to allow extensions to the built-in converters. It does not currently support all the conversion scenarios of TableTest, but that might not be in scope for this change.

• I have a couple of suggestions for improvements:

  1. Supply the value to be converted to canConvert
  2. Provide an abstract TypeConverter with standard default canConvert type check

Long version

I like that it is possible to override the provided built-in converters. In many situations I like to use constants like TODAY, TOMORROW, YESTERDAY as argument values to represent LocalDates to improve readability. I found that when adding the following custom converter it took precedence over the built-in StringToJavaTimeConverter:

public class StringToLocalDateConverter implements Converter<String, LocalDate> {

    @Override
    public boolean canConvert(ConversionContext context) {
        return LocalDate.class.isAssignableFrom(context.targetType().getType()) ;
    }

    @Override
    public LocalDate convert(@Nullable String source, ConversionContext context) throws ConversionException {
        return switch (source) {
            case null -> null;
            case "TODAY" -> LocalDate.now();
            case "TOMORROW" -> LocalDate.now().plusDays(1);
            case "YESTERDAY" -> LocalDate.now().minusDays(1);
            default -> LocalDate.parse(source);
            };
    }
}

I also like that it is possible to convert from other types than String. This allows the ArgumentsProvider to serve other types to the converters. For example for TableTest there is a parser that parses the table syntax and captures collection values as collection types before conversion starts. That means there are not only strings to convert, but also List<String>, Map<String, String>, etc.

However, for complex values there could also be source types like List<Map<String, List<String>>>. In these situations TableTest does a couple of things that makes it easier for providers of custom converters.

Firstly, TableTest will recurse through the source value, and for each level compare the source type at that level with the target type of the parameter.

For example, if the parsed source value is Map<String, List<String>> and the target parameter type is Map<String, List<Integer>>, TableTest will descend the parsed value and convert it level by level:

1. Source Map<String, List> and target Map<String, List> => no conversion necessary
2. Source List<String> and target List<Integer> => find and use converter to convert list elements from String to Integer

Now String to Integer is a built-in conversion, so for this scenario the developer does not have to provide any custom converters. But if the target included a list of some custom type, the developer only have to provide a converter for doing String to Custom, not Map<String, List<String>> to Map<String, List<Custom>>.

Secondly, when invoking custom converters (factory methods in TableTest's case), TableTest will do the same for the converter parameter, allowing multiple converters to work together to do the total conversion.

If the parsed source value is Map<String, List<String>> and the target parameter value is Map<String, Grades>>. When comparing the source List<String> with target Grades, TableTest will look for a factory method returning Grades. Say it finds one that takes List<Integer> as input, TableTest will then convert List<String> to List<Integer> before invoking the factory method to convert List<Integer> to Grades.

This means you could have the same Map<String, List<String>> value converted to StudentGrades by providing the following factory methods:

public static StudentGrades parseStudentGrades(
	Map<String, Grades> input
) {
	// ...
}

public static Grades parseGrades(List<Integer> input) {
	// ...
}

I think this strategy for converting complex types reduces the work for developers providing custom converters and also makes the converters reusable in more situations. However I don't know if this kind of functionality is within the scope of ConversionSupport.

Finally there are two (smaller) things that I think the framework could do to help developers implement custom converters:

1. Providing the actual value that needs conversion to Converter.canConvert. This would allow the converter to inspect the value before deciding if it supports it.
2. Supplying an abstract Converter implementation that provides a generic canConvert method checking if the targetType of the context is assignable to the targetType of the converter:

public abstract class TypeConverter<S, T> implements Converter<S, T> {
    private final Class<T> targetClass;

    protected TypeConverter(Class<T> targetClass) {
        this.targetClass = targetClass;
    }

    @Override
    public boolean canConvert(ConversionContext context) {
        return targetClass.isAssignableFrom(context.targetType().getType());
    }
}

Alternatively there could be a helper method on ConversionContext to ease implementation of canConvert:

public boolean isTargetTypeAssignableTo(Class<?> converterTarget) {
  return converterTarget != null && converterTarget.isAssignableFrom(this.targetType().getType());
}

This concludes my thoughts so far. ☺️

[nchaugen]

I strongly agree! I left that out for brevity as the post was already getting long. ☺️

Thanks for pointing it out!

[jbduncan]

Phew, I'm glad to hear that! Sorry if I came across as a little brusque. 😅

[scordio]

Many thanks for your feedback, @nchaugen!

I'll keep all your comments in mind, especially after we reach an agreement on the redesign we're currently discussing, and give you more precise feedback once I move forward with it.

About:

Providing the actual value that needs conversion to Converter.canConvert. This would allow the converter to inspect the value before deciding if it supports it.

That's currently not working well with generics, see the exchange at #4219 (comment) (nevertheless, it's still an open point).

Supplying an abstract Converter implementation that provides a generic canConvert method checking if the targetType of the context is assignable to the targetType of the converter:

Would SimpleConverter cover the use case you have in mind?

[nchaugen]

Yes, sorry, I missed the SimpleConverter. That should cover it!

I think it would be better to use isAssignableFrom rather than equality to check for type compatibility in canConvert, but I guess that is what the FIXME is pointing too?

[scordio]

Yes, exactly.

[nchaugen]

One additional comment: I think there is a use case for converters that are local to the test class, in addition to the global ServiceLoader converters. The local ones should take precedence over the global ones.

This would allow test developers to add converters for a particular test case, without having to worry about affecting other tests.

[nchaugen]

Yes, right, I forgot about that feature. Good thing the explicit converter takes precedence.

I'm not to fond of the required @ConvertWith annotation though. I find it too verbose. Would have been nice with a simpler way to define local implicit converters, for example static converter methods.

[scordio]

This might be beyond the scope of my changes, but I'll keep that in mind.

[marcphilipp]

That could be already addressed by implementing a "MethodConverter" that uses a custom @ConvertWithMethod("convertMethod") annotation.

[scordio]

That could be already addressed by implementing a "MethodConverter" that uses a custom @ConvertWithMethod("convertMethod") annotation.

... taking inspiration for junit-converters 🙃

[nchaugen]

I have put up some examples on how I use local static converter methods with TableTest to make tests more readable. There is also a blog post if you would like some supporting text. ☺️

[sormuras]

Copied from TableTest.java#L65-L69:

@Target({ElementType.ANNOTATION_TYPE, ElementType.METHOD})
@Retention(RetentionPolicy.RUNTIME)
@ParameterizedTest
@ArgumentsSource(TableArgumentsProvider.class)
public @interface TableTest {
...

This implementation choice makes usages of TableTest incompatible with @ParamterizedClasses, right?

• https://docs.junit.org/current/user-guide/#writing-tests-parameterized-tests

An @TableSource would be neat. 🤓

@ParameterizedClass
@TableSource("""
    Fruit  | Quantity
    apple  | 23
    banana | 42
    """)
record FruitTests(String fruit, int quantity) {

    @Test
    void test() {
        assertFruit(fruit);
        assertQuantity(quantity);
    }

    @Test
    void anotherTest() {
        // ...
    }
}

[lobaorn-bitso]

I agree it would be neat, but that particular example would be achieved with:

@ParameterizedTest
@CsvSource(
    textBlock = """
        Fruit  | Quantity
        apple  | 23
        banana | 42
        """,
    delimiterString = "|",
    useHeadersInDisplayName = true
)

right? But I like the idea also, keep it simpler and then enhance with more behaviours :)