JUnit Test Categorisation With Maven

To group by test types, we annotate our test codes with JUnit's org.junit.experimental.categories.Category and customise Maven's test plugin to allow certain groups to run before others.

But just before we start, we need to provide group names to our test codes. As an example, we'll model these group types as Felicite's early and later life.

package org.lyeung.simpleheart;
public interface EarlyLife {}

package org.lyeung.simpleheart;
public interface LaterLife {}

Below is a sample test class:
public class TestFelicity {

    @Category(EarlyLife.class)
    @Test    public void isKnowsTheodore() {
        System.out.println("isKnowsTheodore");
        assertTrue(true);
    }

    @Category(EarlyLife.class)
    @Test    public void isKnowsPaul() {
        System.out.println("isKnowsPaul");
        assertTrue(true);
    }

    @Category(EarlyLife.class)
    @Test    public void isKnowsVirginie() {
        System.out.println("isKnowsVirginie");
        assertTrue(true);
    }

    @Category(LaterLife.class)
    @Test    public void isKnowsFatherColmiche() {
        System.out.println("isKnowsFatherColmiche");
        assertTrue(true);
    }

    @Category(LaterLife.class)
    @Test    public void isParrotLouLou() {
        System.out.println("isParrotLouLou");
        assertTrue(true);
    }
}

As you can see from above, we've categorised our test methods by annotating @Category with the desired group types.

To enable early life tests when Maven runs package goal, we need to update the pom file to customise Surefire plugin:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>2.18.1</version>
    <dependencies>
        <dependency>
            <groupId>org.apache.maven.surefire</groupId>
            <artifactId>surefire-junit47</artifactId>
            <version>2.18.1</version>
        </dependency>
    </dependencies>
    <configuration>
        <groups>org.lyeung.simpleheart.EarlyLife</groups>
    </configuration>

</plugin>

To enable later life tests when Maven runs integration test or after, we need another update to Failsafe plugin:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-failsafe-plugin</artifactId>
    <version>2.18.1</version>
    <dependencies>
        <dependency>
            <groupId>org.apache.maven.surefire</groupId>
            <artifactId>surefire-junit47</artifactId>
            <version>2.18.1</version>
        </dependency>
    </dependencies>
    <configuration>
        <groups>org.lyeung.simpleheart.LaterLife</groups>
    </configuration>
    <executions>
        <execution>
            <id>failsafe-integration-tests</id>
            <phase>integration-test</phase>
            <goals>
                <goal>integration-test</goal>
            </goals>
            <configuration>
                <includes>
                    <include>**/*.class</include>
                </includes>
            </configuration>
        </execution>
    </executions>
</plugin>

What this means is that when Maven runs package goal, only tests with early life are invoked. When Maven runs integration test goal or after, both early life and later life tests are invoked¹.

"From the threshold of the room she saw Virginia, stretched on her back, her hands joined, her mouth open, and her head thrown back under a black cross bending towards her, between motionless curtains, less white than her face."

Below is a snippet log for package goal:
mvn clean package

-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.lyeung.simpleheart.TestFelicity
isKnowsVirginie
isKnowsPaul
isKnowsTheodore
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in org.lyeung.simpleheart.TestFelicity

Results :

Tests run: 3, Failures: 0, Errors: 0, Skipped: 0

Below is a snippet log for package verify goal:

mvn clean verify
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.lyeung.simpleheart.TestFelicity
isKnowsVirginie
isKnowsPaul
isKnowsTheodore
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec - in org.lyeung.simpleheart.TestFelicity

Results :

Tests run: 3, Failures: 0, Errors: 0, Skipped: 0

-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.lyeung.simpleheart.TestFelicity
isKnowsFatherColmiche
isParrotLouLou
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec - in org.lyeung.simpleheart.TestFelicity

Results :

Tests run: 2, Failures: 0, Errors: 0, Skipped: 0

¹http://maven.apache.org/guides/introduction/introduction-to-the-lifecycle.html

Why Test Groups Matter

I'm having poach eggs today.

Upon checking the fridge, to my horror, the egg trays are empty!

Just as I'm about to rush out to the nearest supermarket, I just remembered my stovetop broke down last night.

My instinct kicks in. Before I run towards the door, I quickly check the stovetop.

If it decides to take a holiday today, I'll surely head back to my bedroom and get more sleep.

Well, this is just an analogy.

---

Test tiers are usually made up of unit tests, integration tests and end-to-end tests (E2E).

And if we are to visualise the Agile Test Pyramid...

Unit test, as the name implies, tests for logical and boundary correctness. Most of these tests are in-memory, therefore, they run trail-blazing fast.

Integration test is an aggregation of unit tests that requires external resources. These resources could be transaction boundaries, filesystem or network calls. Resources are usually expensive to setup and perhaps torn down and they need to be made available to the integration test beforehand.

E2E tests for the whole workflow and may touch multiple integration points. It is usually more complex than an integration test. An example of E2E test is a test on checkout workflow. The test must log into the system, select a list of products, put them into the shopping cart then make a checkout.

As you can see, the test cases become more complex as we move up the tiers.

---

Many years ago, I said to my colleague that he needs to run the whole test suite before committing. He replied back saying, "But it takes too long to run the whole suite."

This made me realise that only a handful of people are willing to sit and wait while they twiddle their thumbs as they stare emptily at the screen. As for myself, I make a quick exit to the lift to get another cup of flat white.

---

From my experience, if test codes are grouped and ordered to run from bottom to the top of the pyramid, the development cycle will find and fix defects quicker and minimise waste.

You wouldn't run to the closest supermarket without ensuring your stovetop works, would you?

Below is an inverted test pyramid in running time.

Elwood Build Process High Level Design

Below is a diagram of how the build process works at a conceptual level.

Elwood High Level Design

"A picture is worth a thousand words..."

Elwood Maven Surefire Integration Part 2

I find this solution slightly obtrusive. After the project is checked out, Elwood alters the original pom and adds a small dependency snippet to pull down the custom run listener (elwood-maven-runlistener) during the build. This is evidently found in the pom.xml sitting under the META-INF directory marked with provided scope to prevent this from becoming an official dependency of the artefact.

I could potentially use a custom Maven that has the run listener JAR as one of it's libraries. But then, this has to be another day.

I had to abandon the original idea of making the run listener POST the build statistics using a 3^rd party REST library because I may inadvertently introduce a class loader issue when the official project dependency has this library defined. Same argument goes with Redis. I also wouldn't want roll up my sleeves and start writing network calls on my own as time is a scarce resource.

My other concern with using network calls is that the back-end REST service could be busily serving requests from the browser, which I suspect may slow down the build process. Note that the front-end code retrieves the build status every few seconds.

I don't want to stress the back-end service too much.

I've also tested the idea of keeping the statistics in memory as a JMX bean. However, as each test classes are started up and torn down, the numbers are lost. In the end, I had to come back to my old friend – the filesystem¹.

Note: All changes² related to capturing the Maven build statistics are found in “maven-integration” branch.

¹https://bitbucket.org/lyeung/elwood-parent/commits/f3b45eb95717fc0b48247c69ba0a15918d37e22d

²https://bitbucket.org/lyeung/elwood-parent/commits/822ee1c8343598210c38f3016dbe9a66ec5ca04d