Why Test Groups Matter

I'm having poach eggs today.

Upon checking the fridge, to my horror, the egg trays are empty!

Just as I'm about to rush out to the nearest supermarket, I just remembered my stovetop broke down last night.

My instinct kicks in. Before I run towards the door, I quickly check the stovetop.

If it decides to take a holiday today, I'll surely head back to my bedroom and get more sleep.

Well, this is just an analogy.

---

Test tiers are usually made up of unit tests, integration tests and end-to-end tests (E2E).

And if we are to visualise the Agile Test Pyramid...

Unit test, as the name implies, tests for logical and boundary correctness. Most of these tests are in-memory, therefore, they run trail-blazing fast.

Integration test is an aggregation of unit tests that requires external resources. These resources could be transaction boundaries, filesystem or network calls. Resources are usually expensive to setup and perhaps torn down and they need to be made available to the integration test beforehand.

E2E tests for the whole workflow and may touch multiple integration points. It is usually more complex than an integration test. An example of E2E test is a test on checkout workflow. The test must log into the system, select a list of products, put them into the shopping cart then make a checkout.

As you can see, the test cases become more complex as we move up the tiers.

---

Many years ago, I said to my colleague that he needs to run the whole test suite before committing. He replied back saying, "But it takes too long to run the whole suite."

This made me realise that only a handful of people are willing to sit and wait while they twiddle their thumbs as they stare emptily at the screen. As for myself, I make a quick exit to the lift to get another cup of flat white.

---

From my experience, if test codes are grouped and ordered to run from bottom to the top of the pyramid, the development cycle will find and fix defects quicker and minimise waste.

You wouldn't run to the closest supermarket without ensuring your stovetop works, would you?

Below is an inverted test pyramid in running time.

Elwood Build Process High Level Design

Below is a diagram of how the build process works at a conceptual level.

Elwood High Level Design

"A picture is worth a thousand words..."

Elwood Maven Surefire Integration Part 2

I find this solution slightly obtrusive. After the project is checked out, Elwood alters the original pom and adds a small dependency snippet to pull down the custom run listener (elwood-maven-runlistener) during the build. This is evidently found in the pom.xml sitting under the META-INF directory marked with provided scope to prevent this from becoming an official dependency of the artefact.

I could potentially use a custom Maven that has the run listener JAR as one of it's libraries. But then, this has to be another day.

I had to abandon the original idea of making the run listener POST the build statistics using a 3^rd party REST library because I may inadvertently introduce a class loader issue when the official project dependency has this library defined. Same argument goes with Redis. I also wouldn't want roll up my sleeves and start writing network calls on my own as time is a scarce resource.

My other concern with using network calls is that the back-end REST service could be busily serving requests from the browser, which I suspect may slow down the build process. Note that the front-end code retrieves the build status every few seconds.

I don't want to stress the back-end service too much.

I've also tested the idea of keeping the statistics in memory as a JMX bean. However, as each test classes are started up and torn down, the numbers are lost. In the end, I had to come back to my old friend – the filesystem¹.

Note: All changes² related to capturing the Maven build statistics are found in “maven-integration” branch.

¹https://bitbucket.org/lyeung/elwood-parent/commits/f3b45eb95717fc0b48247c69ba0a15918d37e22d

²https://bitbucket.org/lyeung/elwood-parent/commits/822ee1c8343598210c38f3016dbe9a66ec5ca04d

Elwood Maven Surefire Integration

I thought of adding a feature to Elwood to display failed Maven tests in real-time.

Below are my options:
1. Constantly watch and scrape the test results produced by Surefire plugin
2. Augment a test listener¹ to Surefire plugin and expose the failed statistics back to Elwood

Since this is an experimental project, I decided to take the slightly complicated approach - option two. From my original thinking, I need to write a Maven model converter, hunt down the Surefire plugin and add the necessary test listener before I feed this pom back to Elwood (see Figure 1)

Figure 1 - Maven Model Converter to add a Surefire test listener

There are a few challenges here. Firstly, writing this converter is not a straightforward exercise as I originally anticipated. It requires deep a knowledge on Maven internals, plus an extra hurdle on the lack of API documentation. I've searched high and lows and luckily found what I was looking for from a test code found in Maven Model Builder project². Secondly, I need to find a way to pass back the failed stats to Elwood through some mechanism. I can choose to do this by writing to a file, but then, how is this different from scraping the result as described in option one?

I've created a local branch containing the model converter and yet to push this to the remote repository as I still have to figure out how to hand the numbers back to Elwood. At this point, I'm not certain if this solution is achievable, so please stay tuned...

Update (24/11): The initial implementation is now available at:
https://bitbucket.org/lyeung/elwood-parent/commits/4b64304870bf58bf6041cdaf7bf6460575c2d04d?at=master


¹https://maven.apache.org/surefire/surefire-api/apidocs/org/apache/maven/surefire/report/RunListener.html ²https://github.com/apache/maven/blob/master/maven-model-builder/src/test/java/org/apache/maven/model/building/DefaultModelBuilderFactoryTest.java

Elwood sources now available in Bitbucket

For the past few weeks, I've been squeezing every last drop of my own time to get Elwood working. From a simple process executor to a functional web application, where I could trigger a build by clicking a link. I have to say that this app is simple and crude, but is really fun to work on!

Think about this for a moment - we're writing a tool to build the tool itself!

Elwood is made up of a back-end REST service and a front-end AngularJS app. The REST service requires Redis 3 (never tested this on Redis 2+). The app keeps a list of projects and tracks the build results. Currently, it only supports Git with SSH (with or without passphrase).

"Elwood threw down his fork. He pushed his chair back and stood up. Liz stared up in astonishment at him, her cup half raised. Bob's mouth hung open, his sentence unfinished. Little Toddy said nothing."

The sources are available in Bitbucket:

• https://bitbucket.org/lyeung/elwood-parent.git
• https://bitbucket.org/lyeung/elwood-ui

 Looking ahead, this is going to be a very long journey...

Elwood - Building the build server (a prototype)

Things are coming together right now! It all started from an imaginary concept filled with dragons, knights and fairy god-mothers - well yeah, you know what I mean.

At the most basic level, a build server has to satisfy 3 requirements:

• Get the source code
• Build the code
• Report if the build is successful

What I have right now is a work in progress. It's quite rudimentary and clunky. It clones your sources (currently Git only) and runs your build command (e.g. mvn clean package). The UI constantly polls for the log but goes endlessly. I have to manually kill the poller to stop this activity - how shameful of me!

He builds a boat in his back yard, to great annoyance to his wife and neighbors. When they ask him why he builds it, he tells them he doesn't know.

What's next?
I need to capture and store the build number, as well as the build result.