Test Analytics – Measuring Automated Test Coverage Part 1: Mind the Gap

Test Analytics – Measuring Automated Test Coverage
Part 1: Mind the Gap


Here at Piccadilly Labs, we don’t write much code, but when we do it is usually to test that things work properly.  Building tools to accelerate and improve manual testing is a pretty important part of every test specialists’ tool kit.

Of course, when we build testing tools, we have to be extremely confident that they work as specified in order to avoid false positives in test results.  We like to try out the latest testing techniques on our own code, to “eat our own dog food”, so to speak.

Like everyone else, we automate our unit tests.  We also automate functional tests using various tools, including Cucumber.  Even more importantly, we use tooling and metrics to target our test coverage.  Something that is rarely looked at in detail in functional test automation but is extremely powerful.

To use a simple example, we have a small tool which exhausts CPU or memory resources for testing purposes, called the “Exhauster”.  It has some input validation code, some functions which determine the current resource usage and a few loops which ramp up usage as required.

It has some unit tests which do some validation, but they really only check that a non-zero value is returned for each method, like this:


Obviously, that is not much coverage, but we also have some more functional tests written in Cucumber:


These are better, they show that we are doing more of a sense check based on requirements.  But are they really enough?  It doesn’t seem like a lot of tests and they don’t seem to address the core purpose of the tool, but then again it can be unstable to try and automate tests which do destructive things to your system, like exhaust memory!

To get under the skin of the problem, we use a few tools.  First – Jacoco and Sonar (both open source).  Jacoco instruments the code, Sonar consumes the output and gives you key facts on your code base.  We are working with a fairly small code base here, 140 lines of code which forms 9 functions (including the tests).

We can quickly see in Sonar that our combined unit and functional tests are only covering 23% of the lines of code – that’s much smaller than we would expect.  With Sonar we can access the code itself and see what is and isn’t covered.

For instance, the below screenshot shows lines highlighted in red where there is no test coverage.  We can easily see that on the startup function of the application (where it validates incoming parameters and displays errors) when tests aren’t covered at all:



All the tests have used API entry points to test methods – what we need in order to get good coverage is to do an end to end functional test.  We added the below in Cucumber:

Scenario: Overload CPU

Given I overload the CPU to “0.1”

Then CPU usage should increase to at least “0.1”

This time, the test calls the main object constructor with the same parameters that would be used in real life.   We need to re-factor the app to get the test to run, but it does run and it passes.  We need to build in a wait time into the main app to ensure CPU usage increases.  The test pack previously took a second or so to run, which is comparatively an expensive test.

The test does however, deliver significant improvement in line coverage – increasing coverage from 23% to 70%!  Condition coverage is increased from 17% to 56%.  This was driven by about twenty lines of Java changes and test code.

This really shows how effective reviews of automated test suites can drastically improve value with minimal change, particularly when combined with tooling and metrics.  The obvious next step is to take code coverage up to 90%+ which can be easily driven from the Sonar reports.

Stay tuned for the second installment of this blog series.

Adam Smith – Director