How many unit tests are enough?

For some time now, I’ve had concerns about the large automated test suites that are being written; some with hundreds and hundreds of unit tests. It’s great that people are taking test automation seriously and writing lots of unit tests. Once developers get on the bandwagon they tend to write large numbers of unit tests and it feels good to see the number rise. It gives you a sense of confidence. I’ve seen this in my own team. Once the developers started writing unit tests they didn’t know when to stop and we ended up with a large unit test suite. The problem is that unit test suites need to be maintained, just like production code. Also it takes time and effort to write the unit tests in the first place.

So the questions in my mind are: are all these unit tests actually catching defects and do we need to write quite so many? Of course others have thought of these same questions and I’ve come across a couple of blogs that have some great ideas on how we can get the most out of a unit test suite with as little effort as possible.

The first is by Steve Sanderson: http://blog.stevensanderson.com/2009/11/04/selective-unit-testing-costs-and-benefits/

His central argument for only writing unit tests for particular areas of the code base makes sense to me. He summarises his thoughts in the following diagram:

There are four broad categories of code:

• Complex code such as algorithms and business logic (in the top left quadrant). These should be unit tested as they are core to the business. There is a high benefit to writing unit tests for this type of code and a low cost if the business logic or algorithms are separated from controller type code; i.e they have few dependencies.
• Trivial code with few dependencies has a low cost to writing unit tests but provides low benefit. This code should not be unit tested.
• Coordinators relate to controller type code that has lots of dependencies. As these dependencies often change, the cost of unit testing is high. Because this type of code does not (or should not) contain business logic the benefit of unit testing this code is low and it should not be unit tested.
• The top right quadrant refers to overcomplicated code, which would generally be a combination of business logic and controllers. This type of code needs to be unit tested because it contains business logic but there is a high cost because it is complicated with many dependencies. The only way to deal with this quadrant is to refactor the business logic out and unit test that. The remaining code should be coordinator type code which does not need to be unit tested.

So as you can see, the key to writing less and more valuable unit tests is to refactor your code base into code that separates algorithms and business logic from coordinator type code. You can then unit test the valuable business logic and not worry about the rest of the code.

Vladimir Khorikov has similar ideas. He discusses them in more detail on his blog http://enterprisecraftsmanship.com/

He is an advocate for domain driven design, which fits in well with the idea of separating out business logic. He also discusses the most important reason for writing unit tests; which is to gain confidence in the code base. A good set of unit tests will allow you to make changes without breaking existing functionality, allow you to move at a faster pace and maintain a low amount of technical debt.

Khorikov defines four attributes that make an automated test valuable:

1. Has a high chance of catching a regression error
2. Has a low chance of producing a false positive
3. Provides fast feedback
4. Has low maintenance cost

The more code that is covered, the more chance there is of catching a regression error. E2E tests are good for this but feedback is too slow. Unit or integration tests that are tightly coupled to implementation details are brittle and prone to false positives. Unit tests provide fast feedback and will have a low maintenance cost if they are decoupled from implementation details. As we saw in the diagram, unit tests that focus on testing business logic and algorithms are the most valuable.

In my next blog post I’ll examine the different styles of unit testing and look at which style is best for writing valuable unit tests.