Skip to main content

How many unit tests are enough?

For some time now, I’ve had concerns about the large automated test suites that are being written; some with hundreds and hundreds of unit tests. It’s great that people are taking test automation seriously and writing lots of unit tests. Once developers get on the bandwagon they tend to write large numbers of unit tests and it feels good to see the number rise. It gives you a sense of confidence. I’ve seen this in my own team. Once the developers started writing unit tests they didn’t know when to stop and we ended up with a large unit test suite. The problem is that unit test suites need to be maintained, just like production code. Also it takes time and effort to write the unit tests in the first place.

So the questions in my mind are: are all these unit tests actually catching defects and do we need to write quite so many? Of course others have thought of these same questions and I’ve come across a couple of blogs that have some great ideas on how we can get the most out of a unit test suite with as little effort as possible.

His central argument for only writing unit tests for particular areas of the code base makes sense to me. He summarises his thoughts in the following diagram:

UnitTestQuadrants.png
There are four broad categories of code:
  • Complex code such as algorithms and business logic (in the top left quadrant). These should be unit tested as they are core to the business. There is a high benefit to writing unit tests for this type of code and a low cost if the business logic or algorithms are separated from controller type code; i.e they have few dependencies.
  • Trivial code with few dependencies has a low cost to writing unit tests but provides low benefit. This code should not be unit tested.
  • Coordinators relate to controller type code that has lots of dependencies. As these dependencies often change, the cost of unit testing is high. Because this type of code does not (or should not) contain business logic the benefit of unit testing this code is low and it should not be unit tested.
  • The top right quadrant refers to overcomplicated code, which would generally be a combination of business logic and controllers. This type of code needs to be unit tested because it contains business logic but there is a high cost because it is complicated with many dependencies. The only way to deal with this quadrant is to refactor the business logic out and unit test that. The remaining code should be coordinator type code which does not need to be unit tested.

So as you can see, the key to writing less and more valuable unit tests is to refactor your code base into code that separates algorithms and business logic from coordinator type code. You can then unit test the valuable business logic and not worry about the rest of the code.

Vladimir Khorikov has similar ideas. He discusses them in more detail on his blog http://enterprisecraftsmanship.com/
He is an advocate for domain driven design, which fits in well with the idea of separating out business logic. He also discusses the most important reason for writing unit tests; which is to gain confidence in the code base. A good set of unit tests will allow you to make changes without breaking existing functionality, allow you to move at a faster pace and maintain a low amount of technical debt.

Khorikov defines four attributes that make an automated test valuable:
  1. Has a high chance of catching a regression error
  2. Has a low chance of producing a false positive
  3. Provides fast feedback
  4. Has low maintenance cost

The more code that is covered, the more chance there is of catching a regression error. E2E tests are good for this but feedback is too slow. Unit or integration tests that are tightly coupled to implementation details are brittle and prone to false positives. Unit tests provide fast feedback and will have a low maintenance cost if they are decoupled from implementation details. As we saw in the diagram, unit tests that focus on testing business logic and algorithms are the most valuable.

In my next blog post I’ll examine the different styles of unit testing and look at which style is best for writing valuable unit tests.

Comments

  1. My Statement is, Unit testing is done if you have covered the requirements / Design.

    ReplyDelete

Post a Comment

Popular posts from this blog

Let’s stop writing automated end to end tests through the GUI

What’s the problem?I have not been a fan of Selenium WebDriver since I wrote a set of automated end-to-end tests for a product that had an admittedly complicated user interface. It was quite difficult to write meaningful end-to-end tests and the suite we ended up with was non-deterministic i.e. it failed randomly.
Selenium Webdriver may be useful in very simple eCommerce type websites but for most real world products it’s just not up to scratch. This is because it’s prone to race conditions in which Selenium believes that the UI has updated when, in fact, it has not. If this happens, the automated check will fail randomly. While there are techniques for reducing these race conditions, in my experience it is difficult to eradicate them completely. This means that automated checks written with Selenium are inherently flaky or non-deterministic. Maintenance of these automated checks becomes a full time job as it is very time consuming to determine whether a failing check is actually a de…

How I got rid of step by step test cases

In my last blog post I told you what I think is wrong with step by step test cases. In this blog post I’ll tell you how I got rid of step by step test cases at the company I work for. When I joined Yambay about 18 months ago, the company was following a fairly traditional waterfall style development approach. They had an offshore test team who wrote step by step test cases in an ALM tool called Test Track. Over the past 18 months we have moved to an agile way of developing our products and have gradually got rid of step by step test cases.


User Stories and how I use them to test
Getting rid of step by step test cases didn’t happen overnight. Initially we replaced regression test cases and test cases for new features with user stories that have acceptance criteria. The key to using a user story to cover both requirements and testing is to make sure that the acceptance criteria cover all test scenarios. Often product owners and/or business analysts only cover typical scenarios. It’s the …

Do we need step by step test cases?

Many companies have moved to agile software development but they still maintain large suites of step by step test cases for regression testing. At the company I work for, I’ve managed to get rid of these step by step test cases altogether.
What is wrong with step by step test cases anyway?
If you are in a traditional waterfall setup with high level requirements and test cases, then the test cases are actually quite valuable as they document how the system works. But if you start to move quality to the left and specify requirements in more detail upfront with user stories, then test cases make much less sense. You are essentially duplicating the requirements in test cases and end up with a whole set of assets that need to be maintained in a separate test management or ALM tool. A big problem with that is that you do not have a single source of truth on how the system behaves.
A second problem with step by step test cases is that they are not visible to the business. I can’t think of ma…