Skip to main content

How to write valuable unit tests

In my last blog post I discussed what makes unit tests valuable and how to structure your code so that you can write valuable unit tests. But so far I haven’t got into the nitty gritty of how you should write these valuable unit tests. I’ll address that in this blog post.

There are three styles of unit test:
  1. Output Verification, also known as the functional style, involves checking the output of a method for a given input. This style of unit testing does not concern itself with the internals of a method.
  2. State Verification involves checking the state of an object rather than the output of a method.
  3. Collaboration Verification is where collaboration between classes is tested, and it usually involves test doubles such as mocks.

See Vladimir Khorikov’s blog post for further information and code examples:

So which style is best for writing valuable unit tests? Here are the four attributes of a valuable unit test:
  • Has a high chance of catching a regression error
  • Has a low chance of producing a false positive
  • Provides fast feedback
  • Has low maintenance cost

All three styles have a high chance of catching regression errors (assuming they are testing business logic and not trivial code) and all provide fast feedback. However, when we look at false positives and maintenance costs, the three styles differ significantly.

The functional style offers the best protection against false positives as we are only considering inputs and outputs. As long as the signature of the method remains the same, the test will not break. This is also the simplest style of unit test to write and thus is easy to maintain. However, to use this style of unit testing, the code itself must be written in a functional way.

The state verification style of unit testing offers good protection against false positives as long as you verify against the public API of the class. This style also has a reasonable maintenance cost.

Collaboration verification is good for testing communication between applications and/or external systems but should not be used to test communication between classes within the business domain model as these are implementation details. This style of verification is prone to false positives as implementation details change all the time and it’s high maintenance due to the use of test doubles.

So the rule of thumb with regard to which style of unit testing to use is: adhere to the functional style as much as possible. State verification is the second best choice but make sure you verify state through the public API. Only use collaboration verification to test communication between applications; i.e. integration testing.

You have probably heard of the concepts of black box and white box testing; black box testing focuses on verifying the outwardly observable behaviour of a system, while white box testing looks at the internal structure of a system. According to Khorikov, we should adhere to the black box style of testing at all levels. I agree with this. Even at unit test level you want to test the public API of a set of business domain classes; you never want to test implementation details as these change often.

For a more detailed explanation of what an implementation detail is, have a look at this blog post:

Before you write a unit test ask yourself: ‘is it testing business logic?’ If the answer is no, consider refactoring your code so that business logic is tested or, if this is not possible, a unit test may not be adding any value and shouldn’t be written at all. View your code from the end user’s perspective and write unit tests to verify its observable behaviour.

Khorikov also discusses Unit Testing Anti-patterns. I have summarised his ideas below:

  1. Exposing implementation details - you should not be testing private methods. If you find that you need to then you may need to abstract a private method into another class and test that class. Another example of this is exposing state getters solely to satisfy a test. Always make sure you are testing the observable behaviour of the class and not its implementation details.
  2. Leaking domain knowledge to tests - make sure you do not reimplement an algorithm or calculation in the test in order to check it. Property based testing may be useful if you need to test a complex algorithm.
  3. Code pollution - don't introduce code into your main code base solely for testing purposes. If the code doesn’t offer functionality that you need for testing the system, add this functionality as a test utility.
  4. Non-determinism in tests - this is when unit tests pass sometimes and fail sometimes. Avoidance strategies include not using Thread.Sleep() in your tests, not testing asynchronous code and being careful about time comparisons.

All these ideas really make sense to me and would fix a lot of the problems that I see with the way we currently write unit tests. Let’s improve our unit test suites by writing less tests that are more valuable. In my next blog post I’ll talk about how to apply these ideas to integration testing.


Popular posts from this blog

Let’s stop writing automated end to end tests through the GUI

What’s the problem?I have not been a fan of Selenium WebDriver since I wrote a set of automated end-to-end tests for a product that had an admittedly complicated user interface. It was quite difficult to write meaningful end-to-end tests and the suite we ended up with was non-deterministic i.e. it failed randomly.
Selenium Webdriver may be useful in very simple eCommerce type websites but for most real world products it’s just not up to scratch. This is because it’s prone to race conditions in which Selenium believes that the UI has updated when, in fact, it has not. If this happens, the automated check will fail randomly. While there are techniques for reducing these race conditions, in my experience it is difficult to eradicate them completely. This means that automated checks written with Selenium are inherently flaky or non-deterministic. Maintenance of these automated checks becomes a full time job as it is very time consuming to determine whether a failing check is actually a de…

How I got rid of step by step test cases

In my last blog post I told you what I think is wrong with step by step test cases. In this blog post I’ll tell you how I got rid of step by step test cases at the company I work for. When I joined Yambay about 18 months ago, the company was following a fairly traditional waterfall style development approach. They had an offshore test team who wrote step by step test cases in an ALM tool called Test Track. Over the past 18 months we have moved to an agile way of developing our products and have gradually got rid of step by step test cases.

User Stories and how I use them to test
Getting rid of step by step test cases didn’t happen overnight. Initially we replaced regression test cases and test cases for new features with user stories that have acceptance criteria. The key to using a user story to cover both requirements and testing is to make sure that the acceptance criteria cover all test scenarios. Often product owners and/or business analysts only cover typical scenarios. It’s the …

Do we need step by step test cases?

Many companies have moved to agile software development but they still maintain large suites of step by step test cases for regression testing. At the company I work for, I’ve managed to get rid of these step by step test cases altogether.
What is wrong with step by step test cases anyway?
If you are in a traditional waterfall setup with high level requirements and test cases, then the test cases are actually quite valuable as they document how the system works. But if you start to move quality to the left and specify requirements in more detail upfront with user stories, then test cases make much less sense. You are essentially duplicating the requirements in test cases and end up with a whole set of assets that need to be maintained in a separate test management or ALM tool. A big problem with that is that you do not have a single source of truth on how the system behaves.
A second problem with step by step test cases is that they are not visible to the business. I can’t think of ma…