Skip to main content

Do we need step by step test cases?


Many companies have moved to agile software development but they still maintain large suites of step by step test cases for regression testing. At the company I work for, I’ve managed to get rid of these step by step test cases altogether.

What is wrong with step by step test cases anyway?


If you are in a traditional waterfall setup with high level requirements and test cases, then the test cases are actually quite valuable as they document how the system works. But if you start to move quality to the left and specify requirements in more detail upfront with user stories, then test cases make much less sense. You are essentially duplicating the requirements in test cases and end up with a whole set of assets that need to be maintained in a separate test management or ALM tool. A big problem with that is that you do not have a single source of truth on how the system behaves.

A second problem with step by step test cases is that they are not visible to the business. I can’t think of many product owners or even business analysts who want to trawl through detailed test cases to understand how a product works. So we get to a situation where the business has a high level idea of how the product works through requirements but only the testers know how the product really works.

Another big problem from a tester’s perspective is that regression testing is boring! Who wants to go through step by step test cases over and over again? Surely testers are more intelligent than that. Once a tester understands a product they do not need step by step test cases to test it effectively. I prefer the idea of an intelligent tester who has a good knowledge of the product and is finding issues, rather than blindly following a step by step test case.

To summarise, the problems with step by step test cases are:
  • They duplicate requirements which means you have two sets of assets to maintain and  there is no single source of truth about how the product works
  • Step by step test cases are not visible to the business
  • It encourages the notion that anyone can test by blindly following a step by step test case

How can you get rid of step by step test cases?


If you would like to find out how I got rid of test cases, why not come to the talk I’m giving on this subject at the CASTx18 conference in Melbourne, Australia. CASTx18 is on 1 March 2018 and you can register here:

In the talk I describe the steps I took to change processes and convince management there is a better way to do things. The approach I pioneered hinges on creating user stories to define features and, in particular, writing acceptance criteria  that effectively become the test cases. This approach means that we have one source of truth for how a feature works and what has been tested. I’ll also describe how other types of tests, such as shakedowns, were replaced with mind maps.

If you live overseas or are not able to attend the conference, I’ll try and get a video of the talk and/or write a follow up post that expands on the details of my approach.

Labels:

Comments

Post a Comment

Popular posts from this blog

Let’s stop writing automated end to end tests through the GUI

What’s the problem? I have not been a fan of Selenium WebDriver since I wrote a set of automated end-to-end tests for a product that had an admittedly complicated user interface. It was quite difficult to write meaningful end-to-end tests and the suite we ended up with was non-deterministic i.e. it failed randomly. Selenium Webdriver may be useful in very simple eCommerce type websites but for most real world products it’s just not up to scratch. This is because it’s prone to race conditions in which Selenium believes that the UI has updated when, in fact, it has not. If this happens, the automated check will fail randomly. While there are techniques for reducing these race conditions, in my experience it is difficult to eradicate them completely. This means that automated checks written with Selenium are inherently flaky or non-deterministic. Maintenance of these automated checks becomes a full time job as it is very time consuming to determine whether a failing check is actuall...
6 comments
Read more

Non Functional Mobile App Testing

When you are testing mobile apps there are a number of non functional elements you need to consider (that do not apply to website testing) such as push notifications, device network issues, location services and app installation. In this post I'll cover these and explain how to test these areas. Push notifications Push notifications were pioneered by Apple in 2008 and this technology was subsequently adopted by Google for its Android OS and by Microsoft for its Windows Phone OS. Push notifications allow an app to deliver information   to a mobile device   without a specific request from the app. This means that the app does not need to be launched for the mobile device to get the push notification. Each operating system has their own Push Notification Service. On iOS it's called Apple Push Notification Service (APNs) and Android had Google Cloud Messaging (GCM) but this has been superseded by Firebase Cloud Messaging (FCM). Note that FCM can also be used to send push n...
2 comments
Read more

How to install a test app onto your mobile device

In this post I'll describe how to actually get test versions of the app onto a device. But first let's discuss whether you should test on an actual device or on an emulator. An emulator is a desktop application that mimics the hardware and OS of a mobile device. The developers will generally do their app development on an emulator and you can use them for early stage testing but when it comes to meaningful end to end testing, a device is a must have. There is no other way to get a feel for the performance of the app and how users will use it real life. Of course you will probably need to test on multiple devices as they vary not only by OS (iOS and Android) but also OS version, device make (e.g. Samsung or Motorola), and screen size. We also now have a new OS for iPads called iPadOS. In a future blog post I'll look at device fragmentation and how to handle it. For now let's assume you are only testing on one iOS device and one Android device. Your developers h...
2 comments
Read more