Skip to main content

Automating Regression Testing

In my last blog post I described how we have got rid of step by step test cases but didn’t have any automated regression tests. Since then we have embarked on a test automation journey and we are building up a suite of automated regression tests.

In some of my older posts on unit and integration testing I talked about “valuable” automated tests. In summary, a valuable automated test is one which:
  1. Has a high chance of catching a regression error
  2. Has a low chance of producing a false positive
  3. Provides fast feedback
  4. Has low maintenance cost
The more code that is covered, the more chance there is of catching a regression error. End to End (E2E) tests are good for this but feedback is often too slow. So how do we make E2E automated tests more valuable? They already have a high chance of catching a regression error and a low chance of producing a false positive but they tend to be slow and have a high maintenance cost. How can we improve those two aspects? A good way to do this is NOT to automate through the UI. Instead use a “headless client” and interact directly with the underlying View Model. This reduces the time it takes to run the tests and reduces the maintenance burden because we don’t have to cater for UI changes.

Now of course they will never be as quick as unit tests and you still need those. See my post How to Write Valuable Unit Tests. In this post, I’ll focus on how we have started automating our end to end regression tests.
Our Journey
So how did we go about automating our regression tests? Four months ago a developer, who had experience writing automated testing frameworks, joined the company. He has built a test harness that we now use to write automated integration and end to end tests.

Our System Under Test (SUT) comprises a Field client (a Windows, Android or iOS app) that connects to the Mobility Server via a Gateway. In a full end to end system the Mobility server connects to our customer’s control room product. We had already developed an emulator of the customer’s backend called the Test Proxy Client. This allows us to manually send the customer’s SOAP messages to the Mobility Server which manipulates them and then sends them onto the Field Client. We now utilise this to drive many of the automated regression tests.

 These concepts can be visualised in the diagram below:

The Test Harness is written in C# using Nunit in Visual Studio. The Test Harness instantiates a headless field client i.e. a field client without the UI layer, and talks directly to the View Model of the field client. It then uses the Test Proxy Client to send SOAP messages to the Mobility Server which sends them onto the headless Field Client. The automated test cases then assert against the View Model of the headless field client. The test harness can also drive the headless field client directly and asserts against what the field client returns. The test cases also assert against the mobility server output. The automated test cases are data driven and this is stored in a file system as xml and text files.

Every night we deploy the latest Mobility Server and Field Client (from the latest build) onto the test automation environment. We start up headless field clients and run the suite of tests. Each test sets up its data, triggers the test, and asserts on the results. At the moment the results of the automated test suite are available in Jenkins but we are planning to create a more user friendly view with a general purpose tool such as Power BI (which is a Microsoft product that allows you to view analytics).
Now that we have a test harness setup with a suite of automated tests, it is much easier for all the developers to add new tests as they develop new features. We are also slowly automating our set of end to end regression tests where is makes sense.
Lessons Learned

What do you need to succeed with test automation:

Firstly, management buy-in and support is essential. We didn’t get anywhere with test automation until management made it a priority.

Secondly, you need a strong developer who will be dedicated to creating an automated test framework. This person could come from your existing dev team or bring in someone who specialises in this. Once the test framework is setup then all the developers should add new tests to it as they develop new features.

Finally, make sure that the automation tests that you do write use headless clients i.e. do not run through the UI.
Labels:

Comments

  1. Mohammad Shafeek31 December 2018 at 01:39

    Wow Very interesting article. Thanks for sharing your thoughts. To know more software testing https://softwaretestingboard.com/blogs/#axzz5bFeem8pn.

    ReplyDelete
  2. Anonymous31 July 2019 at 05:21

    Very impressive article. Can you give brief explanation on Software Testing?

    ReplyDelete
  3. rasheddahocasio4 March 2022 at 11:18

    888Casino Casino - Dr. Maryland
    888casino 전주 출장샵 Casino is one of the 제주 출장샵 first casinos in Maryland, with over 1,300 slots and video poker machines 의정부 출장샵 that you 공주 출장샵 can 파주 출장샵 play at our hotel.

    ReplyDelete

Post a Comment

Popular posts from this blog

Let’s stop writing automated end to end tests through the GUI

What’s the problem? I have not been a fan of Selenium WebDriver since I wrote a set of automated end-to-end tests for a product that had an admittedly complicated user interface. It was quite difficult to write meaningful end-to-end tests and the suite we ended up with was non-deterministic i.e. it failed randomly. Selenium Webdriver may be useful in very simple eCommerce type websites but for most real world products it’s just not up to scratch. This is because it’s prone to race conditions in which Selenium believes that the UI has updated when, in fact, it has not. If this happens, the automated check will fail randomly. While there are techniques for reducing these race conditions, in my experience it is difficult to eradicate them completely. This means that automated checks written with Selenium are inherently flaky or non-deterministic. Maintenance of these automated checks becomes a full time job as it is very time consuming to determine whether a failing check is actuall...
6 comments
Read more

How I got rid of step by step test cases

In my last blog post I told you what I think is wrong with step by step test cases. In this blog post I’ll tell you how I got rid of step by step test cases at the company I work for. When I joined Yambay about 18 months ago, the company was following a fairly traditional waterfall style development approach. They had an offshore test team who wrote step by step test cases in an ALM tool called Test Track. Over the past 18 months we have moved to an agile way of developing our products and have gradually got rid of step by step test cases. User Stories and how I use them to test Getting rid of step by step test cases didn’t happen overnight. Initially we replaced regression test cases and test cases for new features with user stories that have acceptance criteria. The key to using a user story to cover both requirements and testing is to make sure that the acceptance criteria cover all test scenarios. Often product owners and/or business analysts only cover typical scenarios. It...
3 comments
Read more

How to write valuable unit tests

In my last blog post I discussed what makes unit tests valuable and how to structure your code so that you can write valuable unit tests. But so far I haven’t got into the nitty gritty of how you should write these valuable unit tests. I’ll address that in this blog post. There are three styles of unit test: Output Verification , also known as the functional style, involves checking the output of a method for a given input. This style of unit testing does not concern itself with the internals of a method. State Verification involves checking the state of an object rather than the output of a method. Collaboration Verification is where collaboration between classes is tested, and it usually involves test doubles such as mocks. See Vladimir Khorikov’s blog post for further information and code examples: http://enterprisecraftsmanship.com/2016/06/09/styles-of-unit-testing/ So which style is best for writing valuable unit tests? Here are the four attributes of a v...
2 comments
Read more