Skip to main content

Non Functional Mobile App Testing

When you are testing mobile apps there are a number of non functional elements you need to consider (that do not apply to website testing) such as push notifications, device network issues, location services and app installation. In this post I'll cover these and explain how to test these areas.

Push notifications

Push notifications were pioneered by Apple in 2008 and this technology was subsequently adopted by Google for its Android OS and by Microsoft for its Windows Phone OS. Push notifications allow an app to deliver information  to a mobile device  without a specific request from the app. This means that the app does not need to be launched for the mobile device to get the push notification. Each operating system has their own Push Notification Service. On iOS it's called Apple Push Notification Service (APNs) and Android had Google Cloud Messaging (GCM) but this has been superseded by Firebase Cloud Messaging (FCM). Note that FCM can also be used to send push notifications to iOS devices. There are also a number of platforms e.g. Amazon Web Services, that allow push notifications to be sent to mulitple operating systems.

Push notifications don't work on a simulator so you will need an actual device to test them. You will also be prompted to accept push notifications when you install the app. You should check the scenario where you don't accept them i.e. no push notifcations are sent, and the case where you do accept them i.e. push notifications should be sent.

Some push notification test scenarios to consider:
  • User receives a notification when device is asleep (online) and app is in the background        
  • User receives a notification when device is asleep (online) and app is in the foreground        
  • User receives a notification when app is in the background and device is online
  • User receives a notification when app is in the foreground and device is online        
  • User receives a notification when app is in the background and another app is running in foreground        
  • When the user receives a push notification, there should be sound and vibration
  • When the user receives multiple push notifications, all the notifications should be displayed in the notification panel with the most recent one on top

 Consider what happens when the user taps on a notification:
  • When the user taps on a notification, the app should open at the expected screen
  • When the user taps on a notification, the push should be removed from the notification panel

 Some boundary scenarios to consider:
  • When the user is not logged onto the app, they should not receive any push notifications
  • Also consider what happens when the device is offline i.e. no WiFi or network coverage:
  • Does the Push Notification Service store and send the push notification once the device returns to network coverage?
  • Is there a time limit for this i.e. what happens when the device is offline for a long time or the push notification is no longer relevant?

Network Issues

I touched on network issues in the Push Notifications section and it's important to consider how the app is expected to behave when it is out of range i.e. no WiFi or mobile coverage. If you are in the middle of a transaction, is the app smart enough to store and then send the information back to server once it's online? It's best to speak to your app designers and developers about how they plan to handle offline scenarios and then tailor your testing to cater for them. For example, does the app actually tell the user that they are offline? This is also a good time to do exploratory testing to uncover scenarios that no-one has yet thought of. For example, what happens when you logout of the app when it is offline?

Location Services

A lot of apps use Location Services to determine the device's approximate location. As with push notifications these need to be accepted when you install the app. If your app usually uses a location service, you may want to test the scenario where the user turns off the location service. Does you app notifiy the user that the location service is off and is it still able to function without a location service? Does it prompt the user to turn on location settings? All these scenarios should be considered.

To turn off the location service on an iPad or iPhone:
  1. Open Settings, swipe down and tap on Privacy.
  2. At the top, tap Location Services.
  3. Tap the toggle next to Location Services.
  4. Tap Turn Off to turn off location services for all apps or select the particular app you are testing and turn off its location service.

 To disable location tracking on an Android device:
  1. Go to Settings.
  2. Tap on “Apps & notifications.” Open the Advanced heading and look for “App permissions.”
  3. Tap on Location. You’ll find here a list of all your current apps with a toggle showing whether they have location tracking activated. Choose the test app and toggle off the location tracking.

App installation

Another mobile specific feature to test, is how the app behaves when installed for the first time versus when it's updated. For example, when you do a fresh install all user information would normally be deleted and the user would need to register again. But when the user simply updates the app to a new version, you would generally want at least the username to be preserved so that the user can simply logon without having to register anew.

The simplest way to test these scenarios is to use an app that allows over-the-air installation and testing of mobile apps such as Testflight or App Center as this will allow you to test both scenarios. See my blog post How to install a test app onto your mobile device for further info on Testflight and App Center.

I hope this post has got you thinking about the mobile device specific test scenarios that you will need to consider when testing a mobile app.


Popular posts from this blog

How to install a test app onto your mobile device

In this post I'll describe how to actually get test versions of the app onto a device. But first let's discuss whether you should test on an actual device or on an emulator. An emulator is a desktop application that mimics the hardware and OS of a mobile device. The developers will generally do their app development on an emulator and you can use them for early stage testing but when it comes to meaningful end to end testing, a device is a must have. There is no other way to get a feel for the performance of the app and how users will use it real life.
Of course you will probably need to test on multiple devices as they vary not only by OS (iOS and Android) but also OS version, device make (e.g. Samsung or Motorola), and screen size. We also now have a new OS for iPads called iPadOS. In a future blog post I'll look at device fragmentation and how to handle it.
For now let's assume you are only testing on one iOS device and one Android device. Your developers have de…

Testing Mobile Apps versus Websites

You all no doubt own a mobile phone, most likely an iOS or Android device, so you probably think it wouldn't be too difficult to test mobile apps. While the principles of testing remain the same, there are significant differences between how you test websites and mobile apps. For example: You can't just open a browser on your laptop to test the latest version of the application. Somehow you need to get the latest version of the app onto your test device.iOS and Android have completely different design patterns.It's more difficult to get under the hood of an app to see what's going on. You can't just open Chrome DevTools.
This is the first in a series of posts where I'll give you some background on how to test mobile apps and include a number of tips and tricks that I've learnt the hard way in my 10 years of mobile testing experience. I'll focus on iOS and Android since those are the most common apps but I also test Windows apps.…