Non Functional Mobile App Testing

When you are testing mobile apps there are a number of non functional elements you need to consider (that do not apply to website testing) such as push notifications, device network issues, location services and app installation. In this post I'll cover these and explain how to test these areas.

Push notifications

Push notifications were pioneered by Apple in 2008 and this technology was subsequently adopted by Google for its Android OS and by Microsoft for its Windows Phone OS. Push notifications allow an app to deliver information  to a mobile device  without a specific request from the app. This means that the app does not need to be launched for the mobile device to get the push notification. Each operating system has their own Push Notification Service. On iOS it's called Apple Push Notification Service (APNs) and Android had Google Cloud Messaging (GCM) but this has been superseded by Firebase Cloud Messaging (FCM). Note that FCM can also be used to send push notifications to iOS devices. There are also a number of platforms e.g. Amazon Web Services, that allow push notifications to be sent to mulitple operating systems.

Push notifications don't work on a simulator so you will need an actual device to test them. You will also be prompted to accept push notifications when you install the app. You should check the scenario where you don't accept them i.e. no push notifcations are sent, and the case where you do accept them i.e. push notifications should be sent.

Some push notification test scenarios to consider:

• User receives a notification when device is asleep (online) and app is in the background        
• User receives a notification when device is asleep (online) and app is in the foreground        
• User receives a notification when app is in the background and device is online
• User receives a notification when app is in the foreground and device is online        
• User receives a notification when app is in the background and another app is running in foreground        
• When the user receives a push notification, there should be sound and vibration
• When the user receives multiple push notifications, all the notifications should be displayed in the notification panel with the most recent one on top

 Consider what happens when the user taps on a notification:

• When the user taps on a notification, the app should open at the expected screen
• When the user taps on a notification, the push should be removed from the notification panel

 Some boundary scenarios to consider:

• When the user is not logged onto the app, they should not receive any push notifications
• Also consider what happens when the device is offline i.e. no WiFi or network coverage:
• Does the Push Notification Service store and send the push notification once the device returns to network coverage?
• Is there a time limit for this i.e. what happens when the device is offline for a long time or the push notification is no longer relevant?

Network Issues

I touched on network issues in the Push Notifications section and it's important to consider how the app is expected to behave when it is out of range i.e. no WiFi or mobile coverage. If you are in the middle of a transaction, is the app smart enough to store and then send the information back to server once it's online? It's best to speak to your app designers and developers about how they plan to handle offline scenarios and then tailor your testing to cater for them. For example, does the app actually tell the user that they are offline? This is also a good time to do exploratory testing to uncover scenarios that no-one has yet thought of. For example, what happens when you logout of the app when it is offline?

Location Services

A lot of apps use Location Services to determine the device's approximate location. As with push notifications these need to be accepted when you install the app. If your app usually uses a location service, you may want to test the scenario where the user turns off the location service. Does you app notifiy the user that the location service is off and is it still able to function without a location service? Does it prompt the user to turn on location settings? All these scenarios should be considered.

To turn off the location service on an iPad or iPhone:

1. Open Settings, swipe down and tap on Privacy.
2. At the top, tap Location Services.
3. Tap the toggle next to Location Services.
4. Tap Turn Off to turn off location services for all apps or select the particular app you are testing and turn off its location service.

 To disable location tracking on an Android device:

1. Go to Settings.
2. Tap on “Apps & notifications.” Open the Advanced heading and look for “App permissions.”
3. Tap on Location. You’ll find here a list of all your current apps with a toggle showing whether they have location tracking activated. Choose the test app and toggle off the location tracking.

App installation

Another mobile specific feature to test, is how the app behaves when installed for the first time versus when it's updated. For example, when you do a fresh install all user information would normally be deleted and the user would need to register again. But when the user simply updates the app to a new version, you would generally want at least the username to be preserved so that the user can simply logon without having to register anew.

The simplest way to test these scenarios is to use an app that allows over-the-air installation and testing of mobile apps such as Testflight or App Center as this will allow you to test both scenarios. See my blog post How to install a test app onto your mobile device for further info on Testflight and App Center.

I hope this post has got you thinking about the mobile device specific test scenarios that you will need to consider when testing a mobile app.