Testing any mobile app presents a wide range of challenges. The often repeated but rarely followed software best practice of test early, test often is harder to adhere to than usual due to the fragmentation of the target environment and the relative maturity of tools. The increased acceptance of apps by mainstream consumers and intense competition have raised the bars for user experience and quality. There is more to test than ever, yet often very limited budget for doing so. Fortunately every challenge presents an opportunity and a vast array of tools vendors are racing to fill the gaps.

What to test?

Much of the traditional software testing literature focuses on various forms of functional testing – ensuring the system does what it’s meant to do. With a strong trend towards simpler, single purpose apps, this is often the easiest thing to verify in a mobile app project. There is now a much stronger focus on the user experience and this requires testing of an entirely different nature. The most effective way to test that an app is easy (or even fun) to use is to get feedback from real users. Doing that and finding major issues after the app has been built is a very expensive mistake to make, so most developers and designers will want to create mock-ups or prototypes for early feedback. There’s a wide range of tools to help with this task from simple wireframing through to full interactive prototyping. Given the importance of animations within mobile apps to enable users to discover interface interactions and learn to navigate, more complete prototypes are becoming increasingly desirable. As users become more sophisticated and specialist tools reduce the time and effort required to create interactive prototypes this trend is likely to continue.

With the majority of app store revenues coming through in-app purchases, another more specialized form of testing the design of an app is becoming increasingly important – split testing. On the desktop web, tools for trying out design and copy variants to optimize sites for specific user behaviours are very mature and the best of them can be used by staff with no development skills. In the mobile world most of the tools in this space are still very immature and developer-focussed. The responsive design trend on the web and the more restricted deployment options for native apps make this a more challenging problem for mobile devices but we expect the tools in this sector to mature rapidly.

[sectors slugs=’prototyping-mockup’]

When to test?

The earlier you find problems with software, the cheaper it is to fix them. As such, it makes sense to start testing as early as possible. How about testing the idea for the app via a mobile market research service before you even create your first wireframes? It’s worth considering – if you can’t generate interest in your app idea with a simple pitch it’s not going to be easy to get people to download it from the store either.

For most apps (particularly native apps) it’ll be worth using one of the mock-up or prototyping tools mentioned above and test the design before you start coding the real app. It’s much cheaper to iterate a simple design prototype than a native app. However, you’ll still want to try out the actual app with real users before you launch it. To help with that there’s a range of beta testing services that can help you distribute your beta app and find and/or manage testers. There are also services to help you get feedback from your users before and after the app launches. Providing a highly accessible feedback channel for users in the app is your best hope for preventing the inevitable disgruntled few from leaving bad reviews.

Ideally an app will be developed and tested iteratively with functional testing of new features and full regression tests for the existing functionality run for each iteration. This level of testing can get extremely expensive and time consuming unless it is automated. Fortunately there are several tools, open source frameworks and third party services that can help out there too.

[sectors slugs=’beta-testing,feedback-helpdesk,automated-app-testing’]

Where to test?

Another major problem for mobile developers is the scale and fragmentation of the market they’re trying to serve. Collecting a full library of test devices with major firmware variants is way beyond the budget of most developers, let alone the effort that would be required to test manually on all of them. Automated testing solutions can help here also and some services provide access to a large set of devices for remote testing too. However, it’s simply not feasible for most developers to test every version of their apps on all the device and firmware combinations they support. This limitation means some bugs are almost guaranteed to escape into the wild; the important thing then becomes how quickly you discover and fix them. For this reason, crash analytics and bug tracking tools are becoming increasingly important. Another useful weapon in this battle is your usage analytics data – it can enable you to focus testing on the devices which are most popular amongst your user base and also spot changes in use on particular device models that might signal a non-fatal error that’s causing users to abandon the app.

Finally, for some apps, where they are tested geographically may also be important. Do you know what the performance of your app is like for users who are far from your servers? If you use SMS, do you know how long it takes to get to users on different networks around the world (or if it even gets there). Have the localisations for your app been tested by native speakers? Our automated testing and app certification sectors include companies that can crowdsource beta testers or provide access software testing professionals almost anywhere in the world to help you scale globally without leaving your desk.

[sectors slugs=’crash-analytics-bug-tracking,user-analytics,automated-app-testing,app-certification’]