With the advent of responsive Web design, most people can browse on their mobile devices and see a website that appears very similar to their desktop browser. Typically, a mobile site looks simpler than its desktop counterpart, giving the appearance that testing mobile applications is easier than testing of other software. However, there are certain challenges that every mobile app quality tester faces that make this type of testing special and difficult in its own way.
The challenges of mobile device testing
Variety of Devices
In a recent study on the diversity of hand-held devices, it is estimated that, by the end of 2013, there were more mobile devices than people in the world. In 2013, over 12,000 different mobile devices were in use by consumers. This year, it is estimated that the number will rise to well over 20,000. As you can imagine, hand-held devices differ greatly in their features, screen sizes, resolution, hardware capabilities, and input methods (for example, touch, slide, QWERTY).
Diversity in Platforms and OS
The market also features a wide variety of mobile operating systems. The major ones include Android, iOS, Windows Phone, Symbian, and BlackBerry. Naturally, each operating system has its own benefits and limitations. That’s why testing even one website with so many devices running on different platforms poses a great challenge for any tester.
Mobile Network Operators
According to Wikipedia, in the United States alone, there are over 75 wireless network operators. Some of these networks use older technology, such as CDMA or GSM. Each network operator utilizes a different kind of network infrastructure, which can limit the flow of information. Network performance can also have a powerful impact on the user experience with mobile browsing.
Evolution of OS Versions
If you look at the difference between Android’s 1.0 and 4.4 Kit Kat versions or iOS versions such as iPhone OS 1.0 and iOS 7.1, you’ll see that it’s really big. Version control is yet another great challenge for mobile app publishers and testers.
Fragmentation of the Green Robot
Earlier, I mentioned the variety of mobile operating systems on the market, but even within individual operating systems, there is a great deal of diversity. Nowhere is this more evident than in the mobile OS by Google: Android. The website OpenSignal.com provides a report each year about the fragmentation of Android. Some of the highlights of this report include the following:
- There are hundreds of manufacturers of phones that carry the Anrdoid OS, the largest being Samsung.
- Samsung is also the largest brand to carry Android phones, with 43% of the mobile market.
- There are currently 19 different versions of the Android OS, the newest being 4.4, Kit Kat.
- There is a clear correlation between fragmentation and GDP per capita.
- The proliferation of sensors in Android devices allows for increasingly sophisticated apps and networks.
- There are almost 100 different physical screen sizes for Android devices vs. just four for iOS.
One Solution: Scripting
Since most devices vary in input methods, keystrokes, display properties, and menu structure, a single test script won’t function on all devices. Because of this, it is important to leverage the power of automation. With this powerful tool, testers can, to a certain extent, automate testing practices, known as smoke testing or regression testing. A variety of mobile testing applications are available that replicate human interaction to make life easier for the tester.
As you can see, quality testing for mobile devices is no walk in the park. Fortunately, compared to the PC industry, people tend to upgrade their mobile devices more frequently. This makes it easier to target the newer models, operating systems, and mobile browsers.