Mobile App Testing Landscape – Strategy, Tactics for High Quality Applications

In recent years the growth of mobile apps has grown exponentially. Due to the diversity of operating systems, devices, use cases it is critical to empowering users and business with a secure, convenient, effective and glitch-free application which can act as an efficient solution for their problems. Always remember people who browse your website are visitors, but who browse your App are customers.

Any FAILURE or compromise in quality can ruin your brand reputation and result in revenue loss. Always remember for each category, there is a huge number of Apps, so in any circumstance, you fail to give them optimum experience, productivity or if they found it faulty – User may uninstall your application with a bad review.

So what are the Challenges in Mobile App Testing?

• Different range of mobile devices with different screen sizes and hardware configurations like a hard keypad, virtual keypad (touch screen) and trackball etc.

• Wide varieties of mobile devices like HTC, Samsung, Apple, and Nokia.

• Different mobile operating systems like Android, Symbian, Windows, Blackberry, and IOS.

• Different versions of operation system like iOS 5.x, iOS 6.x, BB5.x, BB6.x etc.

• Different mobile network operators like GSM and CDMA.

• Frequent updates – (like Android- 4.2, 4.3, 4.4, iOS-5.x, 6.x) – with each update a new testing cycle is recommended to make sure no application functionality is impacted.

Testing of mobile apps is different than traditional desktop and web applications. There are lots of factors which affect the end user experience. The testing should be carried out considering mobile devices -their Operating System, hardware and software configurations, network conditions- e.g.: Wi-Fi, internet speed, service providers, standards and technology, user expectations and trends, the further scope of versions and additional functionality enhancements are key elements of effective testing.

While devising the testing strategy for your mobile App you need to focus on:

Device

1. Device Assortment –We use an optimal mix of duos for device selection- A range of Device Model emulators and physical devices. Considering the demographics of users, various operating systems; selection of device heavily relies upon::

• Version– Whether the application performance is optimal on the latest and previous versions of the operating system.

• Screen Resolution– There is a diverse screen resolution for different devices. App testing should carry out for various screen sizes.

• Compatibility Factor– Depending upon whether the App is Native, Hybrid or Web, a mobile application behaves differently on a smartphone, tablet, and any other mobile devices. Here you should check compatibility.

2. Emulators – Emulators don’t substitute for physical devices. It gives an extra edge to development by providing the added advantage of the pseudo-live environment, network by-pass, and text scripting. During Agile development; device emulators are extremely effective due to rapid and efficient testing.

Device emulators are equipped with instruments and the network so the testing can be done similar to the real testing environment. It does necessary diagnostics to identify the problem and the flexibility in network stacks needed to test different network options.

Automation

To achieve compatibility between users, devices and the market, including test automation as a part of the agile development process is typical. Automated software testing is the best solution for developing a robust, scalable secure, reliable and high performing mobile app which satisfies the perpetual convolution of technology and under immense competitive pressure.

Here are some automation scenarios:

1. User agent-based automation: User agent add-ons that come with popular browsers who render the Web content which is exhibited on the device with a desktop browser. The functionality can be automated by using popular tools such as QTP, Selenium or RFT etc.

2. Cloud-based automation: Cloud-based test automation vendors support Web-based QA across platforms. These automation tools unleash several advantages over traditional test automation tools.

3. Lab-based automation (devices/simulators): In this method, we take leverage of physical devices and device in a test environment and set up popular open-source tools such as Selenium.

The test automation should be done in the following scenarios:

• The mobile development lifecycle is long.

• The application is growing and embryonic.

• The scale and frequency of regression testing are high.

• A large portion of test cases includes obtainable functionality test cases

With automation, mobile application testing becomes easy towards:

• Verify application compatibility when a new OS version is released

• Check backward compatibility when the application is upgraded

Network Environment –

Performances for mobile applications have a significant impact on the user’s overall experience or impression and commemoration of that business/brand and its products or services – So on brand reputation and revenue. The overall objective is to test the application in such conditions that realistically reflects the environment wherein the application is ultimately expected to operate

There are more than 400 mobile networks worldwide, which support multiple technologies, e.g. – LTE, CDMA, GSM. There is also diversity in network standards such as Eden, FOMA, and TD-SCDMA. Also, several networks use web proxies to define how, when users can access a site. Some proxies limit the sites accessed via phone and only after having the approval of an operator known as “walled garden.” Few proxies use “transcoding” which scale down fixed web content which better fit onto mobile phones. These diverse network challenges occur due to different locations and compliances and signal strength

Network Bypass

Device emulator is used for network bypass. It uses an operator’s proxy for a realistic test. There are many test proxies available on the internet which is used to test the app in the native environment of networking.

Real Networks

This is the best possible way to do testing. However, many a time it seems a bit impractical as you can roam over the world for a global app.

Types of Mobile App Testing

• Acceptance of any app in the court of the customer depends on various factors. These factors address multiple complexities in mobile apps. The specific types of testing required for each application depending on various factors such as:

• The type of application (banking, gaming, social, or business)

• Audience type (consumer, enterprise) and volume

• Distribution channel (e.g. Apple App Store, Google Play, direct distribution)

To be well suited in all eyes of customers the following test should be carried out:

Operational Testing: This testing is carried out to determine inbuilt back-up and recovery operational functions of mobile applications that may occur due to battery drainage, data loss or any unspecified reason. We use Operational testing to test whether the backup and recovery process is working properly and responding as per the requirement.

Functional Testing: This type of testing is used to check the functional behavior of the application to check App is acting as per the specified requirements. Here we emphasize on user interface and call flows. The success of a mobile app depends on lots of human consideration. A mix of manual and automated tools and techniques is used to perform functional testing.

Performance Testing: There are lots of factors which affect the performance of any application. In performance testing, we consider the performance and actions of the mobile app challenges like; power consumption, network out of coverage area/poor bandwidth/changing internet connection mode (2G, 3G, or WiFi) /changing broadband connection, data size, memory issues, simultaneous access through various devices.

Usability Testing: Usability is one of the critical aspects of the success of a mobile app. Here the testers test the app, not as professional bug testers; rather they examine the application to see how end-user experience with the app, how much it is convenient to use etc.

Security Testing: In the mobile app development landscape security is the main concern. We have apps ranging from entertainment and gaming to utility and banking apps. The purpose of security testing is to test an app with regards to any vulnerability, possible misuse, data loss, and prescribed guidelines.

Installation Testing: People download apps from the respective app store for their various compatible devices for a particular operating system. Different devices have different configuration, user interface, and operating system versions. Installation testing ensures that an app can be smoothly installed or uninstalled and intimates user for a new version of the app.

Memory Leakage Testing: Memory leakage slows down the speed of Application and finally can shut down the device. Mobile devices have limited memory so most of the mobile OS stop functioning which requires extreme memory for processing; memory leakage ensures that application is using optimized memory for processing.

Interrupt Testing: Have you ever faced a situation where at the time of using an app you got a call, or suddenly it got switched off? Interrupt testing is about to test the behavior of an app in an interrupted situation.

Criterion Areas / Types of testing

Multi-Platform Support Consider your current and future target platforms and ensure that the tool can support them

Script Reusability Object-based tools such as TCS Hy5 Test™ provide a high degree of script reuse. Image-based tools generally have lower script re-use resulting in higher cost.

Jailbreak Requirement If the tool uses jailbroken or rooted devices, it may not support the latest OS versions and may be incompatible with MDM policies required for enterprise apps.

Source Code Changes Sharing source code may not be always possible and changing the source code increases the risk of missing some defects.

Lead Time for New OS version / Device How soon can the tool support new iOS/Android/other OS versions?

Test Workflow Does the tool provide an easy to use workflow?

Results and Reports Does the tool archive results and provide reports?

Integration Capabilities Can the tool integrate/interoperate with other test management/bug tracking/build management systems?

Deployment What deployments options are available? Can it be deployed in-house/in a ‘cloud model’?

Price How much does it cost?

Service and Support What services (e.g. script development) and support are offered?

Tools:

There are various tools to test an app over multiple platforms. In the mobile development landscape as different brands are flooding with a huge number of devices with multiple configuration and complexity; to counter this problem developers are coming with different test automation tools to test the quality, compatibility of the mobile application. Depending upon criterion you can select a suitable test automation tool.

A note for Testers:

Testers play an important role in mobile app development. As you are always up-to-date with the progress of the project, so it is easy for you to identify the challenge at the very beginning. Always get ready to ask these questions:

• What information exists? Specifications? Project conversations? User documentation? Knowledgeable team members? Could the support forum or an online company forum be of help? Is there a log of existing bugs?

• What OS, platform and device should this app work on and be tested on?

• What kind of data is processed by the application (i.e. personal, credit cards, etc.)?

• Does the application integrate with external applications (APIs, data sources)?

• Does the app work with certain mobile browsers?

• What do existing customers say about the product?

• How much time is available for testing?

• What priorities and risks are there?

• Who is experiencing pain, and why?

• How are releases or updates made?

Based on the information, prepare a plan on how to approach the testing.

Conclusion:

Devising the right test strategy, selection of proper simulators, devices and mobile testing tools including security, usability, performance, functionality and compatibility based tests ensure a high-class mobile app.