What is Mobile Testing? Definition, Tools, Best Practices

We are living in a mobile-first world. It dominates the world, when there are over 10.37 billion mobile connections worldwide, which is even higher than the global population. This number will only increase in the upcoming years. Mobile testing is therefore more critical than ever to accommodate for the ever-rising bar for mobile app quality.
 

This article is written to help testers of any level and expertise to get started with mobile testing. Here are the items we are going to include:

1. What is Mobile Application Testing?

2. Why Mobile Testing?

3. Types of Mobile Testing

4. Manual Mobile Testing

5. Automated Mobile Testing

6. The Mobile Testing Process

7. Mobile Testing Best Practices 

1. What is Mobile Application Testing?

Let’s start with the basic question.
 

Mobile testing is a comprehensive process that evaluates a mobile app's functionality, user experience, security measures, performance under various conditions, and more.
 

Let’s use Flappy Bird - a one-hit-wonder mobile game - as an example of the process of mobile testing. In Flappy Bird, the player guides a bird through an endless series of SuperMario-esque pipes by tapping on the screen to flap its wings.

We all know that it is a notoriously difficult game, but that will not be our focus in this article. Despite its simple mechanism, this indie game still needs to be tested before releasing to Google Play and AppStore. Imagine you are the developer - Dong Nguyen. You would have done these steps to test this mobile application:
 

1. List down Flappy Bird’s core features
2. Prepare the necessary devices to test the app (including Android & iOS. If you do not have these physically available, you’ll need to do it on-cloud).
3. Test the pipe generation mechanism. Is the gap between the pipes a reasonable distance?
4. Test the controls. Does the bird flap its wings and ascend a specific distance upon tapping on the screen?
5. Test the collision detection. You can control the bird to collide with the pipes on purpose to see if the system can detect it.
6. For each passing pipe, the player gets one point.
7. Once all features have been checked, you have completed the functional testing for the game. You can move to non-functional testing, such as performance testing, load testing, or security testing.
8. After that, you also need to test the app across devices (as you have already prepared in Step 2)

Read More: Performance Testing vs. Load Testing: A Complete Guide
 

That is mobile app testing in action. Flappy Bird is a simple game, so all of those steps can be done manually. However, for the more complex mobile apps, manual testing would be a huge challenge.
 

For example, Facebook would need automation testing to cover all of the necessary test cases. For the Login page alone, there would have been hundreds of different scenarios to consider.

Read More: 100+ Test Cases For The Login Page That You Will Need

2. Why Mobile Testing?

Mobiles are diverse. There are 4 major reasons why we need mobile testing:

1. Variety of device specifications
2. User experience & satisfaction
3. Security & Privacy
4. Competitive Advantage
    

Not all mobiles are born equal. Some have 480x800 screen resolution, some have 720x1280, some have 1080x1920. You can check the screen dimensions of the recent iPhone generations in this table. Such diversity is great for users but turns out to be a challenge for testers since the UI can break if mobile responsiveness best practices are not followed.
 

User experience is key to a winning mobile app. With approximately 52,000 mobile apps released on Android alone every month, it is hard to stand out. Good UX is a must-have, and testing plays a role in shipping that. One single bug creating friction in the process of using the app is more than enough to turn users away.
 

Security & privacy should always be considered. As your app grows in popularity, so does the risk of cyberattacks. Applications for sensitive industries, including Finance, Healthcare, Insurance, or even eCommerce, require an even higher standard for security and privacy.
 

All of the above translates into a competitive advantage. If you are dedicated to making an application that will make storms, you’d better have all of those boxes checked. With rigorous mobile testing in place, it is entirely possible.

3. Types of Mobile Testing

There are many mobile testing types to check off the list:

1. Functional testing: an application is meant to deliver a certain set of features, and the goal of functional testing is to check exactly that. However, note that functional testing goes beyond just “testing the individual features” of the app, but should also be how the data handling works at the backend.
2. UI testing: here we want to ensure that the visual aspects of the application align with whatever feature is currently running. There is always an “expected” version of the UI, which testers compare with the actual version of the UI to see if it is consistent or not. This is why it is also known as visual testing. 
3. Integration testing: features don’t exist in a vacuum. They work together, and integration testing ensures that: seeing if different application components trigger conflicts upon being integrated.

Read More: 15 Different Types of QA Testing
 

There is also performance testing where we check how well a device with certain specifications can handle the app, and of course, security testing to identify vulnerabilities of attacks. QA teams usually define what types they’ll do during the planning phase.
 

After that, we need to choose the approach to execute those tests: either manual or automated.
 

• Manual mobile testing is when a tester manually interacts with the mobile app to perform the test cases, without any assistance from tools or scripts.
• Automated mobile testing is when the tester creates an automated script or leverages a tool to execute the test cases on the mobile device automatically and at scale.

Let’s analyze the pros and cons of each approach

4. Manual Mobile Testing

The Good

• Manual testing is ideal for simple and hobby projects. For the Flappy Bird game, for example, it would be quite overkill to build an entire automation test suite. However, if the application scales and becomes more complex over time, there will be the need to automate some of the more repetitive test cases.
   
• Manual mobile testing truly tests, while automated testing simply checks. When doing manual testing, testers can explore the mobile app with a greater degree of freedom. This allows them to discover new bugs. Meanwhile, with automation testing, testers can only “check” for bugs that they anticipate to potentially be there. Manual mobile testing usually helps testers find more complex bugs.
   
• Manual testing has a lower learning curve. No need for complex coding and technical expertise, all you have to do is download, install, and launch the app to start testing.
   
• Manual mobile testing also has lower maintenance costs. Testers who first adopt automation testing can be disheartened by the effort it takes to update the automation scripts after each app updates. After all, test scripts must reflect the current state of the codebase, or else the results it returns will not be reliable.
   

The Bad

• For complex applications, it is a different story. Let’s say you are testing the Amazon mobile app. There are millions of products and hundreds of functionalities with complex workflows. Manual testing, in this case, is just counterintuitive. Leveraging automation testing, in this case, saves a lot of time and effort.
   
• Manual testing is also more time-consuming. Human testers can only perform one test at a time, and their movement speed has a limit. There is not enough to cover all possible scenarios. This also leads to scalability issues, since it becomes impractical to have hundreds of human testers to take care of the large volumes of test cases.
   
• In the long run, manual testing becomes resource-intensive and outweighs the initial benefits it provides.
  
   

The Best Practices

Of course, it does not mean you should just ditch manual testing altogether. A practical testing strategy is to combine both: have a little bit of manual testing to discover bugs, and automate it once it becomes repetitive.
 

• When you are just testing the app for the first time, you can manually interact with the app to simultaneously learn about it and discover bugs on the fly. This practice is known as exploratory testing.
   
• Prioritize the complex use cases. Manual testing is the ultimate playground where testers are given almost limitless possibilities. This is also an opportunity to check for UX-related issues, such as unintuitive workflows or confusing UI.

5. Automated Mobile Testing

The Good

• Automation testing is the key to shipping mobile apps faster. With the click of a button, you can execute hundreds of test cases, saving so much time and effort compared to manual testing.
   
• Automated testing is more scalable. You don’t have to scale the testing team as much even when your application grows in complexity.
   
• After each application update comes the need for regression testing i.e. executing test cases for old features to see if they are still working as expected. Those are the ideal targets for automation testing.

The Bad

• The biggest challenge of automated mobile testing is the effort to maintain the test scripts. Initially, you save time, but then if you don’t update the scripts to reflect the code changes, eventually they produce wrong test results, which is an even worse scenario.
  
   
• Not everybody knows how to write automated test scripts. You need technical expertise to start automation testing. There will be some upfront investment required.

6. The Mobile Testing Process

Step 1. Choose which test cases to do manually and which to automate

The ideal approach is to blend manual testing with automation testing i.e. a hybrid approach to get the best of both worlds. This phase is known as the Test Planning phase. These are the main criteria to decide if a certain test case is suitable for automation:
 

1. It is frequently executed
2. It requires a high level of accuracy
3. The entire test scenario is data-driven (e.g. inputting 1,000 different queries into a search bar of an eCommerce mobile app to check if the result page serves the intent satisfactorily)
    

It is recommended to have all of those test cases listed in a test case management system. This is where they are categorized, with specific tags assigned to each, and columns dedicated to types of taxonomy, as you can see here in Katalon TestOps
 

Step 2. Choose the suitable framework/tool

When it comes to mobile testing, you have a lot of options to go with:
 

1. The first option usually would be to choose an open-source mobile testing framework. They provide pre-built syntaxes that abstract away the complex underlying coding involved with automation. If you have access to the source code, the best bet usually would be to go with a native framework, with the most common choices including:
   1. Espresso: the native testing framework to write UI tests on Android. As it is optimized for Android testing, testers have so much more control over UI elements during testing (compared to generic frameworks). Its integration with JUnit within Android Studio simplifies test execution for developers, as tests reside within the same project as the application code.
   2. XCUItest: this is the native iOS testing framework, and you get the same benefits: faster testing, access to helpful libraries and easy to integrate into CI/CD projects.
   3. Appium: a comprehensive choice would be Appium, which is built on Espresso and XCUItest. If you want to test on both OS, Appium can be a good choice to minimize duplicated work, but there will be some sacrifice in terms of speed and efficiency.
       
2. Another great option would be to go with an automated mobile testing tool. What makes it different from a framework like Espresso or XCUItest? It is about convenience. A tool can simplify the test creation process to a point-and-click mechanism, while still leaving the choice of scripting open for any testers wanting customization. CI/CD integration, cross-browser execution, and other features are also incorporated.
    

Put simply, go with a framework if you have a team of experienced devs and developers. Having a tool is beneficial for teams of any technical expertise. If you’re just switching from manual to automated testing, the simplicity that comes with a tool can speed up the process a lot. If you have an experienced team, a tool boosts efficiency and allows your team to get the best of both worlds.
 

Read More: Top 10 Mobile Testing Tools Your Team Will Love
 

Step 3. Choose your testing environment

1. Test on real mobile devices with a dedicated test environment for your application. You can both perform manual tests and execute automated scripts on these devices and receive accurate results. However, the resources needed to buy and maintain these devices can grow to be astronomical.
    
2. A good alternative would be to go with emulators/simulators. They use virtualization technology to mimic the hardware/software configuration of the mobile device. They are easy to set up and launch.
    
3. QA teams can also use an on-cloud device farm where they can test the mobile apps remotely on real devices. It solves the issue of cross-device testing while providing a high degree of realism compared to emulators/simulators. It is truly powerful to have a TestCloud delivering on-demand mobile testing environments.

Step 4. Think about test case management (TCM)
 

Done right, TCM does wonders for the entire project.
 

If you went with the manual approach, make sure to have a test case management system in place. A simple spreadsheet works well for a small team. For more advanced needs, you may want a ticket-based task management system like Jira or a dedicated test management tool like TestRail.
 

Such systems help QA teams track what test cases are being executed, note down their results, and from that generate detailed reports. The downside? It makes test case management a siloed process, without any connection stages to other stages.
 

If you go with the automated approach, you can incorporate test case management into the testing process. This means from the very moment you write a test case, its taxonomy should have been simultaneously updated in a dedicated test case management system. This system should also update test results after each successful execution, creating a streamlined process.
 

Read More: A Complete Guide To Test Case Management

Step 5. Write your test cases
 

This is when the real work begins.
 

If you choose to use a framework, read through the documentation and familiarize yourself with the syntax. Here are the Espresso docs and the XCUItest doc.
 

Let’s talk about Espresso. The power is its rich set of view matchers and actions to interact with UI elements. Essentially these view matchers allow developers to locate UI elements based on their attributes. There are also “Actions” to help testers simulate actions like click, text, swipes, etc.
 

Here you can see we use “ViewMatchers.withId” to choose the Username field, then perform the Action of typeText and to which we assign the “username” value. After that, we once again use the ViewMatchers to find the Login button ID and perform a Click action. It is really simple and straightforward.
 

 @Test
    public void testLoginSuccess() {
        // Type username and password
        Espresso.onView(ViewMatchers.withId(R.id.editTextUsername)).perform(ViewActions.typeText("username"));
        Espresso.onView(ViewMatchers.withId(R.id.editTextPassword)).perform(ViewActions.typeText("password"));
 
        // Click on the login button
        Espresso.onView(ViewMatchers.withId(R.id.buttonLogin)).perform(ViewActions.click());

 

Let’s see how that’s done with a tool. In Katalon Studio, the process is even simpler. To start, let’s download it.

Download and Witness The Power of Katalon  
 

Launch it, and go to File > New > Project to create your first test project. The cool thing about testing in Katalon Studio is that you can test anything, from Web and API, to mobile apps and desktop apps. There are keywords ready for you to use for any AUT. Let’s choose Mobile since we are doing Mobile Testing.
 

Once you have your project created, it is time to create a new Test Case by clicking on the “+” button on the navigation bar.
 

You now have hundreds of keywords to simulate mobile interactions.

The mechanism is similar to that of Espresso and XCUITest, but you don’t have to write as much code. You have the list of keywords ready for you to choose from, and you can set the values for them as if you’re filling in a form. Talking about simplicity!
 

Even better, you don’t need to write anything. Just activate Recorder mode, choose your device and start interacting with the system as you would like a manual tester. Katalon records all of your actions and turns them into a functional test script, without you having to code anything.
 

Step 6. Execute and create reports

Once you have created the tests, it is time to execute. On the navigation bar, there is the “Run” button, and you can see the list of options available. For Android & iOS specifically, you have to configure the SDK with Katalon to gain access to its core functionality.

You can see Katalon Studio in action here:

7. Mobile Testing Best Practices

• Set out a clear mobile testing strategy and define objectives before performing the actual tests.
• Carry out UI tests on real mobile devices, not just on emulators or simulators.
• Make sure tests are applied in the initial phase as well as progressively throughout the development process.
• Prioritize testing on certain devices and operating systems as it is impractical to test on all devices, OSs and network combinations.
• Execute performance, stress, and security testing sufficiently.
• Cover the app interoperability, battery consumption, and fault tolerance with suitable tests.

Want more best practices? Watch a webinar from our Product Support specialists filled with insightful tips for mobile testing & web testing here:

 Your Mobile Testing Journey Starts Here