Every now and then, it’s great to deep dive with all of our data, to gather and analyze the most interesting aspects of how people use their mobile test automation environments and frameworks.
Here are some facts and numbers from our public cloud during the last year, particularly when it comes to used test automation frameworks with it.
Test Automation and Unlimited Concurrency Are the Enablers
First of all, getting a huge number of test runs on device farm is not possible without test automation. Test automation is the enabler that makes concurrent use of mobile devices possible and enables users to run their app and tests simultaneously on any number of devices. This is also one of our sweet spots as we haven’t ever limited the number of devices that users can use simultaneously for their tests.
It’s also great to see that native app and game developers use the same device roster — yes, slightly differently, and test scripts and used frameworks are different — but the same foundation serves both user groups extremely well.
Especially during the past two years, test automation has come into play and helped mobile game developers to quickly and easily see how their games work across all possible device variants and get that vital debugging information on which things should be fixed (before releasing). This has produced tons of great data of how game data can be optimized for end-user devices.
Overview of Test Runs in Bitbar Public Cloud
During the year 2016, we provided the most diverse (devices all around the world, with OS combinations, etc.) and the largest mobile device farm for Android and iOS app, game, and web developers. Among this group of users, there are Fortune 500 companies, a dozen of the top 20 mobile game developer companies, as well as lots of SMBs and indie developers.
Their mobile test runs hammered thousands of our devices every day and produced an enormous amount of data, results, screenshots, recorded videos out of devices runs, and performance statistics on how their mobile apps/games work on these different handsets.
The total number of test runs exceeded 184 million unique device runs. The majority of those were done on Android devices, but with a significant growing trend with iOS devices (quarter by quarter). This yields that fragmentation is far from beeing solved on both major mobile platforms — and testing must take its place before top companies want to push their apps and games for consumers.
From the data, close to 84% of test runs were done on Android devices (and less than 16% on iOS devices) which provided just a bit higher failure rate (17.8%) than on iOS devices (15.5%). This sort of finding states that there are lots of issues also on iOS devices, with an increasing trend. Mainly, this is because of new iOS versions and some new form factors with iOS devices (plus new APIs, new notification system etc).
What’s really awesome to see is that concurrency has been going up quarter by quarter. The average number of used mobile devices per test run is now over 10 devices per a test run. However, there is still lots of outliers and variation here as many of users still use one-device-at-time and some test automation experts use tens (or sometimes even hundreds) of devices simultaneously for their test runs.
The average test time also varies a lot. For example, quick and fast Espresso tests typically complete quickly and some extensive, thorough Appium test runs may take 10x time. Naturally, it’s all about the scope of testing and typically more logic is included in those Appium tests (those Appium runs cover functional aspects better than unit-like tests with Espresso).
The Most Popular Test Automation Frameworks
Appium has been a framework for functional testing of mobile apps, games and, to a certain extent, mobile web as well (we didn’t include Selenium in these numbers as a majority of all those Selenium tweaks are more or less Appium-based). There are lots of good reasons why people have been using Appium (cross-platform, support for literally any programming language, great coverage with API, etc.) and frankly, it has worked extremely well until recent changes with iOS UI automation.
On Android, Espresso and UIAutomator have been very popular, as well. There are good reasons why people use and love these frameworks. Espresso provides very fast test execution and UIAutomator provides a light-weight API that is easy to adopt and use for your native apps. Both of these frameworks, however, are somewhat limited to native apps only. Again, the majority of game developers either use Appium or some internal grown and developed framework.
Another (promising) new framework that has come into play is the Robot Framework. It quickly took over the Robotium, which used to be one of the mainstream frameworks in the early days of Android test automation.
On iOS Appium and Calabash, both being cross-platform frameworks, have been popular choices for iOS test automation. As many game developers have been using Appium, it’s also obvious from the data that usage has been on both Android and iOS.
Another highly used framework on iOS has been UI Automation (until the last quarter of 2016). Since that there has been (and will be) a clear replacement movement to some other framework and I’ll cover this with a bit more details in the next chapter.
On both platforms, there are also lots of other frameworks that didn’t get listed here. Some of those are internal, proprietary or just niche frameworks that their users happen to use. As stated, there is no wrong choice for a framework if it does the job efficiently and provides exact results on how apps, games and web stuff work on real mobile devices.
New Trends With Mobile Test Automation Frameworks
Okay, let’s speculate a bit.
Currently, we don’t see any major changes taking place in Android test automation. Appium, Calabash, and Espresso will be definitely in high use and probably even grow stronger, as Google tends to favor Espresso — and there are lots of Appium fans out there, as well. Calabash hasn’t provided much new to the ecosystem lately, but one of its strengths is that “if you speak English, you can write a mobile test.”
The most significant changes are happening on iOS side of things. As UI automation got deprecated by Apple, XCTest and XCUITest got an excellent start to be the next big frameworks in iOS test automation. However, the change by Apple hit Appium, as it was relying on UI automation as its foundation.
Now, the big question is that what happens to Appium with UI automation deprecation. If things start to work well with Appium, there is a good chance that many UI automation users will go to Appium (instead of XCTest or XCUITest, which we see going on now). It’s always worth it to consider which framework (i.e., XCTest vs. Appium) could provide a more stable environment for test automation runs for the next years.
If you are looking some guidance on what’s going on with Appium (and how to use the latest version of it efficiently), take a look at this awesome post by my colleague.
Alright — let’s speculate a bit more. Let us know in the comment section below which frameworks will fall, which will flourish, and why!