In the following post, we’ll answer a few key questions: What is beta testing? What’s the difference between beta testing and alpha testing? How can startups beta test successfully?
What Is Beta Testing?
Beta Testing is the last phase of tests your web or mobile app goes through before it gets into the hands of your potential users.
To put it another way, it’s a final opportunity for you as a startup founder or product manager to catch bugs and improve the UX (user experience) before your app is out in the wild.
Beta Testing vs Alpha Testing
Alpha Testing is considered a form of “internal acceptance testing” conducted in-house by developers and testers.
Unlike the alpha phase, the focus of beta testing is not on designers or developers who were involved in creating the app, but instead on consumers or businesses that are the target user base for the product.
Some projects do invite external users as early as the alpha phase, but it’s generally in a more limited capacity than during the beta phase.
The simulated environment of alpha tests can’t account for all combinations of conditions, whether they be network conditions, device variations (screen sizes/resolutions, different device or OS versions), or use cases not thought of during in-house testing.
It’s not just the concept but also the execution that will ultimately lead to a great product, and beta testing puts stress on your product differently than the in-house testing of the alpha phase.
Alpha testing has traditionally been said to be more rigorous and systematic, with a reliance on continuous integration tools. But that doesn’t necessarily mean that beta testing should be relaxed.
When the product graduates to the beta testing phase, continuing the team’s systematic and controlled strategy is just as vital as it was during the alpha.
What Are the Benefits of Beta Testing?
Beta testing provides users early access to a new app (or to new features of an existing app). The process, in turn, provides startup founders and product managers with the opportunity to iron out any problems that weren’t caught during the alpha phase.
Crowdsourcing your testing to hundreds or even thousands of real users in real environmental conditions allows for a greater net to catch any glitches or edge-cases that may not have been spotted previously.
Identifying and fixing those issues during the beta phase can reduce your development costs significantly, in addition to helping you avoid a potential PR crisis associated with a bad public launch.
Beta testing also provides customer validation beyond technical glitches, providing startup founders and product managers with feedback and analytics as to whether to include specific features.
How to Create a Successful Beta Test
Constructing a meaningful, strategic beta test involves a few components, including scoping out targeted testing devices, choosing the right segment of users to test your product, deciding which distribution model to utilize, and figuring out how to track and analyze your testers.
Let’s look at each factor individually.
Choosing Your Targeted Platforms
Before choosing your testers, you first need to define the range of platforms that you will include in your beta test. This is usually guided by the technical specifications you and your team set out at the start of the development phase.
For mobile apps, the variation of platforms will include:
- Device iterations (e.g. iPhone 6S Plus, iPhone 6S, iPhone SE)
- Operating System (OS) iterations (e.g. iOS 9.3.2, iOS 8.4.1)
On Android, we already have dozens of different devices and screen sizes, as well as processor speeds. And with the introduction of the 4-inch, 4.7-inch and 5.5-inch iPhones, the app ecosystem has become even more size-agnostic.
Indeed, there has never been a greater period of device fragmentation as exists today.
The sheer range of combinations means it’s generally not feasible to dedicate enough time and resources to test all of the variants. But, if you approach beta testing as an iterative process rather than a one-off, when newer testers come on board, your sample size will increase and new insights will be discovered.
If you already have existing users, you can match your device test sample proportionally to the distribution of devices and OS versions they’re currently using.
A Note Regarding iOS Apps
Apple’s best practices recommend that you design your apps for all available screen sizes. Therefore, it’s a wise approach to include in your test all devices released during the last two years that are capable of running the targeted operating systems.
Keep in mind that Apple does not allow developers to preclude devices from installing an app based on hardware, but only on software (i.e. the operating system).
Selecting Your Beta Testers
After deciding which devices and operating system versions you want to support, you should ensure your pool of beta testing candidates matches those platforms.
It’s important to note, however, that choosing your beta testers should not be based solely on whether they have the compatible technology, but also how keen they are to use your software.
That is, there is no point finding beta testers who don’t have a genuine personal interest in using your app. They should be(come) your app advocates, people who are excited to get their hands on your app before it’s available to the general public.
A/B Testing
You should arrange your testers into groups categorized by operating system versions and devices (such as iOS 9/iPhone 6S Plus vs iOS 9/iPhone 6S vs iOS 8/iPhone 6S, etc.), which will enable you to conduct A/B testing.
A/B Testing, also known as Split Testing, is a methodology of comparing two versions of the same product or service against one another in order to validate certain UX/design elements and test specific features.
Depending on the testing sample size, this can prove to be a powerful tool to test your hypotheses and ensure that you launch the best possible version of the app. This in turn will help with downloads, ratings, user retention, etc.
How to Find Beta Testers
If you have existing users already, you can send out individual invites by finding the specific users who not only match the platforms you want to test, but are also avid users of your app.
As an interesting experiment, you might also want to find a sample of users who downloaded your app, but aren’t active users. This would allow you to test if your new features or other updates would change how they feel about the app.
If you’re working on a new app launch and don’t have prior users, you can seek out beta testers using a variety of services.
UserTesting provides a paid portal for accessing users, testing your app, and receiving analytics, as well as written and video feedback from real users.
Another service is ErliBird, which delivers qualitative feedback on your beta distribution campaigns, allowing you to recruit targeted participants from within their community.
Distributing Your Beta
There is a range of software available for both iOS and Android to help you distribute the beta version of your app to your beta testers and their devices.
TestFlight
TestFlight, which got acquired by Apple, has been integrated into the Xcode platform to make the task of inviting external users to test your app — something was historically rather tedious — a lot easier.
By simply including an email address, developers are now able to invite up to 2,000 external beta testers.
Additionally, up to 25 internal members of your team can be included in the internal beta testing phase, each of whom can test the app on up to 10 devices.
The testers will receive an email prompting them to join your beta testing group, as well as an invitation to install the free TestFlight mobile app.
Each time you upload a new app version to iTunes Connect, the registered testers will automatically be notified of new versions, as well information on what to test.
For each beta version you release, the TestFlight period lasts for 30 days, and the number of days remaining is indicated in the TestFlight app for each app that the user is testing.
Users will not be charged for any in-app purchases made during the beta period, which are automatically free.
One last note: apps using TestFlight for external testing must go through Apple’s Beta App Review process, which is less strict than the full app review process that takes place when you launch your app to the App Store.
Nevertheless, a review is needed for each new version of the app made available to the public.
HockeyApp
Another prominent beta distribution tool for iOS is HockeyApp.
Unfortunately it’s not free. But it does provide a lot of automative tools that TestFlight lacks, including direct integration with GitHub for continuous integration (CI) testing.
Google Play Developer Console
On the Android side, the Google Play Developer Console does a great job, allowing not just distribution to your beta testing groups, but also alpha testing.
After opting in to participate, your testers will receive a link to subscribe and will receive future alpha or beta release versions automatically.
Interestingly enough, you can also do what is called Staged Rollouts, where you can control the number of users who download your app. This allows you to implement a gradual scaling, while measuring the impact on your servers and gauging user feedback.
Studying Your Beta Testers
Once you have successfully distributed your beta version to your testing groups, you need to ensure you have the correct mechanisms to acquire and analyze feedback from your testers.
Explicit Feedback
Explicit Feedback refers to the mechanism by which your users can provide feedback via email or a message within the app.
It’s important that users are encouraged to provide feedback on how easy and intuitive the app is to use, as well as any weird quirks or bugs they have discovered.
Using Apple’s TestFlight, prior to distribution of the beta, developers must submit a specific set of instructions explaining what to cover during testing. Users would then use the TestFlight mobile app to send feedback.
While this mechanism is okay, it is important that more sophisticated feedback methods are also implemented, such as having a button directly within the app or perhaps a feedback form that pops up when users shake the device.
Implicit Feedback
Implicit Feedback refers to the use of analytics and crash reports to study beta testers.
Crashlytics by Twitter
In addition to providing a method for explicit feedback, it is also useful to observe app crashes through crash logs. One prominent service that can help with this is Crashlytics, which is now part of Twitter’s suite of Fabric tools.
(Another tool in the Fabric suite is Beta, which makes distribution on both iOS and Android easy and intuitive.)
Crashlytics provides more than just a stack trace, but also a filtered, clean and easy-to-read crash item that developers can follow.
Crashlytics aggregates crashes by source issue — particular lines of code that seem to be causing the same type of crash.
If, for example, you had 30,000 crashes reported, these might be attributable to just 20 different source issues. Crashlytics will identify those 20 and assign them a level of severity called an impact level.
Crash Reports by Apple
Apple also provides crash reporting via an Xcode-integrated crash reporting tool called Crash Reports.
This service allows Xcode to provide technical insights into crashes, including the conditions and environment under which the app terminated and stack information, which help developers hone in on exactly where and why the crash occurred.
In terms of usability testing, implicit feedback falls into the space of analytics.
The good news is that there is no shortage of tools for both iOS and Android that cater to analytics. And while analytics is more prominent during public releases, they can serve just as well during the beta phase.
Parse
Parse Analytics goes a bit further than Crashlytics and allows for custom event tracking.
That is, you can send specific, custom calls to track when certain screens were accessed or certain buttons were pressed, as well as duration spent on a specific screen.
Google Analytics for iOS
Google Analytics for iOS is an alternative measuring tool that provides crash reporting, session tracking, and custom event tracking.
Regardless of which service you utilize, metrics that track funnels of user action are important as they allow startup founders and product managers to understand whether certain screens and functionalities are intuitive, over-used, or under-used.
Conclusion
Beta testing is a quintessential aspect of the product development cycle.
A sound external testing strategy will ultimately mean the difference between gaining meaningful insights and capturing glitches missed during the internal alpha testing, and a poorly-executed launch of your app or new feature that may cost you the confidence of potential and/or existing users.
A successful beta program requires paying attention to various components, from the scope of targeted platforms to selecting testers who are diverse and will take the initiative to find edge-cases within your app.
In this article, we’ve also identified different tools that you can use to distribute your app and to solicit both explicit and implicit feedback from beta testers.
After all, beta testing won’t have any meaning unless your developers can get precise information on crashes and glitches and designers/product managers can gain insight into which screens are confusing, under-used, or over-used.
Happy testing! 🙂
