Thoughts on More Effective Beta Testing

I recently ran a successful two cycle beta test for a greenfield mobile application.  After the beta test, the app was successfully released to a national sales organization.  The steps I took to run the beta test are based on what I consider common sense.  However, I was recently on the other side as a beta tester for a major streaming service provider and I was reminded that “common sense is not so common”.

From my perspective, this “beta” test was so poorly run that I pretty quickly gave up on attempting to help them.

Hopefully this article will give you some ideas to improve your next beta test.  Most of these suggestions can be summed up as:

• Be respectful
• Be efficient

Beta is not QA, Beta is not Alpha

Beta testing is not a substitute for proper QA testing.  If you release your code with a lot of obvious bugs, you are going to be overwhelmed with bug reports or you may end up losing your beta testers out of frustration.   I’m still getting emails about the “beta” test for the streaming provider and it is still going on three months later!

Properly Handle Unimplemented Features Correctly

You really shouldn’t have features that are not yet implemented visible at this point in development, but if you must, make sure there is a positive indication that you know the feature is not yet implemented.  Use a message such as “This feature has not been implemented”, instead of letting a generic catch-all error message like “There was an error processing this request”.

Keep in mind too that if a feature is visible you are creating an implied contract with the beta users that this feature will be available when this release becomes generally available.

Pick Your Participants

If your app is an internal line of business system, there’s a good chance that you know your users and there may also be multiple user roles.  Resist the urge to pick your most technically savvy “power users”.  Instead, pick users who understand the business and who have influence in the organization and will be champions for adoption of your application by other users.  Also, if the app has multiple role support, pick at least one user for every supported role

If you let beta users sign themselves up to test, be prepared to trim the tester pool on each cycle.

Use Custom Support Channel

There are several reasons you should not use your normal support channel to receive beta responses:

• Beta testers are working for you for free so you should respond to them very quickly
• You need to determine which participants are not giving you feedback

Make Reporting Issues Easy

Give your testers multiple ways to give you feedback in case they have trouble with the primary feedback method.  If possible create a throw-away beta test email such as “BetaTest123@yourcompany.com”  so they can send you reports if other feedback methods don’t work.  Make sure you can tie the feedback from each channel back to the specific user.

Value of a Picture

You absolutely must have a way for testers to easily submit screen captures.  Often during beta testing many of the issues are with quirks in how the user interface looks, such as display of names that are longer than ones used in testing.  Even if it’s not a UI formatting issue, screen captures have a way of giving a developer subtitle clues to what caused the problem.

Be Agile!

Frequent bug fixes and releases during Beta will ultimately make for a more successful test because:

• You will spend less time dealing with duplicate bug reports
• Your testers will see progress and they will be more likely to stay engaged
• Existing bugs may keep users from testing parts of the app

Engage with the Disengaged

I’m a huge fan of process improvement, having been a willing adopter of Continuous Quality Improvement (CQI) in the 1990s.  In the spirit of CQI, monitor which beta users are testing and communicate with the ones that aren’t to figure out why.  There may be something that you didn’t think of keeping them from testing.  For example, in the beta test where I was a tester, I could use the mobile app but I was never able to log into the web application.  Because they were using their regular support department, they were unable to help me get logged in and I was never able to test the web app.

Respect the Engaged

Keep the testers that are giving you feedback fired up by giving them feedback in return:

• Acknowledge receipt of their report
• If possible give them a direct response to their report when the issue is fixed
• If you can’t respond directly to them, make sure you are providing release notes for each update to all of your testers

Adapt, Improvise, Overcome

If you are doing a multiple cycle beta test consider adjusting your strategy before each new cycle.  If you have a large group of beta testers a new cycle may be the time to thank the less active testers and remove them from the next cycle.  This might allow you to have more one-on-one communication with your star testers during the next cycle.

Avoid Scope Creep

This is the enemy of any person responsible for meeting deadlines.  You absolutely should not be adding new features during Beta testing because by doing so you effectively have bypassed all of the software development life cycle (SDLC) steps before beta.  If you don’t understand why that’s a bad thing, do a web search for “SDLC” and do a bit of research.

Reward your Star Testers

Remember that your testers are working for you for free and you should figure out a way to reward them.  If a beta tester asks for a new feature, one of the best ways to reward them and to keep them engaged in making your product better is by adding that feature in the next release.  If you are testing an internal application, recognition of your stars might be a good motivator for future participation in beta testing.