Photography: Science Museum London
Before an advertising company takes a client’s millions and rolls out its new campaign, it first tests its ideas on a focus group. A small number of people are pulled into a room and, depending on the product and the campaign, used as guinea pigs to discover whether an idea is likely to fly. It’s not an exact science. Focus group members are paid, so biased, they’re not always entirely reliable and when they do have an honest opinion to contribute, that opinion might not be one worth hearing. As carmaker Henry Ford famously said, if he had asked people what they wanted, they would have requested a faster horse. That doesn’t mean that focus groups are useless. But it does mean that there are lots of other tools that you can stuff into an ideas testing lab to make sure that your concepts are sound before you throw money at them.
Perhaps the most used test today is Beta testing, a kind of in-market testing in which a product is released to a limited number of people to play with before general release. In theory, they should come back with usable feedback on the concept and usability, and make suggestions for improvements before general release.
But Beta testing is a late-stage test. It’s the last step before release, when plenty of money and time has already been invested in creating a working prototype. A Beta test in which the most common response is that users didn’t get the idea is one in which something has gone very wrong indeed. It’s also little different to a kind of focus group but one in which users actually get their hands on the product and are asked what they did with it rather than being asked in general what they think of it.
And like any focus group, the most reliable results come from the largest numbers of people which means that Beta releases — like Google’s Gmail — are often used by huge numbers of people. In effect, a Beta test becomes a way of declaring that a release is cautious. It’s often less of a test and more of an excuse for the presence of bugs in a system that its creators know has been released too early.
Your Choice of Two Sales Letters
A more reliable kind of test, and one that’s also easier to easiest to implement online, is the A/B test. Create two Web pages, switch traffic between them at random, and compare the results when users reach the call to action.
The advantage is that you’re getting a true picture of actual responses. You’re not asking people what they think and wondering whether they’re telling you what they think you want to hear. Users aren’t even aware that they’re being tested so you can be certain they’re responding entirely to the options you’ve placed in front of them.
But A/B tests only test as small number of options (usually two) and those options tend to be marketing copy. Offer the same product on two different sales letters written in two different ways, and you’ll be able to count up the number of orders to see which is the most effective. Offer two different products or pitch two different ideas and you’ll need two different pieces of marketing copy. At that point, you can’t be certain whether it’s the copy or the concept that’s influencing the responses.
What you can do though is use the same concept to launch a simple pre-release sales pitch. That doesn’t have to be more than a single page that invites users to leave their email addresses so that they can be notified when the product is ready. Again, it’s not entirely accurate. More people are willing to hand over an email address than hand out their credit card details, but if you have the results of other pre-release pitches you’ll be able to compare results and perhaps even predict final conversion rates if you decide to go ahead and create the product.
Ask the Crowd
All of those tests though take time, and sometimes you want to know right away whether a concept that has you excited really is as good as it feels. You want to ask as many people as possible what they think, whether you should go ahead with it and what the response is likely to be. Social media can be a good place to bounce ideas around. Bad Panda Records, for example, a netlabel that releases regular free songs, recently posted this tweet on its timeline:
thinking of starting to release full length albums. in a new way: free download+buy merchandise and get the cd for free. what do you think?
That was a test of a marketing idea rather than a product idea, and the response wasn’t huge. But as a gentle form of encouragement for a concept that doesn’t cost much to develop and carries few risks, it’s not a bad way to begin. You might not want to try it before throwing millions into production or sharing an idea that others can pick and run with faster than you can but your fans and followers can sometimes make for useful sounding boards.
The best tests though combine all of these approaches. Twitter itself is a useful example of a firm that has grown cautiously, testing the ground before even the tiniest of steps. The original service was used in-house before opening up to a small group of users. It had no features beyond the most basic needed to get the job done, which would have taken time and money to develop, allowing the users themselves to build what was needed. And each development stage, whether it’s sponsored search results, a new retweet method or a promoted trending topic, is first rolled out in a real closed Beta to a small number of trusted users before being made available to the public — a system also used by Google before the company released its instant search results.
But none of these methods is entirely reliable. Every test has its limitations. Each test takes time that allows competitors to reach the marker earlier, and every test skews the results. At best, testing will reduce the risk of failure but the only way to know for sure whether a product is going to fail is to release it.