Founder & President, Infosurv, Inc.
In 2003, an article titled The One Number You Need to Grow was published by Frederick Reichheld of Bain & Company – an article that eventually revolutionized the customer loyalty measurement industry. The Net Promoter Score (NPS) proposed by Reichheld in this article captivated business executives around the world with its simplicity and predictive power. “That number,” wrote Richheld, “is the one number you need to grow. It's that simple and that profound.”
Regardless of what one thinks about the NPS methodology, there is no arguing that it led more companies to adopt a customer loyalty measurement process than ever before.
After months of painstaking research and development, our firm has developed and filed patent on an innovative new online concept testing system that we believe will similarly revolutionize the market research industry. This methodology allows market researchers of all stripes, novice and sophisticated, to measure the future success of new product, logo, packaging, and advertising concepts by looking at a single number. Further, they are able to do so more accurately, less expensively, and much more quickly than ever before.
This system was inspired by the best-selling book The Wisdom of Crowds by James Surowiecki. In this book, the author recounts the amazing power of markets to make accurate predictions about the future. In a market, the wisdom of a diverse, decentralized, independent crowd is aggregated into one enormously important number: a price.
Before we discuss how market prices can be applied to online concept testing, let’s take a moment to appreciate how much information is reflected in a price. We’ll start with a price we’re all too familiar with – the price of gas.
Pondering at the Pump
Imagine that you’re standing at the pump of your local gas stations and notice that a gallon of unleaded has increased in price by $0.10 since yesterday. Why did this happen?
It could be that a violent storm off the Gulf Coast has shut down a major oil refinery, or even that oil companies fear such a storm and are hoarding oil reserves in preparation. It could be that an oil pipeline was harmed in an armed conflict, causing the transportation of crude from a Russian oil field to a Western European refinery to be delayed, negatively impacting world oil supply. Perhaps it was something more local, like one of the gas stations in your area going out of business, causing less competition for the remaining stations.
Then again, it could have been something on the demand side of the equation. Perhaps a larger-than-usual proportion of car buyers have taken an affinity to luxury SUV’s, driving up demand for premium unleaded. Perhaps a cold front moving through the Northeast has caused a temporary shortage in heating oil. Perhaps an oil speculator in New York simply thinks one of these outcomes will occur.
Of course, that $0.10 price increase may have been caused by something seemingly unrelated, like the election of a new governor in West Virginia. Perhaps the new governor is more environmentally conscious than his predecessor, and has decided to tighten environmental regulation on the coal mining industry. This has meant increased compliance costs for coal miners, which they must pass along to their customers in higher prices. This has prompted many of the coal-fired power plants across the country to switch over to petroleum-burning instead, thus driving up oil demand enough to cost you another $0.10 at the pump.
The point is that many variables, perhaps thousands of them, can influence that one number – the price of gas. Though you can’t easily see why the price of premium unleaded has jumped by a dime, you can rest assured that the knowledge and experience of countless individuals, including oil miners, refiners, distributors, speculators and buyers, is reflected in that price.
When Prices Predict the Future
We can easily see how much information is packed within one number -- a price. But what do prices tell us about the future? In the case of gas prices, they don’t tell us much except to the degree that oil speculators affect current prices. Investment prices, however, are another story.
Investors are by nature focused more heavily on the future than the present. Every time that we sink our hard-earned money in an investment asset, we’re publically proclaiming our opinion regarding the future value of that asset: it will be more tomorrow that it is today.
As a group, investors are quite good at making such predictions and markets are remarkable efficient at aggregating them. Reams of academic literature on public markets show that the value of investment assets, usually expressed in share prices, is an excellent predictor of the future worth of the underlying asset.
For example, examining the long-term average price of a share in Google is a great way to predict the future worth of Google. As new information reaches the market about Google, its products and its competitors, it doesn’t take very long for that information to be reflected in Google’s share price. Though the price can fluctuate quite a bit day-to-day depending on investor mood, in the long-term these daily price fluctuations are averaged out and the true value of Google shines through.
The same phenomenon applies to long-term bond prices, commodity prices, housing prices, and just about every other asset under the sun. The famous investor Benjamin Graham once said, “In the short-term, the market is a voting machine, while in the long-term it's a weighing machine.” Turns out it’s a very accurate weighing machine indeed.
Our Client Investors
Wall Street traders and private investors aren’t the only people concerned with the future performance of investments. In the marketing research world, our clients are also investors. Instead of investing in stocks and bonds, however, they invest in product concepts, logos, packages, and advertising campaigns. As with any investor, our clients care deeply about the future performance of their investments. Much is at stake – money, reputation, career path, and sometimes job security.
As researchers we often don’t notice what monadic concept tests and stock markets have in common. However, they serve the same purpose: to figure out how much stuff is worth.
Markets for the Future
When we think about markets, we tend to focus on those in which tangible assets like companies, commodities, houses, or used cars are up for trade. However, intangible items can also be traded on a market – including a wide variety of future outcomes.
In recent years there have been a number of highly innovative markets created with the purpose of predicting future outcomes. One of the most famous amongst them is the Iowa Electronic Market, sponsored by the University of Iowa, which was setup to predict election outcomes. It has been proven to out-predict Gallup polls regarding presidential elections 75% of the time.
As you’ll see in the graph to the right, the Iowa Electronic Market shows uncanny accuracy in predicting election outcomes, especially as the election date approaches. This is despite the fact that anyone can trade on this market without prerequisite.
A lesser known example is the Hollywood Stock Exchange (HSX) prediction market, which allows anyone to wager play money on who wins the Academy Awards. In 2000, a group of Wall Street Journal reporters went head to head against the predictions of the HSX when they interviewed 356 Academy judges on their vote before the awards were announced. The WSJ accurately predicted 5 out of 6 awards – not too bad. However, the HSX prediction was 100% accurate with 6 out of 6 predicted correctly, once again relying entirely on novice investors.
Google has tested a prediction market internally, using Google employees as unpaid traders, to predict future business outcomes like how many Gmail subscribers there will be next year or when Google will open an office in Russia. As you’ll see in the graph above, the market prices for outcomes are extremely strong predictors of the probability of the event actually occurring. The black line represents perfect predictive validity, the red line actual market performance, and the blue line a regression equation obtained via OLS.
Appling Markets to Market Research
We’ve seen how much information is contained within a price, how predictive investment prices are of the future, and how many of the things marketing researchers are hired to measure are essentially investments. If markets are so good at predicting the future value of investments, why haven’t market researchers tried to harness that power?
We scratched our heads at this question and set out to design our own predictive market, with the sole purpose of helping market research clients predict the future success of a wide variety of concepts.
The result of our efforts is the patent-pending Infosurv Concept Exchange, also known as iCE. This innovative, high-tech system presents a new and highly accurate way to predict the success of new products, packages, logos, advertisements, and anything else traditionally measured with monadic concept tests.
iCE is an online market where respondents are given virtual dollars to buy “shares” in the concepts that our clients wish to test. Just like in real world markets, share prices fluctuate based on the perceived market value of the underlying asset. By observing how share prices move over time, we can make very accurate predictions regarding the real world success of the concepts being tested.
Due to various advantages markets provide over traditional concept tests, iCE delivers not only more accurate predictions, but various other benefits as well:
Cost savings – Because iCE can recruit respondents and win their engagement more easily than traditional market research panels, our research costs are much lower. We can also test far more concepts simultaneously on iCE than a traditional concept test, so the cost savings are especially dramatic on a per concept basis.
Speed – The typical time required to design, field, and analyze a traditional concept test is 4-6 weeks.
Turn-around time on iCE concept tests is much less – just 2 weeks from project kick-off to presentation of final results. iCE projects are completed more quickly due to ease of setup, respondent recruitment, and data analysis.
Better respondent engagement – One of the chief concerns in the market research industry today is
respondent engagement. Since surveys are often long and uninteresting, respondents have the tendency to rush through just to collect their incentive. Our respondents, on the other hand, view participating in our market as an entertaining game with both intrinsic and extrinsic rewards. Since respondents can buy real world prizes with the iCE dollars they’ve won, they have ample motivation to remain engaged in the process.
No risk of misrepresented or “professional” survey takers – Other major concerns in the market
research industry today are respondents who misrepresent themselves to qualify for a survey, and
“professional” survey respondents who participate in lots of survey panels and have a disproportionate voice in samples. iCE avoids both of these issues. Since iCE respondents never have to qualify for a particular study based on demographic or behavioral criteria, they have no reason to misrepresent themselves. In addition, “professional” respondents are not a concern because if some iCE participants are disproportionately active in our markets, the validity of the market’s conclusions is only enhanced.
We’re still in the process of validating iCE across a wide range of client industries and concept types, by studying how closely iCE market results reflect the real world performance of the various concepts tested. So far, it’s looking very promising.
If you’d like to learn more about iCE or participate in an iCE validation study free of charge, just send me an email at
. To qualify for a validation study, you must be a market research end user with either 1) historical sales data or 2) concept test results data, on competing consumer product or marketing concepts .
Once market researchers recognize how incredibly predictive the market price of a future concept can be, we believe that this one number will accomplish for market research concept testing what NPS has accomplished for customer loyalty measurement. It will also bring cost savings, greater speed, better respondent engagement, and many other advantages that the online concept testing industry has been demanding for years.