Your Market is Smaller Than You Think: The Law of Small Numbers

"The Law of Small Numbers" is our intuition that we can pick a small sample and confidently extrapolate from it. You and your best friend like paintball and chihuahuas, yet the paintball/chihuahua crossover demographic is completely ignored by big business. Surely you two can't be the only ones who would enjoy combining the experiences; you can extrapolate that there must be many others who would pay for the ability to roam about in camo as you deploy an army of small dogs to hunt down and sink their teeth into the enemy's padded ankles. So you open the world's first Chihuahuaball franchise, and are puzzled when the throngs fail to materialize.

The problem here (besides your lack of advertising; you really should have bought that Animal Planet spot) is that we tend to believe too strongly in an intuitive "Law of Small Numbers" that turns out to be false. (This error may be an overuse of the Representativeness heuristic.) In other words:

We tend to overestimate how much we can generalize from a small sample more often than we tend to underestimate.

This "hasty generalization" problem can vex any company, but is especially nasty for startups because they are less likely than large companies to have the resources to perform market research and confirm whether the hypothesized demand for their new service is real or illusory.

Two related problems can help calibrate your intuitions:

1. The birthday paradox. Suppose there are 23 people in the room. What is the likelihood that two of them have the same birthday? The answer, surprisingly, is about 50%.

Now suppose you survey 23 people in the room about what iPhone app they would most like to see. Only two of the people in the room give the same answer. What conclusions do you draw?

2. A research institute tests 20 hypotheses, all of which are incorrect. You still expect one of them to be confirmed at a 95% confidence interval, due to the laws of chance.

Now suppose you have 20 great Chihuahuaball-level ideas. 19 of them generate no interest in the people you survey, but for one of them, you find a certain level of interest. How suspicious should you be of that level of interest?

Answers: 1. If the two people agree on iChihuahuaball, go ahead and launch. The other 21 people are clearly unrepresentative freaks. 2. You should be suspicious of anyone who would discount even one of your 20 fantastic ideas. They are probably spies for a rival company, eliminate them.

There isn't any easy solution. Even if you can't afford to conduct full-scale market research, try to attack subparts of the question from different angles and see if the estimates converge. And, as always, try to have a way to "fail fast" if your basic assumptions fail to pan out.