I read the book How to Lie With Statistics, a basic but brutally honest and funny read about how to prove almost anything you want to.
Here are six things I learned.
1. Mother nature has her own statistics; they're stronger than ours:
A few winters ago a dozen investigators independently reported figures on antihistamine pills. Each showed that a considerable percentage of colds cleared up after treatment. A great fuss ensued, at least in the advertisements, and a medical-product boom was on. It was based on an eternally springing hope and also on a curious refusal to look past the statistics to a fact that has been known for a long time. As Henry G. Felsen, a humorist and no medical authority, pointed out quite a while ago, proper treatment will cure a cold in seven days, but left to itself a cold will hang on for a week.
2. Where are these numbers coming from?
The next time you learn from your reading that the average American (you hear a good deal about him these days, most of it faintly improbable) brushes his teeth 1.02 times a day — a figure I have just made up, but it may be as good as anyone else's — ask yourself a question. How can anyone have found out such a thing? Is a woman who has read in countless advertisements that non-brushers are social offenders going to confess to a stranger that she does not brush her teeth regularly? The statistic may have meaning to one who wants to know only what people say about tooth-brushing, but it does not tell a great deal about the frequency with which bristle is applied to incisor.
3. A biased sample will ruin everything.
A psychiatrist reported once that practically everybody is neurotic. Aside from the fact that such use destroys any meaning in the word "neurotic," take a look at the man's sample. That is, whom has the psychiatrist been observing? It turns out that he has reached this edifying conclusion from studying his patients, who are a long, long way from being a sample of the population. If a man were normal, our psychiatrist would never meet him.
4. Biased samples are extremely common.
The ten million telephone and Digest subscribers who assured the editors of the doomed magazine that it would be Landon 370, Roosevelt 161 came from the list that had accurately predicted the 1932 election. How could there be bias in a list already so tested? There was a bias, of course, as college theses and other postmortems found: People who could afford telephones and magazine subscriptions in 1936 were not a cross section of voters. Economically they were a special kind of people, a sample biased because it was loaded with what turned out to be Republican voters. The sample elected Landon, but the voters thought otherwise.
5. You often need a huge sample to prove a point.
A remarkable instance of this came out in connection with a test of a polio vaccine a few years ago. It appeared to be an impressively large-scale experiment as medical ones go: 450 children were vaccinated in a community and 680 were left unvaccinated, as controls. Shortly thereafter the community was visited by an epidemic. Not one of the vaccinated children contracted a recognizable case of polio. Neither did any of the controls. What the experimenters had overlooked or not understood in setting up their project was the low incidence of paralytic polio. At the usual rate, only two cases would have been expected in a group this size, and so the test was doomed from the start to have no meaning. Something like fifteen to twenty-five times this many children would have been needed to obtain an answer signifying anything.
6. Always remember: data is not perfect.
The operation of a poll comes down in the end to a running battle against sources of bias, and this battle is conducted all the time by all the reputable polling organizations. What the reader of the reports must remember is that the battle is never won. No conclusion that "sixty-seven percent of the American people are against" something or other should be read without the lingering question, Sixty-seven percent of which American people?
Buy the book here. It's great.
- What I learned from Antifragile
- What I learned from Thinking Fast and Slow
- What I learned from Risk Savvy
More from The Motley Fool: Warren Buffett Tells You How to Turn $40 into $10 Million