Bad news, investors. Some of you must fail.

Not probably, or unfortunately, but must. Professional, amateur. Hedge fund manager, day trader, indexer, 401(k) saver. Some of you must fail. It's a necessity of how markets work.

According to Dalbar and other research groups, the average U.S. stock investor has underperformed the market by between three and seven percentage points per year during the last 30 years, depending on how it's calculated. Most of this is due to what New York Times columnist Carl Richards calls the behavior gap: a parade of dumb decisions where the average investor buys high and sells low. This classic Carl sketch sums it up:


Avoiding this behavior is the holy grail of successful investing. And some people -- many people -- can be taught to behave better. But the reality is that, as a group, we never can, never will.

Why? Because markets must always crash. Decades ago, economist Hyman Minsky wrote about a paradox. Stability is destabilizing, he said. If stocks never crashed, we'd all think they were safe. If we all thought they were safe, we'd rationally bid up prices and make them expensive. When stocks are expensive, the inevitable whiff of danger, uncertainty, or randomness sends them crashing. So, a lack of crashes plants the seeds for a new crash.

And what is a crash? It's people who bought high succumbing to selling low, falling for the same doom-loop behavior Carl's sketch portrays.

The reason stocks offer great long-term returns is because they are volatile in the short run. That's the price you have to pay to earn higher returns than non-volatile assets, like bank CDs. Wharton professor Jeremy Siegel once said, "volatility scares enough people out of the market to generate superior returns for those who stay in." Those are inspirational words for investors who assume they are brave enough to stay in; but not everyone can. The volatility that sets the stage for superior returns is just a reflection of someone getting scared out of the market in real time. 

Put this together, and you get an unfortunate truth that stocks offer superior returns for some because they offer a miserable experience for others. Without the misery, markets wouldn't offer big returns, and without the prospect of big returns, markets will crash and cause misery. That's why some investors must fail. 

All investors I know say they'll be greedy when others are fearful. They never assume that they, themselves, will be the fearful ones. But someone has to be, by definition. With stocks at all-time highs, few people will tell you, "If my portfolio falls 20%, I'm going to panic sell and cash out." They're more likely to say that a 20% decline would be a buying opportunity. This is the right attitude, but the reason there will be a 20% crash is specifically because investors choose panic selling over opportunistic buying. My experience is that most investors who say they'll be greedy when others are fearful soon realize that they are the "others." It has to be this way: When everyone thinks they're a contrarian, at least half will be wrong.

Soon after market meltdowns, journalists and financial advisors come together and ask: When will investors learn from their mistakes? How many times must we buy high and sell low before we learn to behave better?

I've come to realize the answer is "never." As a group, at least. If people stopped acting dumb, markets would be stable, and if markets were stable, they'd be expensive; and when they're expensive, people act dumb. It's a feedback loop that was true 1,000 years ago and will be true 1,000 years from now. I think it's an accurate description of what a market actually is.

Coming to terms with this taught me a couple things.

One, nothing that most people fail at is easy, so we shouldn't assume it's easy to watch our portfolios crash and remain unshakably calm, or stay levelheaded during a big rally. When the average investor doesn't come within hailing distance of an index fund, you're fooling yourself if you think investing will be an emotional cakewalk. Over time, the investor willing to endure the steepest emotional roller coaster will win.

Two, the best thing you can do to up your odds of success is to repeat to yourself, over and over again, that a market crash is not a bug or an indication that something is wrong. It is, quite literally, the admission fee for being able to earn superior long-term returns -- like a stressful work project that lets you earn a big bonus.

Best of luck to all of you, but my deepest sympathies to some of you.

Check back every Tuesday and Friday for Morgan Housel's columns on finance and economics.