The S&P 500 (SNPINDEX: ^GSPC ) is the most popular stock index in the world. But it's also incredibly flawed.
Why? Because it weights its 500 components by market capitalization, which can skew it toward some of the market's most expensive stocks.
Last week, I asked Robert Arnott, CEO of Research Affiliates and a pioneer of alternative forms of indexing, a simple question: If weighting by market capitalization is so flawed, why is it so popular?
Here's what he had to say. (Transcript follows.)
Morgan Housel: If weighting by market capitalization is so flawed, why has it been so popular?
Robert Arnott: Weighting by market capitalization came along as a strategy in the early '70s. Cap weighting was first popularized by the S&P. It really came into its own with the launch of the S&P 500 in 1957. The S&P 500 was never intended as a strategy; it was intended to measure how the market is performing. And it's only 15 years later that people latched on to this. This beats most active managers, or, put a different way, most active managers can't beat this index.
Well, that's true, because the market consists of passive indexers and active managers. Take the passive indexers out of the picture, and what's left is going to look the same as the passive indexers, by definition. So active managers, by definition, have to collectively match the market minus their costs. So of course, most will underperform.
Now, because it beats most active managers, it gained a lot of traction. It also gained traction from finance theory; the efficient market hypothesis came along and said stock picking is a waste of time. Capital Asset Pricing Model was published in '64, and in a footnote, actually, it noted that the cap-weighted market was the mean variance efficient market portfolio, and you cannot beat it on a risk-adjusted basis if all of the underlying assumptions of cap M are true.
That's fine, but those underlying assumptions aren't true. And so cap weighting enjoyed a tailwind from finance theory, saying this is the right way to do it. By beating most active managers most of the time, by having finance theory back it up and say this is the right way to do it, it gained tremendous traction and people didn't look at alternatives. Just imagine if in 1957 the S&P had said, "Here's an S&P 500 that tracks the market. It's cap weighted, and by the way, we're doing a sister index, S&P Profits 500, that weights companies by their profits." The latter would have trounced the former in live experience and cap-weighted indexation would never have gained popularity.