Can individual investors beat the market?

That's a good question. It also happens to be the title of a now-famous academic paper. And if you ask most investors, the answer is an emphatic "No" -- at least not by any meaningful degree. More on that later.

Fortunately, you have options when trying to beat -- or at least have success in -- the market. One such option is to buy an index fund, which is something we recommend for a good chunk of your portfolio. With index funds, you essentially bet with the house. But it has its downsides, not the least of which is lost opportunity.

Let's look at an example. Imagine that on top of your core index holdings you earmark an additional $50,000 as market-beating ammo. If that $50,000 earns 10% yearly -- something you might hope for from the Standard & Poor's 500 -- you'll have $336,375 after 20 years.

Sounds good, but I want more.

A half-million more
If you can muster 15% instead of 10% with the more aggressive portion of your portfolio, you'll walk out of the gates with $818,327 at the end of the same 20-year span. That extra 5% per year gets you an additional $481,952. I'm going out on a limb by saying: We'd all love to grab half a million dollars extra.

Of course, that extra 5% isn't much good if you don't actually get it. Sadly, most individual investors don't. Here are a few reasons why.

Never letting go. In The Courage of Misguided Convictions: The Trading Behavior of Individual Investors, Barber and Odean declare that we are 50% more likely to sell a winner than a loser. Our tendency to avoid pain -- in this case, refusing to take a loss that already exists -- is just one psychological weakness that leads us to poor investing decisions.

That irrational exuberance. This is sort of a self-perpetuating prophecy: We pursue exciting opportunities to the point where they may no longer be such good investments. Hitachi (NYSE:HIT) may indeed have potential as an investment, but is its lofty valuation justified?

The poker analogy
Given the choice, where will you sit? At the table of visor-clad sharks or at the table of Hawaiian-shirted tourists? Sounds simple, but odds are you've been sitting down with the sharks in the stock market, playing a tougher game than necessary.

Lakonishok, Shleifer, and Vishny -- three professors I'll call "LSV" for short -- suspected as much. In 1994, the trio set out to investigate why certain types of stocks consistently outperform. Using a variety of factors, they started by dividing stocks into two groups:

  1. Unloved, low-expectation nobodies (value stocks)
  2. High-priced, high-expectation glamour stocks

Picture shoelace makers and semiregional banks versus, say, EDS (NYSE:EDS), Marvell Technology Group (NASDAQ:MRVL), or Schering-Plough (NYSE:SGP). Tell me, where do you think the sharks are playing?

So how about that extra 5%?
Breaking stocks into a 10-group spectrum ranked by earnings yield (earnings divided by price, or E/P -- academics prefer to rearrange familiar measures to maintain an air of sophistication), the trio shows that the high-E/P, value end of the spectrum bested the glamour end by about four percentage points from 1968 through 1994.

That's not pocket change. And it wasn't news to many of history's greatest investors. The likes of Dodd, Graham, Buffett, and many others (will we hear Tom Gardner in there someday?) have espoused buying stocks with low P/Es. It's great to see confirmation of the low-P/E ideology from both academia and investors past, but there's work to be done.

You don't have to be Warren Buffett to notice that E/P, however powerful, is a simple and imprecise measure. LSV knew this -- in fact, the study aimed in part to expound on a study (by Fama and French) that harped on the book-to-market ratio (B/M), a variable described by LSV as "unclean" for its oversimplicity and inability to capture other forces -- namely, growth.

Investors gone wild
Indeed, growth is sorely missing from measures like B/M (again, an academic inverse of the familiar price-to-book value ratio). So LSV gave growth special attention.

The trio again broke the universe into deciles, this time using a formula based on historical sales growth. Amazingly, value stocks -- the low-expectation, low-growth nobodies -- again walloped glamour, this time by 7.3 percentage points per year.

It's important to understand why. The gang explicitly rejected the knee-jerk academic explanation that if value stocks do better, they must somehow be riskier. Instead, they identified -- and politely termed "suboptimal" -- inferior behavior on the part of investors as the culprit.

Investors overpursue today's hot performers to the point that returns are no longer lucrative. And, to be more specific, LSV's glamour stocks are high-sales-growth stocks that don't have much in the way of assets or cash flows. In other words, stocks such as St. Jude Medical (NYSE:STJ), Coach (NYSE:COH), or Pearson (NYSE:PSO), which have experienced strong sales growth in the past, wouldn't fit the "glamour" moniker. Moreover -- and I'm probably butchering once-eloquent thoughts -- investors don't adequately understand mean reversion, a concept best explained by the following:

Odds are that a car going 100 mph on the freeway will be traveling closer to the speed of traffic five minutes later. Similarly, a driver clocked at 35 may be going slowly only temporarily (unless he's driving in front of me) and will, the odds dictate, also be traveling closer to the speed of traffic five minutes later.

It's not impossible to make money in glamour stocks. In fact, some people make a killing in them. But it's a mistake to assume that current trends will continue indefinitely -- and clearly, the glamour table is a tougher game.

Now I'll drop the bomb
The real secret of the LSV study was the magic of combining factors. A portfolio favoring high (cheap) E/Ps and low growth outperforms its glamour opposite by 11% per year. Now that's astounding. Wouldn't you be eager to sit down at a card table knowing you had an 11% advantage?

And remember the study about individual investors failing to beat the market? There's more to the story, both good and bad. A precious few -- the top decile -- investors actually beat the market and earn "excess returns" of 0.12% to 0.15% a day. But that's largely offset by the bottom decile's 0.11% to 0.12% loss. In other words, for every couch potato investor doing well, one does poorly -- ironically, so poorly that we'd do well to sell short his favorite picks and profit from their declines.

Bottom line for you
If you're just jumping into the game, odds are you won't win. Your odds of consistently beating the market are about the same as those of consistently letting the market beat you. Of course, we'd all like to think we're in that special 10%, but with your retirement at stake, isn't a little honesty in order?

One option is to find someone in that 10% -- a proven top dog -- and keep him or her on a short leash. If you either (a) don't have the time, effort, skill, or inclination to beat the market, or (b) don't know anyone who does, I highly recommend you take a look at Tom Gardner.

Cheating off Tom makes all kinds of sense for the "bottom 90%" of us. Since he launched his Motley Fool Hidden Gems newsletter service in July 2003, Tom's picks have returned 38% -- a mind-blowing 24 percentage points above the S&P 500. If you like those odds, Tom is offering a no-obligation, 30-day free trial to Hidden Gems, full privileges included -- simply click right here to learn more.

This commentary was originally published on Jan. 14, 2005. It has been updated.

James Early owns none of the stocks mentioned in this article. The Motley Fool has a disclosure policy .