On Oct. 31, 2007, the S&P 500 reached its high point, closing at 1,549.38. On March 9, 2009, it closed at a low of 676.53, dropping 56% in just more than 16 months.

Over the course of 2008, the index as a whole dropped 37%. Of the 500 companies that finished the year in the index, a mere 25 saw an increase in their share price over the year.

Twenty-five.

We should have seen it coming -- but most of us didn't. In fact, most investors were whooping it up in 2006 and 2007 instead of looking to see if they were about to careen off a cliff.

And it's not just this bear market, either. Individual investors routinely underperform the market -- and there are two important reasons why.

I'm a genius!
In his book Investment Madness, finance professor John Nofsinger argues that overconfidence -- the belief that you're more skilled than you really are -- is one reason we get ourselves into investment hot water. Overconfidence can make you unwilling to recognize bad news for what it is -- much less how it affects your investment.

Overconfidence can come from early success, which you're likely to attribute to your own skill instead of the dumb luck it's likely to be. For example, if you bought shares of Vale (NYSE:RIO) or Southern Copper (NYSE:PCU) a couple of years ago and watched the shares climb, that's probably more because there was a general increase in commodity prices than because you were a superstar stock picker.

But even if your early investing results weren't spectacular, you can still fall prey to overconfidence, because information -- however irrelevant to the question at hand -- provides the illusion of control. For instance, how much will the price of McDonald's (NYSE:MCD) change this year? Up 10%? Down?

If I tell you that it had increased at an average annualized rate of 17.5% over the past five years, does that change your answer?

It shouldn't, because that information has nothing to do with what the stock price will do this year.

The erratic investor
That tendency toward overconfidence gets magnified when it's combined with our tendency to use past situations to evaluate risks in the here and now.

Experiments have shown that when people risk their own money (say, on an investment) and succeed, they're likely to take on even more risk the next time around. Why? They don't think of that money as theirs, exactly -- it feels like they're playing with house money.

Jason Zweig, in Your Money and Your Brain, tells the story of Russell and his investment in CMGI Inc., a hot Internet stock of the late 1990s. As Russell "won" more from the price going up, he invested more -- only to end up losing it all. Zweig then writes:

Let's say you put $1,000 into a stock that triples; now that it is priced at $3,000, you've got $2,000 of "house money." So long as any of that $2,000 gain is left, you may shrug off any losses as a reduction of the house money -- rather than a depletion of your own. Somehow, losing the house money hurts less than losing your "own" -- even though, strictly speaking, all the dollars are the same. As Russell found, this "house-money effect" can egg you on into taking an ever-escalating series of risks until you get wiped out.

On the down side, Nofsinger points out, when you lose money on an investment, you're more likely to avoid risk. Consider how investors have been behaving lately. The market is nearly 50% off its October 2007 high -- but investors have been leaving the market in droves, despite the fact that buying when prices are low gives investors the best chance of success.

An investment in consumer-staple giant Procter & Gamble (NYSE:PG) or stock market exchange NYSE Euronext (NYSE:NYX), both of which have fallen to multi-year lows, will likely produce solid returns over the next several years -- but investors aren't necessarily jumping on the bandwagon.

It doesn't happen overnight
If you want to beat the market, you have to battle your tendencies toward overconfidence and judging risks in relation to recent failures and successes. Nofsinger suggests that you control your environment by limiting your exposure to information overload. You can also compare your investment performance to market averages to help keep it contextualized. And finally, develop an investing strategy to help you assess each risk on its own terms.

It truly takes years to become good at investing, to learn what information is important versus what is just noise, and to get over risk aversion and away from the idea of house money. Just like master carpenters are not made in a day, neither are master investors. But with patience and a little self-awareness, you can do it.

David and Tom Gardner have been doing this since 1993. And in the spring of 2002, nearly seven years ago, they launched Motley Fool Stock Advisor, our flagship investing service. In it, they are always looking for both the bear and the bull sides of an investing thesis, constantly reevaluating their recommendations in light of new information. This helps them avoid both the overconfidence and house-money traps. It's also led to recommendations like Netflix (NASDAQ:NFLX) and Disney (NYSE:DIS), both of which are handily outperforming the S&P 500.

In fact, over the past seven years, their picks have outpaced the S&P 500 by more than 32 points on average. If you want to learn the secret to that success, just click here and give it a free, 30-day trial.

Jim Mueller is averse to losing money in the market and thinks all the information supplied on CNBC every day is too much, hurting more than helping. He owns shares of Southern Copper and Netflix, but no other company mentioned. Netflix and Disney are Motley Fool Stock Advisor choices. Disney is also an Inside Value pick. NYSE Euronext is Rule Breakers recommendation. The Fool owns shares of P&G. Our disclosure policy has been around the block a few times.