This article was written by Whitney Tilson in 1999, a year when investor confidence was at an all-time high. Cocktail party stories about fortunes made in Cisco (NASDAQ:CSCO), Qualcomm (NASDAQ:QCOM), and Juniper (NASDAQ:JNPR) gave investors a strong sense of security. Savvy marketing campaigns from discount brokers added to the frenzy. Although dot-com mania is now history, these timeless lessons about investor overconfidence deserve to be repeated.

The topic I'd like to discuss today is behavioral finance, which examines how people's emotions affect their investment decisions and performance. This area has critical implications for investing; in fact, I believe it is far more important in determining investment success (or lack thereof) than an investor's intellect. Warren Buffett agrees: "Success in investing doesn't correlate with I.Q. once you're above the level of 25. Once you have ordinary intelligence, what you need is the temperament to control the urges that get other people into trouble in investing."

Numerous studies have shown that human beings are extraordinarily irrational about money. There are many explanations why, but the one I tend to give the most weight to is that humans just aren't "wired" properly. After all, homo sapiens have existed for approximately two million years, and those that survived tended to be the ones that evidenced herding behavior and fled at the first signs of danger -- characteristics that do not lend themselves well to successful investing. In contrast, modern finance theory and capital markets have existed for only 40 years or so. Placing human history on a 24-hour scale, that's less than two seconds. What have you learned in the past two seconds?

People make dozens of common mistakes, including:

  1. Herding behavior, driven by a desire to be part of the crowd or an assumption that the crowd is omniscient
  2. Using mental accounting to treat some money (such as gambling winnings or an unexpected bonus) differently than other money
  3. Excessive aversion to loss
  4. Fear of change, resulting in an excessive bias for the status quo
  5. Fear of making an incorrect decision and feeling stupid
  6. Failing to act due to an abundance of attractive options
  7. Ignoring important data points and focusing excessively on less important ones
  8. Reluctance to admit mistakes
  9. Believing that their investment success is due to their wisdom rather than a rising market
  10. A tendency to seek only information that confirms their opinions or decisions
  11. Confusing familiarity with knowledge
  12. Overconfidence

Have you ever been guilty of any of these? I doubt there's anyone who hasn't.

This is a vast topic, so for now I will focus on overconfidence. In general, an abundance of confidence is a wonderful thing. It gives us higher motivation, persistence, energy, and optimism, and can allow us to accomplish things that we otherwise might not have even undertaken. Confidence also contributes a great deal to happiness. As one author writes (in an example that resonated with me, given the age of my daughters), "Who wants to read their children a bedtime story whose main character is a train that says, 'I doubt I can, I doubt I can?'"

But humans are not just robustly confident -- they are wildly overconfident. Consider the following:

  • 82% of people say they are in the top 30% of safe drivers.
  • 86% of my Harvard Business School classmates say they are better looking than their classmates. (Would you expect anything less from Harvard graduates?)
  • 81% of new business owners think their business has at least a 70% chance of success, but only 39% think any business like theirs would be likely to succeed.

Importantly, it turns out that the more difficult the question/task (such as predicting the future of a company or the price of a stock), the greater the degree of overconfidence. And professional investors -- so-called "experts" -- are generally even more prone to overconfidence than novices, because they have theories and models that they tend to overweight.

Perhaps more surprising than the degree of overconfidence itself is that overconfidence doesn't seem to decline over time. After all, one would think that experience would lead people to become more realistic about their capabilities, especially in an area such as investing, where results can be calculated precisely. Part of the explanation is that people often forget failures and, even if they don't, tend to focus primarily on the future, not the past. But the main reason is that people generally remember failures very differently from successes. Successes were due to one's own wisdom and ability, while failures were due to forces beyond one's control. Thus, people believe that with a little better luck or fine-tuning, the outcome will be much better next time.

You might be saying to yourself, "Ah, those silly, overconfident people. Good thing I'm not that way." Let's see. Quick! How do you pronounce the capital of Kentucky: "Loo-ee-ville" or "Loo-iss-ville"? Now, how much would you bet that you know the correct answer to the question: $5, $50, or $500? Here's another test: Give high and low estimates for the average weight of an empty Boeing 747 aircraft. Choose numbers far enough apart to be 90% certain that the true answer lies somewhere in between. Similarly, give a 90% confidence interval for the diameter of the moon. No cheating! Write down your answers and I'll come back to this in a moment.

So people are overconfident. So what? If healthy confidence is good, why isn't overconfidence better? In some areas -- say, being a world-class athlete -- overconfidence might, in fact, be beneficial. But when it comes to financial matters, it most certainly is not. Overconfidence often leads people to:

(1) Be badly prepared for the future. For example, 83% of parents with children under 18 said that they have a financial plan, and 75% expressed confidence about their long-term financial well-being. Yet fewer than half of these people were saving for their children's education and fewer than 10% had financial plans that addressed basic issues such as investments, budgeting, insurance, savings, wills, etc.

(2) Trade stocks excessively. In Odean and Barber's landmark study of 78,000 individual investors' accounts at a large discount brokerage from 1991-1996, the average annual turnover was 80% (slightly less than the 84% average for mutual funds). Incidentally, I suspect that the number of hyperactive traders has increased dramatically, given the number of investors flocking to online brokerages like E*Trade (NYSE:ET), TD Ameritrade (NASDAQ:AMTD) and Charles Schwab (NASDAQ:SCHW). Odean and Barber have done another fascinating study showing that investors who switch to online trading suffer significantly lower returns. They conclude this study with another provocative quote: "Trigger-happy investors are prone to shooting themselves in the foot."

(3) Believe they can be above-average stock pickers, even though there is little evidence to support this belief. The study cited above showed that, after trading costs (but before taxes), the average investor underperformed the market by approximately two percentage points per year.

(4) Believe they can pick mutual funds that will deliver superior future performance. The market-trailing performance of the average mutual fund is proof that most people fail in this endeavor. Worse yet, investors tend to trade in and out of mutual funds at the worst possible time as they chase performance.

(5) Have insufficiently diversified investment portfolios.

OK, I won't keep you in suspense any longer. The capital of Kentucky is Frankfort, not "Loo-ee-ville," an empty 747 weighs approximately 390,000 lbs., and the diameter of the moon is 2,160 miles. Most people would have lost $500 on the first question, and at least one of their two guesses would have fallen outside the 90% confidence interval they established. In large studies when people are asked 10 such questions, 4-6 answers are consistently outside their 90% confidence intervals, instead of the expected one of 10. Why? Because people tend to go through the mental process of, for example, guessing the weight of a 747 and moving up and down from this figure to arrive at high and low estimates. But unless they work for Boeing, their initial guess is likely to be wildly off the mark, so the adjustments need to be much bolder. Sticking close to an initial, uninformed estimate reeks of overconfidence.

In tests like this, securities analysts and money managers are among the most overconfident. I'm not surprised, given my observation that people who go into this business tend to have a very high degree of confidence. Yet ironically, it is precisely the opposite -- a great deal of humility -- that is the key to investment success.

Charles Schwab is aMotley Fool Stock Advisorpick. Joey Khattab, who updates this article, does not have a position in any of the companies mentioned. The Fool has a disclosure policy.