No doubt you are all familiar with the Efficient Market Hypothesis. University of Chicago professor Eugene Fama first proposed it in his groundbreaking work titled "Random Walks in Stock Market Prices," which appeared in the September/October 1965 Financial Analysts Journal. In case you missed it, I'll quote an extract:

In an efficient market, competition among the many intelligent participants leads to a situation where, at any point in time, actual prices of individual securities already reflect the effects of information based both on events that have already occurred and on events which, as of now, the market expects to take place in the future. In other words, in an efficient market at any point in time, the actual price of a security will be a good estimate of its intrinsic value.

I recall studying this concept in graduate-school finance courses. When you view it through the long lens of history, it makes all the sense in the world. But how can investors reconcile this theory with Peter Lynch's market-dominating run with Fidelity, or the record of Berkshire Hathaway's (NYSE:BRKa) (NYSE:BRKb) Warren Buffett, who over 40 years has more than doubled the S&P 500's average rate of return?

The simple answer, I think, is that you can't. They are two different animals. The first is a theory of how overall markets react, while the second is the real-life experience of two individuals who proved that there is just enough inefficiency in stock markets for an intelligent and perceptive investor to make some serious money. But perhaps we can get some greater insight by examining the principles of what's known as behavioral finance. This competing methodology shows that there are little quirks inside all of us -- quirks that cause investors, who have access to an amazing array of information, to make irrational decisions. A few examples will illustrate some of the work being conducted in this field.

Let's begin with "overreaction bias," the idea that people tend to overreact to bad news and are slow to react to good news. Richard Thaler, professor of behavioral science and economics (also at the University of Chicago), demonstrated this idea by conducting a simple test. He asked a group of students to divide a hypothetical portfolio between stocks and Treasury bills for an investment horizon of 40 years. Half of the students were given reams of information covering a 25-year history of market performance, some of which displayed extreme volatility in equity markets. This group of students allocated only 40% of its portfolio to stocks. The other group of students was given only periodic performance data measured in five-year increments. This second group allocated nearly 70% of its portfolio to stocks. The first group reacted to potential volatility in a very different way from the second group, although both saw the same overall 25-year history of equity performance vs. Treasury bills.

A stock market example of overreaction bias I like to keep in mind is the one that affected Wal-Mart (NYSE:WMT) between 1993 and 1998. Company founder Sam Walton died in 1992, and from then through the end of 1997, the stock hovered at around $20. Then in 1998, the stock began a two-and-a-half year run during which it more than tripled in value. What was the difference between the Wal-Mart of 1998 and the 1995 version? Not a lot. In fact, sales growth was actually lower in 1998, at 12%, vs. 22% in 1995. Comparable sales in 1998 were 6%, vs. 7% in 1995. And EPS growth was 16% in both years. True, the company's international operations were starting to improve, but that shouldn't translate into a tripling of the company's value. I believe that investors overreacted to the death of the founder and to fears of slowing growth that weren't borne out in the actual results. You could argue that Wal-Mart is suffering from the same type of thinking today.

A second finding of behavioral finance relates to "mental accounting," which demonstrates that people change their perspective toward money based on the circumstances they are presented with. Once again, Thaler performed an experiment. He offered one group of people $30 and two choices -- to pocket the money or gamble on a coin flip, where a win would gain $9 and a loss would subtract $9. Seventy percent of the people in this group chose the gamble because they saw the $30 as found money. The second group was offered a slightly different choice -- a gamble on a coin toss that would yield $39 for a win and $21 for a loss, or to walk away with $30 and no coin toss. Only 34% of this group chose the gamble, even though the potential economic outcomes for both groups were identical.

Many people react to moves in the stock market the same way: They invest more confidently when their stocks are up, but they're much more cautious when their carefully selected investments aren't moving in the right direction. Better to follow the advice of Buffett, who makes big plays based on sound research and is patient to wait for other investors to recognize the value that he has already seen, even if it takes a few years.

And don't forget about herd mentality -- our tendency to rely on the safety of being part of a group, even when logic would indicate that we are acting irrationally. We only have to remember the irrational exuberance of the Internet stock craze to know that herd mentality is a powerful influence for investors. Why do you think Merrill Lynch (NYSE:MER) changed its commercials from showing a thundering herd to now presenting a lone steer?

Finally, I'd like to touch on the Theory of Reflexivity from George Soros. "In certain cases," he says of his theory, "the participants' bias can change the fundamentals which are supposed to determine market prices." He cites as an example the international lending boom of the late 1970s that led to the bust of 1982. Soros states that during the boom, "banks relied on so-called debt ratios, which they considered as objective measurements of the ability of the borrowing countries to service their debt, and it turned out that these debt ratios were themselves influenced by the lending activity of the banks." The important point here is the circular relationship between information and the reaction of market participants to that information. Over the short term, in particular, this can result in an overreaction that can move stock prices outside an appropriate valuation range.

What does all of this mean for members of the Motley Fool community? Simply that the stock market is not completely efficient, at least not in the short term, because the way investors react to information is not necessarily rational. In recent years, even Fama has acknowledged this to be the case. In a 1997 paper titled "Market Efficiency, Long-Term Returns, and Behavioral Finance," he defended his Efficient Market Hypothesis against the ideas of behavioral finance by declaring that, in the stock market, "apparent overreaction to information is about as common as underreaction."

That translates into undervalued buying opportunities every day for an investor willing to find them. Like Coke (NYSE:KO) in 1989 and Wal-Mart (NYSE:WMT) in 1998.

How do you find them? Try starting out with the 13 Steps to Investing Foolishly, where you'll find a wealth of information about getting information and evaluating companies. This week at The Fool is all about Rule Breaker investing. It's a perfect time to avoid the herd mentality and look for value where others aren't, by checking out the Motley Fool Rule Breakers newsletter. You can try it free for 30 days. Do some mental accounting on that.

Fool contributor Timothy M. Otte can frequently be found demonstrating the herd instinct at his local Starbucks. He also owns shares of Wal-Mart and Berkshire Hathaway. The Motley Fool has a disclosure policy.