"Admittedly, it is preposterous to suggest that stock speculation is like coin-flipping. I know there is more skill to stock speculation. What I have never been able to determine is -- how much more?"
-- Fred Schwed Jr., Where Are the Customers' Yachts?

I can think of no question that is more important to the average investor than this: Can an individual beat the market through skill as opposed to luck? If the answer is yes, then there is indeed reason to spend time and energy picking stocks. But if the answer is no, then you'd have both more time and more money adopting a simple strategy of passively investing in, say, the S&P 500 (^GSPC 1.20%).

While it seems like the answer to this question must be yes -- why else would so many people strive day in and day out in an effort to do so if it were impossible? -- the reality is that nobody actually knows whether it is. Or, to be more precise, the handful of companies that do know -- that is, retail brokers such as Charles Schwab and E*TRADE Financial -- aren't spilling the beans (I even called them and asked).

The theory and practice of investing
At least in theory, there seems to be little reason to conclude that somebody couldn't beat the market if he sat down and gave it his all. If you wanted to do it with individual stocks, you would just need to identity companies that are sure to rise faster than the broader market. These are touted ad nauseam by television and Internet commentators masquerading about as experts. Or, if securities analysis isn't your thing, then you could just buy and sell the S&P 500 when the market is respectively down and up. Easy enough, right?

Anybody who's been at this for a while can rattle off numerous examples of investors who have succeeded at doing just this. Peter Lynch, the former head of Fidelity's famed Magellan Fund, comes to mind. As does John Templeton, who managed the eponymous Templeton Growth Fund and has been called "arguably the greatest global stock picker of the [20th] century." But perhaps no one epitomizes the potential for stock market success better than Warren Buffett, the famed chairman and CEO of Berkshire Hathaway (BRK.A -0.34%) (BRK.B -0.01%).

It just so happens, in fact, that nearly 30 years ago, the very question I'm addressing here found itself at the center of a debate between Buffett and Michael Jensen, a onetime professor of finance at the University of Rochester. Jensen claimed it was impossible for an investor to systematically beat the market because stock prices proceed as if on a random walk -- that is, absent luck, nobody can predict where they'll go tomorrow, next week, or next year. Buffett, on the other hand, argued that a certain subset of investors -- namely, those like him who had studied under Benjamin Graham -- had, in fact, beaten the market consistently over time. The implication of Buffett's position was that the success couldn't have been random.

So, who won the debate? The simple answer is that it depends on whom you ask. If you're in the financial industry, then you have a vested interest in dismissing Jensen as hogwash and concluding that Buffett most certainly prevailed. This is understandable, though it's also the result of confirmation bias, one of the more prevalent cognitive biases identified by psychologists over the past few decades. If you're an academic, on the other hand, then you probably still adhere to the party line on efficiency -- though, this, too, is a likely consequence of confirmation bias.

The truth, it turns out, appears to lie somewhere in the middle. If you pick up a modern textbook on finance, it distinguishes between a general rule and an exception. The general rule continues to be dominated by the efficient market theory, under which the typical investor isn't able to beat the market absent luck. At the same time, however, it recognizes that "[t]here are enough anomalies in the empirical evidence to justify the search for underpriced securities that clearly goes on." It bridges the gap between these otherwise conflicting positions by observing that those who have "differentially superior information or insight" will indeed be able to outperform the market.

Along the same lines, my colleague Morgan Housel recently recounted a lunch he had with Wall Street Journal columnist Jason Zweig, who is perhaps more familiar with Benjamin Graham's thinking than anyone else. In response to Morgan's question about whether Graham would have all of his money in a Vanguard index fund, Zweig responded, "No, I don't think so." Instead, Zweig believes that Graham would advise those who have "an edge at stock-picking" to do so, while telling those who don't to opt for a passive approach with index funds.

Are you as exceptional as you think?
If you've followed along thus far, Zweig's response raises the question: Do you fall under the exception or the general rule? Or, more specifically, do you have an edge at stock-picking?

For reasons that are beyond me, nearly all people (myself included) have a remarkable tendency to believe that they are exceptional at everything they do. "You've probably heard of the Lake Woebegon effect with drivers [in which] the vast majority of drivers have above-average driving skills," Morgan addressed in a separate column about investor overconfidence. "This even holds true for drivers surveyed in the hospital after being injured in car accidents that they caused." And, just to be clear, the same type of delusion spills over into investing.

While it's true that there are certain inherent advantages that individual investors have over their institutional counterparts, such as time and the freedom to think beyond the next quarter, we also have an unfortunate tendency to get in our own way. "The investor's chief problem -- and even his worst enemy -- is likely to be himself," says James Montier in The Little Book of Behavioral Investing. We tell ourselves that we're contrarians -- that we'll be "fearful when others are greedy and greedy when others are fearful" -- only to then buy in the euphoria of a bull market and sell in the druthers of a bear market.

The point is that, as much as it pains me to tell you this, as it did to acknowledge myself, we are not exceptions to this rule. Indeed, if you were to go through your brokerage statement, as I've done, my guess is that you haven't beaten the broader market over the course of your investing career. And if you have, it's more than likely a result of luck. This is harsh medicine, I know. But it's medicine nonetheless.

So, where does this leave you?
I want to be very clear about this point, because I suspect it has yet to sink in. If you're reading this article, as opposed to vacationing on your 24-meter yacht somewhere in the British Virgin Islands, then it's highly likely that you're among the vast majority of us mere mortals who don't have the time or the "differentially superior information or insight" that's needed to outperform the market in the absence of random chance.

That's the bad news.

The good news is that you don't need this type of insight. You can buy it. Over the past five years, the recommendations of our top investing newsletter, Stock Advisor, produced an average annual return of 15%, or nearly double that of the broader market's 7.2% (as measured by the Willshire 5000 index with dividends reinvested). And since 2002, it's outperformed the S&P 500 by 83.6%. Thus, to revisit the question that prompted me to write this article: Can you, as an individual investor beat the market absent luck? The answer is "yes." And you don't even have to do the work.

To try a free 30-day trial of Stock Advisor, click here now.