I'm not going to lie: I wanted to be just like Jim Cramer and the other besuited prognosticators on CNBC.

Years ago, when I started writing for The Motley Fool, I thought that I was headed in that direction. Not that I'd necessarily have my own TV show or even be on TV, but that, like those CNBC regulars, when asked about any particular stock, I would be able to confidently tell the inquirer exactly where that stock was headed over the next year, six months, or even few weeks.

That hasn't happened. In fact, as I continue to spend much of my waking hours studying companies and the stock market, I find that with each passing day I feel less confident trying to tell someone where a stock is headed over short time periods.

This still leads to moments of disappointment. Occasionally, someone will ask me what I think about some stock over the next few months and it's not particularly fun to honestly answer, "I have no idea." But at the same time, I'm glad to answer that way, because to do otherwise would be to dangerously ignore some of the most important realities of the stock market.

People are unpredictable
Ben Graham, the father of value investing, famously said that over the short term the market is like a voting machine, but over the longer term it's like a weighing machine.

His point was that over shorter time frames, a stock's price can be moved -- sometimes drastically -- by emotions and how investors feel about a stock, even if that perspective doesn't reflect what the underlying company is really worth. Over longer time periods, though -- five, 10, or 20 years -- a stock's price will inevitably move toward and in relation to the value of the underlying company.

For a good example of this in the real world, look no further than Green Mountain Coffee Roasters (Nasdaq: GMCR). Over the long term, the price of the stock will reflect the success of the company's business -- enduring demand for Keurig coffee machines and K-Cups as well as changes to the competitive environment. However, over the shorter term, the stock will continue to get whipsawed by the strong emotions of both bulls and bears as they continue their gruesome tug-of-war.

The future is unpredictable
Some people like to say that the massive BP (NYSE: BP) oil spill was predictable because of the company's approach to safety and a less-than-stellar track record. While the full legal verdict is still not in, there has been some very concerning evidence against the company. Still, to say that the nature and extent of the Macondo spill was predictable is silly.

The future is unpredictable and that presents a very serious challenge for investors. For instance, how much is Berkshire Hathaway (NYSE: BRK-A)(NYSE: BRK-B) worth? Flip through the finance news and you'll find plenty of commentators who slap a specific number on the valuation. But the value of Berkshire could be affected by a wide variety of unknown variables. Major natural disasters that need to be covered by its "big-cat" insurance unit could put a dent in the company's value. On the other hand, Warren Buffett adds value to the company every year he's still at the helm. If he's around longer than anyone expects, that could have a very positive impact on the value of the company.

There are few assurances and guarantees about the future and that means that investing is about probabilities and possible scenarios, not set-in-stone outcomes.

Overconfidence kills
And what of investors who think they have special insight into the two unpredictable market realities above? Well, the bottom line for all investors is that being overconfident is a great way to shoot yourself in the foot.

Investors who are overconfident make mistakes, like trading more than investors who aren't overconfident (beware, men; you're more likely to fall into this trap!). And investors who trade more, well, they tend to end up with worse performance. Classic behavioral finance research out of the University of California concluded:

Individuals turn over their common stock investments about 70 percent annually ... Mutual funds have similar turnover rates ... Yet, those individuals and mutual funds that trade most earn the lowest returns. We believe that there is a simple and powerful explanation for the high levels of counterproductive trading in financial markets: overconfidence.

The easy answers
One way to deal with the market's unpredictability and the damage that overconfidence can do is to simply forget about trying to beat the market. It may be tough to brag about owning a reasonable collection of low-cost index funds at cocktail parties, but it's likely that that "match the market" approach is beating most of the braggers.

That said, there's still plenty of opportunity to take advantage of Mr. Market's many miscues and earn market-beating returns. But in doing that, the best approach isn't the one where you pretend to know all of the answers and have the market figured out over all time frames, short and long. Instead, it's the approach that focuses on probabilities and possibilities and is only overconfident in the idea that much is unknowable.

For those ready to embark on the latter course, The Motley Fool is offering up free copies of the special report "Secure Your Future With 9 Rock-Solid Dividend Stocks." While there are no guarantees -- even with dividends -- buying dividend-paying stocks can up the probability that your investment will work out in your favor over time.