At the start of every Berkshire Hathaway (BRK.A -0.76%) (BRK.B -0.69%) annual report, the company provides a list that compares its stock performance against the S&P 500 since 1965, the year Warren Buffett took control of the company. The track record is impressive.

Investors who came on board with Buffett back in 1965 have seen their money compound at an average annual rate of 19.8%. The S&P 500 has returned an average of 10.2% during that same period. Over 59 years, that translates into a cumulative return 140 times greater than holding the S&P 500.

Going forward, Buffett himself doesn't think investors should expect those types of market-crushing returns. He still thinks it's a great company to own, but there's a big thing holding back its ability to grow like it used to.

A close up of Warren Buffett.

Image source: The Motley Fool.

Moving the needle at Berkshire

The biggest challenge Buffett faces in driving outsize returns for shareholders comes down to the law of large numbers. Berkshire Hathaway isn't the $20 million company it was in the 1960s when Buffett first bought shares. It's not even the $100 billion company it was at the start of the century. It's now rapidly closing in on a $1 trillion market cap.

What's more, the vast majority of Berkshire's businesses and holdings aren't in massive growth industries that can support strong growth. The company invests in and/or owns steady cash-generating businesses like insurance and railroad transport.

For a long time, Buffett generated growth for shareholders by taking the cash from those businesses and buying all or part of a new business. Describing what exactly Berkshire Hathaway does, Buffett writes in his most recent letter to shareholders, "We want to own either all or a portion of businesses that enjoy good economics that are fundamental and enduring."

But when you're already the eighth-largest business by market cap in the world, it's hard to find a business that can have an outsize impact on the value of the holding company.

"There remain only a handful of companies in this country capable of truly moving the needle at Berkshire, and they have been endlessly picked over by us and by others," Buffett wrote to investors. "Some we can value; some we can't. And, if we can, they have to be attractively priced."

So, not only are there only a handful of companies in the investable universe for Berkshire that have potential within Berkshire's portfolio, but Buffett also suggests a lot of the market is currently overpriced. That's led him to build a record cash hoard of $167.6 billion, which he's primarily invested in short-term Treasury bills.

Is it time to ditch Berkshire Hathaway stock?

Buffett's somewhat pessimistic outlook for the future of Berkshire Hathaway may have some investors wondering whether it's time to sell.

While Buffett doesn't think his company is about to double in value within the next five years, he's not completely down on his own company. He likes the present mix of businesses owned by Berkshire, describing them as having "somewhat better prospects than exist at most large American companies." (He's never been one to be too boastful.)

As such, Buffett suggests returns for Berkshire Hathaway shareholders should be "a bit better than the average American corporation." He also notes Berkshire stock comes with materially less risk of permanent loss of capital.

But investors shouldn't discount Berkshire Hathaway's cash position. Not only does it offer great downside protection, but it also puts Buffett and his team of investment managers in a spot to take advantage of a market opportunity, should one present itself. There are only a handful of companies Berkshire couldn't afford to make a sizable investment in, if not purchase outright.

But as Buffett noted in his 2009 letter to shareholders, "Big opportunities come infrequently." A decade and a half later he might add that massive opportunities are even more infrequent.

That's why Buffett thinks Berkshire will still do a bit better than average. "Anything beyond 'slightly better,' though, is wishful thinking," he warns.