No one in modern financial history comes close to matching Warren Buffett's record as an investor.

Hedge fund managers like John Paulson and Kenneth Griffin have had huge years along the way, and more conventional money managers such as Peter Lynch have turned in stellar performances as well, but none has performed at such a high level over such a long period of time as the Oracle of Omaha.

When Buffett took control of Berkshire Hathaway (NYSE:BRK.A) (NYSE:BRK.B) in 1965, it was an ailing textile company with a bleak future. Today, thanks to Buffett's brilliance at capital allocation, Berkshire is the fourth-biggest company in the S&P 500 as measured by market capitalization.

The compound annual increase in Berkshire's book value per share has averaged 19% since Buffett grabbed the reins. That's nearly double the compound annual gain of the S&P 500, which comes out to 9.7% over the same 52-year stretch.

A chart of Berkshire Hathaway's annual increase in book value per share.

Data source: Berkshire Hathaway. Chart by author.

But Berkshire's success comes at a price. The conglomerate's now-enormous size, combined with its vast and widely diversified stable of subsidiaries, makes it all but impossible for Berkshire to continue growing at such a rapid pace.

A 19% increase in 1995 translated into $3 billion worth of earnings or investment gains. Today, that figure is $54 billion. Even Apple, the most profitable company in America, doesn't generate that much net income.

The result is that, as you can see in the chart, Berkshire's annual returns are almost certainly doomed to more closely approximate the broader market in the years ahead.

Offer from The Motley Fool: The 10 best stocks to buy now
Motley Fool co-founders Tom and David Gardner have spent more than a decade beating the market. In fact, the newsletter they run, Motley Fool Stock Advisor, has tripled the S&P 500!*

Tom and David just revealed their ten top stock picks for investors to buy right now.

Click here to get access to the full list!

*Stock Advisor returns as of 1/30/2017

This article represents the opinion of the writer, who may disagree with the “official” recommendation position of a Motley Fool premium advisory service. We’re motley! Questioning an investing thesis -- even one of our own -- helps us all think critically about investing and make decisions that help us become smarter, happier, and richer.