Many see Warren Buffett as an investing guru who can do no wrong. Yet even Buffett himself would be quick to admit that he's made plenty of mistakes. As his company, Berkshire Hathaway (BRK.A -0.28%) (BRK.B -0.68%), has grown, it's become increasingly difficult for Buffett to work his magic and find opportunities for dramatic outperformance.

Recently, many people have started to criticize Buffett, suggesting that his best days are gone and that his illustrious track record is coming to an end. The one thing that longtime Berkshire shareholders can remember that most of these critics have forgotten, though, is that the stock has gone through these cycles before -- and inevitably will again.

Warren Buffett close up, with other people behind him.

Image source: The Motley Fool.

Berkshire's underperformance

Buffett has long advised that many investors would be best served to invest in an index fund. Each year in his annual letter to shareholders, Buffett compares Berkshire's performance to that of the S&P 500, this year using Berkshire's stock price as the gauge. Over the entire 55-year period of Buffett's track record at Berkshire, the stock has averaged an annual gain of 20.3%, compared to the 10% average annual return from the S&P 500, including dividends.

More recently, though, that outperformance has reversed itself. Below, you can see how the stock has done compared to the broader market over the past 10 years.

Year

Berkshire Return

S&P 500 Return

2010

21.4%

15.1%

2011

(4.7%)

2.1%

2012

16.8%

16%

2013

32.7%

32.4%

2014

27%

13.7%

2015

(12.5%)

1.4%

2016

23.4%

12%

2017

21.9%

21.8%

2018

2.8%

(4.4%)

2019

11%

31.5%

Data source: Berkshire Hathaway 2019 annual letter to shareholders.

When you take the total return over the past 10 years, you get 242% for Berkshire stock, compared to 257% for the S&P 500. Over the past five years, the difference is even more extreme: 50% for Berkshire versus 74% for the S&P 500.

A history of bouncing back

With Buffett turning 90 this year, this underperformance feeds a narrative of the Berkshire CEO being past his prime. Yet when you look at the broader context of how cycles like these have played out earlier in Berkshire's history, you'll see that periods of underperformance aren't anything new. Consider the following:

  • From 2003 to 2005, Berkshire largely missed out on the bull market in stocks, trailing the S&P 22% to 50%. Berkshire then reversed the tide, gaining 9% from 2006 to 2008 versus the S&P's 22% loss.
  • In 1999, Berkshire's stock lost 20%. Critics said that Buffett was out of touch with the tech revolution, given the S&P's 21% gain that year. When the tech bust happened, Berkshire stock rebounded while the broader market plunged. Looking at the period from 1999 to 2002 as a whole, Berkshire stock ended up rising 4%, while the S&P lost almost 25%.
  • From 1972 to 1975, Berkshire did a horrible job of navigating the oil shock and resulting bear market, losing 45% even as the S&P bounced back to rise 2%. Look beyond that to 1976 to 1979, however, and Berkshire's stock crushed the S&P 680% to 44%. Buffett saw the share price more than double in two of those four years.

Turning a bigger ship

As it's grown, Berkshire has had some difficulty capturing meaningful investment opportunities. Those heady returns in the late 1970s were a lot easier for Warren Buffett to achieve when he had a much smaller portfolio to manage. With as much money to put to work as Berkshire has, it's unrealistic to expect a return to the huge gains that the insurance giant earned early in its history.

What investors can take away, though, is that despite Berkshire's size, it should continue to see the same sorts of relatively strong and weak periods compared to the broader market. Berkshire shareholders therefore need to prepare to endure stretches of underperformance -- but with the belief that they'll be rewarded for their patience in time.