All signs are pointing to Berkshire Hathaway (BRK.A -0.30%) (BRK.B -0.26%) seeing its book value growth over the last five years fall short of the total return of the S&P 500 (^GSPC -0.46%).

This has always been a key proxy for Berkshire Hathaway and how Warren Buffett gauged the success of his efforts at the helm of the company. When speaking about the likelihood of this happening in 2013 in his most recent annual letter to shareholders in March of last year, Buffett noted:

It's our job to increase intrinsic business value -- for which we use book value as a significantly understated proxy -- at a faster rate than the market gains of the S&P. If we do so, Berkshire's share price, though unpredictable from year to year, will itself outpace the S&P over time. If we fail, however, our management will bring no value to our investors, who themselves can earn S&P returns by buying a low-cost index fund.

In the video below, Motley Fool financial sector writer Patrick Morris takes a look at what Berkshire Hathaway not outpacing the S&P 500 may mean for investors and whether or not it should be cause for concern about Buffett's investing prowess. Yet when you consider the nearly 20% annual return posted by Berkshire Hathaway versus the 9.4% return of the S&P 500 over its lifetime -- the answer may already be on the table.