The stock market has rebounded nicely. After plunging by about 40% from its record highs in March as the COVID-19 pandemic worsened, the S&P 500 is now about 5% shy of where it started the year.

One surprising underperformer is Berkshire Hathaway (NYSE:BRK.A) (NYSE:BRK.B), which historically has done rather well in turbulent markets. In fact, the S&P 500 has had 11 negative years since Warren Buffett took control of the company in 1964 and Berkshire outperformed the market in all but two of them.

With that in mind, here's a look at how Berkshire has performed so far in 2020, the reasons behind the relatively poor performance, and if it might be a good idea to buy the stock while it's down.

Warren Buffett smiling.

Image source: The Motley Fool.

Berkshire Hathaway has experienced a rough 2020

I don't think anyone was surprised when Berkshire plunged by about 30% as the market crashed in March. But it is a bit surprising that the stock hasn't really recovered yet. While the S&P 500 is now down by just 5% in 2020, Berkshire is still 18% lower than where it started the year.

BRK.B Total Return Price Chart

BRK.B Total Return Price data by YCharts.

Why is Berkshire performing so poorly?

At first it might seem odd that Berkshire has underperformed the market by so much. After all, many of its operating businesses were relatively immune to the effects of the pandemic. For example, people still needed to make their auto insurance payments to GEICO and Berkshire's utility businesses continued to provide an essential service. However, there are a few good reasons behind the lousy stock performance this year.

First off, most investors who follow the company know that Berkshire has a massive stock portfolio, but many don't realize just how big it is. Berkshire's stock portfolio is worth about $206 billion as of this writing -- which means that its stocks make up more than 45% of its market cap. It might surprise you to learn that with $137 billion in cash (another 30% of the company's valuation), Berkshire's operating businesses make up about one-fourth of the current market value. In other words, while many of Berkshire's businesses are indeed fine, they aren't the biggest factor.

Berkshire's stock portfolio is also highly dependent on stocks that have been beaten down, particularly from the financial sector. Berkshire owns 10 different bank stocks with a total market value of $65.5 billion. With the financial sector still down more than 20% year to date, plus the losses on Berkshire's recently sold airline stock positions, it's fair to say that the company's stock portfolio has been a bit of a drag on Berkshire.

The other major issue is Berkshire's growing cash stockpile and lack of investment activity. With more than $120 billion on the company's balance sheet at the beginning of 2020, investors had been waiting to see how Buffett would deploy some of this capital when the market started to plunge. To the surprise of many, not only did Buffett not make any large purchases or stock buybacks, but it turns out Berkshire was actually a net seller of stocks in the first quarter. Berkshire's cash hoard has grown to its highest level ever at $137 billion -- and that was before Buffett decided to sell the airline investments.

In a nutshell, a big reason why many investors own Berkshire is Buffett's track record of making savvy investments during market crashes. But that hasn't been the case so far in 2020.

Why you might want to consider buying

There are two big reasons Berkshire Hathaway is still on my watch list.

First is valuation. Berkshire trades for just 1.2 times its book value at the end of the first quarter. With the rebound in the stock market, the company's book value has almost certainly increased since that time, but even at this multiple, Berkshire is very cheap. If you recall, this is the level that Buffett has previously said would represent a large discount to the intrinsic value of the business, as it was the former threshold at which buybacks were allowed to occur. Over the past few years, Berkshire has spent most of its time between 1.3 and 1.6 times book, so it's almost certainly trading for an attractive price relative to the true value of its business.

Second, and most importantly, keep in mind that this is an unprecedented situation, especially when it comes to governmental support. Think of it this way -- when businesses can get partially forgivable loans with virtually no interest from the U.S. government, why would they need financial help from companies like Berkshire? I'm more interested to see what opportunities present themselves when government support dries up. Recall that Buffett's most successful investment of the financial crisis era, the purchase of Bank of America preferred stock and warrants, was actually made two years after the market bottomed.

In short, I'm not giving up on Warren Buffett and his team. Historically, that's been a bad idea. While I'm as disappointed as anyone that Berkshire didn't take advantage of the March market plunge, I'm also optimistic about what Buffett can still do with over $100 billion in usable cash to work with.