What happened

On paper, it would appear the first half of 2020 set up nicely for Warren Buffett, the investor who famously said, "Be greedy when others are fearful, and fearful when others were greedy." Investors were certainly fearful in February and March as the COVID-19 pandemic swept across the globe, with the S&P 500 falling more than 30%.

But Buffett and Berkshire Hathaway (BRK.A 0.80%) (BRK.B 0.92%) mostly sat on the sidelines as the markets dropped, surprising investors. And questions about that inaction have lingered over the stock in the months since, causing Berkshire shares to miss out on the recent rally and leaving the stock down 21% in the first half of the year, according to data provided by S&P Global Market Intelligence.

BRK.A Chart

Berkshire data by YCharts.

So what

Berkshire has gained a reputation for using downturns to amass large stakes in long-term winners, famously doing a number of deals with large banks during the 2008-2009 downturn that have paid off well.

But as markets fell in 2020, Berkshire's most notable action was to sell, taking losses to exit large positions in the nation's four largest airlines. At his company's annual meeting, Buffett seemed full of doubt about his decision making, admitting he sold out of the positions because "I just decided that I'd made a mistake."

Warren Buffett

Warren Buffett. Image source: The Motley Fool.

Berkshire's lack of activity during the downturn seemed to surprise market watchers and drained the momentum from the shares. As the broader indexes recovered in May and into June, Berkshire Hathaway shares remained stuck in neutral.

Now what

Many an investor has been left looking like a lower-case fool for questioning the acumen of Warren Buffett over the years, and I'm loath to do it now.

Although broader markets have recovered most of what was lost during the early days of the pandemic, there are worrisome signs that cases are spiking in many states, which could lead to a new round of business closures. If so, Buffett might yet have another chance to fire his so-called "elephant gun" and look smarter than all of us who questioned him.

All of the deal talk also distracts from the solid collection of businesses that Berkshire has accumulated over the years. The portfolio of wholly owned assets throws off more than $35 billion in operating cash flow annually. It also owns a number of top-performing stocks, including Apple, and still has a huge cash pile to eventually put to work to create shareholder value.

This wasn't the best six months to be a Berkshire Hathaway shareholder, but Buffett and his company have always been about the long game. It's way too soon to write off this longtime outperformer.