Berkshire Hathaway's (BRK.A -0.59%) (BRK.B -0.74%) annual shareholder meeting and its marathon question-and-answer session have once again come and gone, offering investors another glimpse into the extraordinary mind of legendary investor Warren Buffett. Some of the questions that came up in Saturday's livestream dealt with moves Berkshire made during the early stages of the pandemic last year -- including what led the company to exit most of its banking positions while loading up on Bank of America (BAC 0.45%). After months of speculation, Buffett gave a little bit of insight into what drove his decision making.

Too much exposure

Prior to the pandemic, Berkshire owned a slate of bank stocks, and Buffett seemed to have a bullish stance toward the sector. However, as stay-at-home orders and economic shutdowns spread across the country and banks braced for heavy loan losses, Buffett seemed to do a 180 and exited many of his holdings.

Berkshire eliminated its stake in investment bank Goldman Sachs and eventually JPMorgan Chase, America's largest bank by assets. Berkshire also sold its position in regional banks like PNC Financial Services Group and M&T Bank. And Buffett appears to be looking for an exit on former favorite Wells Fargo, selling shares gradually for the past several quarters.

While all of this was going on, Buffett and Berkshire pumped more than $2 billion of stock into Bank of America and increased the company's stake to 11.9% of outstanding shares, a move that required special regulatory approval.

Warren Buffett

Image source: Getty Images.

While we don't yet know what moves Buffett has made in 2021, he doesn't appear to have done much, since the cost basis of Berkshire's bank, insurance, and finance stocks after the first quarter was just slightly higher than it was at the end of 2020.

During the Q&A part of Berkshire's shareholder meeting, Buffett was asked why he had sold most of his bank stocks last year.

"I like banks generally, I just didn't like the proportion we had compared to the possible risk if we got the bad results that so far we haven't gotten," Buffett said. "We overall didn't want as much in banks as we had."

The Oracle of Omaha added: "The banking business is way better than it was in the United States 10 or 15 years ago. The banking business around the world -- in various places -- might worry me. But our banks are in far, far better shape than 10 or 15 years ago. But when things froze for a short period of time, the biggest thing the banks had going for them was that the Federal Reserve was behind them, and the Federal Reserve is not behind Berkshire. It's up to us take care of ourselves."

What to make of this

Buffett is one of the greatest investing minds of all time, if not No. 1, so it's important to realize that it will be difficult for us to see these moves exactly as he does. And I can certainly appreciate his remark about having too much exposure. Keep in mind, Buffett is managing an equities portfolio of hundreds of billions of dollars, so a few percentage points one way or the other is huge. He needs to think about safety much more than your standard responsible investor (Buffett sometimes refers to himself as Berkshire's "chief risk officer," and he did so again on Saturday). But it really is difficult for me to understand all these moves.

Goldman Sachs is not very loan-heavy, so it didn't really face the same degree of risk as other large banks like Bank of America and JPMorgan Chase. Investment banks also tend to do better in periods of volatility, and Goldman is working to build up its consumer bank and asset and wealth management divisions to generate steadier revenues. Goldman's stock is nearly 40% higher than it was prior to the pandemic, and many think it's still trading cheap.

Buffett's decision to load up on Bank of America and dump JPMorgan as an even bigger mystery to me. JPMorgan not only has a more diversified business, but its earnings power is greater, as well. The bank is also extremely safe. Its reserves for loan losses peaked at $34 billion during 2020, but the bank was prepared for scenarios where its reserves would have peaked at $52 billion. JPMorgan also engineered its way through the Great Recession better than any of the largest banks in America. Buffett officially eliminated Berkshire's stake in JPMorgan in the fourth quarter of the year. Meanwhile, the stock is up nearly 21% year to date at Tuesday's prices, and Buffett likely missed out on even more appreciation, considering he sold the bulk of his JPMorgan position in the third quarter of 2020.

Hindsight is, of course, 20/20, but the numbers are also the numbers, and Buffett missed out on huge gains on his former bank holdings that could just be getting started. Banks are potentially poised for a strong multiyear run, with earnings expected to jump in a rising-rate environment. Buffett did say Saturday that he is concerned about "very substantial inflation," so maybe he thinks too much inflation may kill loan demand, or that rising rates might reveal bad credit quality. However, cyclical stocks like banks and financials tend to perform better during inflation than high-growth tech stocks, which Berkshire has recently made a bigger part of its portfolio than financials.

Is Buffett done with banking?

Buffett has not abandoned the sector altogether. Bank of America is still Berkshire's second-largest holding behind Apple. Berkshire also still has large positions in American ExpressU.S. Bancorp, and Bank of New York Mellon. Buffett also may not be pulling all the strings here -- he's ceded a lot of Berkshire's investing authority to others who may be making the decisions. It has long been known that two of Buffett's lieutenants, Todd Combs and Ted Weschler, are much more involved in Berkshire's portfolio decisions now. In addition, Berkshire recently confirmed that Berkshire's Vice Chairman Greg Abel is expected to succeed Buffett as CEO when he eventually steps aside.

But with all that has recently happened, it does seem like it could be a while until we see Berkshire buy traditional bank stocks again.