Image source: iStock/Thinkstock.

Even though Warren Buffett's Berkshire Hathaway (NYSE:BRK.A) (NYSE:BRK.B) is one of the nation's most successful bank investors, the 85-year-old chairman said at this year's annual meeting that he wouldn't invest in 90% of the world's biggest banks.

"If you take the 50 largest banks in the world, we wouldn't even think about probably 45 of them," Buffett said while talking about the dangers that lurk in large derivative portfolios. "There can be enormous gaps in things that you thought were fully protected by collateral, or netting arrangements and that sort of thing. So I regard very large derivative positions as dangerous."

This may seem odd given that roughly a third of Berkshire Hathaway's $140 billion equity portfolio consists of bank stocks, such as Wells Fargo (NYSE:WFC), U.S. Bancorp, and M&T Bank, among others. And this doesn't include Berkshire's $5 billion position in Bank of America's (NYSE:BAC) preferred stock, which gives Berkshire the option to purchase 700 million shares of Bank of America common stock for $7.14 a share by 2021.

Importantly, however, most of the banks Berkshire has invested in are only minor players in the derivatives market. You can get a sense for this by comparing the notional amount of derivatives Wells Fargo has on its books relative to JPMorgan Chase (NYSE:JPM), Bank of America, and Citigroup (NYSE:C).

Data source: JPMorgan Chase, Bank of America, Wells Fargo, and Citigroup.

JPMorgan Chase leads the way with $50.7 trillion in derivatives, followed by Citigroup and Bank of America with $48 trillion and $42.2 trillion, respectively. Wells Fargo brings up the rear with only $5.7 trillion in notional exposure to derivatives.

These figures include derivatives that are offset by other derivatives, reducing the net exposure to a more manageable level. However, to Buffett's point, it's reasonable to assume that even the most sophisticated banks don't have a complete grasp on their derivatives portfolios, both because derivatives can be complicated, and because it's impossible to predict what will happen to the assets or events they're tied to.

We saw this at AIG during the 2008-09 financial crisis. The operating unit that amassed AIG's near-fatal derivatives exposure was one of the most sophisticated groups on Wall Street. The same was true at Long-Term Capital Management in 1998, a hedge fund that imploded after its highly leveraged bet on derivatives tied to foreign exchange rates went awry as a result of the Russian Ruble crisis.

We saw this even more recently at JPMorgan Chase. The nation's biggest bank by assets lost more than $6 billion three years ago after a trader's derivatives-based bet tied to the health of large American corporations went against the bank. The trader responsible for the loss was known among hedge funds as the "London whale."

Berkshire Hathaway's position in Bank of America is the exception to Buffett's aversion to banks with huge derivatives exposure. However, the structure of Berkshire's stake in Bank of America is unique, given that it doesn't own the bank's common stock outright. It instead owns preferred shares and was given the warrants to purchase common stock for free, essentially as consideration for Berkshire Hathaway's endorsement at what was then the peak of Bank of America's post-crisis travails.

This aside, the point is that Buffett thinks banks with big derivatives portfolios are too risky for his money. Given the source, this is probably worth considering for anyone who invests in big bank stocks.

This article represents the opinion of the writer, who may disagree with the “official” recommendation position of a Motley Fool premium advisory service. We’re motley! Questioning an investing thesis -- even one of our own -- helps us all think critically about investing and make decisions that help us become smarter, happier, and richer.