Buffett's Berkshire Hathaway (BRK.A -0.36%) (BRK.B 0.11%) has long considered American Express (AXP 0.07%) an integral part of its portfolio. He includes it in the list of so-called "big four" investments, which together dominate its holdings of public securities.

Berkshire Hathaway currently holds 151.6 million shares of the company, about as many split-adjusted shares as it held in 1995 (148.4 million). Thanks to share repurchases, Berkshire's stake in the card company has only grown as a percentage of ownership, despite the fact that the number of shares owned by Berkshire has hardly grown at all over the past 20 years. (Buffett alluded to the fact that Berkshire cannot buy more American Express because it is a bank holding company at the 2013 annual meeting.)

What may be more interesting is the extent to which Buffett has defended American Express. ValueAct reportedly acquired more than $1 billion of American Express shares, but blew out of the position when Buffett wasn't open to management changes at the card company. It's rumored that ValueAct wanted to replace key insiders like Ken Chenault, Amex's CEO, who has been with the company for decades. Buffett wanted nothing to do with it.

Cashing in
Increasingly, American Express appears to be a has-been company -- a once-great company that has slowly lost its competitive position and is destined to generate lower returns for investors going forward. Even Charlie Munger, Warren Buffett's right-hand man at Berkshire Hathaway has said just that.

Munger, who once said that "it would be easier to screw up American Express than Coca-Cola or Gillette, but it's an immensely strong business" at the 2000 meeting of shareholders recently sang a different tune. Munger opined that "Amex had a long period of achievement and prosperity. It doesn't look quite so easy going forward as it once did" at the 2015 meeting of Daily Journal shareholders.

Why doesn't Berkshire sell its American Express stake? Taxes certainly play a role. Berkshire's 151.6 million shares were acquired at an average cost basis of $8.49 per share. Selling at today's depressed price would trigger a huge tax bill, something Buffett has preferred to defer for as long as "forever."

Of course, Buffett could minimize the tax consequence by pairing American Express with a loser. The company's stake in IBM is both sizable and underwater. With an average cost basis of more than $171 per share at the end of 2014, Berkshire could take losses in IBM to offset gains in American Express.

The net result is that Berkshire would free up billions of dollars in cash, divest from a company that Munger (and many others) view as having a shrinking moat, and allocate the capital elsewhere. IBM shares, if truly a bargain, could be repurchased later.

Food for investing thought
While it's sometimes seen as sacrilege to disparage the decisions of one of the world's greatest investors, one has to wonder if, at times, a desire to defer taxes might lead to sub-optimal outcomes for Berkshire Hathaway's public stock portfolio.

This has been debated for years. You can go all the way back to the late 1990s, when many debated the rationality of owning Coca-Cola stock at 50 times (low-quality) earnings. It's become a classic case in earnings quality, as the soda company made numbers by tapping into gains by buying and selling its bottlers, a finite source of one-time earnings help. More recently, Buffett took a short-term round trip in Exxon shares, buying and later selling to fund an acquisition. There is some precedent for Buffett making active portfolio management decisions, albeit with smaller stakes.

That's neither here nor there. Buffett isn't much for change. Deep down, I think we all know it's unlikely that he becomes more active in managing Berkshire's biggest positions. But it's good food for thought to ask yourself if maybe, just maybe, Berkshire's shareholders might be better off from a little more activity.