Image source: The Motley Fool. 

American Express (AXP -0.18%) has been one of Warren Buffett-led Berkshire Hathaway's (BRK.A -0.41%) (BRK.B -0.78%) stock holdings for some time, but its performance lately hasn't been stellar. Even after a recent rebound, shares are down nearly 20% over the past year as a result of losing its Costco partnership and a few other negative developments. However, Buffett defended his Amex investment at Berkshire's annual meeting, and I think he's got a good point.

AXP Chart

American Express' troubles

The biggest reason for American Express' decline is the loss of its relationship with Costco, which made up about 8% of Amex's total card spending in 2015. In addition, AmEx's co-branding deals with JetBlue and Fidelity ended, and although this was a much smaller piece of the company's revenue, it's still a significant setback. There's also the looming threat of losing the Starwood Hotels partnership, as well, once the hotel company is acquired by Marriott, which has a co-branding deal with Chase.

Competition in the credit card business has never been higher. Banks are offering unprecedented introductory bonuses and 0% APR deals to attract customers, and this could create a problem for AmEx, which will have to offer its own attractive incentives (read: "spend more money"), thereby cutting into its profits.

Buffett still loves American Express

American Express is one of Berkshire's "big four" stock investments, and the company's stake now represents 16% of the credit card giant.

At Berkshire's annual meeting in late April, Buffett said American Express' business model is "under attack," although it's not enough to get him to sell. He called American Express an attractive business and said he is happy with the company and still likes the stock as an investment.

Buffett has repeatedly praised American Express' talented, shareholder-friendly managers, who return tons of cash to investors in the form of dividends and buybacks. I really can't stress enough the value Buffett places on great management -- he believes the right management team can add billions to a company's intrinsic value. Buffett also loves AmEx's rock-solid brand and ability to generate cash.

A great value and a durable competitive advantage

Admittedly, it will take some time to replace the Costco revenue, but the company plans to trim expenses by about $1 billion this year to help make up the difference.

Additionally, AmEx has had some interesting positive developments lately, including a deal with Sam's Club to start accepting the company's cards. AmEx is increasing its product offerings and pushing for more merchants to accept its cards, hoping to be as universally accepted as Visa and MasterCard products by 2019. AmEx has increased its marketing and promotions spending by 19% during the first quarter, so it's not taking its challenges lightly.

Results are promising so far. During the first quarter of 2016 the company increased its number of issued cards by three million, and increased revenue by 4% on a constant-currency basis. And, when you exclude the Costco and JetBlue portions of the portfolio, AmEx's loan portfolio actually grew by 11% over the past year.

American Express has the competitive advantage of a strong brand and an affluent, attractive group of cardholders, which allows it to charge merchants more for payment processing than rivals Visa and MasterCard. In fact, one report found that AmEx cardholders spend an average of 71% and 60% more than Visa and MasterCard customers, respectively. And, AmEx's widespread closed-loop network allows it to consistently deliver a return on equity in the 25% range.

AXP Return on Equity (TTM) Chart

Also, there is the global trend toward a cashless society, which benefits the entire payment processing sector.

Stick with AmEx

The bottom line is that although revenue will certainly suffer in the short term, American Express is likely still a long-term winner, and I believe investors with the stomach to make it through the in-between time will be handsomely rewarded.