American Express (NYSE:AXP) has a long history of being a pioneer in the payment card business, and its membership model has given the company a cachet among high-income customers that its rivals lack. Yet the company has run into some strategic challenges lately, not the least of which came when warehouse retail giant Costco (NASDAQ:COST) chose to terminate its partnership with American Express. Even though the loss of the Costco relationship has hurt AmEx's business, the financial giant has nevertheless started to build up more positive momentum both in its U.S. and international business segments. The question investors have to ask is how risky American Express is right now and whether future return potential justifies that risk.

1. Credit risk

One thing that investors in American Express have to remember is that unlike card networks that don't issue their own cards, AmEx has retained exposure to potential defaults from its cardholders. For instance, during the financial crisis, default rates skyrocketed, and that left American Express vulnerable to losses that non-issuing card networks didn't have to address.

AmEx's upper-end clientele lessens credit risk to some extent. Yet the company's customers also have higher demands, and even the wealthy can get into serious credit trouble. Moreover, American Express has aggressively sought new customers, and the danger is that a lack of underwriting diligence could lead to bringing on less-creditworthy cardholders that in turn could boost default rates in the future.

People paying with AmEx.

Image source: American Express.

2. Competitive risk

American Express got an early start in the card business, but its competitors have grown even larger than it is. Visa (NYSE:V) and Mastercard (NYSE:MA) have been able to grow their payment networks by leaving behind the potential profit from issuing cards directly, instead entering into partnerships with financial institutions, which issue cards and are responsible for any credit risk problems.

The result has been aggressive expansion by Visa and Mastercard, especially into underserved overseas markets. American Express has a well-known brand around the world, but Visa and Mastercard have become even more ubiquitous, and that could continue to pose competitive threats for AmEx well into the future.

3. Merchant risk

Historically, one of American Express' competitive advantages has been its ability to charge its merchants more than its rivals do. Fees that are sometimes 50% higher than what Visa or MasterCard charge haven't been uncommon in the past, but the contracts that merchants sign with AmEx prevent them from urging their customers to use an alternative that would be less costly for their businesses.

However, litigation against American Express has been underway for years, in which the federal and various state governments allege that provisions limiting merchants from recommending other cards amount to impermissible antitrust violations. Recently, the U.S. Justice Department chose not to appeal an adverse judgment to the Supreme Court, but several state governments plan to keep fighting the case. Regardless of the outcome, the risk for AmEx is that merchants will finally grow tired of its restrictions and stop accepting American Express entirely.

4. Buffett risk

Finally, one thing to remember about American Express is that Warren Buffett is a huge investor in the card company. As of March, Buffett's Berkshire Hathaway (NYSE:BRK-B) owned about 17% of AmEx's stock, and recent stock buybacks from American Express have led to that percentage rising over time.

Berkshire recently asked the Federal Reserve to boost the maximum amount of American Express stock it can own from 17% to just under 25%. Berkshire has committed to be a passive investor in AmEx in order to get around another rule that would have limited its ownership stake to just 10%.

If the Fed grants Buffett's request, then American Express shareholders will be able to breathe a sigh of relief. Buffett has said that he wants to hold onto shares of AmEx. But if the central bank doesn't comply, however, then Berkshire might have to trim its position in the card giant, and that could put downward pressure on share prices in the immediate future.

American Express is recovering from its Costco loss, and things look increasingly favorable for the company. However, being aware of these potential risks will help you be better prepared for whatever happens in the future -- good or bad.

Dan Caplinger owns shares of Berkshire Hathaway (B shares). The Motley Fool owns shares of and recommends Berkshire Hathaway (B shares), Costco Wholesale, Mastercard, and Visa. The Motley Fool recommends American Express. The Motley Fool has a disclosure policy.