American Express (AXP -0.84%) has seen its stock do well lately, with gains of more than a third over the past year. After suffering dramatic losses when the company lost out on its past card partnership with big-box warehouse retailer Costco (COST -0.12%), AmEx shares have climbed as the company has worked on moving forward with growth initiatives.

Yet even with the gains that shareholders have seen, American Express still faces some challenges ahead. Any three of the following things could lead to AmEx giving up some of its recent advances and losing ground.

1. Stronger competition in payments

American Express has done a good job of carving out a niche in the lucrative high-end card market, attracting customers with cards that capture the attention of status-seeking cardholders with rich benefits and the promise of exclusivity. Yet one issuer of rival Visa (V 0.05%) cards has answered with a black card of its own, and some believe that the rewards are superior to what AmEx Centurion offers. Any erosion of American Express' reputation in serving high-end customers could take away one of the company's main competitive edges.

Meanwhile, broader competition in the payments space could eat into AmEx's core business. With companies like PayPal creating their own payment networks, American Express faces the prospect of getting cut out of the equation. Admittedly, PayPal and similar services have made partnerships with AmEx and other card giants in a way that could prevent full replacement. But having to share profits with more parties will almost certainly eat into margin figures and hurt results at American Express and its peers.

American Express serve credit cards and the app on a smartphone.

Image source: American Express.

2. Weakening creditworthiness

Another benefit that American Express gets from serving high-end customers is that credit risk tends to be somewhat less for its clientele than for more mainstream card issuers. That offsets some of the added risks involved in issuing cards directly in the first place, which is a practice that Visa and peer MasterCard (MA -0.08%) have avoided.

The U.S. economy has been expanding at a good clip for years, and default rates have been relatively low. However, with the prospect for higher interest rates ahead, borrowers could start to see more pressure on their personal balance sheets. That, in turn, could lead to an increase in late payments and defaults, and any resulting need to boost credit loss reserves would eat into American Express' profits.

3. Adverse litigation results could hurt the company

Finally, American Express has had to deal with litigation for years concerning its relationship with merchants. Earlier this month, the U.S. Department of Justice (DOJ) decided to drop its antitrust case against the card company, which had questioned whether American Express could ban merchants from encouraging their own customers to use cards from rival networks like Visa and MasterCard. After winning a lower-court decision, the government had its victory reversed at the appellate court level.

The DOJ decision resulted from its choice not to petition the Supreme Court for review of the appellate court judgment. However, several state governments had been parties to the litigation against American Express, and they have moved forward with filing for certiorari in asking the Supreme Court to hear the case. What if the Supreme Court hears the case and decides that the appellate court was incorrect in letting American Express off the hook? According to the card company, an adverse ruling could have a material downward impact on its business going forward.

American Express has done well lately, and its stock has reflected its recent success. Yet with so many issues outstanding, there's always the possibility that the rebound in AmEx shares could come to an abrupt end. Investors need to keep their eyes on these concerns to see if any of them grow into larger threats to the card company.