Image source: Getty Images.

This should be a great time to be a credit card company, but you wouldn't know it from the stock charts. American Express (NYSE:AXP), Capital One (NYSE:COF), and Discover Financial (NYSE:DFS) are all off their highs, posting annualized returns of -21%, -8%, and -8%, respectively, since the start of 2015.

Favorable industry trends

Data from the Federal Reserve shows that credit card receivables and other revolving loan balances were recently growing at a rate of 7.7% annually, the fastest pace since the Great Recession. On an absolute basis, the amount of credit outstanding has grown to more than $950 billion.

Consumers, it seems, are loosening their grip on their wallets and increasingly using credit to fuel consumption. Importantly, borrowers are also paying their bills on time, resulting in low loss rates.

American Express's credit card book remains pristine, with charge-offs roughly half that of Capital One and Discover. Capital One's charge-offs have risen recently as a result of growth in its loan book. (The company is issuing more cards to new customers, which typically have a higher charge-off rate before leveling off as newly issued cards season and bad credits are terminated, according to conference call commentary.)

Despite favorable industrywide growth in balances, and relatively benign credit metrics, credit card stocks simply haven't kept pace with the broader market.

More than meets the eye

American Express's stock performance is perhaps the most justifiable. Having lost its relationship with Costco, and thus shedding 8% of its network spending and 20% of its loan book, the stock price merely reflects a declining business.  

The key challenge going forward is profitably deploying the $1 billion in proceeds it expects from selling its Costco book into marketing to drive card growth. Though the company reported strong card issuance in the first quarter, it admitted many of its cards were issued to soon-to-be former Costco cardmembers, who have little reason to stay with the company when Costco's new Citi card takes effect on June 20.

When it comes to Capital One, the commercial banking unit is the real troublemaker. The bank's net charge-off rate jumped from 0.29% in the first quarter of 2016, up from 0.02% during the year-ago period. Likewise, non-performing assets grew to 1.64% of assets, up from 0.31% last year.

This business simply cannot afford losses in the same way the card business can. Seeing as Capital One earned just 3.38% on its commercial banking loans in the latest quarter, a slight increase in nonperforming assets and charge-offs has an incredible impact on its bottom line.

Underperformance stems from the bank's foray into taxi medallion lending, a loan book that has increasingly run into trouble as taxis fail to compete with ride-sharing services like Uber. Meanwhile, the company's 2005 acquisition of Hibernia Corporation, a bank headquartered in Louisiana, greatly increased the company's exposure to oil credits, which have come under stress as oil prices plunged from $100 a barrel in 2014 to a recent price of just $51.

Discover Financial's book of riskier credits has performed well, despite its focus on lower-quality borrowers who are more likely to carry a balance. If there's any weakness, it's in its credit card receivables, which grew just 4% over the last year despite 7.7% growth in revolving balances across the industry.

The company noted that gas prices had a particularly strong effect on card spending, noting that 4% credit sales growth in the first quarter was actually more like 5%, if gas sales and the benefit of a leap year are thrown out.

This article represents the opinion of the writer, who may disagree with the “official” recommendation position of a Motley Fool premium advisory service. We’re motley! Questioning an investing thesis -- even one of our own -- helps us all think critically about investing and make decisions that help us become smarter, happier, and richer.