Warren Buffett is one of the greatest investors of all time, so it certainly grabs your attention when he has held a stock for almost 60 years. Credit card company American Express (AXP 0.07%) has become one of Warren Buffett's most significant holdings -- the stake is worth roughly $25 billion and is more than 8% of Berkshire Hathaway's portfolio, Buffett's holding company.

But the past can't speak for the future, and there are numerous signs that American Express could be past its glory days. Here are three reasons why 2022 could finally be the year to sell American Express.

1. Not enough growth

Warren Buffett originally bought his stake in American Express in the early 1960s when the young company got caught up in a Salad Oil scandal that scared the market. He purchased 5% of the company for approximately $20 million, a stake that is worth billions today.

Person cutting up a credit card.

Image source: Getty Images.

Over six decades since then, American Express has evolved into one of the most iconic credit card brands globally. However, as a large company today with a $126 billion market cap, American Express' best days could be behind it.

AXP Free Cash Flow Chart

AXP Free Cash Flow data by YCharts

The company's key operating numbers have struggled over the past decade. Despite business essentially recovering to pre-pandemic levels, American Express has grown revenue a total of 34%, free cash flow 35%, and net income 57% over the past 10 years. This total growth breaks down into low-single-digit annual rates.

A valuation can influence a stock's returns, but American Express is likely to produce returns that mirror the pace the business is growing at over the long term. In other words, low-single-digit growth should eventually produce low-single-digit returns, which might not be good enough for many investors.

2. Continued travel industry struggles

American Express faces some challenges that could make it harder to rediscover a faster growth rate moving forward. The company's credit cards are popular for people who spend on travel and entertainment, which accounted for roughly a quarter of American Express' U.S. business before the pandemic.

After travel began to rebound throughout 2020 as more of the population got vaccinated, the recent surge of the coronavirus omicron variant has already started to put a damper on travel once again.

Nobody can predict the future; what is known is that the COVID-19 virus continues mutating, producing new variants. While vaccines and other medications continue arriving to help combat the pandemic, nobody can be sure how the coronavirus will continue to impact travel spending in the near term.

American Express has reported that travel spending still lags pre-pandemic levels through three quarters of its 2021 fiscal year. Travel billings contributed 10% in the first quarter of 2021, 12% in the second quarter, and 14% in the third quarter. Investors will want to see fourth-quarter numbers to see how omicron has influenced travel. The coronavirus has been around for two years, so it's harder to discard the pandemic as a short-term business threat at this point.

3. The buy now, pay later threat

Looking further out, American Express remains threatened by the buy now, pay later (BNPL) competition rapidly heating up. Society is pivoting -- older consumers are quickly retiring, and younger generations such as Millennials are entering their prime earning and spending years. 

American Express reported that Millennials and Gen Z are now driving growth in its business. Spending among this age group is up 40% compared to pre-pandemic 2019 levels, while other age groups are at 10% growth or less. This underlines the importance that these young customers have to the long-term success of American Express and its competition.

However, BNPL companies like Affirm, Klarna, and Afterpay have become very popular over the past year, especially with younger consumers. According to a study, 36% of BNPL users in the U.S. are under 25, and 41% are Millennials. Meanwhile, some experts predict that the BNPL market will grow 22% per year through 2028, and at least a chunk of this growth may be at the expense of traditional credit card companies like American Express.

Get with the times

American Express seems like an American classic that is past its prime. It may have staying power, continuing its low-single-digit growth rate, but in such a big consumer spending market, there should be better opportunities out there.

By contrast, Affirm is one of my favorites. It has partnered with many of the leading e-commerce sites in the U.S., and that could place it among the largest benefactors of BNPL's rapid growth moving forward.