Earlier this year, Warren Buffett invested in Capital One Financial (COF 0.16%) for the first time and cut back on other bank stocks, like US Bancorp. The move may have taken some by surprise as the bank -- the ninth largest in the country with some $469 billion in assets -- has been a spotty performer over the years. 

Over the last 20 years it has posted an annualized return of about 3.6%, and over the past 10 years, the annualized return is just 4.6%. Both of these numbers trail the S&P 500. 

However, during the past three years, Capital One has doubled the large-cap benchmark index, posting an annual return of 17.2%. As one of the largest credit card issuers, Capital One has benefited in recent years from a pandemic-led surge in digital spending, and it should continue to take advantage of the gradual shift to digital payments. It is also dirt cheap, with a price-to-earnings ratio of less than 8 while trading with a price-to-book ratio of just 0.73. Maybe those factors appealed to Buffett, the chairman and chief executive officer of Berkshire Hathaway. 

So, let's look at the past three years. If you invested $10,000 in Capital One just after the start of the pandemic in August 2020, how much would you have today?

Beating the benchmark -- and its rivals

On Aug. 20, 2020, Capital One closed at $63.80 per share. Three years later, on Aug. 20, 2023, the stock opened at $105.27. That's a 65% increase with a roughly 17.2% annualized return per year. If you reinvested the dividend over those three years, that's about a 20% return per year. And that includes the bear market of 2022, when Capital One fell 35%.

Over that same three-year period, the S&P 500 was up 8.9% on an annualized basis, and 10.6% with dividends reinvested. So, Capital One has far outperformed the benchmark in the past three years.

If you invested $10,000 in Capital One back on Aug. 20, 2020, you'd have seen that increase to about $17,280 -- without adding a cent to the investment. If you added $100 per month to that initial $10,000 investment, you'd have about $22,035.

Among the top 10 U.S. banks over that three-year period, only Wells Fargo and Goldman Sachs had a better annualized return as of Aug. 20. It topped all of the others, including such banking stalwarts as JPMorgan Chase and Bank of America.

Buffett buys more

Based on these numbers, Capital One has clearly been a buy over the past three years. But is it a buy right now? Buffett certainly thinks so, as he added Capital One in the first quarter for the first time and then boosted his position in the second quarter.

I think it is a stock worth considering. But it has been very cyclical, even more so than typical banks because of its reliance on consumer spending, since credit cards are its primary revenue generator.

While the economy has performed better than expected, uncertainties remain. In the second quarter, Capital One has seen increases in net charge-offs and nonperforming loans, and it has boosted its provisions for credit losses. I certainly wouldn't sell it, but there may be better options out there among banks or credit card issuers.