I'll admit it -- I'm as bad as everyone else. When it comes to banking, I spend most of my time talking about big players like Bank of America (NYSE: BAC) and Citigroup (NYSE: C).

For a lot of reasons, that focus makes sense. The big bank stocks are much more widely held than the smaller banks; they were the nuke nobody wanted to go off during the financial meltdown; and they'll be hit the hardest by the forthcoming financial reform bill.

But when it comes to talking about trouble in the banking sector, the big boys may no longer be the place to look. Most of the biggest banks also have huge trading arms that have been steadily subsidized by low interest rates from the Federal Reserve. That's helped them shore up their balance sheets and prepare to take a crack at the next cyclical upswing.

But when we look at the industry's smaller players ... well, let's just say the picture doesn't look quite so pretty.

Be afraid -- be very afraid
When the TARP program began, the public reserved most of its ire for the big banks. After all, they were primarily responsible for holding our financial system for ransom. But today, most of the big-name banks have repaid their TARP borrowing, while numerous smaller banks that received their own TARP funds are still hanging onto them.

Worse yet, while many smaller banks haven't rushed to pay back TARP, a growing number have stopped making their minimum payments altogether. In all, 91 banks missed the most recent TARP dividend payment, up from 74 banks that missed the prior dividend. Add that worrisome stat to the 82 banks that have failed so far this year -- versus 140 for all of last year -- and you end up with a picture of many smaller banks continuing to limp along.

The tale of the tape
Most investors are unlikely to find themselves digging into the TARP-dividend-dodging banks. Pacific Capital Bancorp and Anchor BanCorp Wisconsin are two of the larger deadbeats, and they carry market caps of $50 million and $15 million, respectively.

However, that doesn't mean that everyone else is A-OK.

When judging the safety of a bank, I like to look at the allowance for credit losses as a percentage of nonperforming loans. It's basically a measure of how much cushion the bank has if conditions don't improve.

At least as far as that measure is concerned, the smaller banks seem to offer far greater cause for concern. From an investor's perspective, it's particularly interesting that while the big banks have built a bigger safety cushion than many smaller banks, their valuations still trail many of the little guys'.

Compare the four largest U.S. banks to four smaller banks that haven't been as aggressive about provisioning for losses.

Bank

Market Cap

Allowance for Credit Losses / Nonperforming Loans

Price-to-Book Value

Bank of America

$159 billion

121%

0.75

JPMorgan Chase (NYSE: JPM)

$153 billion

224%

0.98

Wells Fargo (NYSE: WFC)

$146 billion

92%

1.35

Citigroup

$115 billion

171%

0.76

New York Community Bancorp (NYSE: NYB)

$7 billion

19%

1.29

Hudson City Bancorp (Nasdaq: HCBK)

$7 billion

22%

1.21

TFS Financial

$4 billion

37%

2.28

East West Bancorp (Nasdaq: EWBC)

$2.5 billion

33%

1.24

Source: Capital IQ, a Standard & Poor's company.

Certainly, a couple of simple metrics don't make an entire investment case -- or undo it, for that matter. Historical loan loss experience, potential recoveries, loan portfolio composition, and a host of other factors go into determining loan loss allowances. A low percentage doesn't necessarily mean a bleak future.

Still, let's pick Hudson City Bancorp as an example. Back in 2005, the bank had loan loss allowances of $27.4 million at year's end, against non-performing assets of $20.4 million. That 142% coverage ratio declined to 102% in 2006, and then started plummeting as the real estate market tanked, going to 44% in 2007, then 23%, and finally inching down to 22%. At the end of the first quarter, Hudson City had $166 million in loan loss allowances, against nonperforming loans of $745 million.

Why is this such a problem? As loans continue to sour, the bank may have to continue significantly stepping up its loan-loss provisions. In the first quarter of this year, the bank recorded $50 million in loan loss provisions, up drastically from $20 million in the prior year. As these provisions continue to run through the bank's income statement, investors might find themselves increasingly disappointed with the bottom line.

Buyer beware
If all you read is the big-print, front-page stories, you might be inclined to think that you're much safer avoiding the big banks in favor of the little guys right now. To be sure, I think there are definitely good investments among the vast number of smaller U.S. banks.

However, if you're going to venture into the world of small- and mid-cap banks, be sure to put your research cap on -- right now, I see a lot more danger among the small banks than the larger ones.

Since the financial crisis, the banking industry hasn't been quite so generous with the dividends. But this stock has.

Try any of our Foolish newsletter services free for 30 days. The Motley Fool has a disclosure policy.

Fool contributor Matt Koppenheffer does not own shares of any of the companies mentioned. You can check out what Matt is keeping an eye on by visiting his CAPS portfolio, or you can follow Matt on Twitter @KoppTheFool or on his RSS feed. The Fool's disclosure policy assures you no Wookiees were harmed in the making of this article.