It's been a pretty good year for banks as a whole. Through yesterday, the KBW Bank Index has topped the S&P 500 by a reasonable margin even though it's taken a dip over the past week.

Looking at the individual stocks, Bank of America (NYSE: BAC) was dragged down more than most during "foreclosuregate," and it's lost more than 18% so far this year. Both JPMorgan Chase and Goldman Sachs have spent most of the year drifting and are currently slightly down. Wells Fargo (NYSE: WFC) tacked on a modest 10%, but, interestingly, the best performer so far this year among the biggest banks has been Citigroup (NYSE: C) with a sweet 39% gain. Go Uncle Sam!

But stock-picking isn't a backward-looking pursuit, so we want to know what's ahead for the banking sector in the year ahead. My fellow Fools and I have already covered where to look for the best banking buys in 2011, so let's turn to what areas of banking you're going to want to avoid in 2011.

Investment banking
To start with, I'm not crazy about investment banks as an investment in general. The reward system seems to be forever tilted in favor of insiders, with investors bearing much of the risk for a fraction of the rewards. At the same time, the biggest of the investment banks -- Goldman, Morgan Stanley, JPMorgan -- still have very opaque operations and are difficult to analyze.

Looking specifically to next year, as I noted in a recent look at the investment banking outlook, there are definite opportunities, overseas in particular, but there are also going to be some stiff headwinds. These headwinds will provide drag for the focused investment banking companies like Goldman and Morgan Stanley as well as the diversified banking houses like JPMorgan and Bank of America.

Meanwhile, the smaller IB shops like Lazard and Greenhill, which I find much more interesting from a business perspective, don't have terribly compelling valuations.

Knocking at death's door
The banking sector is improving, but it's not quite out of the woods. As such, there are still plenty of banks that are on very shaky ground. Earlier this year, we saw Wilmington Trust hastily agree to sell itself to M&T Bank (NYSE: MTB) at a significant discount. Flagstar Bancorp (NYSE: FBC), meanwhile, came to market with a painfully dilutive capital raise to shore up its balance sheet.

So whether you're avoiding a fire-sale of the entire bank or a crippling dilution, you'll want to shy away from the banks with the worst vitals. What do these banks look like? They look a lot like Doral Financial (NYSE: DRL) and Hampton Roads Bankshares. Both banks are in the double digits in terms of nonperforming loans as a percentage of the total loan book, neither has taken overly conservative loss provisions, and neither has a particularly comforting capital base.

With plenty of banks out there that have stronger capital bases and credit trends moving in the right direction, I think investors are best served avoiding speculation on banks that aren't on a relatively clear path to recovery.

Mind the valuation
The banking sector got absolutely hammered during the financial crisis, and there were some legitimately solid banks that were trading at highly attractive prices. There were also some slightly lesser quality banks that were knocked down to truly absurd prices.

Investors as a group still seem to be pretty skeptical of the banking sector, but there has been significant recovery from the lowest levels of the downturn. Shares of some of my favorite banks -- US Bancorp (NYSE: USB) and M&T Bank among them -- are already trading at fairly lofty multiples. If investors continue to warm back up to the industry, the group with premium valuations will only increase. As a result, investors will need to be even more mindful in the coming year of what price they're paying for the banks they're buying.

More good than bad
In all, I think there are a lot of opportunities in the banking sector heading into next year. A big part of this reasoning is the investor skepticism that I noted above. By avoiding the three areas mentioned earlier -- investment banking, banks with severe balance sheet issues, and overvalued banks -- I think investors can sift through the banking universe to get to the most out of the industry in the year ahead.

My fellow Fools have ventured into the non-bank financial sector to find three picks that could pop due to the resurgence of the individual investor. Claim your free copy of the report and check out the picks.