The banking sector had a tough year in 2020, as challenges from the coronavirus pandemic proved formidable. But there were banks that were able to overcome the challenges from higher credit costs and the ultra-low-rate environment to generate stellar returns for shareholders.

And the main thing the majority of these banks have in common is that they serve a niche segment of the market or offer niche lending products. Let's take a closer look at a few of them.

Niche banks

I recently wrote about some of the top-performing banks of the year in terms of stock price appreciation, and it quickly became clear that almost all of these banks cater to niche customers. This helped them steer clear of some of the main problems that most standard commercial banks ran into this year.

The entrance to a Silicon Valley Bank

Image source: Silicon Valley Bank. 

For instance, Silvergate Capital (NYSE:SI), which only went public in 2019, is a bank for those in the cryptocurrency space. It has built a real-time payments system for institutional investors and crypto exchanges so they can trade crypto assets like bitcoin faster. These types of customers help bring in lots of non-interest-bearing deposits and fee income. The bank has seen its stock price rise by more than 150% this year.

Another bank on the list is Live Oak Bancshares (NASDAQ:LOB), which specializes in loans administered through the U.S. Small Business Administration (SBA). When Congress passed the Paycheck Protection Program (PPP), which is essentially built on top of the SBA, Live Oak was uniquely positioned to take advantage.

Other top-performing banks included SVB Financial Group (NASDAQ:SIVB), which caters to the start-up, venture capital, and private equity community, and Triumph Bancorp (NASDAQ:TBK), which does a type of specialized lending in the transportation sector called factoring.

The pandemic forced the Federal Reserve to drop interest rates to near zero, which hurt the revenue outlook for banks. And unlike a normal low-rate environment, there has been very little loan demand in the commercial sector because most businesses are still worried about the uncertain economic outlook. The niche banks operate in segments that still had loan demand during the pandemic, which therefore allowed them to generate enough loan volume to offset the drop in loan margins. 

For instance, SVB Financial found more demand in the venture capital and private equity community for its specialized lending product, capital call lines of credit, as investors searched for higher yields in the private markets. While there is certainly competition for this kind of lending, it is not as common as your standard commercial and industrial or commercial real estate loan. And it is a specialized type of lending, so it likely requires some expertise to do it, presenting a barrier to entry. The same goes for Live Oak. While the SBA allowed every bank to participate in the PPP program, many had never done it before. Live Oak had already been the No. 1 SBA lender in the country, so it was ready to go.

Another great thing about niche lenders is that most of them also offer standard banking products like residential mortgages and commercial real estate loans. These banks are able to leverage their unique product sets to meet a special type of customer and form a unique relationship. Then they can use these inroads to potentially cross-sell standard banking products and create a more holistic relationship with the customer.

Look for niche banks to invest in

Even when regular commercial loan demand returns, the competition is going to be fierce, so these niche banks will still have an advantage. Look for banks whose niche can still thrive during the pandemic and in the new world after. And that niche doesn't have to be as specialized as those mentioned above. You might find a standard commercial bank that specializes in, say, lending to grocery stores or utility businesses, which have all fared pretty well during the pandemic, and will likely continue to after as well. This kind of advantage may not be as obvious, but if you find it, even if the bank isn't offering a specialized lending product, it could still do very well.

This article represents the opinion of the writer, who may disagree with the “official” recommendation position of a Motley Fool premium advisory service. We’re motley! Questioning an investing thesis -- even one of our own -- helps us all think critically about investing and make decisions that help us become smarter, happier, and richer.