I have to admit that when I noticed that 45% of BofI Holding Inc.'s (AX 1.91%) outstanding shares are currently sold short, I had to check a second data source just to be sure the figure was correct (it is). So, to say that many investors likely have an unfavorable opinion of the bank right now would be a major understatement.

However, if you take a moment to look past the short-sellers' pessimism, it's easy to see that BofI's business is doing just fine.

Why shareholders might be worried

The proportion of outstanding shares that are sold short has risen considerably over the last few months. As of May 15, 2017, roughly 45% of BofI's outstanding shares are sold short. Allegations of misconduct have resulted in an ongoing legal saga, which is yet to be fully resolved. You can read a thorough discussion of the matter here, but to make a long story short, an ex-employee accused the bank's CEO of money laundering, and of creating a toxic culture in which the risks taken on by the bank were much higher than what was reported to shareholders.

Then in March, the New York Post reported that the bank was the subject of a Justice Department investigation. The company has denied knowledge of such an investigation, but given the rise in short-sellers, it appears the market is taking the allegations seriously.

To make matters worse, the bank's growth rate has slowed down a bit recently, which may be a cause of concern for some shareholders.

Relaxed man sitting at a desk.

Relax -- BofI's business is doing fine. Image source: Getty Images.

Why they shouldn't be worried

For starters, nearly all of the allegations against the bank have been proven false at this point. Plus, the bank has undergone several regulatory reviews, has had its financial statements audited multiple times, and generally appears to be firing on all cylinders.

As my colleague Jason Hall wrote in April, BofI's most recent quarter produced the highest net income in the bank's history, including strong growth in net interest income and loan originations.

Also, keep in mind BofI's key advantage -- its online-only platform -- which translates into industry-leading efficiency and profitability. In fact, the bank produces a return on equity of 21.1%, more than double the 10% industry benchmark, and its efficiency ratio of 31.73% is roughly twice as strong as those of most of the big U.S. banks.

Here's a good illustration of how BofI compares with its peers. The bank generates net interest income equivalent to 3.93% of its assets. Its expenses, such as salaries and benefits, equipment, and other noninterest expenses add up to 1.44% of assets, which translates to a margin of 2.49%. Meanwhile, the average U.S. bank with between $1 billion and $10 billion in assets produces net interest income of 3.44% but spends 2.96% of assets on noninterest expenses, for a margin of just 0.48%. In other words, BofI operates at five times the profit margin of its peer group.

In addition, BofI is actively expanding and diversifying its revenue streams away from its core portfolio of mortgage loans. For example, the bank's commercial- and industrial-lending business is growing rapidly. And in January 2017, BofI launched a pilot version of its unsecured-lending initiative, designed to compete with peer-to-peer lenders such as Lending Club, as well as with other banks that make such loans. The bank also plans to launch a retail auto lending business, and to expand the geographic reach of some of its current business lines.

Finally, BofI has achieved its growth without taking on too much risk. In fact, the bank's charge-off and non-accrual rates are significantly below its peer group's average.

BofI non-performing loan statistics.

Image source: BofI investor presentation.

What about the slowing growth rate?

To be fair, the bank's growth has slowed down a bit from the breathtaking pace of the past several years. Since 2011, BofI has grown EPS at an annualized 32% rate, and recent data reveals much slower growth than that.

However, I'd still call it an impressive level of growth -- 13% year-over-year asset growth, 12% deposit growth, and 13% EPS growth is hardly a sluggish pace. And as you can see, the bank's loan portfolio continues to grow steadily, with no signs of slowing down yet.

BofI loan portfolio growth, past 5 quarters.

Image source: BofI investor presentation.

The Foolish bottom line

Don't get me wrong -- BofI isn't a risk-free stock. However, I believe that when you look at the big picture, the potential for growth and long-term returns greatly outweighs all of the risks involved -- litigation-related and otherwise.