Despite losing some altitude in the last few months, the stock of Bank of Internet (NYSE:AX) has still more than quadrupled over the last three years. If you bought B of I stock back in 2011, I'd love to come visit your summer home.

BOFI Chart

BOFI data by YCharts.

But if you're looking at the bank as an investment opportunity today, caution is required. A dive into the numbers reveals two potential issues that might constrain future growth.

Bank of Internet's story
Unlike the behemoth big banks, Band of Internet has a relatively simple business. B of I offers personal and business checking and savings accounts and uses the deposits to make mostly real estate loans for single-family homes and multi-family properties. Real estate lending accounts for around 80% of B of I's loan portfolio, with about half of its real estate loans in southern California.

Bank of Internet's high growth hurdle
Rapid deposit growth of 33% per year since 2010 has fueled the company's ability to make loans. But the company is showing signs of a slowdown in deposit growth.

Backing out the approximately $500 million in customer deposits it purchased of H&R Block's bank assets, through three quarters, B of I is only on pace to grow deposits around 15% this fiscal year. 33% growth would have been impossible to sustain long term, so a slowdown isn't entirely unexpected, but it suggests Bank of Internet will have to slow its loan growth unless deposit growth picks back up.

B of I's stretched loan portfolio
The only thing that has grown faster than B of I's deposit base is its loan portfolio.

The company's total loans have a compound annual growth rate of 45% since 2010, outpacing even its extraordinary deposit growth. This has stretched B of I's loan to deposit ratio all the way to 110%, which is way out of whack compared to the rest of the industry.

Bank of Internet has been extremely aggressive in lending at a time when many of the other big mortgage lenders are holding more dry powder.

With interest rates at historic lows and the Fed intimating that rate hikes could be here as early as next year, Bank of Internet looks the least prepared of its competitors to take advantage of the higher rates.

Although B of I's CEO said on the most recent conference call that the company stands to benefit when interest rates rise, unless deposit growth picks up, the company that is already lending 110% of its deposits will likely have to look to alternative and higher-cost funding if it wants to continue lending at such a prodigious pace. 

As an investor with a long time horizon, I love coming across a small but rapidly growing business that is shaking up its industry. But in banking, there is something to be said for a long-term track record. When I see first-rate institutions like US Bancorp and Wells Fargo staying conservative while interest rates are low, I worry that B of I is being a little too aggressive for what the interest rate climate dictates.

Foolish final thoughts
Bank of Internet has been a rocketship over the past few years. Its branchless model gives the company a cost advantage that is the envy of the banking industry.

But declining deposit growth and aggressive lending makes me skeptical that the bank can keep up its impressive post-crisis run. Despite being 30% off its March highs, the bank trades at a relatively expensive 3.1 times book value.

As much as I like what this business could be, I'm going to watch how it handles higher interest rates and wait for a lower price.