Real estate has been a great source of wealth for plenty of today's millionaires -- but it wasn't always like that.

According to data from Robert Shiller in his book Irrational Exuberance, inflation-adjusted home price between 1890 and 1990 rose by a mere 0.21% per year. Between 1950 and 1997 the return shrank even more, to just 0.08% per year.

Source: Robert Shiller, Irrational Exuberance

However, in 1998 the home prices got a dose of steroids and hardly looked back, practically doubling in just a 10-year span. Unfortunately, we all know how that story ended, and it wasn't pretty. Subprime mortgages went bust in record numbers, home prices collapsed, new homes sales slumped more than 70%, and some large banks and investment firms went under or were forced to sell their assets for pennies on the dollar.

We're now more than five years removed from the low-point of the market and it would appear that everything is peachy within the homebuilding sector again. Last summer homebuilder confidence hit a seven-year high as historically low lending rates continued to allow consumers to purchase new homes and refinance existing loans at very attractive rates.

But all is not what it appears.

The one figure that should have homebuilders and investors terrified
Housing data has been mind-numbingly weak over the past couple of months despite mortgage rates for March averaging just 4.34%, which is historically quite low.

Source: St. Louis Federal Reserve.

Just this month we've seen new home sales plummet 14.5% to lows not seen since July 2013, existing home sales dipped 0.2% from February, and building permits fell nearly 3% from the previous month. 

Yet none of these results represents the scariest figure for homebuilders. That comes courtesy of research firm Inside Mortgage Finance via The Los Angeles Times, which on Friday reported that new loan originations for the mortgage industry hit a 14-year low in the first quarter of 2014. That's right, folks: People were taking out loans at a faster pace to buy homes during the height of the housing crisis when interest rates were considerably higher than they are now. Based on Insider Mortgage Finance's figures, new mortgage originations of just $215 billion were down 58% from the year-ago period, and 23% from the sequential fourth quarter. 

This is troublesome in a number of ways for both the U.S. economy, and for investors.

Who could this hurt the most?
Perhaps the biggest concern is that interest rates are historically low to begin with. What's going to happen when lending rates begin to normalize to historic norms around 6%, or perhaps even higher, for a 30-year mortgage?

There's reasonable concern that the Federal Reserve's tapering of QE3 will eventually lead to a rise in interest rates. As part of the Fed's stimulus plan it's been purchasing near-equal amounts of mortgage-backed securities and long-term U.S. Treasuries. With the Fed easing up on these purchases $10 billion at a time per month it's possible that bond prices will fall, causing yields to rise (remember, bond prices and bond yields have an inverse relationship). Were this to happen I suspect it would be devastating for the housing industry and a number of other sectors given how rapidly loan originations dropped off when rates rose around 100 basis points from April to July of last year.

Source: US Navy, Wikimedia Commons.

Specifically, I see this as a problem for homebuilders that have less pricing leverage and shakier balance sheets. Hovnanian (NYSE:HOV), a homebuilder that blankets the East coast and Southwestern U.S., for instance, has a scary $1.6 billion in net debt and has only produced one meaningful year of profits since 2006. Since 2006 its shareholder equity has completely disappeared with its share count more than doubling. If lending rates rise, I suspect it will be one of the most directly affected homebuilders. Considering that Hovnanian's deliveries already dropped 4% in the first quarter, we could be witnessing the beginning of a fresh downturn for weaker builders.

But homebuilders aren't the only sector primed to be whacked by weakening loan originations. Consider the adverse impact that lower mortgage originations will have on banks. If lending rates were rising significantly this wouldn't be an issue because higher yields would net financial institutions more interest income. At the moment, however, this isn't the case.

Take Wells Fargo (NYSE:WFC), the nation's largest U.S. mortgage lender as the perfect example of a bank that's being hurt by falling loan originations. In its first-quarter report from just two weeks prior, Wells Fargo announced that residential mortgage originations tumbled 28% to just $36 billion from $50 billion in the year-ago quarter. Furthermore, its gain on sale margin dipped 16 basis points to 1.61%. This combination of stagnant lending rates and weakening mortgage originations is bad news for big lenders like Wells Fargo.  

Broad-market investors in tracking ETFs that follow the S&P 500 (SNPINDEX:^GSPC) may want to be on full alert as well. Combined, the financial services and real estate sectors make up about 17% of the S&P 500's value, but they aren't the only companies affected. Think about how a slowdown in the housing sector trickled down to affect other sectors during the recession, such as home-improvement stores, building specialists such as roofing and flooring suppliers, and even home appliance makers and retailers.

Skeptics have been looking for the next viable reason why the market may head lower, and this just might be it.