While testifying before Congress earlier this year, Federal Reserve Chairman Ben Bernanke was asked if there was precedent for the economy recovering from a recession while the housing market stayed weak. He looked puzzled, thought about it for a moment, and replied: "It's normal for housing and construction to be an important part of the recovery."

No, in other words.

The key to getting the economy back on track is deleveraging -- paying off debt accumulated during the bubble years. For households, the vast majority of that debt is in the form of mortgages. In that sense, housing not only caused the recession, but it's by far the biggest impediment to recovery.

But it goes beyond paying off debt. In a report released last week, the Fed asked a simple question: How many of the 8.7 million jobs lost during the recession were tied to the housing market?

Its answer: about 40%. And that probably understates it.

Of all the jobs lost between December 2007 and February 2010, 2 million -- or 22% of the total -- were construction jobs. The Fed estimates another 1.6 million jobs were lost due to knock-on effects of those construction workers cutting back on spending after they lost their jobs. Add those together, and nearly 40% of all jobs lost during the recession were due to the soured housing industry.

That's where the Fed's analysis ended, but I think you can take it a step further.

Since peaking in 2006, the financial industry has lost about 500,000 jobs. Most of those jobs were tied to housing in one way or another. There are actually fewer financial jobs today than there were in 1999, even though the overall population has grown by 13%. And that trend doesn't look like it's letting up anytime soon. More than 2,500 banks -- or one-third of the total -- cut jobs last quarter, according to The Wall Street Journal. Big banks like Citigroup (NYSE: C) and Goldman Sachs (NYSE: GS) are slashing headcounts as once-lucrative trading avenues disappear -- partly because of new regulations, but partly because the housing industry that used to fuel such trades is now a fraction of its former self.

And then there's the impact lower housing prices have on consumer spending. According to the Congressional Budget Office, every $1,000 decline in home values causes households to cut spending by $20 to $70 a year. That's the power of the wealth effect, or the theory that people's willingness to spend is based on how wealthy they feel. 

Since 2006, U.S. housing values have declined by $7 trillion. Using the CBO's assumptions, that translates to a decline in consumer spending of $140 billion to $490 billion per year.

How many jobs has that cost? It's hard to say. As a rough rule of thumb, every 2% decline in GDP causes unemployment to rise 1%, eliminating about 1.5 million jobs. At the high end of the CBO's estimate, a $490 billion decline in consumer spending could be responsible for 2.4 million lost jobs.

Add that to the 3.6 million jobs lost from construction and the 500,000 lost finance jobs, and you get up to 6.5 million jobs lost directly or indirectly to the housing market -- nearly 75% of the total lost during the recession.

Now you know what Bernanke is talking about.

As ugly as that is, the reality is many housing-related jobs never should have existed in the first place. When thinking about housing-related job losses and the potential recovery, you have to make a distinction between jobs tied to the housing market and jobs tied to the housing bubble. Many fall into the latter group. According to economist Mark Zandi, 23% of all new jobs created from 2003-2006 were housing-related. Many won't come back -- nor should they.

But things may be turning a corner. We're now nearly six years past the housing bubble's peak, home prices are back to 2003 levels, and housing construction is far below the level of household formation. Housing may be closer to the bottom than some think. And with so many jobs tied to the industry, even a small upturn could boost the entire economy.  As Warren Buffett said this summer, "We will come back big time on employment when residential construction comes back. ... You will be surprised, in my view, how fast things change when that happens." It can't come soon enough.

Check back every Tuesday and Friday for Morgan Housel's columns on finance and economics. 

This article represents the opinion of the writer, who may disagree with the “official” recommendation position of a Motley Fool premium advisory service. We’re motley! Questioning an investing thesis -- even one of our own -- helps us all think critically about investing and make decisions that help us become smarter, happier, and richer.