Source: Flickr / Lee Forrest.

It's common knowledge that the refinancing boom is pretty much over, and that housing isn't as peppy as many hoped it would be in the sixth year of the economic recovery., however, has put the issue in stark relief. According to a recent press release by the site, the fourth quarter of 2013 was the worst three-month period for the mortgage industry since 2007, with nearly 19,000 mortgage sector workers losing their jobs.

For all of 2013, almost 32,000 positions  in the mortgage industry were lost – nearly as much as 2008, when nearly 37,000 people were given their walking papers. The sector experienced a veritable massacre in 2007, when about 89,000 employees lost their mortgage-related jobs.

The biggest banks top the list
As you might expect, the biggest players in the mortgage business led the way in layoffs for the last three months of the past year. Wells Fargo (NYSE:WFC), which commands one-third of the home-mortgage market, let over 6,000 employees go, while Bank of America (NYSE:BAC) gave pink slips to 4,000. JPMorgan Chase (NYSE:JPM) rounds out the list with nearly 3,900 terminated workers.

What happened? Much of the slowdown was due to rising interest rates, which squelched any further interest in the refinance rush by mid-2013. Another, more positive factor, is the decrease in troubled loans that banks must process. Bank of America, inarguably the king of the distressed-loan portfolio, dismissed approximately 3,000  in its legacy unit in the fourth quarter.

The cuts aren't over yet. B of A recently laid off 103  refinancing and default workers in Richmond, Virginia, in January. Wells Fargo  announced in February that 700 more nationwide job cuts are on tap for 2014, while JPMorgan said it plans to lay off an additional 6,000  full-time workers and contractors this year, as well.

Purchase loans sluggish
New loans for home purchases aren't taking up the slack, either. The refinancing boom seems to have had an unfortunate side effect: homeowners with newly minted refinanced loans don't want to part with them . They are sitting tight, reveling in their lower monthly payment – which is keeping them from buying up, and selling their current homes, furthering the housing market's low-inventory problem.

The new qualified-mortgage rules may be making it more difficult for buyers to obtain a mortgage, as well. The changes, meant to guarantee that a buyer will be able to afford his or her mortgage payments, went into effect in early January of this year. Borrowers need higher credit scores to obtain a conventional mortgage these days, with the average score jumping to 755  in February from last October's 711, an incredibly huge increase.

The release noted that layoffs have moderated for the first quarter of 2014, but the industry's workers still aren't out of the woods. There are those big layoffs being planned by JPMorgan, and Wells Fargo's estimated 700 terminations for this year could well mushroom if mortgage demand doesn't step up. It's early yet, but the perfect storm engulfing the mortgage sector might not be over just yet.

This article represents the opinion of the writer, who may disagree with the “official” recommendation position of a Motley Fool premium advisory service. We’re motley! Questioning an investing thesis -- even one of our own -- helps us all think critically about investing and make decisions that help us become smarter, happier, and richer.