Saving up for a down payment on a home is easier said than done, especially when you're grappling with bills, debt, and other expenses that eat up most of your income. But for millennials who are already strapped for cash and trying to make progress in their financial lives, there's a disturbing tendency to tap into their retirement savings for a down payment on a home.

On the one hand, raiding a retirement plan for a down payment on a home may not seem like the worst idea. After all, that money is yours, and if you can withdraw it penalty-free, why not use it?

And to be clear, you can remove up to $10,000 from an IRA for the purpose of buying a first-time home without incurring an early withdrawal penalty, even if you're nowhere close to 59 1/2 (the age at which those funds become yours to withdraw unrestricted). If you and your partner each have an IRA, you can withdraw up to $20,000.

House with large grass lawn.

IMAGE SOURCE: GETTY IMAGES.

The rules are different with 401(k)s. Though you can't take a direct withdrawal for a home down payment without incurring a penalty, you can borrow against your 401(k) and use that loan for that same purpose. Still, taking funds out of a retirement plan is generally a bad idea, so if that's the strategy you're planning to use to buy your home, you'd be wise to rethink it.

You need that money for retirement

The reason you shouldn't take funds out of an IRA or 401(k) prematurely boils down to this: The purpose of that money is to serve as an income stream during retirement. The more you remove ahead of time, the less you'll have available when your golden years roll around. And that, in turn, could cause you to struggle later on.

Now you may be thinking: "What's the big deal if I remove $10,000 from my retirement savings? It's not that much money."

Oh, but it is. When you take an early retirement plan withdrawal, you don't just lose out on the principal sum you remove, you also lose out on potential growth on that money.

Let's assume your IRA investments give you an average annual 7% return and you remove $10,000 of your balance to buy a home at age 30. If you were to keep that $10,000 invested at an average annual 7% return for another 35 years, you'd turn it into almost $107,000. And that's a far more significant sum to be missing.

That's why you're better off leaving your retirement savings alone and finding other ways to come up with a down payment for a home. Those could involve slashing expenses for a year or two and living frugally to save money or getting a side job to drum up extra cash. You might even consider moving back in with your parents for a year, if that's possible, to save money on what's probably your biggest expense: rent.

Taking money out of your IRA or 401(k) for non-retirement purposes is a bad idea across the board, so don't do it. Delay homeownership for a year or two if you need to, or take the above steps to come up with your down payment more quickly. If you raid your long-term savings to buy that home, you may be happy you did in the near term, but you're apt to regret that decision down the line.