Without independent savings, retirees risk struggling financially during their golden years. That's why workers are advised to sock away funds in an IRA or 401(k) -- to ensure that they have enough money to cover their living expenses when they're older. But data from J.P. Morgan Asset Management reveals that workers across all income levels are borrowing money from their retirement plans. And that's a troubling trend.

The problem with borrowing from your retirement plan

Though middle-income earners are more likely to take a loan from a retirement plan than lower and higher earners, it's a habit all three groups are upholding. Now at first glance, borrowing against your own retirement savings might seem to make sense. After all, that money is yours, so why deal with a bank when you can access funds that are already in your name and restore them when you can?

Here's the problem, though: Borrowing against a retirement plan is risky, and if you're not careful, you could wind up getting hurt financially.

Man covering his face

IMAGE SOURCE: GETTY IMAGES.

Now to be clear, only 401(k) plans actually allow you to take out a loan, so let's discuss those first. Whether you can get a loan from your 401(k) will depend on your plan's rules, but usually, you can borrow the lesser of $50,000 or 50% of your vested balance (the balance you own outright). And some plans allow you to borrow up to $10,000 even if that represents more than your vested balance.

Once you borrow that money, you'll generally need to repay it within five years, paying interest all the while, but to yourself, which isn't such a terrible thing. What is terrible, however, is that if you fail to repay your loan on time, it will be treated by the IRS as an early withdrawal, and that means you'll be subject to a 10% penalty for removing funds from your 401(k) prematurely, assuming you haven't yet reached the age of 59-1/2. (If you are 59-1/2, you're probably not contemplating a 401(k) loan, because you have the option to just take a withdrawal as you wish.)

Another thing: If you lose your job while you're in the process of repaying a 401(k) loan, you'll usually get just 90 days from that point to replenish those funds. Fail to do so, and that 10% penalty is back on the table. As such, 401(k) loans can be dangerous, and should be treated as a last resort only (though it's worth mentioning that you're better off borrowing against your 401(k) than taking an early withdrawal and being penalized off the bat).

Now let's talk about IRAs. Though there's no such thing as an IRA loan, you can technically borrow money from an IRA for a limited period of time via a rollover.

Rollovers come into play when you move funds from one tax-advantaged account to another. For example, you can roll money from a 401(k) into an IRA, or from one IRA to another.

When you do a direct rollover, you never see that money yourself; it lands in your new account automatically. With an indirect rollover, however, you get a check for the funds you've moving, and it's on you to get them deposited into your new account within 60 days. Fail to do so, and the amount you roll over will be treated as an early withdrawal, which means you face the aforementioned 10% penalty if you're not yet 59 1/2. Because you're limited to a 60-day period to repay your IRA with a rollover, it's not the most viable solution for borrowing money when you need to.

Be careful with your retirement savings

The purpose of socking funds away in an IRA or 401(k) is to have that money available to you in retirement. If you borrow against your nest egg and can't pay it back, you'll not only be penalized, but also, you'll have less money on hand during your golden years, when you need it the most. And that by itself is a good enough reason to leave your long-term savings alone, and find other ways to access cash when you need it. You may, in many cases, be better off borrowing against the equity you have in your home, or even applying for a personal loan if your credit is strong and you're able to qualify for a reasonable interest rate.