Whether you're just starting out or have plenty of experience, you know how tough it is to accumulate the hefty savings you'll need in order to retire when you want. That's why you really can't afford to take a huge step backward by undoing the work you've already done.

Unfortunately, though, that's exactly what a staggering number of people have done lately. According to a Hewitt Associates survey of 170,000 retirement plan participants, 46% of all those who left their jobs chose to cash out their 401(k) plan balances, take the money, and run. The less workers had in their plans, the more likely they were to cash out.

A bad move
What many workers don't understand is that there are dire consequences to taking money from your old 401(k). Because you got a tax break when you first added money to the 401(k), the IRS forces you to include that cash as taxable income, which usually results in your paying higher income taxes. Moreover, if you're under age 59 1/2, then you'll generally have to tack on another 10% in early-withdrawal penalties. That's a steep cost for some quick cash.

Even more importantly, by taking money out of your 401(k), you undo years of hard work establishing your retirement nest egg. Sure, it may not seem like a huge amount of money right now. But even if you start small, the impact of compounding combined with continually adding money month after month can eventually build up huge amounts of wealth. Consider: If you'd invested just $100 in Dell (NASDAQ:DELL) 20 years ago, your shares would be worth nearly $25,000 today. The same $100 invested in Wal-Mart (NYSE:WMT) back in 1979 would bring a cool $55,000 if you sold your shares now.

Better to stick with it?
You might wonder, though, whether it really makes sense to leave your money with your old employer. 401(k) plans have gotten a lot of negative press in the wake of the market meltdown, as horrible returns combined with high hidden fees and other expensive charges have taken a huge bite out of investors' retirement savings. Moreover, you often have a limited menu of investment options, severely restricting your ability to tailor your retirement portfolio to your own needs.

It's true that many companies have 401(k) plans that don't offer the best investments at a good price. Many employers, including Ford Motor (NYSE:F), Merck (NYSE:MRK), and JPMorgan Chase (NYSE:JPM), have actually faced lawsuits from employees alleging various breaches of companies' fiduciary duties to employees. In some cases, as with Southern Company (NYSE:SO) and AIG (NYSE:AIG), claimants actually received settlements from their employers.

Staying with your existing 401(k) sometimes does make sense, though. If you own company stock, for example, you may have to stay within the plan in order to keep it. For most investors, though, the problems that 401(k) plans pose make it easier to get out and seek better options elsewhere.

Your best choice
In order to avoid unnecessary fees and get access to a virtually unlimited range of investments, the easiest thing to do is to roll your 401(k) balance over to an IRA. Those direct transfers don't incur any taxes or penalties, and many discount brokers can help you make a transfer easily without any hassle. Some will even give you special deals like money back or free trading.

If you've just taken a cash distribution, you may have a second chance to get it right. If you can get the money into a rollover IRA within 60 days of when you received the money, then it's treated much the same way as a direct transfer, and you won't face any taxes or penalties.

Raiding your 401(k) may seem like an easy way to get cash, and it may seem like a waste of time to move small balances to new accounts. But the long-term impact to your retirement prospects of cashing out is huge. By rolling over an old 401(k) to an IRA, you can stop yourself from making a mistake you'll regret.

Need help doing a rollover to a new IRA? Find out everything you need to know about IRAs right here.

Fool contributor Dan Caplinger always tries to move forward. He doesn't own shares of the companies mentioned in this article. Dell and Wal-Mart are Motley Fool Inside Value recommendations. Southern Company is a Motley Fool Income Investor selection. Try any of our Foolish newsletters today, free for 30 days. Unlike tricky 401(k) plans with their hidden fees, the Fool's disclosure policy puts everything right out in the open.