Source: Flickr user Hobvias Sudoneighm

Retirement planning can seem complicated, but often it's the simple, avoidable mistakes that hurt your returns the most. Whether you're fresh out of college and just getting started, or only a few years from retirement, here are three common mistakes that can make a big difference in your nest egg. 

1. Cashing out your 401(k) early 
When you change employers, don't make the mistake of cashing out your 401(k) early. Not only do you lose up to 20% in tax penalties, but you're also giving up the benefit of compounding returns. Even cashing out a small amount, say $5,000, early in your career can have major repercussions:

9.75% CAGR since 1900; 11.5% CAGR since 1983

By just rolling it over to an IRA or your new employer's 401(k), it will keep growing and compounding. That's a lot of catching up to do if you don't replace those funds early. It's best to avoid cashing out if at all possible. 

2. Ignoring "free" money
There are two ways you can get "free" money toward your retirement.

The first is your employer's matching funds. According to a Department of Labor study, employers contributed $122 billion to defined contribution plans, while employee contributions totaled $185.7 billion. If you're not contributing at least enough to get your employer's full match, you're leaving money on the table. 

The second way is a result of those contributions, which are often pre-tax deductions. For the median household, every $100 you contribute to your 401(k) would reduce your federal income tax by about $15. What does that look like in real terms? Increasing your contribution by an extra 2% of your pay -- only about $39 every two weeks -- would cut your federal taxes by $150 a month. Of course, things could be different based on your pay and tax situation.

3. Paying too much in fees 
This is the dirty secret of the mutual fund industry. Mutual funds and ETFs -- exchange-traded funds -- charge fees called expense ratios, and these expense ratios can range from the very low 0.17% Vanguard 500 Index Fund (NASDAQMUTFUND:VFINX), to more than 2% per year. Since you never see a bill for this fee -- the fund managers take it out of the fund's assets every year -- this hidden cost is easily overlooked but can reduce your returns by tens of thousands of dollars.  

Here's how much a 1% expense ratio can take out of your returns, as compared with a low-cost index fund, assuming same performance:

Based on actual past 30 year average returns of S&P 500, $5,000 per year invested

What can you do about it? 

Ideally, your employer should offer index funds like the Vanguard fund above in your 401(k). These funds typically have the lowest expense ratios and are pegged to a popular index, like the S&P 500, which has averaged around 11.5% in annual returns (not accounting for inflation) over the past 30 years. Instead of investing in actively managed mutual funds, go with index funds. Not only are the fees much lower, but they are far more likely to perform better than active funds -- the majority of which underperform the index they are measured against. 

Take control of things you can control
Nobody can predict the future, but there are things that you can do to put yourself in the best position to retire with the kind of money you'll need to have a great retirement. It starts with working on the things that you can control, like the three things above. 

These may seem like relatively small things, but compounded over 20 or 30 years, they add up to pretty big sums of money. Avoid these mistakes, because it could make all the difference when you're ready to retire.

Jason Hall and The Motley Fool have no position in any of the stocks mentioned. Try any of our Foolish newsletter services free for 30 days. We Fools don't all hold the same opinions, but we all believe that considering a diverse range of insights makes us better investors. The Motley Fool has a disclosure policy.