What if I said your 401(k) could be losing thousands or even tens of thousands of dollars each year? I'm not talking about bad investments. I'm talking about an often overlooked and seemingly innocuous factor that's been silently gouging peoples' retirement savings for decades: 401(k) fees.

You're not alone if you thought your 401(k) was free, but you're sadly mistaken. All 401(k)s charge some fees, though many people don't realize it because they never receive a bill. The money comes out of your account automatically without you knowing. Unfortunately, there's no way to stop this outward flow of dollars, but there are ways to slow it.

401(k) jar with coins

Image source: Getty Images.

The true cost of 401(k) fees

Your 401(k) fees cover administrative costs, like record keeping. The investments you choose may also have their own costs, like the expense ratios (annual fees) mutual funds charge shareholders. While some things like account rollovers may cost a set dollar amount, 401(k) fees are usually a percentage of your assets. That means the more you have in the account, the more you pay every year.

A 1% 401(k) fee may not sound that bad until you consider it's $10,000 per year on a $1 million portfolio. And you'll keep paying that 1% every year as long as you have the account. Over a lifetime, that could easily add up to tens of thousands or even hundreds of thousands of dollars lost.

Consider two individuals, both 25 and planning to retire at 65. Each makes $50,000 per year and puts 10%, or $5,000, of their earnings toward retirement each year. They both anticipate a 7% annual rate of return. The only difference between the two is that one has a 1% 401(k) fee and the other has a 0.5% 401(k) fee.

After 40 years, the person with the 1% 401(k) fee will have about $774,000 in the retirement account. But the person with the 0.5% fee will have $878,000. That's a difference of $104,000. The person with the 1% 401(k) fee would have to save an additional $674 per month for 40 years to retire with the same amount as the person with the 0.5% fee.

How to minimize your 401(k) fees

So what can you do to stop your 401(k) from hemorrhaging money? The first step is to figure out how much you're actually paying in fees. Check your plan summary or the prospectus for your investments. Your plan administrator or company's HR department may be able to assist you as well.

Typically, larger companies have more affordable 401(k)s because they can spread administrative costs among more employees. But your fees will also depend on what your money's invested in.

Try moving your money to lower-cost investments if you're paying more than 1% of your assets in fees annually. Talk to your employer if there aren't any lower-cost investments available. The company may be willing to add more options, like index funds. These are mutual funds known for their solid returns and low expense ratios, often less than 0.5%.

Consider moving your money to an IRA if your employer refuses to offer lower-cost investment products. IRAs typically have much lower fees and give you greater freedom to invest your money in more investment products. The only time you may want to keep your high-fee 401(k) is if you're getting an employer 401(k) match that's enough to offset what you're paying in fees each year.

Think about rolling your 401(k) over to an IRA if you leave your current job. This will reduce the number of retirement accounts you have to manage and it may also reduce the amount you pay in fees. If your new employer offers a 401(k), ask about its fees so you understand what you're signing up for, and take the above steps to minimize your fees, if possible.

Saving for retirement is hard enough without giving up tens or even hundreds of thousands of dollars to 401(k) fees. If you don't know what you're paying in fees, check into this today. It could make a huge difference in your retirement savings.