The average U.S. worker is still struggling to save for retirement -- for instance, nearly half of baby boomers have no money set aside for their golden years at all, according to a report from the Insured Retirement Institute. It's just one financial concern among many for Americans: Money is our most common source of stress, according to a survey from Northwestern Mutual.

Saving for retirement, though, can be particularly stressful because of its complexity. Given that our individual financial goals can vary widely, there's no one right way to do it.

That said, there are a few wrong ways to prepare for retirement -- choices that you might not think would have a significant impact, but that could end up costing you loads of cash over the long term.

Young man with cash flying out of his wallet

Image source: Getty Images

1. Not contributing enough to your 401(k) for the full employer match

Employer matching 401(k) contributions are essentially free money. So if you're not routing enough money into your retirement account to earn the maximum amount your company will give you, you're missing out. At first, those matching contributions may not seem like they make a big difference in your retirement account balance. But over time, they do add up.

For example, say your company matches 100% of employees' contributions up to 3% of their salary. If you earn $50,000 per year, that's $1,500 per year extra you could be getting. But if you only contribute $1,000 per year to your 401(k), you only earn a $1,000 match. Here's what your total savings would look like if you continued saving at that rate versus boosting your contributions to $1,500 per year to earn the full match (assuming you're earning a 7% annual rate of return on your investments):

Years of Contributions 401(k) Balance When Contributing $1,000 per Year (+$1,000 Match) 401(k) Balance When Contributing $1,500 per Year (+ $1,500 Match)
10 $27,688 $41,449
20 $82,155 $122,986
30 $189,299 $283,382

Source: Author's calculations

In other words, in this scenario, contributing an extra $500 a year to your 401(k) can add nearly $100,000 to your net worth after a few decades. This calculation also doesn't account for any salary raises you may get down the road, which would increase the amount your employer will match. In that case, you could miss out on even more money by failing to capture the full match.

2. Not investing in stocks

The stock market can be intimidating, and many people associate it with risk. For that reason, some people try to play it safe with their savings by avoiding the stock market to some degree. In fact, 53% of those saving for retirement say they're stashing at least some of their cash in a savings account, according to a survey from Morning Consult.

On the surface, that doesn't seem like a bad idea. Savings accounts are safe, and they avoid the market downturns that inevitably come with investing in the stock market. However, even the highest-yielding savings accounts only have interest rates of around 2% per year. If you keep your cash in one of these accounts over several decades, your savings may not even keep up with inflation -- meaning your money could actually lose value the longer you keep it there.

Of course, savings accounts do still have their merits. When you're saving for short-term financial goals and may need to pull your cash at a moment's notice, you can't beat a high-yield savings account. But if you want to be able to accumulate hundreds of thousands of dollars in savings over a few decades, the stock market is the way to go. This is especially important if you're struggling to save, because even a few thousand dollars can go a lot further in the stock market than a savings account.

It's easier (and less scary) than you may think to invest in the stock market -- in fact, if you're putting your money in a 401(k) or IRA, you're already doing it. While you can't completely eliminate risk when putting your cash in the stock market, there are ways to limit it -- particularly by investing in index and mutual funds, which are essentially large collections of stocks. While annual returns vary greatly based on the individual fund and the overall market, you'll typically see returns of anywhere to 6% to upwards of 10% per year while still limiting your risk.

3. Waiting too long to start saving

When you're in your 20s and just starting your adult life, retirement is likely the last thing you're thinking about. However, the earlier you start to save, the easier it will be to establish a robust retirement fund.

The average age workers start saving for retirement is 31 years old, according to a survey from Nationwide, although nearly one-quarter of Americans started saving in their 40s or later, Morning Consult found. While it's never too late to save at least something for retirement, it becomes exponentially more difficult to save enough the longer you wait.

For instance, say you want to save $750,000 by age 65. If you started saving at age 25, you'd need to save just over $300 per month, assuming you're earning a 7% annual rate of return on your investments. But if you wait until age 40 to begin saving, you'd have to stash away approximately $1,000 per month to reach your goal, all other factors remaining the same. For many people, saving that much is simply not feasible, so they're potentially leaving thousands of dollars on the table by waiting too long to start saving.

Saving for retirement can be difficult, and there's no one-size-fits-all approach to preparing for the future. But by avoiding these potentially harmful financial habits, you can make it easier on yourself and save more money at the same time.