If you've managed to save up $100,000 for retirement, you're already doing great. But you're going to need a lot more if you want to retire comfortably.

Here's the good news: By getting to that $100,000 milestone, you're already done with a lot of the heavy lifting. Growing your $100,000 into $1 million is relatively straightforward. Here's how to get there.

Three piggy banks: one small, one medium, one large.

Image source: Getty Images.

1. Don't just save -- invest

If you want to grow your retirement savings from $100,000 to $1 million, you need to invest.

Investing is the act of buying an asset that will allow your money to work for you instead of you working for your money. You can invest in paper assets like stocks and bonds, hard assets like real estate or artwork, or you could invest in yourself, perhaps building your own business.

There are dozens of different styles of investment. Some require practically zero work, like passive investing, while others are more involved, like owning individual rental properties. What's more, each asset has varying expected returns and volatility. A highly volatile investment has a higher chance of producing returns that are greatly different than the expected returns.

The closer you are to retirement, you likely want to invest in assets with lower volatility in order to preserve capital. That said, low-volatility assets generally have lower expected returns. So, if you need to grow your retirement funds and you're willing to work longer if need be, you may explore a more aggressive portfolio than other people your age.

2. Avoid high fees

One of the big killers of investment returns is high fees.

If you work with a financial advisor who charges you 1% of your assets under management, that's $1,000 per year on your $100,000 portfolio. If that advisor puts you in high-fee mutual funds that charge another 0.75%, that's another $750. And considering stock investments return around 7% or 8% per year, that's a huge drag on your returns. That could be as much as a quarter of your total annual returns eaten up by fees.

If you take the time to learn to manage a basic portfolio yourself and use simple index funds to establish a diversified portfolio, you'll end up paying very few fees. Index funds generally have expense ratios less than 0.15% with many much lower than that.

Those savings will compound over time. Imagine adding 1.6% to your annual returns every year, guaranteed. That'll add up to tens of thousands of dollars over an investment career. If you want to get to $1 million sooner, you need to be mindful of fees.

3. Minimize your taxes

Staying mindful of how the government taxes your investments will help you reach $1 million much sooner.

Investing in a retirement account like a 401(k) or IRA will protect your investments from incurring taxes. In fact, contributions to traditional retirement accounts are tax deductible, enabling you to save even more.

Investments outside your retirement accounts are taxable but only on dividends and interest or when you sell an investment. With that in mind, it makes sense to put your investments paying high dividends or interest payments in a tax-deferred retirement account. More importantly, it behooves you not to trade out of your positions very frequently, which can also help improve investment returns.

If you end up having to pay a 22% income tax on your investment returns every year, because you're trading in and out of stocks, that's a significant cut to your net returns. If you end up having to take money out of your portfolio to cover your tax bill every year, it's going to be much harder to reach $1 million.

Putting it all together

If you want to grow your $100,000 in savings to $1 million, it all comes down to choosing appropriate investments and managing a portfolio to keep fees and taxes low. If you can put all three of these paths to $1 million together, the last thing you'll need is enough time for your investments to do the work.