When you're in your 20s, you might think you don't have enough money to start putting away for retirement, whereas when you're in your 40s or 50s, you might worry that it's too late to start. In a perfect world, you would start putting money aside as early as you can.

But it doesn't always have to be that way, and even if you invest later, it might not be too late. People normally make more money as they get older, and so it could be easier to put aside a big lump sum into an investment at that stage of your life.

Below, I'll show you how much you will need to invest at various ages to get to $1 million by the age of 65, without putting your money at risk.

Low-risk investments are key

Billionaire investor Warren Buffett says the first rule of investing is to not lose money; his second rule is to not forget the first one. It can be tempting to buy riskier stocks as a way of making up for not investing earlier, but the danger is that losses can set you even further back in your retirement plans.

When it comes to low-risk investments, two great examples are Johnson & Johnson (JNJ 1.49%) and AbbVie (ABBV -2.04%). These two drugmakers are among the top healthcare companies in the world.

Johnson & Johnson is spinning off its consumer health business so it can focus more on growth, specifically in pharmaceuticals and medical devices, which should allow it to pad its already strong results. Last year, the company netted a profit of $17.9 billion, down from net income of $20.9 billion the year before. It pays a dividend that yields 2.8% and has been increasing its payouts for 60 consecutive years, making it a Dividend King.

AbbVie is another highly profitable business to invest in. The company behind the drug Humira, which treats rheumatoid arthritis and other conditions, reported a profit of $11.8 billion last year. That was a slight improvement from the $11.5 billion it reported a year earlier.

Some investors are concerned about Humira's loss of patent protection this year. But AbbVie has a strong pipeline and is confident that it can more than replace the lost revenue in the long run with the promising immunology drugs Skyrizi and Rinvoq. AbbVie pays an even higher dividend yield of 3.9%, and like Johnson & Johnson is a Dividend King.

Both stocks are good examples of relatively safe businesses that could outperform the S&P 500, which averages long-term returns of around 10%.

How much would you need to invest to get to $1 million?

Investing in stocks like AbbVie and Johnson & Johnson can put you in a good position to earn an average total return of 11% per year.

If, however, you want a more diversified option, you could consider investing in exchange-traded funds that hold a range of healthcare stocks. That way, you aren't dependent on the success of a single stock. That also makes it easier if you need to invest a large lump sum.

Here is a breakdown of how large a one-time investment you would need to make at different ages to get to $1 million, assuming you retire at 65 and earn an 11% annual return.

Chart by author.

As you can see, by waiting to 50, you'll need considerably more money to invest than if you started 10 years earlier at 40. But there's no rule that you have to invest it all at one time; you can invest whatever you can over the years, and that can help accelerate your portfolio's growth. 

Buy and hold is the best strategy for retirement

Whether you're investing in individual stocks or an ETF, having discipline and being able to remain invested for the long haul is an important trait to develop.

Meme stocks and high-risk investments can quickly undo any gains you have achieved over the years. Even if your annual gains are modest, there might be a year when your investments outperform and make up for below-average years.

By buying and holding, you stay the course, and provided that you've invested in low-risk, quality investments, odds are that you'll be much better off by the time you get to retirement than if you were frequently trading and chasing the next hot growth stock.