Success is simple. Do what's right, the right way, at the right time.
-- Arnold H. Glasow 

The advice above is amusingly unsimple. Most of us would like to do the right thing the right way at the right time, but it's not always so clear what to do, how to do it, or when to do it. When it comes to retirement, the right thing to do is to save and invest during your working years.

But when is the right time to stop working and retire? That answer is different for everyone, and the average retirement age was recently 63, but many people should consider holding out until at least age 66.

Middle-aged woman at her retirement party, with a cake in front of her and balloons, presents, and coworkers.

Image source: Getty Images.

Here are three reasons why you might not want to retire until age 66.

Reason No. 1: You can't afford to retire before age 66

Many people can't retire as early as they'd like because they're not where they should be with their retirement savings and investments. According to the 2017 Retirement Confidence Survey, about 24% of workers said they had less than $1,000 saved for retirement, and a whopping 55% had less than $50,000. Only 20% had socked away $250,000 or more. That suggests that millions are not likely to have enough to retire on at age 62, 65, or perhaps even age 70.

The well-known "4% rule," which suggests that you might withdraw about 4% of your nest egg annually in retirement -- adjusting for inflation over time -- can help you think about how much the money you've saved might last. A nest egg of even $400,000 would give you just $16,000 for the year. The table below offers other possibilities:

Nest Egg Size

4% Withdrawn in First Year

$250,000

$10,000

$300,000

$12,000

$400,000

$16,000

$500,000

$20,000

$600,000

$24,000

$700,000

$28,000

$800,000

$32,000

$900,000

$36,000

$1 million

$40,000

Source: Calculations by author.

Clearly, many of the sums above will seem insufficient as retirement income. Of course, most of us should have Social Security benefits as part of our retirement income, but with the average annual Social Security retirement benefit recently at about $16,400 a year, it will only help so much.

Another reason to not retire too early is if you're risk averse. By delaying retirement, you can also reduce risk -- such as the risk of running out of money late in life, the risk of getting hit with an unexpected financial curveball when you don't have your current income to help you deal with it, and so on. The more you're able to save for retirement, the better you'll be able to deal with any unexpected financial needs that life throws your way, such as a costly health crisis, or a sudden need to help out your child.

A piggy bank wearing glasses, on top of a stack of books, in front of a blackboard on which is a graph going upward, with the word "retirement"

Image source: Getty Images.

Reason No. 2: A bigger war chest

Here's another reason to aim high when you're stockpiling funds for retirement and considering retiring a bit later: Your retirement might last a very long time. If you retire a little early, such as at age 62, and you live to age 95, an age that plenty of people will reach, you're looking at a retirement that's 33 years long. You'll probably need a lot of money socked away for that.

Fortunately, working for a few more years can do wonders for your financial health. Imagine, for example, that you have $400,000 in your retirement account at 64. If you work three more years and retire at 67, and your nest egg grows by an annual average of 8%, it will turn into more than $500,000. That's a big difference -- enough to generate an additional income of about $4,000 if you follow the 4% rule. And that's assuming you didn't even add anything to your nest egg during those three additional years of work. Add $10,000 annually, and your grand total might be more like $525,000 or $540,000.

Working a few more years offers additional benefits: Yes, you'll be able to save and invest a lot more money, resulting in a bigger nest egg. But you'll also be delaying when you start drawing down your nest egg, too. Thus, it will have to sustain you for fewer years, and it will last longer.

In addition, if you're enjoying employer-sponsored health insurance, you can keep doing so for a few more years, helping you save money on the healthcare expense front, as well. Fidelity Investments has estimated that a 65-year-old couple retiring today will spend, on average, a total of $275,000 out of pocket on healthcare during their retirement.

If you're somewhat close to retiring, the table below will show you how much more you might amass over several relatively brief time periods if your money grows by an annual average of 8%:

Growing at 8% for

$10,000 Invested Annually

$15,000 Invested Annually

$20,000 Invested Annually

3 years

$35,061

$52,592

$70,122

5 years

$63,359

$95,039

$126,719

10 years

$156,455

$234,682

$312,910

12 years

$204,953

$307,429

$409,906

15 years

$293,243

$439,864

$586,486

Source: Calculations by author.

two signs pointing in different directions -- one says "more" and one says "less"

Image source: Getty Images.

Reason No. 3: Bigger Social Security checks

Let's return to Social Security. Yes, the average annual retirement benefit is around $16,400. But if your earnings were above average during your working life, your benefits will be higher than that average, too -- though still not a princely sum. On top of that, you can make your monthly benefit checks even bigger by delaying starting to collect Social Security.

For every year between your full retirement age -- which is 67 for those born in 1960 or later -- and 70 that you delay, your payout will grow about 8% larger -- up to a total of 24% larger if you delay from age 67 to 70. That can turn an expected monthly check for $2,000 into one for $2,480, a meaningful difference.

That bigger check can make delaying retirement seem like a no-brainer move, but things are not quite what they seem. Yes, your checks will be bigger, but remember that by delaying starting to collect them, you're ending up with many fewer checks.

The system is actually designed so that if you live an average-length life, it won't really matter whether you start collecting at 62 -- the earliest age at which you can do so -- or 70 -- the latest age to start collecting. It will be a wash. So take your level of health and your family's general longevity tendencies into account when deciding when to start collecting Social Security. There are other strategies you can employ to increase your Social Security benefits, too.

There are lots of solid reasons not to retire too early, but depending on your situation, it might still make good sense to do so. Keep the considerations above in mind and do your own calculations. Or consult a financial professional.