Roughly two-thirds of Americans have already retired by the time they reach age 66, according to a survey by LIMRA Secure Retirement Institute, while only 18% leave the workforce before age 61.

For some people, retirement can't come early enough, and they're counting down the minutes until they can walk out of work and never come back. And after you've been working for 40-odd years, you may tell yourself that retiring a year or two early won't make a difference.

Woman lying in hammock reading on an e-reader

Image source: Getty Images

But early retirement could make the difference between living comfortably in your golden years and scraping by on what's left of your nest egg.

If you're on the fence between retiring early and working a few more years (or if you're just desperate to say goodbye to your boss forever), consider a few situations where it's not the best idea to retire early.

1. You may be expecting too much from Social Security

When you're planning for retirement, it's perfectly fine to count on Social Security benefits -- after all, you've earned that money by paying your payroll taxes for many years, and you will receive at least some benefits once you're eligible to apply.

However, Social Security is likely to undergo some form of cuts over the next couple of decades, which means your benefits could be lower than your current estimates. The problem is that baby boomers are retiring in droves, and pretty soon Social Security will be paying out more in benefits than it takes in through taxes. As a result, Social Security is facing a cash shortage. If lawmakers don't make changes to the program, its cash reserves will be depleted by 2034, which would force the program to cut benefits by up to 23%, according to the Social Security Board of Trustees.

Congress will likely find a way to patch the program and prevent drastic benefit cuts. However, even assuming you'll receive your full expected benefit, it's better to save aggressively enough that you could get by without Social Security in retirement. Then your Social Security benefits will serve their intended purpose: acting as a financial safety net. And if you don't need those benefits whatsoever, they can serve as "fun money," allowing you to enjoy your newfound freedom even more. Reaching that stage of financial readiness may take a few extra years of work, but it will be more than worthwhile.

2. You're not eligible for Medicare yet

You'll become eligible for Medicare at age 65, so if you choose to retire before then, you'll have to provide your own health insurance.

When it comes to insuring yourself, you have a few options. The easiest (and likely least expensive) option is to use your spouse's plan, assuming you're married and your spouse is still working. The downside is that, of course, it requires a working spouse, and you may be hoping that you and your one-and-only can retire at the same time.

COBRA is another option, and it allows you to stay on your employer's plan even after you retire. However, it's typically more expensive than what you paid while you were an employee, and the benefits only last for 18 months. So if you're planning on retiring a few years before 65, this option may not be the best. Your employer may offer a retiree health insurance plan, but this benefit is increasingly rare, and these plans tend to be more expensive because they provide coverage for a segment of the population that requires more healthcare.

You could also enroll in a healthcare plan through the Affordable Care Act marketplace. This could be one of the more, well, affordable options available, but it's still not cheap: While plans and premiums vary by state, the average 60-year-old, non-smoking couple with no dependent children and an annual income of $40,000 can expect to pay about $273 in monthly premiums for a silver-tier plan.

3. You're still paying off (a lot of) debt

Some debt isn't a bad thing. In fact, 70% of baby boomers are still paying off debts during retirement. The key is differentiating good debt from bad debt.

Consider your mortgage, for example. It's not necessarily a bad thing to still be paying off a mortgage when you retire, especially if you can comfortably afford your monthly mortgage payment and all your other necessities. But there are other types of debt -- such as credit card debt -- that can put a serious crack in your nest egg.

The average American age 55 to 64 carries about $8,158 in credit card debt, according to a study by ValuePenguin. Say you're paying 17% APR on that debt and you're putting an extra $200 per month toward paying it off. It will take you over 5 years to pay off all that debt, and you'll end up paying over $4,000 in interest during that time -- which can be especially tough if you're retired and living on a fixed income.

Paying off credit card debt before you retire has two major benefits. First, it helps you avoid the minimum payment trap and can save you hundreds (or thousands) of dollars in interest. And it can also free up your credit line in case of an emergency. While maxing out your credit cards isn't necessarily advised, if you ever get to the point that you really need the extra credit, at least you'll know it's available.

Most people dream of retiring early, and that dream may very well be within reach. But the last thing you want is to retire early and then realize you're not as prepared as you thought you were. So before you hand in your two weeks' notice at work, make sure you've thought everything through -- that hammock on the beach will still be there in a few years if you decide to wait.