When people are planning for retirement, most of their worries usually center around not being able to build a large enough nest egg. That's a valid concern, but it's not the only thing that you have to worry about. You also have to consider the lifestyle you're bringing with you, and how it could impact your expenses in retirement. 

The consequences of putting a low priority on your health may not feel terribly dire when you're in your prime, but as your body ages, bad habits start to catch up with you -- and your wallet. Here are three ways that poor health could destroy your dreams of retiring comfortably and some advice on how to prevent that from happening.

Stethoscope lying on $100 bills

Image source: Getty Images.

1. It could force you to retire earlier than you planned

If you know that you're behind the curve on savings, and unlikely to have enough to retire comfortably at the traditional age of 65, you probably plan to continue working longer -- both to increase the size of your monthly Social Security payments, and to give yourself more time to set money aside. An Associated Press-NORC Center for Public Affairs Research study found that 23% of workers (including nearly 20% of those 50 and older) said they don't ever plan to retire, and an additional quarter of workers intend to work past 65.

That's a good way to reduce your retirement savings deficit -- if you can do it. The trouble is, you can't be sure that your health with allow it. If you become disabled, chronically ill, or simply physically unable to perform the tasks involved consistently, you might have to quit your job, whether you're ready or not. After that, you'll have to stretch out whatever retirement savings you have for a longer period than you'd planned. 

2. It could force you to start collecting Social Security early

If poor health forces you to retire ahead of your planned schedule, you may also have to file for Social Security benefits as soon as you're eligible at 62 in order to cover some of your living expenses. While many people choose to begin taking their retirement benefits at 62 regardless of their health, that choice so permanently reduces the size of your monthly checks. And if your lifespan is longer than average, that will also diminish the total amount you receive from Social Security.

The size of your Social Security checks is calculated based on the average indexed monthly earnings (AIME) for your 35 highest-earning years, and adjusted based on the age at which you start collecting your retirement benefit. To receive what the agency defines as the full benefit you're entitled to, you must wait until your full retirement age (FRA) to claim it -- an age that today falls between 66 and 67. If instead you begin at 62, you'll only receive 70% of your scheduled benefit per check if your FRA is 67 or 75% if your FRA is 66. You can also postpone the time you start collecting your benefits past your FRA. Doing that earns you delayed retirement credits that increase the size of those monthly payments. Workers who wait until they're 70 will receive 124% of their "full" benefit if their FRA is 67 or 132% if their FRA is 66. (Delayed retirement credits stop accruing once you hit 70 -- there's no benefit or value to waiting later than that to file.)

Filing for Social Security early might still be your best bet if your poor health has shortened your life expectancy because, in that case, you might not have enough time to reap the benefits of the larger checks you'd get by waiting. But if the conditions that keep you from working don't compromise your life expectancy, you could miss out on tens of thousands of dollars in benefits over your lifetime by starting Social Security prior to your FRA.

3. It could significantly raise your healthcare costs in retirement

Different studies have come up with a range of figures for how much healthcare will cost the average American retiree -- but they all agree that it's not cheap. Most studies looking at costs for a 65-year-old couple retiring in 2019 have found that they should expect to spend, over the rest of their lives, anywhere from $285,000 to $363,000 on healthcare. That doesn't include the portions of your expenses that are covered by Medicare, and it doesn't cover many common expenses that Medicare doesn't cover, like hearing aids or long-term care.

In particular, long-term care should be a big concern for those who neglect their health because it carries such high costs. Hiring a home healthcare professional can run more than $20 per hour, and the average semi-private room in a nursing facility costs about $225 per day or $6,844 per month, according to the U.S. Department of Health and Human Services. That adds up to more than $80,000 a year -- enough to drain most people's retirement savings within a few years.

How to prevent your health from hurting you financially in retirement

The most important piece of advice is also the most obvious: Make an effort to stay healthy. Exercise regularly, watch what you eat, make sure you're getting enough rest, and visit your doctor regularly for checkups and preventative care. Doing what you can to remain mentally and physically fit has the potential to save you serious money, both in the short term and the long term.

Also, be sure you're properly factoring healthcare costs into your financial plan for retirement. You can use the figures mentioned above as estimates, but you might want to use a more pessimistic number, especially if you're many years from retirement because inflation will drive these costs up further. Bump up your retirement savings rate to account for these extra costs if you hadn't fully planned for them. And consider adding a long-term care insurance policy to your retirement plan as well.

The best place to start saving for retirement healthcare costs is in your health savings account (HSA), if you have one. Individuals are allowed to contribute up to $3,500 annually in 2019 and families may contribute up to $7,000. Those 55 and older may add an extra $1,000 above these limits. These contributions reduce your taxable income in the year you make them, and if you use the money in those accounts for qualifying medical expenses, you won't pay taxes on it in the year you spend it, either.

One way or another, healthcare is likely to eventually become a major expense for you in retirement. But taking better care of yourself today can go a long way toward reducing those bills later.