When planning for retirement, choosing where to live is almost as important as deciding when to finally retire. While many people choose to stay put and age in place, 40% of Americans age 55 and older say they'd like to move at least one more time, according to a survey from Freddie Mac.

Although it may be tempting to pack your bags and follow wherever your heart leads, not all retirement destinations are created equal. Every state is different when it comes to taxes, quality of healthcare, and the cost of living. And unless you have a very healthy nest egg (and let's face it, most people don't), choosing the wrong place to retire could quickly drain your retirement fund, leaving you penniless at the end of your life.

Hammock on the beach

Image source: Getty Images.

Before you close on your dream retirement locale, make sure it isn't in one of the least-friendly states for retirees, ranked by Kiplinger:

1. New York

This may not come as a surprise, considering New York City is one of the most expensive cities in the U.S. And although other cities in upstate New York offer more-affordable costs of living, the state still imposes higher-than-average property and sales taxes, which can take a significant bite out of your retirement savings. Healthcare isn't cheap in this state -- Kiplinger estimates the average total cost of healthcare for a retired couple in New York is around $433,000. Furthermore, New York has one of the highest poverty rates among residents age 65 and older -- at 11.4%.

2. Massachusetts

Massachusetts has both high taxes and expensive healthcare (averaging $450,000 per couple). The cost of living is one of the most expensive in the U.S. -- 38% higher than the national average. And if you're looking to buy a home in Massachusetts, be prepared to spend a pretty penny. The median home costs around $407,000, according to Zillow, and the median rent is $2,600 per month.

3. Maryland

The cost of living in Maryland is 17% higher than the U.S. average, and like other states on the list, it also has higher-than-average taxes and healthcare costs. Retired couples spend an average of $436,000 on healthcare, and Maryland has an estate and inheritance tax -- so the money you leave behind may be taxed even after you're gone.

4. New Jersey

Property taxes in New Jersey are the highest in the country, with an average rate of 2.40%, versus the national average of 1.19%. The state also has a higher-than-average cost of living, with a median home value of around $326,000. Property tax on that median New Jersey home costs nearly $8,000 per year. 

5. Connecticut

If you decide Connecticut is the place you want to enjoy your golden years, prepare to be hit by hefty taxes. Not only does the state have some of the highest property taxes in the country (with an average property tax rate of 2.02%), but you'll also have to pay taxes on your Social Security benefits (which you're exempt from in many other states). On top of that, the cost of living is 24% higher than the national average, and the average healthcare cost for a retired couple living there is a whopping $439,000.

So where should you retire?

When you're deciding on an ideal retirement destination, don't consider just one or two factors, but look at the big picture instead. For example, a state may have high property taxes, but if you plan to rent rather than buy, it's a moot point. Similarly, perhaps a state has high income tax rates, but not being taxed on your Social Security benefits is more important to you.

Certain states offer perks to retirees in hopes of encouraging them to relocate there. For instance, Georgia doesn't tax Social Security benefits, and it also offers residents age 64 and up a $65,000 tax deduction on all types of retirement income. One of the most popular retirement destinations, Florida, is also a tax haven that doesn't tax any type of income in retirement -- including Social Security benefits, retirement account withdrawals, and pensions.

It's also important to look beyond taxes. What's the sense of community like in your desired retirement city? What about the crime rate? How far will you be from friends and family, and how often will you be visiting? Who will you depend on if you require long-term care or if you become sick? Travel costs can add up quickly, so make sure you've accounted for airfare or other transportation costs in your retirement budget.

If you're not sure where to start, begin by imagining your dream retirement location. Maybe it's closer to your grandkids, or perhaps it's right on the beach, or in a quaint town in the countryside. Once you have a place in mind, start looking into its tax rates, crime rates, cost of living, proximity to healthcare, and recreational amenities. Then see how these things will impact your budget. If you're looking to buy a home in your new city, can you afford a house and the property taxes that come with it? If you're renting, what's the average rent in your preferred neighborhood?

Once you've done a deep dive into your budget and decided you can afford retiring in your ideal city, it's time to take it for a test run. The last thing you want is to retire, sell your home, and move to a new place only to become miserable six months later. Long before you retire, spend a month visiting your prospective new city and living like the locals do. Truly think about whether you could see yourself living there. Do you feel safe? Are there enough restaurants, museums, shopping centers, and other amenities to keep you busy? Can you see yourself fitting into the community or do you feel isolated?

Retirement is a major life milestone, and moving to a new city can make it all the more daunting. But it can also be one of the best decisions you'll ever make. Do your research to make sure you're making the most informed decision you can, so your golden years will be great.