It's no secret that debt can hurt your financial health and make it harder to build wealth. High-interest debt, in particular, is especially toxic and can eat away at your budget until you don't have any extra money to put toward your retirement savings.

As if it's not challenging enough to deal with debt while you're trying to prepare for the future, if you're in deep, the effects of this toxic debt can carry over into retirement. In some cases, your Social Security benefits might even be withheld if you're in arrears on certain types of loans.

Person buried under pile of paper holding a sign that says HELP.

Image source: Getty Images.

More seniors are struggling with debt

First, it's important to clarify that not all types of debt will have an effect on your Social Security benefits. The majority of creditors, such as credit card companies and loan agencies, have no control over your benefits. But if you have unpaid federal student loans, your benefits could be at risk.

It's becoming more common for older Americans to carry student loan debt, particularly as they help their adult children pay for college. More than half (53%) of parents say that helping their kids pay for college is more important than saving for their own retirement, a survey from T. Rowe Price found. Nearly 3 million adults age 60 and older carry student loan debt, according to a report from the Consumer Financial Protection Bureau, and of that group, the average borrower owes around $23,500.

If you're able to pay off this debt, there's no harm in financially helping your children. But many parents, especially those nearing or already in retirement, are struggling to keep up with loan payments and still pay the bills. Close to 40% of borrowers age 65 and older with federal student loans are in default, according to the CFPB, and they're more likely than those without student loans to skip basic healthcare needs like prescription medication and doctor's visits.

Student loan debt can make it difficult to afford even basic necessities, and it's no secret retirees need all the income they can get. But sometimes student loans can even take a chunk out of Social Security benefits, too.

How student loan debt affects Social Security

If you've defaulted on federal student loans, you may be forced to give up a portion of your Social Security benefits to pay them back. Typically, borrowers will see up to 15% of their benefits offset to go toward student loans, according to the U.S. Government Accountability Office.

You won't see your benefits reduced after just one missed payment; you'll need to be severely behind before the government steps in. The GAO found most borrowers whose benefits were reduced had held their student loans for decades, and roughly a third were in default for five years before their Social Security benefits were affected.

While it's relatively unlikely your Social Security benefits will be garnished because of student loan debt, it is becoming more common. Approximately 114,000 borrowers age 50 and over had their benefits offset because of student loans in 2015, according to the GAO. The majority of those people were receiving Social Security disability benefits, but it's possible for the government to withhold retirement benefits as well.

Losing Social Security benefits has the heaviest impact on low-income beneficiaries. Approximately half of unmarried retirees rely on Social Security for at least 90% of their income, according to the Social Security Administration, and seeing benefits reduced by 15% to repay student loans could make it even more difficult to make ends meet. In 2015, approximately 70,000 borrowers receiving Social Security benefits were living below the poverty line, according to the GAO.

Don't let student loans eat away at your benefits

When you don't make a payment on your federal student loan for at least 270 days, you're considered to be in default. Once you've defaulted, the government will try to get its money any way it can -- whether it's by withholding your tax refunds, demanding that your employer withhold your wages, or taking a chunk of your Social Security benefits.

Defaulting on your loans comes with a long list of other consequences, too. For example, it's much more difficult to get another loan, your credit score takes a significant hit, and you might even have to go to court -- which, of course, comes with a slew of legal expenses.

To avoid any of these potential consequences, it's best to avoid defaulting on your loans in the first place. If you're struggling to make your payments, contact your lending organization to see if you can come up with a mutual agreement for paying the money back. FOr example, you might be able to consolidate your loans or enroll in an income-driven repayment plan.

If you've already defaulted, loan rehabilitation may be an option. When you rehabilitate a loan, you make a certain number of payments in a certain time period in order to remove the default status.

Regardless of how you choose to manage your debt, it's important to get it under control as soon as possible to avoid major financial consequences. Failing to repay your debt will not only impact your credit score and overall financial health, but it can also hurt your chances at a comfortable retirement if you can't rely as much on Social Security benefits. By paying off your debts on time, you can ensure your loans won't affect your retirement income.