Social Security benefits are vital for tens of millions of Americans. In retirement, Social Security makes up the vast majority of income for many people, and even those who have other sources of financial support nevertheless value the benefits that they've worked their careers to collect. Yet one of the most surprising things about Social Security is that its benefits can be subject to income tax -- effectively meaning that the federal government reclaims a portion of the benefits it pays out.

The way that Social Security benefits get taxed prevents most recipients from having to pay income tax if they have absolutely no other way of supporting themselves. However, 20 million households -- representing roughly one in three Social Security recipients -- do get taxed on a portion of their monthly checks. Below, we'll look at this issue and how the amounts involved have risen in recent years.

Keyboard with blue tax button.

Image source: Getty Images.

How did Social Security ever get taxed?

Taxing Social Security goes back to the early 1980s, when the program faced a major financial crisis. Lawmakers decided to try to collect more revenue to support Social Security, and taking benefits was an easy way to do so. However, it didn't want to tax everyone's benefits, but rather only a select few.

The way lawmakers set up the rules originally had an impact on just a tiny number of retirees. To calculate income for these purposes, you'd take all your sources of income other than Social Security, such as wages, investment income, and pension benefits, and then add in half of what you got from Social Security. If that number was above a certain threshold, you'd have to include as much as you're your benefits as taxable income.

Income thresholds of $25,000 for single filers and $32,000 for joint filers might not sound all that high right now, but 35 years ago, the top tax rate kicked in at just over $55,000 in income for singles. However, the law governing taxation of benefits didn't have inflation adjustments, so those initial thresholds are still exactly the same currently. Moreover, thanks to more recent laws, as much as 85% of your benefits can get taxed if your income exceeds higher thresholds of $34,000 for singles and $44,000 for joint filers.

Here's how much of Social Security gets treated as taxable income

The total amount of Social Security that gets taxed is substantial. Almost $286 billion in benefits was listed on tax returns as being taxable income, according to the latest available IRS data for the 2016 tax year. Close to 20 million taxpayers included taxable Social Security benefits on their returns that year, making the average American's total Social Security income included for tax purposes $14,321 per household. That's up about $200 from the corresponding amount the previous year, and the number of people affected by the provision has risen gradually but regularly over time.

The IRS doesn't reveal which tax bracket these taxpayers were in, so we can't be sure exactly how much extra tax that works out to. However, with current tax rates ranging from 10% to 37%, an extra $14,321 in taxable income would mean additional taxes of anywhere from about $1,400 to almost $5,300.

Is there anything you can do?

If you don't want to pay taxes on your Social Security benefits, then you have to have some control over the income calculation that goes into determining what portion of your benefits is taxable. If you're still working and getting Social Security at the same time, it's tough to do much to change the situation. In some cases, it's better not to claim Social Security in the first place, waiting until after you retire before starting to receive benefits. Similarly, it's hard to do much to change pension income if it's paid out on a monthly basis in fixed amounts.

For other types of income, though, there are things you can do. Timing sales of investments to avoid capital gains tax liability can be smart, as can managing the timing of money you take out of tax-favored retirement accounts like traditional IRAs and 401(k) plans. Retirement plan withdrawals can be a big contributor to income in retirement, and they're often the culprit in making Social Security taxable.

Social Security has grown to rely on taxes on Social Security benefits, and it'll be difficult for lawmakers to make major changes to eliminate those taxes. Therefore, you're the only one who can take steps to try to avoid having your benefits taxed -- or at least to reduce the impact of their taxation going forward.