Image: Social Security Administration.

Americans hate taxes, and having to pay tax on your earnings is a bitter pill for most taxpayers to swallow. Yet if there's anything worse than paying taxes, it's paying them twice. Unfortunately, when it comes to Social Security tax, that's exactly what happens to many people, as two completely separate taxes often end up hitting you over the course of your lifetime. Let's take a more in-depth look at these two taxes with an eye toward figuring out what you can do to pay as little tax as possible.

The Social Security tax that hits you during your career...
The tax that Social Security imposes on nearly everyone who earns wages is the Social Security payroll tax. Under this tax, employers have to withhold 6.2% from the first $118,500 of your pay in 2015. Employers then have to match that amount with an equal employer contribution. The two pieces of the payroll tax feed into the Social Security Trust Fund, which provides the financial support for the retirement benefits of current retirees and other recipients.

Recent proposals have suggested making changes to the Social Security payroll tax in order to increase the amount of revenue it raises. Several proposals center on raising or eliminating the $118,500 wage-base limit, subjecting more income to the tax from high-income earners. Others would increase the current 6.2% rate over time. Regardless, though, the Social Security payroll tax is one of the most visible, with most workers seeing it spelled out every time they get a paycheck.

... and the one that hits you in retirement
The Social Security tax that fewer people are familiar with affects those receiving benefits in retirement. The way the system is set up is confusing, but what it boils down to is that, if you have a high-enough income, you have to include a portion of your Social Security benefits in your taxable income -- and that will raise your eventual tax bill.

To figure out whether you owe tax, start by taking your income from other sources, including wages, investment income, other taxable pensions, and withdrawals from retirement plans that are included in income. Then, add in half of your Social Security benefits. If you're single and that total is above $25,000, you'll have to include half of the excess in your taxable income, up to half of your total benefit. For singles with income above $34,000, that percentage rises to 85%. The corresponding income limits for joint filers are $32,000 for the 50% threshold and $44,000 for the 85% threshold.

A simple example might make things easier. Say a single person gets $15,000 per year from Social Security and $25,000 from other sources, including a pension and investment income. In that case, you'd take the $25,000 in income from other sources and add in half of the $15,000 in benefits, which totals $32,500. The portion over $25,000 is $7,500, and so half of that amount, or $3,750, would be added to the person's taxable income. Depending on tax rate, that could equate to tax of as little as $375 for those in the 10% bracket all the way up to nearly $1,500 for top-bracket taxpayers.

Note that married couples often end up with an unintended tax hit if one spouse takes Social Security benefits while the other is still working. In those cases, it's easy to go above the thresholds and have a large portion of your benefits subject to income tax.

How to minimize Social Security tax
Planning to reduce the impact of Social Security tax is really only effective for the tax on income in retirement. For most people, the payroll tax is automatically taken from their paychecks, and so short of working for less pay, there's no obvious way to reduce the tax.

For the Social Security income tax in retirement, though, there are things you can do to reduce its impact. By waiting to take Social Security benefits until your overall income will be lower, you might be able to avoid or reduce taxes on those benefits. Similarly, taking less money from traditional IRAs and 401(k)s, and instead favoring Roth retirement accounts can keep your income lower, making it less likely that you'll move above the threshold amounts.

These two types of Social Security taxes seem draconian because you end up having to pay both when you earn wage income and when you get your benefits. Regardless of whether that's fair, it's important to understand what you should do to minimize the impact of the taxes, and keep as much money as possible for yourself.