Many seniors pledge never to work again after retirement, but others opt to work. For some, it's a matter of needing the money. For others, it comes down to not wanting to be bored. 

There are definitely perks to working during retirement, but if you choose to go that route, you should know that it could end up impacting your Social Security benefits -- and not for the better.

How a higher retirement income could come back to bite you

If you hold down a job and collect Social Security before reaching full retirement age, your income from that job will be subject to the earnings test, and if it's too high, you may have some of your Social Security benefits withheld. Once you reach full retirement age, however, that risk disappears. At that point, you can work and earn as much as you'd like without impacting your benefits.

Older person with concerned expression holding document.

Image source: Getty Images.

So what's the problem with working during retirement? It's simple: Doing so could raise your provisional income, and the higher that income, the more likely you are to have your Social Security benefits taxed.

Provisional income is what's used to determine whether your Social Security benefits will be subject to federal taxes. It's calculated by taking all of your non-Social Security income and then adding in 50% of your annual benefits. If that total falls between $25,000 and $34,000 and you're a single tax filer, you could be taxed at the federal level on up to 50% of your Social Security benefits. If it exceeds $34,000, up to 85% of your benefits could be taxed.

If you're a married couple filing jointly, you may be taxed on up to 50% of your benefits with a provisional income between $32,000 and $44,000. Beyond that point, up to 85% of your benefits could be taxed.

That's why working during retirement could end up hurting you from a Social Security perspective. The average senior on Social Security today collects $18,168 a year in benefits. If you're single and that's your only income, your benefits won't be taxed. But if you take a job and boost your income above $25,000, taxes could come into play.

Now, this isn't to say that you shouldn't work to supplement your Social Security benefits. Most seniors can't live on Social Security alone, and if you attempt to do so, you may wind up miserable. What you should do is make sure that working during retirement actually makes financial sense for you.

Say you're single and entitled to $18,000 a year in Social Security, and you're also able to withdraw another $6,500 from your retirement plan each year for a total annual income of $24,500 -- just below the $25,000 threshold where taxation kicks in. A part-time job that pays you $2,000 a year may not be worth it because what you gain in that paycheck, you may lose in the form of taxes on your benefits.

The bottom line is that you must understand how Social Security benefits are taxed in retirement and what counts as provisional income. It may very well make sense for you to have a job as a senior in terms of your financial and mental health, but be aware of the impact that could have from a tax perspective.