Imagine learning that the government will no longer manage the entirety of your Social Security taxes. Instead, an account is being opened in your name and you're solely in charge of how the money is invested. Whether you have enough money to support yourself in retirement depends on how well you do as an investor.

If implemented, that's how Social Security privatization would work. The details (like whether you'd manage some or all the money currently being paid into Social Security) are fuzzy. Still, fans of Social Security privatization would like to see the government get out of the investment management business.

A change as big as privatization of Social Security is sure to have ramifications, both positive and negative. Opponents of the proposal say one downside is the number of people who could be hurt by the change. Here's who they're concerned about.

Elderly woman taking cash from an ATM.

Image source: Getty Images.

Low-income workers

Those living paycheck to paycheck may struggle to contribute to a privatized account, leaving them with little to fall back on in retirement. Unlike a household bringing in more money, low-income households have less financial flexibility. A single emergency situation, like a flooded basement or medical costs associated with an injury can wipe out their budget and leave them in debt.

Inexperienced investors

Experienced investors will likely have less trouble taking over their retirement account from the Social Security Administration. However, workers who lack financial literacy may find it difficult to make the kinds of investment choices that will build their portfolios. In addition, they may be more likely to panic when an investment is not performing as well as expected and dump the asset before it has a chance to recover.

While it's sometimes necessary to rebalance your portfolio, the decision should never be based on emotion.

Anyone with health issues

Unmanageable medical expenses is the No. 1 reason people file for bankruptcy in the U.S. Those who face serious health problems may also face significant medical costs that divert funds away from their retirement savings and potentially jeopardize their financial security once they've retired.

Older workers

Because Social Security privatization is more of a theory at this point than a well-designed plan, we don't know who would be expected to make the change. For example, we don't know if it would be those entering the job market for the first time, or if older workers would be expected to make the switch.

If older workers are required to make the switch, a sudden shift to a privatized account could disrupt their current retirement planning and complicate the financial strategy they've depended on.

Nearly anyone who puts their money in the stock market deals with an occasion bad investment. The older a person is, the less time they have to wait for their portfolio to rebound after taking a hit.

Single parent households

Statistics show that most households led by a single parent have less wealth than households headed by couples. Those with limited income may face a greater challenge as they attempt to build retirement savings.

The underemployed

There was a time in American history when a person would go to work for a company and remain with that company for decades. The same cannot be said today. Given how commonly companies lay off employees to plump their bottom lines, millions of American workers are one pink slip away from unemployment. That insecurity can make it difficult to invest rather than hold on to money that may soon be needed to buy groceries or pay the light bill.

There are passionate people on both sides of the privatization issue, with most opponents citing the number of people who could be hurt if privatization were implemented. Hopefully, if Congress ever becomes serious about the issue, they'll keep those who desperately need a Social Security safety net front of mind.