Image source: Getty Images.

Social Security represents the single largest source of retirement income for millions of Americans, with about 64% of recipients getting at least half of their income from Social Security. Yet demographic trends threaten the program's sustainability, and lawmakers haven't taken any steps to avoid an impending funding shortage.

There's a lot to worry about when it comes to Social Security. Here are five things in particular that you should be concerned about.

No. 1: The Social Security trust funds are running out of money

According to the 2016 Social Security Trustees Report, the trust funds that support the program's payments are on track to be exhausted by 2034. When that happens, absent any changes in the law, Social Security's payments will drop by about 21%, and the cuts could eventually reach about 26%. This chart says it all:

Image source: 2016 Social Security Trustees Report. 

With that day of reckoning less than two decades away, even some current Social Security recipients are at risk of seeing their benefits slashed -- not just those counting on the system sometime in the future.

No. 2: The upward trends in Social Security Disability add strain to the system

According to the chart below from the Congressional Budget Office, the number of Social Security disability recipients has been on an upward trend for decades. That increase is both in total and as a percentage of working-age Americans.

Image source: Congressional Budget Office.

That increase is a problem for two key reasons. The first is that Social Security's disability-oriented trust fund is in even worse shape than its retirement-oriented trust fund. In fact, were it not for emergency legislation passed late last year that shifted money from the retirement trust to the disability trust, the Social Security Disability Insurance Trust Fund would have run out of money this year.

The second reason is that people receiving Social Security Disability are typically working-age people who are not working for pay. The few who are working are working for a minimal amount of pay. This is an issue because Social Security is funded by payroll taxes, and people who don't work don't generate payroll taxes, thus hastening the decline of the program's funding.

No. 3: Higher taxes haven't solved Social Security's shortfalls

When Social Security taxes first started, the rate was 2% on the first $3,000 of earnings -- half paid by employers and half paid by employees. Today, the rate is 12.4% on the first $118,500 in earnings. Even adjusting for inflation, Social Security Taxes are more than 14 times as high as when the program started, yet the program is still on track to slash benefits.

No. 4: Real benefits have been cut already -- and continue to shrink

A prior patch to Social Security exposed Social Security benefits to taxes, assuming you're single and have a combined income that exceeds $25,000, married filing jointly with a combined income above $32,000, or married filing separately in most instances. Your "combined income" includes your adjusted gross income, half your Social Security income, and any tax-free interest you receive. Those thresholds are not indexed for inflation, which makes it more likely that your Social Security benefits will be exposed to those taxes over time, reducing your net income from the program.

In addition, the "full retirement age" at which recipients are eligible to receive their promised benefit was originally 65. Now, the full retirement age is gradually increasing to age 67 for those who will reach that milestone within the next decade or so. A higher full retirement age translates directly to lower payments for the Social Security recipients who take benefits early and are thus subject to a permanent reduction. (And by the way, the majority of recipients claim early, with the most popular age being the earliest possible age of 62.)

No. 5: It's not your money, and Congress can change the rules around it

While millions of Americans rely on Social Security for their retirement income, the unfortunate reality is that your benefits are only as good as Congress' ability to make sure they get paid. The Supreme Court case Flemming v. Nestor confirmed that Congress has the ability to change benefits -- and can even deny them altogether.

There's no chance that Congress would vote to eliminate Social Security altogether. But it has a history of raising taxes, cutting real benefits, and even denying benefits altogether in some instances. And that's not just ancient history, either. Just this past year, Congress closed the loophole that enabled married couples to use the "file and suspend" strategy to maximize their lifetime benefits. Not long before then, Social Security severely limited the "do-over" option that let people pay back their Social Security in order to earn delayed-retirement credits and boost their benefit.

Know the risks and plan for your future

Despite these real reasons to worry about the future of Social Security, all signs indicate that American workers will receive some level of benefits when they retire. Still, Social Security was not designed to be seniors' only source of retirement income, and the challenges facing it should give you all the more reason to plan around its shortcomings.

So as you craft your retirement plan, make sure Social Security isn't a major part of it. With a solid end-to-end plan, even if the worst-case scenario comes to pass for Social Security, you can still have a comfortable retirement.