Most working Americans pay into Social Security, and the program pays around $1 trillion in annual benefits to nearly 68 million people, making it the largest single source of retirement funding in the country.

As large and as important as that program is to the people covered by it, it may surprise you to find out that you can lose part of the benefits you've been paying toward throughout your career. These three causes are key reasons you may end up with less than you might otherwise expect.

Social Security cards on top of tax forms.

Image Source: Getty Images.

1. Possible changes in 2035

According to Social Security's annual Trustees Report, the program's trust funds are on track to run out of money in 2035, and while the continued cash influx from payroll taxes means that most payments will continue, without an overhaul to the program, benefits will need to be slashed by 20% to 25% when that happens.  

In the past, when Social Security faced shortfalls, Congress has acted to shore up the program. Those fixes have generally come through a combination of tax increases and benefit reductions. In the 1983 reform, the full retirement age (FRA) was gradually raised to age 67 from 65, benefits were reduced more sharply for starting at age 62, and benefits became taxable if recipients had enough other income.  

Because any fix for the program will likely involve a combination of tax increases and benefit reductions, you should expect your payment to be smaller than your My Social Security statement would indicate. To be on the safe side, consider building your end-to-end retirement plan around the expectation of a 25% cut to benefits. That way, you'll increase your chances of either winding up OK or being able to support a somewhat more robust retirement than you otherwise would have.

2. You earn a lot less than your spouse

Senior man and woman with a picture of a social security card with George Washington's face

Image Source: Getty Images.

Your personal Social Security benefit is based on your highest 35 years of covered earnings, indexed for changes in the overall wage level. If you took substantial time off of work -- to raise kids, to care for aging parents, or simply due to extended layoffs -- that earnings record may include several years with $0 or very low earnings.

If you're married (or were married to the same spouse for at least 10 years), you may also qualify for benefits based on your spouse's earnings record. The typical spousal benefit is half of the primary wage earner's benefit. Social Security will pay you based on the higher of either your own benefit or your spousal benefit, which means that if you're getting spousal benefits, your own personal benefit might as well be $0.

3. You continue working while collecting and you're below FRA

The most common age to begin collecting Social Security retirement benefits is 62, the earliest age you can do so. Unfortunately, collecting at that age brings with it two sets of problems. For one, your monthly benefit is permanently reduced because you took it early.

For the other, if you are below FRA and are still working or go back to work while collecting benefits, you will get penalized. The penalty can be as steep as $1 for every $2 you earn above $1,470 per month. Earn enough, and your monthly benefit can even be cut to $0. That makes it worthwhile to wait to start collecting if you plan to keep working up until full retirement age or beyond.

Once you reach FRA, Social Security will begin paying you back the penalized money. As a result, if you live long enough, you'll be made whole from those penalties. Still, starting early while still working means that your monthly benefit will be permanently reduced.

Despite the risks, Social Security still provides a decent foundation

Although these factors mean that it's possible for you receive less from your Social Security benefit than you thought you would, the program still provides a decent foundation for retirement planning. Just don't expect it to deliver more than its official communications tell you it will and recognize that the risks mean your benefits may not even reach that level. Build your overall retirement plan within those constraints, and you will have a strong chance of reaching your golden years comfortably.