Retiring early is a goal of countless working Americans, yet many aren't able to achieve it. You might think that the folks who get to leave the workforce ahead of their peers are those who earn the big bucks or inherit heaps of money from family, but the truth is that anyone intent on retiring early has the potential to make it happen. If you want to retire at a relatively young age, here are three steps you can take to get there.

1. Start saving as early as possible

The more time you give your money to grow, the more you stand to accumulate. It's really that simple. Therefore, if you begin setting money aside for retirement at the start of your career, there's a greater chance you'll accumulate enough wealth to exit the workforce earlier than most.

Check out the following table. It shows how much savings you stand to amass based on when you start:

If You Start Saving $600 a Month at Age:

Here's What You'll Have by Age 60 (Assumes an 8% Average Annual Return):

25

$1.24 million

30

$815,000

35

$526,000

40

$329,000

TABLE AND CALCULATIONS BY AUTHOR.

Notice the difference between kick-starting your savings efforts at age 25 versus age 30 in the above scenario. Though you're only adding another $36,000 to your nest egg in out-of-pocket contributions, you'll end up with an additional $425,000 because your money gets all of that extra time to grow. And that's precisely why starting early could be the key to exiting the workforce sooner rather than later.

Older couple riding bikes

IMAGE SOURCE: GETTY IMAGES.

2. Invest your savings aggressively

Retiring early doesn't just boil down to saving money; it's also a matter of investing that money wisely. Let's revisit the first row of the table above, which shows that setting aside $600 a month for 35 years could result in $1.24 million. If we do the straight math on those contributions, we see that it means putting in $252,000 out of pocket. The rest of that growth comes from investments that produce an average yearly 8% return. And if you're wondering how to get that sort of return, look no further than the stock market.

Though investing in stocks carries some risk, those who stay in the market on a long-term basis tend to come out ahead. In fact, historically, the stock market has averaged a little bit more than 8%, and that figure accounts for its various downturns. Therefore, if you put the bulk of your retirement savings into stocks, there's a good chance you'll average 8% a year, if not better.

Watch what happens, however, if you play it too safe and stick to less risky investments, like bonds. In that case, you might snag just a 3% average annual return over the life of your investments. That means that if you're putting $600 a month into your nest egg over a 35-year period, you'll end up with just $435,000 -- not a negligible amount of money, but also nowhere near $1.24 million.

The point here is this: If you want your savings to grow enough to allow for an early retirement, you'll need to take some risks in your portfolio. Either that, or you'll need to boost your savings rate substantially, which may not be possible if your income isn't particularly high.

3. Reset your retirement expectations

Many people think of retirement as a period to travel the globe, indulge in fine dining, and experience life's many luxuries. In reality, it takes a lot of money to live like that for a couple of years, let alone several decades.

If you really want to retire early, you may need to rethink your picture of what that actually looks like. Maybe you don't need those fancy cruises or that country club membership -- perhaps you can learn to be content with tending to your garden and spending quality time with your grandchildren, instead. There are so many different ways your retirement might shake out, but if it's important to you to live by your own schedule as opposed to an employer's, then it pays to think about how you might tweak your picture of it to get there early.

It's easy to give up on the notion of early retirement and resign yourself to many more years in the workforce. But why do that? There's a lot to be gained by retiring early, and with the right attitude and approach, you could easily realize your dream.

The Motley Fool has a disclosure policy.