Even people who like their jobs often dream about retiring early. But achieving that milestone is easier said than done. In a recent FinanceBuzz survey, respondents were asked the age at which they'd like to retire, and 50 and 55 were the two most common responses. But when asked what age they'd realistically be able to retire, those answers changed to 65 and 70.

Clearly, Americans don't have a lot of confidence in their ability to retire early. But if that's your goal, you should know that with planning, it can be done.

Set your eyes on the prize

Retiring early is possible if you begin saving for it at an early age -- ideally, as early as your first paycheck. The more time you give your savings to grow, the more wealth you stand to accumulate.

Man with eyes closed sitting at a table overlooking a lake

Image source: Getty Images.

Imagine your goal is to retire at age 55. If you were to start socking away $1,000 a month starting at 25, you'd grow your savings to $1.13 million in 30 years if your investments were to generate an average annual 7% return during that time (which we'll get into in a bit).

Of course, setting aside $1,000 a month is easier said than done, especially with other expenses like rent, transportation, food, healthcare -- and, in many cases, student loans. But if you're willing to live frugally to free up that cash, you may very well be in a position to retire during your 50s, instead of having to wait until your 60s or even later.

But saving money consistently, and from a young age, isn't enough. If you want to retire early, you'll need to invest your money in a relatively aggressive fashion, which generally means loading up on stocks and limiting the extent to which more conservative investments, like bonds, take up space in your portfolio.

While it's true that stocks are riskier, those who keep their money invested for long periods of time tend to come out ahead financially. And by "long periods," we're talking 10 years or more. That's why in the above example, a 7% average annual return is a reasonable figure to work with -- it's actually a couple of percentage points below the stock market's average.

Watch what happens, however, when you play it safe with your retirement savings. If you were to set aside that same $1,000 a month over 30 years, but generate only a 3% average annual return instead, you'd wind up with just $571,000. That's still a lot of money, and more than many people retire with today. But remember: When you retire in your 50s, you're too young to collect Social Security or get on Medicare, so you really need a decent chunk of savings.

What does early retirement look like for you?

Savings aside, another factor that could determine whether or not you retire early is your picture of what retirement actually looks like. If you're willing to work in some capacity and generate a small chunk of income, you might manage to retire from your main career well ahead of your peers. But if you're unwilling to work at all, then you'll really need strong savings to pull off an early retirement.

The work you do in retirement doesn't actually have to feel like work. You can turn a hobby you love into a paying gig, or do something you've always wanted to do, like open your own bakery or cafe.

The fact that Americans think they'll retire a decade or more after they actually want to isn't a terrible thing, because it means that many are realistic about the state of their savings. And it's better to retire in your 60s with a decent level of savings than to retire in your 50s with limited savings and wing it. But you should know that early retirement is possible if you commit to it from a relatively young age, and you're smart about investing.