If you've been looking for a cheap and easy way to build a diversified portfolio, relying on an index mutual fund or ETF is a tried and true method that investors have been using for 35 years. But before you jump into the biggest, most popular index funds that are based on the S&P 500 index, think twice -- because you don't want to miss out on investments that could turn volatile and disappointing returns into big wins with a smoother ride over the long haul.

A great vehicle, but not the best
There's a lot to be said for the S&P 500. It has all sorts of different companies that span across every sector of the U.S. economy. It contains a wide range of different sizes of companies; although all of them qualify as large-cap stocks, you have $300 billion behemoths alongside companies as small as just more than $1 billion in market cap.

Moreover, as an index, the S&P 500 has sustained almost the perfect combination of rigidity and flexibility. Throughout its history, its components have represented the cream of the crop of American business. Yet when companies fall from grace, new up-and-coming stocks usually take their place in short order.

But as investors have discovered over the past decade or so, the S&P 500 isn't always the best way to invest. S&P investors who chose the hallmark SPDR Trust (NYSE: SPY) as their stock vehicle have earned an average annual return of less than 1% since late 2000, failing to match bonds or even Treasury bills for return. That so-called "lost decade" has caused many investors to flee stocks entirely. But as it turns out, that move isn't warranted at all, because the S&P 500 isn't representative of how stocks performed overall.

Pulling in other types of stocks
Over time, various investment strategies and styles go in and out of favor, sometimes outperforming and sometimes underperforming the broad market. Even the broad market indexes themselves go for long periods during which you'll find big disparities among their performance.

For instance, look more closely at the past five years. If you bought iShares S&P 500 (NYSE: IVV) as your choice of index ETF, you would have had a return of about 0.9% annually since 2005. But by broadening your asset base to include other types of stocks outside the S&P, you would have improved your results. In particular:

  • Small-cap stocks did a bit better than large-caps, as the iShares Russell 2000 ETF (NYSE: IWM) posted a 3.3% return over the past five years.
  • Mid-cap domestic stocks did even better, with SPDR S&P Midcap 400 (NYSE: MDY) rising 4.5% per year since 2005.
  • Even in the worst housing market in history, the Vanguard REIT Index ETF (NYSE: VNQ) scraped out a 2.3% annual return.
  • International stocks put in a mixed performance. The flagship iShares MSCI EAFE ETF (NYSE: EFA) only returned 1.4%, although that's more than the S&P 500 could manage. But emerging market ETF Vanguard Emerging Markets Stock (NYSE: VWO) posted a whopping 12.2% yearly return.

As you can see, pretty much whatever other asset you added to your total portfolio, you would have done better in recent years than you would have if you'd held 100% of your money in the S&P 500.

A gentler ride
Similar studies have taken those results out 10 years and come up with even more stunning conclusions. A 60/40 stock/bond mix that uses the S&P 500 as its sole stock investment lost to a more diversified portfolio with a variety of stock ETFs every single year from 1999 to 2010. During one year – 2002 -- the diversified investments outperformed by 10 full percentage points. And surprisingly, the broader portfolio was able to post those extra returns while damping down its volatility; the strategy reduced or eliminated negative returns during bad years.

The point isn't that S&P stocks will always underperform. Sometimes, they'll shine while other asset classes suffer. But even when other investments don't outperform, they'll typically help smooth the ride over the long haul. So unless your retirement plan only gives you access to stocks through an S&P 500 index fund, don't limit yourself merely to the biggest U.S. stocks. With a more diversified portfolio, you can protect yourself against bad markets and avoid big swings that could make you panic if you're not careful.

The right ETFs can put you on your way to a great retirement. Click here to read the Fool's new special report, The 7 Secrets to Salvage Your Retirement Today.