Humans have pursued gold for millennia. The shiny, soft metal has inspired exploration and started wars, but only in recent decades has it become easy for the average person to own it. It's something you can now easily keep in your investing portfolio via equities like the SPDR Gold Shares (GLD -2.47%) exchange-traded fund.

But just because you can buy gold doesn't necessarily mean you should buy gold. After all, it really hasn't been a very good long-term investment, underperforming stocks and bonds in total returns over the past 40 years. It has also generated mixed results over the past decade, depending on when investors bought. If you're thinking about gold, you may want to reconsider your situation and investing time frame.

Here's why three of our smartest investors won't put gold in their portfolios -- at least not now. Their reasons could help you make a better decision with your investing dollars.

A gloved hand holding a gold bar

Image source: Getty Images.

Where are the compound returns?

Tyler Crowe: Investing in gold can be lucrative under certain conditions. It is considered a safe asset that provides a hedge when the value of other assets such as stocks, bonds, or real estate declines. During times of economic or geopolitical uncertainty, investing in or owning gold can offset any losses from other asset classes. When your portfolio is getting devastated by a stock or bond market collapse, having gold in your portfolio can be that security blanket you cling to, letting you know things are going to be OK.

Here's the thing, though: These hedges against market calamity are always short term in nature, because gold isn't a productive asset. While the dollar value of an ounce of gold has changed from 100 years ago, the amount of gold in an ounce never changes. Conversely, stocks, bonds, and real estate produce something, whether it be interest payments, dividends, rent payments, or higher profits per share of a business. As your investment time horizon lengthens, the productivity of other asset classes will produce superior returns compared to that same ounce of gold.

Gold Price in US Dollars Chart

Gold Price in U.S. Dollars data by YCharts.

The road to wealth is paved with compounded returns over decades, and those compounded returns don't come from unproductive assets. With decades of investing ahead of me, I can't justify adding an unproductive asset to my portfolio that will act as a dead weight on returns, just so I can say "well, at least my gold position is going up" during the next market crash.

Don't gamble -- invest

Matt DiLallo: Super-investor Warren Buffett once called gold an "unproductive asset" because unlike bonds, stocks, or commercial real estate, it doesn't produce any cash flow that could be used to compound the wealth of its owners. Instead, Buffett noted that people buy assets like gold (and more recently, bitcoin) in the "hope that someone else will pay more for them in the future."

While it's possible to make money betting on hope, the reason Buffett favors productive assets over gold is that they can become compounding machines over the long term. A prime example of this is precious-metals streaming and royalty company Royal Gold (RGLD -3.19%). The company helps miners finance new developments by agreeing to give them an up-front cash payment in exchange for a portion of the mine's future production, which it then flips to buyers for cash.

Over the years Royal Gold has built up an extensive portfolio of streaming and royalty agreements that generate cash flow for the company, even as the price of gold ebbs and flows. That gives it the money to invest in additional contracts, which grow its cash flow over time. The result has been a masterpiece of compounding magic as Royal Gold has delivered a more than 1,250% total return over the past 20 years, which has vastly outperformed gold's 300% rise in value. That outperformance shows why it doesn't make sense to buy an unproductive asset like gold when productive ones -- even in the gold industry -- could create significantly more value for investors over the long term.

Gold would be a bad choice for my investing timeline

Jason Hall: I think one of the biggest mistakes most people make when they invest is not understanding the difference between certain assets. For instance, bonds, stocks, and commodities like gold can all be valid investments, but they also serve different purposes. And that can mean something that's an ideal investment for one person is a terrible choice for someone else. Let me use my own situation as an example.

I'm in my early 40s with an infant son. The vast majority of my investing dollars are dedicated to two things: retirement, and my son's college education. Both of those things are decades away.

And that makes gold -- an asset that has generated poor long-term returns, as Tyler showed -- a poor investment for me, since I should be focused on long-term growth right now. My long-term returns will likely be far better if I invest in stocks -- that is, businesses that will steadily grow their earnings over time -- than gold.

This isn't meant to say that gold is a bad investment. At some point in the future, when my goals shift from capital growth and it becomes more important that I preserve my nest egg and reduce my downside risks, gold could become part of my strategy. But right now, investing in gold would probably harm my ability to generate the best long-term returns I can. That keeps it off my shopping list and out of my portfolio.