Debt can be vexing. You consider your overall financial picture, and you don't like seeing all that mortgage debt there -- or maybe your student loans are weighing on you plenty of years after you graduated from school. It can be tempting to try to pay off those loans early, but that might not be the best idea.

Here are three solid reasons you might not want to pay off your mortgage or student loans early.

The word "debt" written on lined paper, half erased by a pencil, some of which is visible.

Image source: Getty Images.

1. They're not your highest-interest-rate debts

When you make a payment against debt, you're essentially getting a guaranteed return on that money of whatever the interest rate is. So if you pay off $10,000 on which you were being charged 8% interest, it's like earning an 8% return -- because you won't be having to pay 8% on that sum anymore.

Therefore, if you have a handful of different debt obligations, such as a mortgage, a student loan or two, credit card debts, and/or a car loan, your smartest move is to pay off the highest-interest-rate debt first. That will give you the best bang for your bucks. Credit card debt is often going to be your highest-interest-rate debt, with some cards charging close to 30% in interest! So pay off your debts with the highest interest rates first before putting any extra money toward mortgages or student loans, as those loans tend to have lower interest rates.

2. You're still building an emergency fund

Next, understand how important it is to have an emergency fund -- because if you don't have a fully funded one, building it should be your priority over paying off any debts early.

Consider that 28% of Americans are just one financial emergency away from disaster, with no emergency savings at all, according to a Bankrate survey, while close to half of survey respondents hadn't socked away enough to keep themselves afloat for three months. Meanwhile, a 2015 report from the Pew Charitable Trusts found that fully 60% of American households "experienced a financial shock" over the previous year, with about a third of them experiencing two. The median cost of households' most expensive shock was $2,000, or about half a month of income, and more than half of households had trouble making ends meet after experiencing their shock.

That's why most of us should have an emergency fund ready and waiting -- ideally funded with 6 to 12 months' worth of living expenses, such as food, rent or mortgage payments, utilities, taxes, insurance, transportation, and so on. You might not expect to lose your job anytime soon or to need a new engine for your car, but lots of people who are laid off or experience an automotive catastrophe don't see it coming, either. A costly medical issue can derail you financially, too.

3. You can get better returns for your money elsewhere

Finally, remember how paying off a debt is like earning a return equal to the interest rate? Well, if your mortgage is carrying a 5% interest rate, you'd be earning a guaranteed 5% return on any extra payments you send in. But you might be able to do better. After all, over many decades, the stock market has averaged annual returns approaching 10%. Even if you average 7% or 8% during your investment period, that can serve you better than making extra mortgage payments.

You might do even better than that if you're willing to spend the time studying stocks and you choose some individual stocks that perform well. Here, for example, are some 20-year growth rates for a few familiar companies -- plus what they would have turned $10,000 into over that period:

Company

20-Year Average Annual Growth Rate*

$10,000 Would Become

Walmart

7.68%

$43,924

Clorox

8.77%

$53,726

Johnson & Johnson

9.15%

$57,607

Walt Disney

9.18%

$57,924

Microsoft

9.77%

$64,517

Marriott

13.91%

$135,281

Nike

17.33%

$244,445

Starbucks

18.80%

$313,566

Amazon.com

28.95%

$1.6 million

Source: theonlineinvestor.com. 
* With dividends reinvested

Of course, you're not guaranteed to choose stocks that will deliver outstanding returns over long periods -- though one or two big winners can make up for several laggards. For most people, it's best to just invest in a low-fee broad-market index fund, such as one based on the S&P 500. That will get you roughly the same returns as the overall stock market. A good example is the SPDR S&P 500 ETF (SPY), which distributes your assets across 80% of the U.S. stock market. The Vanguard Total Stock Market ETF (VTI) and the Vanguard Total World Stock ETF (VT) are also sound choices, respectively investing you in the entire U.S. market or just about all of the world's stock market.

If you have no high-interest debt, a loaded emergency fund, and plenty of money invested in the stock market, you might want to go ahead and pay off your mortgage or student loans early. But otherwise, consider just making your required payments and putting any extra money to better use.