U.S. stock markets are up big in 2013. The Dow Jones Industrial Average (^DJI 0.85%) has returned 26.9% to investors this year and the broader S&P 500 index has returned a whopping 30.2%.

^DJITR Chart

^DJITR data by YCharts.

But what's driven stocks higher this year? As fellow Fool Dan Dzombak recently pointed out, it's actually rising P/E ratios, not growing earnings, that have driven stocks higher. That's problematic for investors who want to buy today, especially if you're looking at high P/E stocks. Companies will have to perform flawlessly just to live up to expectations, which is one reason today's market scares me.

Profits don't matter
Some of the market's best stocks this year have gone up primarily because of speculation from investors, not rising profits. Look at the gains for Tesla Motors (TSLA 0.01%), Amazon, Netflix, and 3D Systems this year.

TSLA Total Return Price Chart

TSLA Total Return Price data by YCharts.

Now consider that Tesla and Amazon have lost money over the past year while Netflix and 3D Systems are barely making a profit. These stocks are trading at price/sales multiples of 10.1, 2.6, 5.4, and 18.6, respectively.

These stocks are up because expectations jumped in 2013, which poses problems if results don't eventually live up to those lofty expectations.

Good isn't good enough
One major worry I have going into 2014 is that investors and traders no longer just expect companies to meet expectations, they expect companies to crush them.

Take Nike (NKE 2.29%) as the example this week. The company increased 8% revenues in the fiscal second quarter and beat expectations by a penny, but the stock dropped 1.2% after earnings.

That's not a big drop but when you combine just a small beat with a stock that has lofty expectations, the results can be dramatic. Tesla's stock is down 26% from its 52-week high with most of the drop coming after only beating earnings estimates by a penny.

This will play out over and over next year because stocks have gone so high that even meeting or beating Wall Street's expectations won't be enough. Investors are expecting blowouts.

Interest rates are only going up
I'm not a big believer that the Fed's easy money has been the biggest driver of the stock market over the past few years, but I do believe that rising interest rates are problematic for the market. It's not because companies don't have enough cash to expand without easy money; it's the demand side that I'm concerned about.

The housing market has been boosted by low interest rates and it will suffer as rates rise. Look at the graph below to see how fast housing slowed down when rates rose, a trend we've seen continue in recent weeks.

US 30 Year Mortgage Rate Chart

US 30-Year Mortgage Rate data by YCharts.

The same factors impacting the housing market apply to the auto industry. Cheap loans and leases have been an incentive for people to upgrade vehicles this year and if rates rise in 2014, the pace of sales will slow.

Corporate America has gotten used to easy money, but the bigger concern I have going into 2014 is how much consumers have gotten used to paying almost no interest on big-ticket items. Sticker shock could ensue.

Foolish bottom line
I don't know if stocks will go up or down in 2014, but these are factors that I worry about when looking at buying stocks right now. There are a lot of downside risks, especially after the market's long rise. Keep that in mind when you're investing money next year.