There are some stocks that look to be good investments to hang on to amid a downturn and inflation. Starbucks (SBUX -1.02%) doesn't strike me as one of those stocks. Year to date, it has fallen 27%, proving to be a worse buy than the S&P 500, which is down just 13%.

But as bad as things have been for the business, I wouldn't be surprised if they get even worse. Here's why this could be a bad stock to be holding in this challenging market.

Its growth rate has been slowing down

Starbucks released its third-quarter numbers earlier this month, for the period ended July 3; sales rose by a modest 9% year over year to $8.2 billion. What's particularly telling is that global comparable-store sales rose 3%, and that was mainly due to a higher average ticket (which was up 6% while comparable transactions declined by 3%). Starbucks has raised its prices multiple times since October due to rising inflation.

A quarter earlier, the company's sales grew at a rate of 15% and comparable transactions rose by 3%. Although China's comparable numbers in Q3 were down significantly due to lockdowns (sales declined 44% on a comparable basis) and weighed down Starbucks' numbers, even in North America, comparable transactions were still up just 1% this quarter versus 5% a period earlier.

The danger for Starbucks is that slowing demand, and therefore revenue, is not its only problem.

Costs are accelerating and could go higher

Modest sales growth might be acceptable for investors if the company's costs were tightening, but that isn't the case. Last quarter, operating expenses rose by 13% but revenue by only 9%. The result was that Starbucks' net earnings fell 21% to $912.9 million.

Higher supply costs due to inflation could make the issue worse, as could the threat of more of its stores unionizing. In December, a Starbucks store in Buffalo, New York, became the first in the chain to unionize. Since then, more than 200 stores have voted to do so.

Putting more pressure on its expenses could mean even slimmer profits for Starbucks. And that makes the stock's earnings multiple even harder to justify.

The shares trade at a hefty premium

Even with its declining value this year, Starbucks stock trades at 25 times its future earnings. By comparison, investors are paying 18 times future profits for the average stock in the S&P 500. High multiples can be justifiable for businesses that are growing at high rates, but that isn't the case for Starbucks these days. Its profits face significant headwinds from the potential of both stalling revenue growth and rising costs.

These headwinds may not last for the long term, and as inflation subsides, the business could get back to normal. However, the effects of unionization could lead to costs that remain with the business for the long haul. And it's still too early to tell how much of an impact that will have on the bottom line, as unionization efforts are still fairly new. Starbucks has more than 15,000 stores in the U.S., so there's potential for many more locations to unionize.

Starbucks has become a risky buy

The strength of Starbucks' brand is getting tested right now. Sales growth is slowing down, and if the numbers continue declining, it could be a sign that consumers aren't taking the price hikes in stride. Given the record-breaking inflation numbers in the economy right now, I'm not optimistic that Starbucks will be able to keep prices high while also ensuring that sales don't end up declining.

Starbucks' high valuation only seals the deal: This is a growth stock that looks too risky to be holding right now.