Amazon (NASDAQ:AMZN) is still growing revenue at a double-digit rate in its U.S. e-commerce unit, its web services unit, and in international markets. Yet at its physical retail stores, sales have stalled out.

While the e-commerce giant's overall revenue rose 20% in the fourth quarter -- a rate that has some analysts worried about its slowing growth -- its brick-and-mortar locations posted a 2.7% year-over-year revenue decline.

Chart of Amazon physical store revenue growth

Data source: Amazon.com quarterly SEC filings. Chart by author.

Whole Foods is most of the story

The decline is noteworthy because Q4 provided the first year-over-year comparison for sales at Whole Foods Market after Amazon bought it in late 2017. The high-end grocer's almost 500 stores provide the bulk of sales in Amazon's physical store segment, which encompasses Whole Foods, Amazon Books, Amazon Go, Amazon 4-star, and a number of pop-up stores. Amazon, however, doesn't break out the revenue from any of its store concepts.

It's also important because Amazon believes that Whole Foods has shed its "Whole Paycheck" image effectively enough that the company can stop adding locations to the 365 discount grocery chain, and instead focus more effort on bringing Whole Foods stores into new markets. But consumer surveys don't necessarily back up that view. Furthermore, recent reports say that Whole Foods will raise prices on hundreds of items to keep up with rising costs from suppliers, which will only reinforce the "Whole Paycheck" stereotype.

Whole Foods produce departmne

Image source: Whole Foods Market.

Expanding the number of Whole Foods locations probably won't result in higher comparable sales growth. If customers are not shopping as much at its existing stores -- arguably because of the enduring belief that Whole Foods stores are expensive -- then adding more locations isn't going to change that.

Further, Amazon is targeting a lot of brick-and-mortar growth that may not be justified. The company reportedly plans to open up to 3,000 Amazon Go convenience stores in less than three years. However, there's not much evidence yet that Amazon's retail plan is working, so embarking on a massive expansion may be premature. And if Amazon is to meet its ambitious goal, it will need to accelerate its efforts, because there are only 10 Amazon Go stores in operation now.

The competition responds

The physical retail marketplace may be more challenging than Amazon expected. It's also possible that Amazon didn't anticipate how competitors like Walmart (NYSE:WMT) and Target (NYSE:TGT) would respond to the threat posed by its ever-expanding empire.

Walmart, for example, has invested billions of dollars upgrading its online and digital presence to become a leader in e-commerce, and laid out billions more on acquisitions designed to thwart Amazon's encroachment. It bought online retailer Jet.com in 2016 for $3 billion; Shoes.com, Moosejaw, and Bonobos for a combined $431 million in 2017; and last year acquired a controlling stake in Flipkart for $16 billion in a bid for e-commerce dominance in India.

Target, too, has made several key acquisitions, most notably spending $550 million on Shipt to enhance its delivery services.

Target and Walmart now have much more cohesive physical retail and e-commerce operations. Walmart in particular has made its stores a point of differentiation by employing them as pickup points for online orders. The "buy online, pick up in store" option is only growing in importance for retailers.

Time will tell

Amazon says that Whole Foods' sales decline last quarter can be explained partially as a result of realigning Whole Foods' fiscal calendar with Amazon's, which added five more sales days to the fourth quarter of 2017. Also, when consumers place their orders through Prime Now and then pick up their groceries at Whole Foods, the revenue is credited to the online unit, not the physical location.

Amazon Chief Financial Officer Brian Olsavsky said that if you compare the results "on an apples-to-apples basis," Whole Foods saw a 6% sales increase in the quarter. The company expanded Prime Now grocery delivery from Whole Foods to more than 60 markets, while Prime Now Pickup was expanded into more than 20 markets.

Check out the latest Amazon earnings call transcript.

It may take a few more quarters to get fully comparable numbers, but Whole Foods hasn't been the hot growth story many had envisioned. And with heavier competitive pressure coming at the same time that the rest of Amazon's growth is slowing, there could be more difficult comparisons ahead.