Aspirational brand Coach (TPR 0.96%) has been terrible. The appearance of rival affordable luxury bag makers Michael Kors (CPRI -0.25%) and Kate Spade (KATE) gave consumers (and investors) an outlet for modifying their upwardly mobile tastes away from a brand they became all too accustomed to. While business has been horrible, there may be reason to believe it's only going to get worse.

After nearly reaching $80 a share two years ago, shares of Coach have since consistently traded in a range 25% to 35% below that level. And with good reason. The handbag maker's North American business crumbled, culminating in a disastrous near-14% plunge in same store sales. That's an important retail metric because it gives investors a sense of organic growth in the business and not increases made as a result of new store openings or acquisitions.

Source: Coach SEC filings

In comparison, Kors and Spade have been growing exponentially, with Kors in particular witnessing double-digit revenue and comp growth, and triple-digit margin expansion. Kate Spade, which sold off its Lucky Jeans business to focus on the handbag business (and changed its name from Fifth & Pacific in the process), also seems to have come into its own too, as sales have finally turned higher and same store sales surged 30% in the fourth quarter.

At the same time, it also looked like even Coach's top designer no longer wanted to be associated with what appeared to be a damaged brand as Reed Krakoff, who served as its creative director for the past 16 years, left the company last year and bought back the rights to his name. Although the handbag maker is counting on his replacement Stuart Vevers to reinvigorate the brand when he takes the reins in September, there may already be signs not he will be able to make a difference.

Where the luxury market has been a bright spot in the economy, it may have reached its limit. Prada reported missing analyst expectations in its latest earnings report and said that same store sales, after rising 7% in 2013, will now only rise by low single-digit levels through January 2015 and mid single-digit thereafter. It believes the luxury market is slowing, at least in the short term, even though it believes over longer periods the market for its goods remains intact.

Prada boutique, Milan, Italy. Source: Wikimedia Commons.

Moreover, the Chinese market that has been the one consistent bright spot for Coach as booming sales to the country's new middle class buoyed results, is reporting signs of a slowdown as its economy ebbs faster than analysts expected. Although Coach's comps grew north of 20% last quarter, Prada says the region is cooling. Nevertheless, it still plans on opening more stores there and elsewhere around the globe.

Coach is under attack from all sides, and it muddies its message with a plan to become a "lifestyle" brand selling trinkets alongside its purses. That's a challenge in itself, as it's laden with significant marketing expenses to get out the message of just what this lifestyle is supposed to be. It's more than any one design head can tackle and this could just be the start of the handbag maker going from bad to worse.