Source: Coach

It has been a difficult year for Coach (TPR -1.60%), and an even tougher year for the company's shareholders. In addition to being the worst performing stock in the S&P 500 yesterday, shares of Coach have lost more than 8% year-to-date . The handbags and accessories retailer has suffered at the hands of rivals including Michael Kors (CPRI -1.67%) and Fifth & Pacific-owned (KATE) Kate Spade.

Adding insult to injury, Coach reported lackluster results for its fiscal 2014 first-quarter yesterday . The luxury retailer's net income slid 1.6% to $217.9 million for a profit of $0.77 per share in the period . Same-store-sales were also disappointing. This was primarily because of slowing sales in Coach's North American segment .

Earnings aside, let's look at three key things that are dragging shares of Coach lower today.

A lingering identity crisis
Coach's identity crisis over whether to stay true to its roots as a luxury handbag maker or reinvent itself as a "lifestyle brand" continues to weigh down the stock. Coach is adding new products including women's apparel, shoes, and accessories to stores in a bid to be more like its rival Michael Kors.

The so-called lifestyle category has worked well for Kors. Unlike Coach, Kors' profits nearly doubled for its fiscal 2014 first-quarter, with comparable sales in the U.S. growing as much as 25% in the period . In fact, the stock is up more than 250% since going public in 2011. Kors reports second-quarter results next month .

Here's the problem. Kors' has always been a lifestyle brand, whereas Coach is trying to redefine its image seemingly overnight. On top of this, Coach lacks the namesake factor. High profile fashion designers are behind the top selling brands today including Michael Kors, Ralph Lauren, and Tory Burch. Moreover, Coach's new branding strategy comes at a time when the company is bleeding key executives.

A legend says goodbye
As if things weren't bad enough for Coach, its longtime president and executive creative director, Reed Krakoff, will be leaving the company in June . Additionally, Krakoff is taking his namesake brand with him. Together with a group of investors, Krakoff purchased the stand-alone brand from Coach two months ago . While Coach maintains a minority stake in the Reed Krakoff business , the timing couldn't be worse. While management has tapped Stuart Vevers to be Coach's new creative director, the switch comes at a transitional time for the company .

As a result, some investors are waiting on the sidelines until a clearer picture emerges as to Vevers' creative vision for the company. On top of this, Coach's CEO Lew Frankfort is set to step down in January and will be replaced by Victor Luis . With new management taking over in the months to come, the company's future is far from certain.

North American sales weakness
Weakness in its core market is another problem the company can't ignore. North American comparable store sales at company-owned stores fell 6.8% in its latest quarter . This is particularly concerning because Coach generates nearly 70 % of its total revenue from its North American business. Coach now expects North American sales to decline in the high single digits for the remainder of the year .

To put this in perspective, Kors has grown its North American sales by more than 50% every quarter since its initial public offering in 2011. Coach, on the other hand, has an average growth rate of just 9% over the same period, according to data from the Wall Street Journal . Investors are right to worry.

A smarter retail play for investors
Coach's recovery is going to take time, as new management gets settled and the company transitions to new product categories. As a result, Coach will likely continue to have a tough time defending its market share to Kors and others in the space. These challenges as well as weak quarterly results are the reason investors continue to punish the stock. 

Nevertheless, investors shouldn't miss out on other winning retail opportunities.