Gotta go. That was the message an activist shareholder delivered to Abercrombie & Fitch (ANF 5.74%) yesterday regarding the teen retailer's chairman and CEO, Michael Jeffries.

"We are confident that an independent and objective evaluation of management's performance would result in the conclusion that an immediate leadership change is necessary," the hedge fund Engaged Capital wrote in a letter to the board of directors

As the saying goes, a fish rots from the head, and the hedge fund, which says it owns approximately 400,000 shares of A&F's stock, or about 0.5% of its outstanding shares, says the board of directors needs to either send Jeffries packing when his contract expires on February 1 or seek a leveraged buyout. Where once Abercrombie was a teen fashion icon, it rarely garners much attention other than where its Hollister brand is concerned, and that, says Engaged, is a problem that can be laid at Jeffries' feet.

It notes that five years ago Jeffries recognized that the fast-changing nature of the teen retail market was affecting the company's turnaround and promised to address the issue. Yet just last month, the CEO was once again proclaiming the company will need to respond to rapidly changing trends if it hopes to keep up with the industry.

Apparently, five years hasn't been long enough to come to grips with this reality, and the hedge fund says Abercrombie doesn't have any more time to dither under Jeffries' leadership. "Unfortunately, these are not areas where Mr. Jeffries has demonstrated expertise or competency," it wrote.

The retailer, despite management failures, remains a top player in the space, particularly when stacked up beside peers like Aeropostale (APO), American Eagle (AEO 6.32%), and Urban Outfitters (URBN 4.51%). Engaged highlights that Abercrombie is the largest global teen retailer, has a mammoth direct-to-consumer business that generates approximately $700 million in annual sales, and generates gross margins above and beyond the competition.

Despite that preeminent position, investors have failed to realize any benefit. Over any time period you choose to look at during the past decade, Abercrombie & Fitch has woefully lagged behind its peers on an absolute basis. The juxtaposition, then, of having well-regarded brands but an underperforming stock suggests the problem isn't within the business but rather with those running it.

Source: Engaged Capital letter to Abercrombie & Fitch board of directors.

New leadership could invigorate Abercrombie to regain its balance against retailers like Forever 21, H&M, and Urban Outfitters that have a better fast, fashion-forward sensibility. Aeropostale finds itself in the midst of an identity crisis, too, as it abandons the logo-centric marketing featured on its hoodies, jeans, and tees for a new look and feel, but investors are left struggling to keep above water.

Both A&F and Aeropostale have lost around a quarter of their value this year, and Aeropostale is also feeling the pressure from its investors to sell itself, so much so that it felt the need to adopt a poison pill defense to give it time to sort out its options.

While there is renewed focus on its core Abercrombie and Hollister brands, having been forced to jettison both the Ruehl and Gilly Hicks divisions that proved to be disastrous ventures, A&F's also been forced to close a good 30% of its U.S. stores this year, with the number growing to as much as 35% by 2015. Engaged Capital estimates the mismanagement has cost investors more than $500 million in asset impairments and operating losses over the last six years.

From poor fashion choices to quixotic ancillary ventures -- not to mention Jeffries' incendiary remarks about who best represents Abercrombie's target demographic (in short, overweight, unattractive people need not apply) -- the teen retailer has lost its way and Jeffries is not the one to lead it back from the wilderness. But owning only a tiny fraction of Abercrombie's stock, it seems doubtful Engaged Capital can effect much change on its own.

There is the possibility it could marshall forces behind it for a proxy fight. Considering the teen retailer posted losses of $11.5 million over the first three quarters of 2013 as sales shrunk 7%, the hedge fund operator may just find more shareholders rallying to its cause. And along with saying Jeffries has to go, they may help give him the heave-ho.