From the fourth quarter of 2014 to the first quarter of this year, 31% of the hedge funds that owned Ally Financial (NYSE:ALLY) completely exited their positions.

That's right, according to data compiled by the analytics firm Novus, nearly one-third of hedge fund investors in Ally sold everything they owned in the financial stock.

If the smartest firms on Wall Street are selling, should you?

Breaking down the buying and selling
In the first quarter, 25 hedge funds abandoned Ally. Nine hedge funds opened new positions, 19 bought additional shares, and 25 sold some, but not all, of their shares. In total, 65 major hedge funds currently have long positions in the stock.

Among those owners are a number of high-profile funds managed by some of the best-known investors in the world. Dan Loeb's Third Point LLC owns $581 million of Ally stock. Likewise Richard Perry's Perry Capital owns $492 million. 

Altogether, hedge funds own over 43% of Ally's market cap, totaling more than 212 million shares.  

Having so many hedge funds sell their shares is certainly a bearish sign. However, Ally clearly remains very popular among hedge funds, so we should expect to see plenty of buying and selling by these firms.

Remember that hedge funds are expected to produce market-beating returns quarter after quarter. That means they will often sell a good stock simply because they think they can get a higher return in the short term in another opportunity. That is a high-risk strategy, and it stands in stark contrast to the longer-term buy-and-hold approach recommended for retail investors.

One possible reason hedge funds sold Ally in Q1
Ally Financial lost its largest customer in the first quarter, which to me is the most obvious driver of hedge-fund selling at this time. 

Prior to the financial crisis, Ally was named GMAC and was a wholly owned subsidiary of General Motors. The government bailed out GMAC during the financial crisis and took majority ownership from GM and another private equity owner. The Treasury Department later sold its ownership in Ally via a 2014 IPO.

A legacy of Ally's prior life as a General Motors subsidiary is that GM remained the lender's largest source of auto lease business. But in the first quarter the automaker moved all of its leasing business to its new internal financing subsidiary, GM Financial. For Ally, GM represented about 10% of its lease originations in 2014.

The loss of GM's business likely exacerbated concerns over growth at the company. Revenue growth has been particularly hard to come by since 2012.

ALLY Revenue (Quarterly) Chart

ALLY Revenue (Quarterly) data by YCharts.

But that doesn't make Ally a bad investment today
For investors with a longer-term approach than the typical hedge fund, the case for Ally is a bit more bullish. Equity analysts at Goldman Sachs went so far as to say the impact from the loss of GM's leasing business "has been overly discounted and could be neutral to [earnings per share]."

Goldman pointed to Ally's relatively low price-to-tangible book value ratio, projected return on tangible equity of about 10%, and better-than-expected loan originations as positive drivers of the stock price over the next year.

Longer term, the narrative for Ally also seems optimistic. The bank repaid the government's TARP investments in the fourth quarter of last year. Its total capital ratio increased to a very sound 14.1% as of March 31, 2015. Net income has steadily improved as management focuses on cutting costs and improving efficiency. At the same time, an improving U.S. economy will be a tailwind for the company going forward as increasing disposable income should drive high auto lending activity.  

Revenue remains a concern, but now that the GM relationship is officially over, I see limited risk of future top-line shocks. Ally's loan growth will likely track with the rest of the industry; if that is the case, then management is correct in focusing on costs and efficiency today.

Hedge funds might be selling, but from my point of view as an everyday retail investor, Ally's long-term prospects look pretty good.