U.S. stocks are little changed in early afternoon trading on Thursday, with the S&P 500 (^GSPC -0.88%) and the Dow Jones Industrial Average (^DJI 0.56%) (DJINDICES: $INDU) down 0.07% and 0.20%, respectively, at 12:30 p.m. EDT.

Billionaire investor Warren Buffett has long been a critic of hedge funds, so it was no surprise when he delivered a broadside against the industry at last month's Berkshire Hathaway annual meeting.

What's more unusual is that the hedge fund industry itself is beginning to do some soul-searching -- and the prognostic isn't pretty.

As president and chief operating officer of the Blackstone Group LP, Tony James has a bird's-eye view of the landscape. With over $340 billion in assets under management, of which $68.5 billion are dedicated to hedge funds (as of March 31), Blackstone is the world's largest alternative investment manager and the world's largest allocator to hedge funds.

("Alternative investments" include hedge funds, private equity, venture capital, real estate, etc.)

"We're definitely worried..."

At an investment conference in Canada yesterday, James told Bloomberg Television, "It's kind of a day of reckoning that we face here. There will be a shrinkage in the industry and it will be painful. That's going to be pretty painful for an awful lot of places," before adding, "[w]e're definitely worried about what's going to happen in the hedge-fund world right now."

Following years of underperformance relative to broad market averages, no wonder James says hedge funds' "2-and-20" compensation structure "is hard to justify these days." He predicts the industry could lose a quarter of its aggregate assets under management -- which are estimated at roughly $3.1 trillion.

"2-and-20" refers to a fixed annual management fee of 2% of assets and a performance fee that rakes in 20% of investment profits.

"We are in the first innings of a wash-out..."

One month ago, another high-profile hedge fund manager, activist investor Dan Loeb, came to the same conclusion in his first-quarter letter to investors in his Third Point funds. Remarking that the quarter had been "one of the most catastrophic periods of hedge fund performance that we can remember since the inception of this fund," Loeb wrote that "[t]here is no doubt that we are in the first innings of a wash-out in hedge funds and certain strategies."

At this point, you're probably wondering what any of this has to do with you. After all, hedge funds are reserved for institutions and ultra-wealthy individuals. It's true you're unlikely to be able to invest in hedge funds directly, but consider that data provider Prequin estimated a year ago that the among the largest investors in hedge funds -- those allocating at least $1 billion -- public and private pension funds and insurance companies represent nearly half (48%) of total invested assets.

CalPERS: We're out!

In fact, some of these very institutions have publicly expressed their dissatisfaction with hedge fund fees and performance. Some are withdrawing from the sector -- partially or totally.

In Sept. 2014, the largest public pension plan in the U.S., the California Public Employees' Retirement System (CalPERS), exited all of its hedge fund investments, citing their cost and complexity. Last month, trustees of the New York City Public Employees' Retirement voted to do the same. This trickle of assets could well become a flood.

The capital cycle turns

Here's the irony of this turn of events: The hedge fund industry, built on the promise of being able to swim apart from the market's current, is proving to be subject to the same dynamics of the capital cycle that affect any other industry.

Strong returns -- and the huge fees -- earned by hedge funds early on attracted a flood of capital from investors and a host of new entrants seeking to reproduce those results. Too much capital chasing too few ideas drags returns down, though, and at some point, investors begin to withdraw capital, which is where we are now.

When the capital cycle turns, it's always painful. For these Masters of the Universe, the notion that they can't all be the smartest players is going to take some getting used to.