The Fed's stress testing looks good on surface. In its worst-case scenario, the stress test parameters include a nearly 5% fall in GDP, a 4% rise in unemployment, a 50% drop in equity prices, and a 25% decline in real estate. These numbers bottom out at pretty respectably bad levels before leveling out again. This year's test also includes a significant counter-party default for large banks.

Thus, that all the largest banks passed the tests could be taken as a good sign. 

Except for that fatal flaw
The Fed's tests have one major weakness: they do not take into account the possibility or effects of financial contagion. In a crisis, a bank might not find it as cheap or easy to raise capital as usual, and perceptions of instability in the financial sector can punish even a healthy bank. Further damage can come in the form of credit freezes. As Bloomberg points out, "Credit freezes can force financial institutions to sell assets at a loss, setting in motion downward spirals in which falling prices and banks' woes reinforce each other."  

Sounds jolly. Because these externalities are not incorporated into the Fed's tests, the results are misleading to say the least. In a real crisis of the magnitude the Fed envisions, these kinds of things could very easily transpire, and they would significantly affect the capital position of the tested banks. 

What would a real stress test look like?
New York University researchers designed a similar test using less specific bank data but more appropriate parameters, and their results are hair-raising. Under the NYU test, only Wells Fargo (WFC -0.56%) passed among the top six banks, a group that also includes JP Morgan Chase (JPM 0.49%), Bank of America (BAC -0.13%), Citigroup (C -0.32%), Morgan Stanley (MS 0.10%), and Goldman Sachs (GS -0.23%).

All of the other banks have capital shortfalls. Notably, these would run over $100 billion for JP Morgan, $80 billion for Bank of America, and $75 billion for Citigroup. Wells Fargo squeaks by with a slightly positive capital base of $5 billion, the only one not to drop into negative territory. 

The NYU test captures the effects of contagion in a way the Fed tests ignore, making it an interesting (and troubling) counterpoint to the Fed scenarios. 

So are the Fed tests pointless?
I wouldn't say the Fed tests are absolutely useless, but I don't think you should believe that they'll tell you everything you need to know. This is certainly the case for contagion and crisis. These situations do not produce once-off shocks that occur and suddenly go away, they tend to reverberate through the financial system and the economy until they die down. The fact that the Fed doesn't take this into account is something everyone should know about and understand.  

On the other hand, you can use the tests as a guide of what to expect from banks in the near future. As one Credit Suisse analyst put it, "The tests should be thought of as less of a supervisory tool and more of a way to say who should and should not be allowed to distribute capital." 

Of course, if you look at the tests from the point of view of the NYU analytics, you may find some major bones to pick with that idea as well.