A few days after the Berkshire Hathaway (NYSE: BRK-A) (NYSE: BRK-B) shareholders meeting, Vice Chairman Charlie Munger did a short interview with CNNMoney. Asked about the state of the financial sector, he (as usual) held nothing back.

We would be better off if we downsized the whole financial sector by about 80%. I don't think the rest of us have anything to gain having massive trading between computers which try to outwit one another with their algorithms to the extent that when one succeeds, the rest of us are all paying for it. And why should we want to encourage our brightest minds to do what amounts to code-breaking and electronic trading? I think the whole system is stark-raving mad. Why should we want 25% of our graduating engineers going into finance? ... I don't see any social contribution.

But Berkshire has large investments in Wells Fargo (NYSE: WFC) and Goldman Sachs (NYSE: GS). Isn't that hypocritical?

We buy the investments in the public market that are available. We don't tell the people running them what to do, and we don't allow our thoughts about what the law should be to change our investments. We invest in the world as it is. But if you ask me what the world should be, I would say that the finance sector of the world should be downsized by at least 80%.

I like Munger, but this seems like at least a partial cop-out. Munger has praised Goldman not just for the return Berkshire received from its investment, but for the bank's morality. Pressed last year, he eventually blamed Goldman's mishaps not on the bank's ethical code, but the legal code. "Goldman was in a world where Congress legalized all types of derivatives. It's an inherently dangerous world," he said last year. "Given that world, I see no reason to think Goldman misbehaved in some horrible fashion. Everyone was doing it, and it's only natural to increase your moneymaking activities when you can do so legally."

At any rate, let's put some numbers on this stuff. To get a sense of the magnitude of the problem, consider the profit growth of the financial sector compared with all other sectors:

Source: Bureau of Economic Analysis.

To put Munger's criticism into context, the important point here is not gawking at how much faster the financial sector has grown over other sectors. It's wondering how much faster other sectors would have grown if our best and brightest chose careers in engineering, medicine, or technology instead of derivatives trading. In one of the more depressing statistics to come from the academic world, financial engineering is now the most popular undergraduate major at Princeton's School of Engineering and Applied Science.

Munger is no doubt being hyperbolic when he says he'd shrink the financial sector by 80%, but not as much as you might think. Finance currently makes up about 8.5% of gross domestic product. In the 1960s, the figure was closer to 3.5%; in the late 1940s, it was around 2.4%. So even if today's financial sector were cut in half, its share of the economy wouldn't come close to breaking historical precedents. Cutting out one-third would bring the financial sector back to where it was in the mid-1980s.

There's an argument that growth in the financial sector has helped the broader economy, but this is controversial -- and unsubstantiated -- to say the least. As former Federal Reserve Chairman Paul Volcker once said, "I wish someone would give me one shred of neutral evidence that financial innovation has led to economic growth -- one shred of evidence." That evidence would not only have to point to growth, but growth so large that it compensates for the downsides of financial innovation, such as financial crises, deeper recessions, and bailouts. Good luck with that.

What do you think?