Image by Hidehiro Kigawa.

Few rational people would argue that closer regulation of the banking industry was uncalled for following the calamity of the last few years. But as things play out, it's become increasingly obvious that innocent shareholders in banks like Bank of America (BAC -0.21%), JPMorgan Chase (JPM 0.06%), and U.S. Bancorp (USB 0.32%) are the ones paying the price and not the bankers that caused the crisis in the first place.

The most obvious examples of this are the massive legal fines and expenses that the megabanks have racked up over the last few years. To date, Bank of America has paid $43 billion in fees and expenses related to litigation and settlements, its chief executive officer Brian Moynihan said earlier this week -- the figure for Bank of America in the chart below is based on estimates from The Wall Street Journal. And in the middle of last month, JPMorgan Chase settled a variety of claims for a record $13 billion, the largest fine ever extracted by the government from a single company.

Even more expensive are the increased compliance costs that will haunt the industry going forward. In a presentation on Wednesday, JPMorgan CEO Jamie Dimon told analysts that the nation's largest bank by assets will incur an added $2 billion in expenses related to compliance by 2014. "The $2 billion, for the most part, is permanent," said Dimon. "It's not going to go away."

To be fair, JPMorgan is in a unique position, becoming a favored target of regulators following the 2012 London Whale scandal that eventually cost the bank $6 billion, after being initially dismissed by Dimon as a "tempest in a teapot." But the increased costs of compliance are nevertheless hitting the entire industry.

This was something that Richard Davis, the CEO of U.S. Bancorp, discussed at the Bank of America Merrill Lynch Banking and Financial Services Conference last month. In response to an analyst's question about the "top three structural issues that the industry needs to address," Davis called out compliance (emphasis added):

It then used just to be in the conversation, even two years ago I wouldn't have brought it up. But I think compliance is now the watchword for success. All of your performance financially doesn't get credit unless you're compliant, when you're doing things perfectly all the time. And so, we have to invest a fair amount of expenses into doing things right the first time and to accrue the kind of trust you need to have from your regulators to enjoy, I think, what will be a long-term success. And that's an investment curve right now for all of us that I want you to be paying attention to.

The net result is that almost all of the nation's major banks have seen an uptick in their core efficiency ratios, meaning that there's less revenue left over for shareholders at the end of the day. For its part, U.S. Bancorp, a notoriously efficient operation, reported a 200-basis point year-over-year increase in its efficiency ratio in the most recent quarter.

Is this something that bank investors should be concerned about? I think it is, as it'll inevitably drive down shareholder returns. The real question, in turn, is whether they'll be driven down to the point that investors will start avoiding the nation's biggest banks in favor of their smaller, less affected rivals. That, as they say, is to be determined.