A few years ago I had a medical issue that consumed me. I always assume the worst and naively consult Google, which diagnoses everything as terminal cancer.

I finally saw a doctor. Just making the appointment was a relief. Doctors care about you. They're there to help.

I could have talked to the doctor for an hour, explaining my story, my fears, my hopes. Instead I got about 90 seconds. I'm only exaggerating a little when I say the appointment was:

Doctor: "What's the problem?"

Me: "For a while I've ..."

Doctor: "Here are pills. Bye."

This is common. One 1999 study found patients were allowed to talk for an average of 23 seconds before their doctor interrupted. Another found new doctors spend an average of eight minutes with each patient. Three out of five patients think their doctor is rushing through exams. It frustrates them. It frustrated me.

But you have to balance the desires of a patient with the realities of being a doctor. Dr. Larry Shore once told NPR:

When you have that pressure to see three, four, maybe five patients an hour, you can't wait for the exposition of the patient's story. Which is exactly what you should do. But you can't.

Patients don't want to pay much for primary care, so providers make it up on volume. It's not that doctors don't want to listen to patients. Most desperately do. They just have to practice their craft within the confines of running a viable business. In a perfect market this would correct, with patients paying up for more personalized care. And some do. But the path of least resistance is to carry on as normal, with everyone frustrated. It can force well-meaning doctors to deliver subpar results not because they're incompetent, but because their industry has conflicting realities.

The same thing happens across Wall Street.

Alice Schroeder, a former analyst who wrote Warren Buffett's biography, gave a great explanation of the realities of working on Wall Street (emphasis mine):

The firms essentially want two things that are incompatible. They really do want you to do non-consensus work that's attention-getting enough to be of interest to clients, but it also has to be right to be commercial, or you are punished. The fear of punishment nearly always beats the desire for reward, so this creates constant pressure to pull in your elbows.

A lot of what we scoff at on Wall Street – poor fund performance, high fees, inflated expectations – come from good, honest, people who work in an industry that wants two incompatible things.

Take active fund managers. Most are disastrous at what they do. Ninety percent of active stock funds underperformed their benchmark over the last 10 years, according to Vanguard.

In order to outperform, a fund has to look far different from its benchmark index. ("Active share," in industry lingo.) But when you have a lot of active share, you are nearly guaranteed to have years when you dramatically underperform. And when you dramatically underperform, your clients withdraw their money faster than you can blink, and your career's over. What's left is an industry with very low active share, even among smart managers who know it's the wrong thing to do. This guarantees a fund will never outperform, but it also promises big underperformance will never show its face, which is what the manager needs to run a viable business. It's sad, and I'm not condoning it. But it's the practical reality of two parties wanting incompatible things.

There are so many similar examples.

Bond investors want objective analysis, but rating agencies are paid by, and want the return business of, the companies they're rating. The result is a string of incompetency, not because rating analysts aren't smart, but because the path of least resistance is toward an industry with conflicting goals. Schroeder shared a similar example as a stock analyst: "Once, we were told 'Pat Ryan [then CEO of Aon] is reading your reports, and he's not happy,' as if our job was to make Pat Ryan happy."

Good journalists only want to cover the most important stories, but media outlets need to publish enough stories to pay the bills.

Regulators want to shake things up and do what's right for society, but they also want to keep their jobs. Same for politicians, many of whom want to do the right thing but also need to win the next election in order to do it, with the latter taking precedence.

The good news is technology and access to information is spawning more win-win business models. High-fee funds are dying. Conflicted advice is leaving. As Josh Brown wrote this week, good financial advisors have tons of job security. There's more alignment between customers and advisors today than there probably ever has been. That trend isn't slowing down.

But we should also acknowledge that, just like my doctor, there are conflicts between what customers need and what good, honest professionals are sometimes able to provide. Coming to terms with this might make you less cynical about the people who work in the the finance industry and more enlightened about what needs to change. 

For more:

Performance vs. outcomes

Why does pessimism sound so smart?

Things I'm pretty sure about

Hard truths for investors to wrap their heads around

How the investing industry could change