Justin Fox is the editorial director of the Harvard Business Review Group and author of The Myth of the Rational Market: A History of Risk, Reward, and Delusion on Wall Street. Before his gig at HBR, he wrote a weekly column for Time magazine and created a popular blog at Time.com, The Curious Capitalist. At our recent Motley Fool 2011 Investing Conference, Fox was interviewed by the Fool's Tim Hanson.

You can watch the video of Fox's Q&A at the conference (running time is approximately 35 minutes), and we've included two camera angles. Or you can read the full, lightly edited transcript below. Note: This was recorded on Sept. 22, 2011.

Tim Hanson: Fools, we are here with Justin Fox and we are going to talk about the myth of the rational market, and this is a great day to do it, on a day that the market is down 3.5% at last I checked. Everybody here I think is an aspiring investor, but not everyone is a master investor yet, so I thought we could start by just defining the terms for everybody. So what is a rational market?

Justin Fox: There was this belief, starting in the 60s, 70s, and 80s. This belief came out of academia, University of Chicago, MIT, that financial markets were these kinds of perfect processors of information. Sort of the first leg of this whole belief was the random walk. This idea that all the chart readers out there were full of it and that most of the patterns that they saw in stock prices could pretty easily be replicated by just flipping a coin and then adding up the results. People started doing research on that in the late 50s and early 60s, and poking holes in a lot of the chartist methods.

Next up came starting to look at mutual fund performance and testing it against these new measures of market risk that the professors were coming up with, like beta, the Sharpe ratio, alpha, the Traina ratio. And again, the result was when you looked at American mutual funds as a group, they did not outperform the market after fees. So that was sort of part two.

OK, first of all, they are random. Second of all, these really smart professionals, highly paid professionals as a group can't seem to beat the markets. Then the third part was they started in Chicago doing these things called Event Studies, where you would basically find something that was happening at companies or out in the world and watch how quickly and in which direction markets reacted. Basically the gist of; thousands of these things have been done over the past 30 years and what they find is that wow, the stock market is pretty good at sussing things out and reacting really quickly. So you put those three together and there was this belief in the 70s that that must mean that the market is just right and there is not point at all, ever, in trying to beat it.

Hanson: So this is a "wisdom of crowds" idea, the fact that as millions of buyers and sellers in the market are gradually getting closer and closer to truth basically, in their assessment of the companies that they are looking at?

Fox: Right. There was this; sort of when you start thinking about this whole idea, you realize that it can't be literally true because they are relying on the fact that there are smart, rational investors out there to keep markets rational, and if markets were priced so perfectly, then those smart, rational investors could never make any money, so presumably they wouldn't expend time and resources trying to, and so you would have an irrational market.

And you talk to these finance professor types and they all admit if everybody in the world owned index funds, we agree, markets would be completely crazy. We just think there is a continuum and we think it is a lot closer to markets being right most of the time than most folks out in the world think.

Hanson: You called the book The Myth of the Rational Market. I think we can all guess which side you ultimately come down on, but I thought you did a very fair treatment of the original idea of a rational market and why it came to be. But I thought one of the interesting insights was you said over the years as this idea started, it gained momentum, gained force, gained popularity, but then also lost nuance. What does that mean exactly and what did it lose along the way?

Fox: A classic one is just with the idea of the random walk. The people who were looking at this early on, one of them was Benoit Mandelbrot, famous mathematician and creator of the idea of fractals, who died about a year ago. Mandelbrot was visiting the University of Chicago and Harvard and MIT and Mandelbrot, he has this view that yeah, the markets were random, but it wasn't this easily captured randomness. The movements of the markets didn't fit into the bell curve. There were these fat tails and what have come to become called black swans and all that kind of stuff. And basically everybody who was studying these things at the time, Eugene Fama at Chicago was sort of a disciple of Mandelbrot's.

Everybody knew this was true, but they finally decided they couldn't use Mandelbrot's statistical models to do anything interesting. They couldn't calculate beta with it, they couldn't do all these other things, so they said well, we will set it aside. Within 20 years, any young person studying finance actually believed that markets followed these nice, clean bell curve statistical patterns. They just completely pushed aside this fact that everyone early on knew, that there are different kinds of market randomness and it is actually messier out there than the finance professor's models make it out to be.

Hanson: So it sounds like there is some wisdom, crowds have some wisdom, but they are not 100% right 100% of the time.

Fox: I always thought "wisdom" was the wrong word for it.

Hanson: What would you propose?

Fox: They are greedy, in the market they are greedy, so they are always looking for money, easy money, and they are very quick and they are smart in a way, but "wise" just isn't quite the word for it.

Hanson: Just to take it to sort of the modern day, one of the more interesting, I think, debates happening in the financial markets today is between banks and the market. European banks right now, for example, say that all the bonds that they hold and sovereign debt and these sorts of things are worth X, and the market, in bidding on those same bonds and things, say they are worth a lot less than X. There is no chapter on this in the book, but who, as an investor, as a potential investor, should I believe when I am looking at that huge disconnect between what one bank says about what they own and what the rest of the world says about what they own?

Fox: In general -- and this doesn't just apply to banks. Like when the CEO of a company is saying, Oh, the market is misunderstanding my company?" -- I'd trust the market more than I trust the CEO, because at least with the market it is a bunch of outsiders making judgments, not some totally self-interested judgment from inside. And I think the experience of the financial crisis over the past few years was that early on, especially early on, the people, the market prices which had these subprime securities and things derived from them knocked down to incredibly low prices really quickly, were a lot more correct than the banks were.

But it is definitely true. Markets are volatile and they jump around a lot and I think bankers were used to this system where they sort of perceived the world in a different way and smoothed out a lot of that volatility. So I would say probably the market is right, but I do think there is, at some level there is something to this argument from, and it is not just the bankers. Steve Schwartzman was making it back early in the financial crisis, that if you sort of outsource all your judgment about the value of things to financial markets, you are going to end up really schizophrenic about the value. They are going to jump around a lot.

Hanson: I will admit, I was hoping for an easy system. I was a little bummed when the end of the book said I had to use my own judgment to figure things out. [Laughter.]

Fox: Sorry.

Hanson: There are a lot of great characters in the book. It is a wonderful read of market history and financial history, and I thought one of the chapters I didn't see a lot of other people writing about was the chapter on Michael Jensen, and about how, you alluded to this earlier, the stock market sussing out CEOs who are doing a bad job leading to takeovers and those sorts of things, and also CEOs who are doing a good job.

We have a belief at The Fool that we love CEOs who own a lot of stock in their company, but we also, at the same time, don't really want them paying attention to their stock price. We don't think it matters, but should a CEO be paying attention to the stock price?

Fox: I think they should be looking at it as a potential source of information, but they shouldn't be looking at it as like a box that they have to listen to, get orders from every morning. That is a really difficult balance to get, but I do think markets tell you when you say something in your quarterly earnings conference call and the market goes down, you should learn from that as a CEO. So I don't think it is right to want to completely cloister yourself from it, but I also, that sort of late-90s thing where you would walk into any company in Silicon Valley and they would have the stock price posted in the lobby. That is not really healthy either. That doesn't help you run your company any better.

Hanson: One eye on the market, it's maybe an unbiased advisor?

Fox: Yeah, maybe not even one eye. Check in with it every couple of months and see how it is doing, but don't let it run the show for you otherwise.

Hanson: To go back to the Jensen chapter, I was interested; a lot of those takeovers that you talk about, we were talking about people like Carl Icahn and some of these hostile takeovers where guys come in to force change on a company and create value. Do you think that works as a strategy or should these CEOs who got taken over, obviously they objected strenuously to the potential for takeover and tried to have the government stop it from happening. Should they have been given the benefit of the doubt, given the time to manage things or were these hostile takeovers ultimately good for the companies that they happened to?

Fox: I mean on the margin, I think it is healthy to have that threat of a hostile takeover out there. When you have times when there are lots of them going on, you end up having waste and a lot of financiers who have actually not the faintest idea of how run a company. I mean Carl Icahn is kind of fun with all his campaigns, but he is basically got about three things he is pushing for every time. And that is sort of right. He doesn't claim to have any sophisticated idea of how to run a company; he just looks, OK, you are doing too much of this, stop it.

The thing that happened in the early 80s was stocks were just so cheap and so when Michael Milken kind of opened up another source of financing with junk bonds, there was just this imbalance that you could get the financing through junk bonds and that would allow you to go buy up all of these companies. And in the end, that was healthy. That helped begin the bull market and bring stocks up.

Clearly by the late 80s, these people like Ron Perlman. I shouldn't say this without sort of going through it, but I don't get the feeling that man has created a lot of value for investors in general. I think that is true of a lot of those people.

Hanson: So I want to go back to maybe the beginning of the book, the beginning of the theory, because one of the things that is great to read about as an observer is other people's failure. And I want to talk about Irving Fisher, who said in 1928 and 1929, that stocks had reached a "permanently high plateau." How was he so spectacularly wrong?

Fox: Irving Fisher is just such an interesting character because he was so smart, but he was such a goofball too. And I guess those things are combined a lot. Sylvia Nasar, who wrote A Beautiful Mind, has this giant, new economics book coming out called Grand Pursuit, which has, if possible, even more about Irving Fisher than my book does.

This is the guy who invented the Rolodex. You find in his writings around the turn of the century basically everything that later came to be known as academic finance from in the 50s and 60s, the idea of dividend discount model, the idea of diversification, the idea that stocks should, over time, outperform bonds. He was sort of the first or among the first to say all of these things.

What happened to him in the 20s is he sold his little index card company that became Rolodex. He sold to Remington Rand, which is one of the big, kind of technology companies of the day and he ended up a big shareholder of that, so he is suddenly rich. He had sort of married into wealth already, but he had earned it this time. He'd helped push Prohibition through.

Hanson: I am surprised no one booed that in this crowd.

Fox: He'd had all these great successes. He wrote this long piece for the New York Herald Tribune at the end of 1928, and I think the headline is, "Will Stocks Stay Up in 1929?" and that is sort of the most involved justification for his "permanently high plateau" argument. And it was basically that; first of all, the economy has grown a lot. Companies are earning more money, so it is not all a bubble. Then second of all, people are diversifying more thanks to the innovation of the, at that time it was mostly closed-end funds. There were few modern-style, open-end mutual funds, but it was mostly closed-end funds, but thanks to the just boom of the fund industry, people were diversifying in ways they hadn't before and because when you diversify, it is OK to have a bunch of risky stocks, as long as they are not completely correlated with each other. And that was a good thing, and that was therefore driving prices up. And this thus constituted the "permanently high plateau." If you just look at the history of stock prices, plateaus aren't something markets really do.

Hanson: No, I couldn't believe that that idea even passed the sniff test. When I read about it, my jaw sort of dropped. Basically the idea in its essence was that I own 10 good companies; therefore I can now own five really crappy companies and feel really good about it. Yeah, it was wild. To fast-forward, two of the other characters in the book are characters we know very well, and we have conference rooms actually named after both of them in our office in Alexandria. One is Jack Bogle and the other is Warren Buffett. Both have been spectacularly successful. How do they coexist?

Fox: You listen to Buffett and he says, Well, most people should go buy Jack Bogle's funds. And I think Bogle at the same time; Bogle's argument for why index investing works isn't that the market index is the best possible thing you can invest in. It is that an index fund is the cheapest possible way to invest. He has done all of this research over the past 20 years where he will just look at what is the best predictor of future mutual fund performance, and basically most of the time over most periods it is just the cost. It is the lowest cost funds give the best performance over time because they just don't have that extra hill to run up of having high fees.

The funny thing with Bogle is that he participated in the whole 60s go-go fund thing on the other side, and there were these two guys, two grad students at Chicago in the early 60s wrote an article for The Financial Analyst Journal saying, You know, we ought to have index funds. I could just buy the Dow [(INDEX: ^DJI)] and that would be great. This guy writing under a pseudonym, and it was John B. Armstrong, and I sort of suspected. As soon as I came across that, I was looking through back issues in the library at the business school at Columbia, and when I got back to my office I called Bogle and I was, Is that you? And he said yeah.

John B. Armstrong, a mutual fund company executive writing under a pseudonym says this is a horrible idea, this index fund, because mutual funds give you, at least as they have existed up to now, they have these smart managers who have been through hard times, and if you adjust for risk, they have actually outperformed the indices. I think up through about 1959, that was true of the handful of mutual fund companies that had survived through the 20s, 30s, and 40s. And it immediately started not being true because the fund industry just got caught up in this whole performance craziness with Fidelity leading the way.

And Bogle finally, he was at Wellington and their main thing was the Wellington Fund, it is a balanced fund. It was stocks and bonds and that just wasn't working in the 60s. And so he went and by then in the mid-60s he became president of Wellington and he went and bought or merged with this little go-go fund company in Boston. They ended up basically taking over and throwing him out in the early 70s and Vanguard was his kind of sneaky way to keep a job because mutual funds always have boards that are supposed to be independent of the advisors.

In this case, the boards of Wellington and a couple of other, I guess Windsor, were mostly old Wellington loyalists, not people who had come from this other fund company. I am just spacing out on the name of it. It is totally forgotten now. Ivest I think it was called. So Bogle went to the boards of these funds and said let's break off from these people who now run Wellington. And they wouldn't do that, but they came up with this sort of; he came up with this scheme where OK, we won't do management and we won't do distribution. We will just do like the bookkeeping work for the funds, and that allowed him to start an index fund and to do direct sales by phone, because distribution was brokers and management was actually managing the fund, and that is how Vanguard started.

I love that story. He calls it, when he talked to me, one of the greatest disingenuous acts of opportunism known by man.

Hanson: It worked very well for him.

Fox: Yeah, it did.

Hanson: In terms of the feedback you have gotten from the book, would you say that most of the people out there today are still believers in the rational market, or have more gone over to the other side and are out there picking their own stocks and trying to beat the market like we all are?

Fox: I think in general there are just fewer people trying to go anywhere near the market today, for reasons that have nothing to do with my book. What is kind of funny is after doing this whole book, my takeaway was I am a crappy investor. I mostly am just going to sit in index funds and not try to do all this stuff, because I am hearing these stories of the guy who bought Apple at $8 and I bought Apple at like $11 and held on, because my wife wanted to buy it. She really liked her Apple products, and at around $48 I thought I don't want to be greedy, that's ridiculous, so I sold. And on gold my wife wanted me to buy her some Krugerrands back when it was $150 an ounce or whatever, and I went down to 47th Street in New York, and they wouldn't take charge cards, and all I had to do was go to an ATM and get the money and come back and get it, but I never got around to it. So I shouldn't be allowed to pick my own.

Hanson: Well Peter Lynch was successful because he listened to his wife.

Fox: Yeah, I guess if I just listened to here, I would do a lot better. So I think for a lot of people, if you invest in index funds, at least if they are really cheap or you just have a generally passive investment method, the one thing you can say is you will do slightly better than the average investor because in general, the indexes are going to reflect the averages, and if you are paying less than the average investor, I totally think there are people; there are reasons why people, including individuals, not just hedge fund managers with tons of resources, can outperform the market. I think there are a lot of behavioral reasons why most of us don't.

One of the interesting things is the reality that value investing is successful over time. That is something that finance scholars like Eugene Fama, Mr. Efficient Market, started to realize in the early 90s, and Fama and Ken French's explanation for it was, Oh, it's just because value stocks must be riskier. Because that was just the only way they could fit it into their model. But in the mid-90s and this was kind of an obvious thought and Keynes had written about it before, but it had to be done in academic papers with formulas and all. People started writing papers kind of point out that it is actually really hard at exactly the moment when as a value investor you should be really making a killing or at least preparing yourself to make a killing, is exactly when all your investors are abandoning you. That is just the difficulty of being a professional investor. You are stuck with your customers, and as an individual you can avoid that to some extent.

Hanson: That is absolutely true. Just to change gears here at the last moment before we open it up to audience questions. A lot of people in this room have something in common, and that is that we all write about the market and we write about finance, and you do a very, very good job of it and have for a very long time. So how do you make writing about the market interesting and what's the biggest mistake you see people who aren't as good at it make?

Fox: I guess one of the biggest things, and when I look at the biggest mistakes I have made writing about business and market, just acknowledge that it is OK not to know what the future is going to be. It is fun to make the predictions and I loved this panel before, because there was that element of "it's our job to make predictions, but we understand that a significant percentage of the time we are wrong." And I think just actually often that can be seen as such a breath of fresh air if you do it right.

The other thing, I guess the other element to it is just find something that makes, that you can add to the discussion that 20 other people aren't doing. And my immediate penchant is learn the history. I figure that is what can give me a leg up. I want to learn the history. And I am just interested and I want to read about it. I am definitely not the only person who does that, but that in general is, gee, how do I put this in context? How did things get to be this way? It often is kind of fun to read and I think it does help you understand how the world works.

Hanson: So with that, we will open it up to anybody who wants to ask Justin a question themselves. Or I can keep going. Tom?

Tom Gardner: How do you put this market that we are in in context? Just the incredible dynamic of having pretty much 0% interest rates and $15 trillion in debt. Is there any good context for us to understand the marketplace in the U.S. today from your historical research?

Hanson: Justin, I am supposed to repeat the question which was, "How do you put today's market with 0% interest rates and massive amounts of debt in context, in historical context?"

Fox: The main thing is it really does seem like; it is such a ridiculous thing to say, but it's awfully uncertain. And the reason is, it seems like there are just a lot of issues out there where it could go one way or the other and the difference is really dramatic between those two outcomes. So Europe could muddle through maybe with modest problems, but somehow get through this, or the Euro could fall apart and there is this total breakdown of the European economy. Those are so different. The future is always uncertain, but right now it feels like one of those times when you look at the possible scenarios out there. They are really far apart.

To me, in a lot of ways, that is the most, after the fact, that turns out to be the times that create fortunes and are great times to invest. I don't know in what exactly, but it is definitely an opportunity-creating time whereas when everybody sort of sees the future in the same light and everybody's scenarios for the future, not everybody but most people in the market, their ideas of the future are pretty similar, you figure there is less opportunity there. Whereas right now there is such disagreement that would seem to signal opportunity.

Hanson: Ollen, yeah, next to Tom.

Ollen Douglass: Speaking of opportunity, what is your wife telling you to buy now? (laughter.)

Hanson: The question is, What is Justin's wife telling him to buy now?

Fox: I should ask her. I think she has sort of given up because I don't seem to listen, but that's a good; I will ask her. I might not tell anybody, but I will ask her.

Hanson: Dave?

Dave Meier: In today's cast of characters, who may be flying under the radar, who do you think we'll be talking about in 20 years?

Hanson: The question is, "Which of today's characters in the market are we going to be talking about in 20 years?"

Fox: I don't think I have a good answer to that, because I have been spending the past two years helping run this business publication that barely pays attention to the investing world. I don't know because I am not out there covering money managers, investors, and in terms of companies …

Meier: It could be an official, it could be anyone. You don't have to link it to (unclear).

Fox: Right. I feel like this is the kind of thing where on the Metro ride back I will come up with some good answers, but it is not happening to me now, so I had better stop just hemming and hawing, but maybe talk to me afterwards. Maybe we will come up with something better.

Hanson: Jim?

Jim Mueller: So the answer to rational markets is behavioral finance, but behavioral finance says that humans kind of go all-in on something. So is it possible that the markets are rational most of the time?

Hanson: The question is, "Could the markets be rational most of the time?"

Fox: In some broad sense over time, yeah, I think markets do; they certainly do a better job than any other mechanism I know of to sort of set prices and allocate investment capital over periods of time. I think one of the most difficult things about that question, though, is that sometimes markets create the reality that they then go on to reflect in times of financial crisis, but also times of big bubbles. You reach this point where it is not about whether the market reflects fundamentals or not; it is that fundamentals of the economy start to reshape themselves because of the incentives being created by the market. That just is what it is. I don't know whether it is good or bad, but I think that is one of the core difficulties of this whole idea of this market that rationally reflects fundamental values, because sometimes markets could shape fundamental values.

Hanson: Ron?

Ron Gross: Do you think the rise of things like high-frequency trading have made the market more efficient or less?

Hanson: Has high-frequency trading and all the computers made the market more or less efficient?

Fox: It has made it more efficient in the simple sense that it has lowered transaction costs even more, because basically a lot of the high-frequency trading going on out there has sort of replaced what the market makers used to do. It is people creating markets in these stocks by making half a penny here and half a penny there, partly in fees that the exchanges pay, partly just by timing the trading well. I guess the main issue with it is it, like with the Flash Crash, it seems like no one really has a grasp on it. It is all happening faster than any human can keep track of. Either the machines will rise up and take over our stock markets, or else they will all just go nuts, and that is scary. It feels like the kind of thing that when over a period of years maybe we can figure out what the risks are and make it work more smoothly, although I guess if the trading periods keep getting smaller and smaller, we will never catch up. I don't know.

Hanson: Scott?

Scott Schedler: Same question as Ron, but about derivatives.

Hanson: Same question about derivatives.

Fox: There is this belief, and this was one that I struggled with because you don't find it written up in a lot of academic papers, but clearly one of the big things that came out of academic finance that then was really adopted wholeheartedly by a lot of people on Wall Street was this idea that the more financial instruments you had out there, the more different ways to play the market, to play different sides, the closer you would approach economic perfection. And it is a similar issue with algorithmic trading. At some level maybe yeah, if all the only bets you can take out there in the world are will the economy grow next year or will it not grow? And that is the only thing you can bet on as an investor, then that is not a very smoothly functioning market.

Whereas if there are a million different little bets you can make about companies, about whether these loans will go bad or not, then there is some value to that, but clearly it took on a life of its own. It seems like when you have this ability to make these huge bets on things that don't even exist, like it was possible to have these CDOs built upon CDOs that weren't actually backed up by any real loans, it seems just like an opportunity for more bubbles and busts and volatility. So I think it is one of those things where it was hoped for a while that there was this clear answer for what is the right amount of high-frequency trading, what is the right amount of friction in markets, I guess would be the best word. I think the answer is I don't know, but it is not zero.

And so there is something to be said for not being able to cheaply bet on every last possible outcome in the world. I am sort of losing myself in this discussion. I think that is the trouble a lot of people have when they start talking about derivatives. But I don't think that the sort of standard accepted viewpoint from Bob Martin and a lot of other people in finance and that Alan Greenspan held very deeply, which is that we are approaching this moment of perfection where there will be no more economic volatility because it will all be taken care of by financial markets. It doesn't work that way.

Hanson: Yeah, Alex?

Alex Scherer: Twenty years from now, do you think the market will be more or less efficient?

Hanson: Will the market be more or less efficient in 20 years?

Fox: It will probably be over time. I don't think it will get massively more efficient. I think it is just going to vary. There will be times when it is doing things pretty well and times when it is not. I don't think, at least since, I don't know what the point would be in the 20th century where we sort of move from; because clearly if you are in a market where there are huge transaction costs, there's not much information out there, and you move to a market where transaction costs are lower and there's tons of information available to everybody, that's more efficient. It may still have volatility, but over time it is more efficient. But I think there are clearly diminishing returns and I get the feeling we have reached the point where whatever further gains in efficiency we make by having more information, more processing, computer processing power, lower transaction costs, we will kind of be swamped by the usual rises and falls in investor optimism and pessimism.

Hanson: I will close with a selfish question of my own, which is that I will say that (A) that I really love reading The Harvard Business Review when it arrives on my doorstep, but that is sort of an obvious one for someone working in the business and investing world. Assuming that you have a view under the huge landscape of writers and publications out there writing on this field, what non-obvious publications or people should we all be reading?

Fox: It's funny, because I sort of have, when I was doing The Curious Capitalist in the middle of the financial crisis; I kind of got my crew of people I would read all the time. One guy who, I think he is pretty popular now, but I think he is wonderful. He is this fund manager down in Australia, John Hampton, Bronte Capital, and I imagine a lot of you guys …

Hanson: Fabulous blog, yeah.

Fox: Yeah. My friend Felix Salmon I love to read, so I don't feel like I have kind of moved beyond that bunch. What I am starting to do, I finally realized I have library privileges at Harvard, so I check out weird, old books.

Hanson: You guys have a library up there? I thought that was a party school.

Fox: Yeah, exactly. (Laughter.) I am just sort of realizing, OK, this job is this great opportunity to not have to know the news every day, and so it is more kind of pulling back and finding what is in the library, finding things I should read. The book that I checked out most recently is by Robert Nisbet, and I think it is called The History of the Idea of Progress. I don't know if I will get any investing ideas in there, but I think it will be interesting.

Hanson: Very good. Well Justin, thank you so much for your time. We loved having you.

Fox: Thanks for having me. (applause.)