Late last month, Robert Shiller stopped by Motley Fool Headquarters for an hour-long interview about housing, stocks, bubbles, and more. A Yale professor who just published his 10th book, Finance and the Good Society, Shiller is an expert on a variety of economic topics.

Given his clout within the field, I asked him about the biggest change in his investment thinking over his long career (he's been a professor at Yale since the early 1980s).

In the video below, see why he thinks the efficient markets hypothesis is a "half-truth," and see what he thinks is really driving the markets these days. (Run time is 2:19; a transcript is below.)

Brian Richards: What has been the biggest change in your investment thinking over your career?

Robert Shiller: Over my career? That's a long time. I used to be cowed by the efficient markets hypothesis, and now I think it's a half-truth. It is true that money is harder to make a profit than you'd think, and randomness plays a huge role in your outcomes, investing outcomes. But it's not true that markets are efficient, and in particular, aggregate markets are driven primarily, I believe, by investor sentiment.

So I have been involved for decades now in behavioral finance, which is academic study of psychology and finance together, and I think this is hugely important. The most exciting thing that's happened in academic finance is all this behavioral research. It's now becoming augmented with neuroscience research, and that's the new, new thing. They're putting investors under magnetic resonance imaging and observing their brain during their thought processes about investing. And this will change the way finance is done.

Richards: You mentioned you'd been studying behavioral finance for a number of years. Are you seeing that investors are still falling for the same biases that they were when you started in this field, or has this sort of improved discourse about what we're prewired to think and to do, has that helped change any of the behaviors?

Shiller: Well I think that it's gradually getting better. Markets are getting more efficient. So the momentum effect in the stock market was stronger in the years 1881 to 1941 than they are now. But we still have a momentum effect. We still have new people coming up and making the same mistakes that have always been there. We haven't all become savvy.

For more insights from my talk with Robert Shiller, see: