Last month I interviewed psychologist Daniel Kahneman, who won the Nobel Prize in economics in 2002 and recently authored the book Thinking, Fast and Slow.

In this clip, Kahneman and I discuss how overconfidence affects everyone, and what role it plays in investing. (Transcript follows.)

Dr. Kahneman: Well, overconfidence is generally a human trait, optimistic overconfidence, most people are. But then when you look at investors, you are looking at the self-selected group who are likely to be more optimistic than other people. The pessimists just aren't there, so when you look at investors, you see only the optimists in the investor, the active investors who are looking for ways to get richer. So these two things together, self-selection and the general optimistic bias of mankind, I think they tell the story.

Morgan Housel: So it's more natural for us to be optimistic than pessimistic?

Dr. Kahneman: Oh yes, that's definitely true.

Morgan Housel: But at the worst times, we seem to be pessimistic as investors. In 2008, when everything's falling apart and getting cheap, the times you should be investing are when people get the most pessimistic. What is the decision-making process that people go through in that situation?

Dr. Kahneman: Well, the decision-making process is basically inferring from recent trends as if they were to continue. That seems to be the information that people go on and so when things have been getting worse for a while, you become pessimistic, and when things have been getting better for a while, you become optimistic, and it's those feelings that really control the investment, I think. It's not the reasoning that, well, things are bad, and I shouldn't invest now. That's a very unnatural frame of mind. Professionals have it, but other people don't.

Morgan Housel: So this is recency bias, when we're taking the recent past and assuming that that's going to continue in the future.

Dr. Kahneman: It's not only restricted I think to individual investors; I mean analysts also tend to predict the past to a very substantial extent. But yeah, it's extrapolation from the recent past.