Motley Fool Explorer advisor Simon Erickson recently spoke with Clayton Christensen, who is the world's pioneer and foremost thinker in the theory of disruptive innovation. (You can listen to their conversation above, or scroll down below for the full transcript.)

Christensen was the first to coin the term "disruption" in his 1997 book, The Innovator's Dilemma. Disruption refers to how smaller, nimbler organizations can introduce new products and services that overthrow larger, more established competitors. The theory of disruption has influenced many in the business world and inspired several Motley Fool recommendations, such as Splunk (SPLK 0.04%) and Veeva Systems (VEEV 0.06%)

Christensen's academic theories are incredibly powerful. But perhaps even more impressive is that he's used them to achieve significant investing returns. He claims his personally managed Rose Park Advisors has returned an average of 36% annually since its founding in 2007. How has he managed to achieve these great returns? He shared two of the key qualities he looks for in his investments.


Be disruptive

Wall Street underestimates disruptive companies' ability to move upmarket to increase their sales and profits over time. Be on the lookout for companies whose products are initially inferior to those of the incumbents but are winning customers by introducing different forms of value. Think Southwest Airlines (LUV -0.58%) offering lower-cost regional flights, or Netflix (NFLX 1.08%) offering DVDs by mail. 

Focus on jobs to be done

Companies that maximize customers' experience are difficult to displace, which becomes the true source of a competitive advantage. Watch for companies that nail a nagging customer problem with an incredibly effective solution. That can eventually lead to their brands becoming synonymous with the solution. Examples include Ikea's easy-to-assemble furniture, Intuit's (INTU 0.57%) TurboTax product, and Starbucks' (SBUX 2.04%) coffee. 

Christensen elaborated on each of these during a recent interview. Hear him share his thoughts about the following topics:

  • An overview of Competing Against Luck (0:00-2:27 minutes in)
  • Industries undergoing disruption (2:28-5:54)
  • How to build a competitive advantage (5:55-11:30)
  • How companies should define their market (11:31-18:15)
  • How we should think about big data (18:16-21:09)
  • Two specific things to look for in investing (21:10-23:16)

To conclude our discussion, he played "Buy, Sell, or Hold" (23:17), making calls on several disruptive concepts, including Uber and Bitcoin.

Want a few more disruptive investment ideas? Motley Fool Explorer provides a deep-dive into four companies each month, tied together by a relevant investment theme. Explorer is now open to new members, and you can get a 30-day trial (with a full membership-fee back guarantee) using this link


Erickson: Clay, can you start with giving us a brief overview of your book, Competing Against Luck? For those who haven't read it, yet, why did you write this book and what was the overview that you're getting at with it?

Christensen: Well, it's something that I have seen in my life, and especially as I've tried to help people be better managers. Every time we put a new [product] into the market, we think that our product will be successful. We've done a lot of study of the market and figured out what the competitors are and everything else. So every time we put a product into the market, we think it will be successful. 

But the reality is that nearly 80% of those products that we think will be successful fail financially in the market, so I [wondered] if innovation is intrinsically unpredictable. [That] you have to roll the dice many times in order to get a few wins, but you can't predict which number will come up on top. Or [whether there is] some other explanation for the unpredictability that we see. 

And what we decided is the way people like me teach our students [at business schools] how to think of the structure of markets [means] we've created the problem ourselves, and we concluded that unpredictability is not inherent to innovation. We can actually be successful much more frequently than we thought.

So that was where the question came from, originally, and the reason why we gave it this title, Competing Against Luck. That most of us think that innovation entails competing against luck, but we don't think that that necessarily is the case.

Erickson: Clay, in the book you said it's a book you've been writing for the past 20 years, just in thinking about the business world. We know that you've spent a career molding the theory of disruption. I wanted to ask how your thoughts about disruptive innovation [have] changed over the past two decades.

Christensen: Well, they've changed a lot, and I'm glad that they did. You know, never has a good theory just emerged in a perfect state from the person who developed it at the beginning. It's almost always [that] a theory at the beginning [is] half-baked and incomplete. The way a theory gets to have power is that people who read and use it have to find anomalies or things that the theory can't explain. And the only way the theory gets better is if the people who are the authors of the theory are willing to accept the anomalies as an opportunity to improve it. 

So that's what we did. Some of the anomalies [were] pointed out to us by other academics like us; but by far the most useful ones have been from our MBA students here at HBS. I teach a course that's broadly regarded, and I [tell] the students [the following], as they prepare for a case (because we do it all by the case method).

I say, "You've got three assignments to prepare for every case. The first one is there's a theory that we use in every session of every class, and these are theories about management that emerge from our work or the work of others." 

And then the second is, "You have a case, and I want you to put the theory on like a set of lenses and examine today's case. The question is can you explain why and how this company got in the situation that it's in through the lens of that theory? And then, given that, what does the theory say the management ought to do to resolve the problem?" 

Then the third one is, "I want you to come to class prepared to tell Clay Christensen what's wrong with this theory." 

And so that way, year after year, day after day, more than 100 students [turn] their brains around to show Clay what it can't explain. And I'm really grateful to them that they took it seriously, and they really were trying to find out what the theory can't explain. So as a consequence, the theory of disruption and the theory of jobs to be done, and a few others, really are much better as a result of [having gone] through this crucible. 

Erickson: Are there certain things, Clay, which keep coming up? You said that students were always [asking] what's wrong, or how can the initial theory of disruption [be changed]. Are there certain themes that continually came up in your class? 

Christensen: Yes. Very often they would come up with an example of an industry that didn't get disrupted. Has not ever been disrupted. And maybe the question is [whether] the theory only [applies] to a few situations. For example, hotels for many years have just never been disrupted. Holiday Inn comes up in the 1950s at the low end of every market, but 40 years later, the Holiday Inn has not moved upmarket. The corporation bought Crowne Plaza and part of other businesses at different points in the constellation of hotels, but the Holiday Inn, itself, still is at the bottom of the market. And in a similar way, higher education hasn't been disrupted and we just couldn't explain it. 

And then as we worked with a couple of our students for their paper, we realized that in describing the theory of disruption, I had always assumed that the trajectory of performance improvement that innovating companies provide [means] they keep introducing better and better products. I assumed that it always had the same trajectory or slope. And it turns out that every market has a different trajectory of improvement [and so I did some studies].

The first study about disruption [centered on] the disk drive market and boy, technology moved at a rapid rate in that market. So a company would come in at the bottom of the market and in about eight years they're in the top. I described that market as a place where [00:09:39]. Their whole life was measured in a few years.

But then I [reviewed] our study of the steel industry, where the mini-mills came in at the bottom of the market. It took about 50 years for the industry to get turned upside down. And so the rate of technological change has a big impact on how long an industry will go without being disrupted. 

Then we realized that in hotels and higher education, the trajectory of technological improvement has been zero. There just weren't any new business models that have occurred. And then we realized that oh, my gosh, but wait a minute. Airbnb in hotels is an innovator and a new business model, and they very rapidly have disrupted a significant chunk of hotels. And online learning, in a similar way, is a new business model [and] in just a few years it's turned upside down higher education in many ways. 

So we realized that when we see disruption not occurring in an industry, the result of that is the impact of technological change differs industry by industry, and it isn't the case that it happens in some industries and not in others; but rather, it occurs when [the pace of] technological progress changes. 

Erickson: I'll follow that up, perhaps, with a question about competition, because you say that the pace of change differs in different industries, but the investing community has typically looked at the incumbents (the companies that have the highest margins or the strongest cash flows) as a sign of a competitive advantage against others. 

But then from a lot of your research, we've seen several of these companies are the same ones that get disrupted, whether [it's Eastman Kodak, Cisco, [or] whoever it might be, because of those market shifts and then competition coming in. How can we tell if a competitive advantage is actually sustainable?

Christensen: There are two ways, and the most important is that the theory of disruption predicts that if you have an incumbent leader and an entrant company wants to attack them, if that entrant company tries to attack them head-on by winning the best customers of the established leaders, the theory says that the incumbents are going to turn around and try to beat off the entrants and use its resources in powerful ways to beat the entrants.

And the theory says that in almost every case, the incumbent wins when an entrant tries to win through what we call sustaining innovations, and sustaining innovations make good products better. And if somebody comes in at the bottom of the market with a product that is so much more affordable and accessible that a whole different population of people can now have access to and use a product that historically was beyond their reach, almost always the incumbent leaders fail. 

And it has nothing to do with the power of the incumbent or with a lack of power of the entrant; but rather the incumbent leaders don't want to go after it. That's the essence of disruption. And so your question is answered by disruption. 

Erickson: Yes, it does. It sounds like the competitive advantage of an incumbent for sustaining innovation can't really be understood against a disruptive company that's playing at the lower end [using a] different value in the industry, right?

Christensen: That's right, and it is very predictable. And what's interesting, in the question that you asked, is why do investors not get it?

Erickson: Right.

Christensen: And it's a very interesting question. My son Matthew and I founded an investment company called Rose Park Advisors. It's been operating for about 12 years, now, where we invest in just publicly traded companies. But the policy is if it's disruptive, we buy their stock, and if they are the incumbent and an entrant goes after them, we'll short them occasionally. The greater return on that portfolio for 12 years, now, is 36% on a compounded annual rate of return, and without any shorting or hedging (occasionally we short), but it's not hedged or leveraged. But 36% compounded annual rate of return.

Erickson: That's incredible.

Christensen: And the ability to do that is known. We publish everything. But investors seem to be a little bit arrogant, I think, and not humble to listen to an academic theory in practice.

Erickson: I think it's very interesting when you think about the concept of competitive moats like Warren Buffett has described for decades, versus Jeff Bezos saying your margin is my opportunity. Most of the companies that are capturing all that value are the ones that are destined to be in the sights of disruptive competitors, right?

Christensen: No, that's right. And I mentioned there's another type of innovation in which the incumbent can succeed, and that is when the company follows [the jobs to be done in this new book that we published]. And when the incumbent leader has organized himself around developing products to get a job done for customers, it's very hard for incumbents to win that game; because in many ways it makes disruption impossible to accomplish. So that's why we wrote the book. It took us, as you mentioned, 20 years, but we finally were confident that what we were saying was true.

Erickson: I wanted to highlight that a little bit — the jobs to be done — because that's one of the real themes of competing against luck in your book. Customers are hiring those companies to perform jobs that lead to progress for them. Another thing the investment community has typically done is categorized companies, whether it's based on industry, or selling products that are of a similar function. But based on this jobs to be done theory, and the research you've done, do you think there's a more effective way for companies to really define their own market and then also their competitor's?

Christensen: Oh, absolutely. I think that the way you describe it is exactly right — that the way managers and investors think of the structure of markets is the world is structured by product, category, or customer category. Except that that's not the way that [the world] appears to a customer. 

Rather what happens — the way the world appears to a customer — is every day, darn it, stuff happens to them. Some jobs that arise in our lives are just little, incremental jobs that happen to us every day, and others are dramatic, breakthrough innovations. But when a job arrives in our lives, it forces us to go out and find something, or buy something, or use something to get the job done. And understanding the job is what's critical. 

So it's a silly example. Here I am talking to you. I'm 64 years old, unfortunately, and I'm 6'8" tall, unfortunately. I married a wonderful wife, fortunately. The youngest of our five children, Katy, went to Columbia, unfortunately. And I have all other characteristics and attributes about myself. 

But none of my characteristics or attributes have yet caused me to pick up The New York Times today. Now, there might be a correlation between my characteristics and the propensity that I would have to buy The New York Times. Those characteristics don't cause me to do that, nor do our characteristics and attributes cause us to buy any product or service. 

But rather what causes us to buy something is we have jobs to do, and when we have a job, we've got to find a product that gets the job done. And it's understanding [what the job is] that allows you to harness the causal mechanism behind that choice. That's why we decided predictability was so important to focus on, because if you harness the causal mechanism around a choice, you can predict the future quite well. Do you want me to give you an example?

Erickson: Sure.

Christensen: The third of our five children, Michael, unfortunately went to Stanford to get a Ph.D. a few years ago, and after he had been there for about a week, he called us up and said, "Mom and Dad, I found my apartment and I need to furnish my apartment tomorrow." And what had happened for Michael is he realized he has a job to do, and that is he needs to furnish [his] apartment tomorrow. 

So I'll ask you, Simon. When I described for you this job that Michael had to do (that he furnish the apartment tomorrow), is there a place where you could go to buy what they're selling to get that job done?

Erickson: Sure.

Christensen: Where would you go to get the job done?

Erickson: Oh, I would try to find whoever has got the best posters of the college I'm going to, if I'm at Stanford, something right by the university that's close and that has everything that you need, as quickly as possible. 

Christensen: And what company would that be?

Erickson: Oh, I believe that you referred to Ikea a couple of times as that company, right? 

Christensen: Sure, because it's organized around that particular job to be done. And the reason I refer to it is because anybody who's thoughtful about their lives realizes that that name pops into their mind and they have never been disrupted. In fact Ikea has been rolling its business model out around the world for 50 years, and nobody has disrupted them. 

And it's not that they have any secrets, and it's not that there's no money in it. Their owner is the third richest guy in the world, and yet they have no competition. And the reason is that everybody else that sells furniture is organized around types of furniture. And if I want to compete against those guys, I just have to make better products than they do.

But Ikea is organized around the job to be done, and if you do it in that way, then you can describe all of the experiences — in purchase, in use — that Ikea provides in order to nail that job perfectly. And because they organized themselves in a very unique and complicated way, it's very hard for competitors to compete with them. So that's just an example of my thinking about the market in terms of the jobs to be done rather than products as a useful distinction.

Erickson: And Clay, you refer to a lot of those companies as purpose brands, correct? The companies that become just synonymous with those jobs. Whether it's Ikea, or TurboTax, or Starbucks, it's the company that immediately when I think of it, I think of why I would go to that company, and it's immediately apparent in my head, right?

Christensen: Absolutely right. Absolutely right. V8 Juice. Disney (DIS 0.70%). There are really important ones all around you, when you look at it. 

Erickson: Back to the mention you had of correlation versus causality. We're kind of in the buzzword age of big data and artificial intelligence, right now...

Christensen: Yes...

Erickson: ...where you've got tech that can quantitatively search through a ton of data. It's trying to find correlation between those input factors, but it seems like it misses the qualitative aspects that you've described of what the job is to be done, or what it is a customer wants to hire me to do. Is there some way that we could be using big data more effectively?

Christensen: That's a great question, and as you probably know, it bothers me a lot that people can think that they can dispense with thinking and simply have somebody else develop an algorithm that will provide the answer. I think a much better way is to put data aside for a minute and try to figure out what causality is. And you can't look at causality — you can't observe causality — when you've got 200,000 points of data in a database. 

And [in] the case [of] physical sciences, biological sciences, and management science, to observe things you have to look at an N of 1. And if you can explain what happens in that N of 1, then you can try to explain what happens when the N is two and grow it that way. And data can be very valuable if you use it to validate or show anomalies to a theory. 

But the data doesn't yield the theory itself. The development of a theory comes from an N of 1, and then the validation of the theory, or identifying anomalies to the theory is where data occurs that can be useful. And so big data, to me, [has] nothing big about it. That's the wrong categorization scheme.

Erickson: It sounds like we have the wrong buzzword for big data. It should be go back to small data.

Christensen: Yes, I think that's exactly right. 

Erickson: Clay, one of the other things you talk about in the book is organizational structure. You pointed out a study done by, I believe, Bain Consulting, [which showed that] two-thirds of corporate reorganizations are actually destroying value and not accomplishing what they intended to do. 

Christensen: Uh-huh.

Erickson: Why is it that businesses are so bad at reorging?

Christensen: Boy, that's a great question. To me, the answer is there's a difference between interdependent and modularity in the architecture of systems. Some architectures are interdependent; that is, if you change one thing, it affects the way all of the other elements of the system have to change, as well. You have to do everything in order to do anything. And if you change one thing, you have to change everything, so you have to understand the whole system.

And a modular system is one which each component interfaces with the other components in standardized ways. And when you have a modular system, it's real easy to map that architecture onto an org chart that you can stick up on a wall, because somebody has to be responsible for [every component]: for making it, and designing it, and manufacturing it, and testing it, and replenishing it, and so on. So the organizational chart on the wall reflects the architecture of the product or service. And when you change the org chart, you have to develop a different modular architecture so that people can snap the pieces together in a different way.

In a product that is interdependent, then an org chart almost always creates more problems than they resolve, because if you change one thing, you have to change everything. And I think the key reason for your question (why [organizational] changes eliminate more value than [they create]) is managers don't take the time to understand the nature of the interdependencies of what they're trying to implement. It's a complicated problem.

Erickson: [Is a fair summary of that], then, that markets and products change perhaps faster than organizational structure does? 

Christensen: Yes, that's a great way to frame it. That's a great way to frame it. 

Erickson: Yes, a great point. And the last question before I'd like to get in a game of Buy, Sell and Hold with you, Clay. The last question before that, though, is our audience at The Motley Fool is primarily individual investors. You mentioned a bit about your investing, and I wondered if there were specific things that you look for when you're buying stocks or you invest in companies. Are there certain attributes that you personally are looking for as an investor?

Christensen: The answer is yes. Number one, are they disruptive? 

Erickson: I'm not surprised by that.

Christensen: Because if you invest in disruptive companies, they move up upmarket in very predictable ways. The analysts don't understand that, and so a disruptive company surprises the investors over and over again — that there is more growth in the product or that company than they thought. 

Because what the analysts see at the bottom of the market is products, and they don't realize there's a trajectory of improvement over time that surprises the investors over and over again. And so, that's the way you define what is disruptive. It starts at the bottom [and] they move up. And the analysts are too arrogant. They don't understand the theory, and so they're surprised over and over again.

And in a similar way, if you focus on [investing in] companies that are organized around a job to be done, they can be the incumbents in an industry, but they can't be disrupted because they're providing all the experiences that customers need to have. They can't be disrupted. So those are two rules that we follow when we make investments.

Erickson: OK. Clay, before we let you go, we have to have a little bit of fun. We wanted to spot you up with game of Buy, Sell, or Hold. 

Christensen: All right. 

Erickson: Not talking specific companies, but I'm going to throw you a couple of concepts in a lightning round kind of format where you can conceptually buy, sell, or hold them based on your level of conviction in their future relevance. So larger role in the future is a buy, less of a role is a sell, and more data (you don't really know) is a hold. Are you ready?

Christensen: Tell me those rules again.

Erickson: So a buy would be you are very bullish on this concept in the future.

Christensen: Buy, sell, hold. OK. 

Erickson: Are you ready?

Christensen: Yup.

Erickson: Topic one. Buy, sell, or hold in the big data era companies continuing to use small focus groups for market research?

Christensen: Buy.

Erickson: Why is that?

Christensen: Because I already gave the answer to that question.

Erickson: Yes, fair enough on that. Topic two. Buy, sell, or hold accountable care organizations and just organizations that focus more on outcomes rather than procedural reimbursements as a part of the US healthcare system? 

Christensen: I'd say hold. I'm not sure I could answer your question because the way you frame it, I think is not consistent with itself. Accountable care organizations, overall, I would sell, because I don't think that they are organized in a way that help companies solve problems for customers. Sometimes an accountable care organization collectively develops enough clout that the bigger players can be bigger. But as a solution to healthcare problems, I'd pretty strongly give it a sell. 

Erickson: Next up, Clay. Buy, sell, or hold the future of traditional four-year universities?

Christensen: That's a sell. 

Erickson: Why is that?

Christensen: Including the Harvard Business School.

Erickson: The reason for that? [Are you thinking] more online learning? What's the direction you're taking?

Christensen: That's right. We have not spent any energy at all trying to control the costs of higher education, and tuition has increased at a faster rate than the cost of healthcare, for example. It is so expensive that it is very hard for a graduating student to get a job in an operating company anymore, because operating companies can't pay the salary that our graduating students need in order to pay off their student loans. So our students have to take a job with venture capitalists, or private equity shops, or consulting firms. But operating companies can't hire people with a Harvard MBA. That's why it's a sell.

Erickson: Fair enough. The next one is buy, sell, or hold Uber?

Christensen: Hold. Their model has been a wonderful model, and it's been [disruptive in the past], in that its business model is hard for a taxi to put their arms around. But I think that model is understood, now, and it's not clear how many other industries can be Uberized. There might be some, but there are lot of interdependencies that I [still] need to think it through. 

Erickson: Well, I'll be looking for "uberize" in the dictionary, because I'm pretty sure that's going to be a verb that's going to get a lot of attention.

Christensen: No kidding.

Erickson: Last one for you, Clay. Buy, sell, or hold the future of digital currencies such as bitcoin? 

Christensen: That's a buy in a different way than many people think. The problem is that there is so much money around the world. The graduates of our business schools become leaders of large corporations and they spend their energy on what we call efficiency innovation; that is, squeezing more and more free cash flow out of the companies that they run. And as a result of that, there is just money everywhere, and the cost of money is essentially zero. So the essence of what a bank historically has been becomes obviated, now, when the cost of capital is zero. 

So the very concept of data — the different definition of capital needs to change. And I don't know what that is, but the bitcoin is a concept that I think is something I would invest in. 

Erickson: Clayton Christensen is the Kim B. Clark Professor at Harvard Business School. He's twice been named the world's most influential business thinker in a biennial ranking conducted by Thinkers50 and he is the author of the recent book Competing Against Luck. Clay, this was a lot of fun. Thanks for being with us this afternoon.

Christensen: Oh, it's been fun to do it. Call me again. I like to keep buying and selling things. 

Erickson: That is the most fun we always have, is with the game Buy, Sell, and Hold at the end. 

Christensen: All right. Congratulations.

Erickson: I really enjoyed it. Thanks very much, Clay.

Christensen: You're welcome.