More Fool's Gold
Watch out for falling data miners

by Ann Coleman (TMF AnnC)

Alexandria, VA (May 14, 1999) -- We've been examining the dangers of "data mining" this week in response to an article critical of the Foolish Four in the March/April issue of the Financial Analysts Journal by Grant McQueen and Steven Thorley.

To briefly summarize, McQueen and Thorley are rightfully concerned about the dangers of searching for factors associated with high stock returns and using those factors to form an investment strategy that can be relied on to provide similarly high returns in the future.

Over the past several years, I have received numerous e-mails from people who believe that they have found a "stock indicator" that allows them to predict the direction of the market on a short-term basis. My response to them has always been pretty much a rundown of the points this article makes. Investors do indeed need to be wary of anything that promises returns well above the market. But I have always kept an open mind, because it's obvious that such correlations CAN exist. (If you have one, don't try to sell it! You may kill your golden goose. More about that next week.)

Obviously, I don't think that the Foolish Four is fatally flawed because some of the techniques that were used to develop it are similar in nature to "data mining." They are also time-honored investigational techniques used, with great care, I hope, by social scientists, biological scientists, and medical researchers. One difference between data mining, used in its pejorative sense, and searching for correlations that might prove predictive is "reasonableness."

While McQueen and Thorley attempt to mock our strategy by devising the "Fractured Four," a strategy that invests in the Foolish Four only during even years and the PPP during odd years, that kind of association is patently absurd. If we were truly trying to construct a portfolio that simply had the highest returns, instead of looking for reasonable associations, we would have advised investors to drop the fourth stock on the list as well.

As Chris Rugaber's article yesterday pointed out, for some as-yet-inexplicable reason, the fourth stock only has an average return.

But while dropping the lowest-priced stock is at least somewhat reasonable (because companies that are really in financial trouble can be very beaten down in price), there is no reason for the 4th stock to perform differently than the 2nd, 3rd, or 5th. It is obviously an artifact of the data. If I had wanted to construct a portfolio with the highest backtested returns, I could have left it out. Of course, I would have been laughed off the Internet. (Not all individual investors are as much in need of protection as McQueen and Thorley fear. Check out the debate on this topic on our web message board for ample proof of that.)

To finish up with the data mining charge, let's look at one other point made in the article. In the section labeled "Out of Sample Testing," McQueen and Thorley say:

"When the number of variables and the lack of a coherent theory cast suspicion on a trading rule, the clearest antidote to data mining is to test the rule on-out-of sample data. If a trading rule is claimed to be true and universally applicable, it will work on more sets of data than the set used to create the rule."

(Note: We don't claim that the Foolish Four is universally applicable -- it specifically needs to be restricted to a selection of stocks that are similar to the Dow stocks.)

McQueen and Thorley tested the Foolish Four two ways: January portfolios from 1949 through 1972, and portfolios renewing in July from 1973 through 1996. The early period found that the Foolish Four only beat the Dow by a fraction of a percentage point -- nothing to write home about. We have known since the first Dow spreadsheet was finished that the strategy didn't work well in the '60s, so this was not surprising.

My feeling about the poor performance in the sixties has always been that I would prefer to use a strategy that has worked only for the last 30 years than one that only worked 40 years ago. This is a valid concern, though.

The other out-of-sample test was done over the same period our original spreadsheet covered but using portfolios that started in July as opposed to January. McQueen and Thorley found that this data set only beat the Dow by 3 percentage points.

Now here is where things get sticky. I spoke to Steven Thorley before they submitted their article for publication. They kindly sent us a copy of their study before it was published, two days before it was due at the publisher. As we went through the article point by point, I explained that the lower return in July was something that we had noted when we extended the backtesting to our monthly database (not yet available for sale in FoolMart). What we found, I explained, was a fairly smooth curve peaking in December/January and bottoming out in July. Purely by chance (they picked July because it was "opposite" January), they selected the worst month for their out of sample test.

Thorley volunteered that such a pattern was not what he would have expected if the results of the January test was simply due to chance. If chance were at work, you would expect to see a random distribution, not a nice pattern. He said that such a pattern would lend legitimacy to our claim that something real was going on here.

But that didn't make it into the article. I guess there wasn't enough time for them to verify what I had told them, but I found it disturbing that they completely ignored that additional information and maintained that their test indicated the strategy was not valid.

The original draft of the article also suggested that if the strategy were valid, it would also work in other markets. I told Mr. Thorley that I had heard that some Fools had found evidence that it worked on the Toronto Stock Exchange 35 (the Canadian equivalent of the Dow), and that we had also tested it on the Standard & Poor's 500 Index and found the that the strategy produced higher-than-market returns -- but that our data only went back to 1987.

The suggestion that if the strategy "had legs" it would also work in other markets was subsequently removed from the paper but, again, I guess that they did not have time to investigate that aspect.

Somehow I thought that, in the academic world, truth would be more important than a deadline. Silly me.

Fool on and prosper!

Fools Wanted: Apply Within.

 Recent Foolish Four Portfolio Headlines
  12/28/00  Modifying Mechanical Strategies
  12/27/00  Beating the S&P Year 2000 Recap
  12/26/00  After-Hours Quotes
  12/22/00  Why Include the Foolish 4 Port?
  12/21/00  The Value of Community Input
Foolish Four Portfolio Archives »  

Today's Stock Lists | 1999 Dow Returns

05/14/99 Close
Stock  Change   Last
CAT  +   3/8   61.00
JPM  -5 13/16  140.94
MMM  -   5/8   90.31
IP   -   1/4   53.13
                Day   Month    Year   History
        FOOL-4   -1.09%  -0.21%  28.67%  30.58%
        DJIA     -1.75%   1.15%  19.25%  18.78%
        S&P 500  -2.18%   0.20%   9.15%   9.41%
        NASDAQ   -2.10%  -0.59%  15.29%  16.87%

    Rec'd   #  Security     In At       Now    Change
 12/24/98   24 Caterpillar   43.08     61.00    41.60%
 12/24/98    9 JP Morgan    105.51    140.94    33.58%
 12/24/98   14 3M            73.57     90.31    22.76%
 12/24/98   22 Int'l Paper   43.55     53.13    21.99%

    Rec'd   #  Security     In At     Value    Change
 12/24/98   24 Caterpillar 1034.00   1464.00   $430.00
 12/24/98    9 JP Morgan    949.62   1268.44   $318.82
 12/24/98   14 3M          1030.00   1264.38   $234.38
 12/24/98   22 Int'l Paper  958.12   1168.75   $210.63

              Dividends Received      $29.45
                             Cash     $28.26
                            TOTAL   $5223.27