What Does Overfitting Mean in Machine Learning?

What is Overfitting?

Think of overfitting as the overzealous student in a classroom who, in a bid to outshine peers, learns every peculiarity of the teacher's test patterns but fails to grasp the general principles of the subject. Similarly, overfitting in machine learning is when an algorithm tries too hard. It performs impressively on the training data, fitting it perfectly like a glove. But when faced with new data (the true test!), it stumbles.

Why? Well, because it also learned the noise, outliers, and random fluctuations of the training data. All of this can help the system reconstruct its training data set to perfection, but it can be a problem when applied to new situations and fresh data.

And this potential problem makes its presence known in all facets of machine learning. Developers must account for the overfitting risk, whether they are building systems for supervised learning, unsupervised learning, semi-supervised learning, or designing neural networks for deep learning. All of these approaches can deliver poor results due to a weak or unbalanced bucket of training data.

Why should you care about overfitting?

Let's think about overfitting in the context of investing. You wouldn't put all your money on a single stock just because it's had a winning streak recently, right? Of course not. That's because you understand the principle of diversification, and you know that a strong set of earlier results don't necessarily lead to similar gains in the future.

Overfitting is the algorithmic equivalent of betting it all on one stock, based on financial data and stock charts from the past, with no regard for recessions and other changing market environments, new innovations, more or fewer competitors, etc.

In the machine learning case, the algorithm bets it all on the specific training data. The system ends up building a prediction model that fits that data but does poorly when facing information from outside that specific data set.

This is important because an overfit model is a deceptive model. It may look like it's doing wonders as long as your tests stick to the training data. But in reality, its predictive power is about as reliable as a magic eight-ball. Overfitting undermines a machine-learning model's ability to generalize the lessons learned so they can apply to new input from previously unseen data sets. In the world of machine learning, generalization is the name of the game.

How to skirt the overfitting quagmire

Now you know why an overfit machine learning system is about as helpful as a solar-powered flashlight without batteries -- it only works when you need it the least. So what can you do about it? As it turns out, the inventors of modern machine learning systems have come up with some nifty techniques to avoid this problem.

• Cross-validation: This technique involves dividing the data set into a number of subsets. The model is trained on some of these subsets and then validated on the remaining ones. This process is repeated several times with different combinations, providing a robust estimate of the model's performance on unseen data.
  
• Early stopping: This involves halting the training process before the model starts to overfit. Essentially, you monitor the model's performance on a validation set during training and stop training when performance begins to degrade. The most advanced machine learning model isn't always the best one.
  
• Dropout: This is a technique used specifically in neural networks. It involves randomly "dropping out" or turning off certain neurons during training. The system must route the data flow through a different digital neuron path, which helps prevent an undesired focus on seemingly helpful but potentially misleading data relationships.
  
• Data augmentation: This involves creating new synthetic training examples by applying transformations to the existing data. For instance, you might flip or rotate images in a computer vision task. This increases the size and diversity of the training data and helps the model generalize better. If a landscape-tagging algorithm can tell a mountain range apart from an upside-down ocean wave, you might be on the right track.
  

Remember, each of these techniques has its place and might be more or less effective depending on the specific nature of the data set and the problem at hand. A successful machine-learning system will probably use several anti-overfitting techniques because there is no silver bullet that stops every potential problem.

Like most things in machine learning (or investing, or life), it's an art as well as a science.