Overfitting: why fitting old data is not enough

A model can earn excellent scores on the examples it learned and still fail on new cases. The useful question is whether its performance carries over to unseen data. Overfitting explains one way that transfer can break down.

🧠 1. What overfitting means

A model may follow the quirks or noise of its training set too closely, fitting old examples while generalizing poorly. The scikit-learn polynomial example shows how increasing complexity can make a curve fit noise. Underfitting and overfitting example.

This does not mean that learning more is inherently harmful. The question is whether the model captures a useful pattern or adapts too specifically to its development examples.

🧮 2. Equal training fit, different new predictions

In this original toy example, the true rule is y = x. We observe four points at x = -2, -1, 1, 2. Both functions below match those four points perfectly:

def simple(x):
    return x

def wiggly(x):
    return x + (x + 2) * (x + 1) * (x - 1) * (x - 2)

def mse(fn, xs):
    return sum((fn(x) - x) ** 2 for x in xs) / len(xs)

train = [-2, -1, 1, 2]
test = [0, 0.5, 1.5]

for fn in (simple, wiggly):
    print(fn.__name__, round(mse(fn, train), 4),
          round(mse(fn, test), 4))

Output:

simple 0.0 0.0
wiggly 0.0 9.5651

The numeric columns show mean squared error on the old points and the test points. At x = 0, the simple function returns 0 while the wiggly function returns 4.

This is a mathematical illustration of multiple functions fitting the same observations, not an experiment that trains a real AI model. Zero training error alone cannot tell us which predictor to choose.

🗂️ 3. Separate data by its decision role

One approach uses train for fitting parameters, validation for choosing settings and test for final evaluation. Repeatedly adjusting a model against test scores turns that set into selection data. Cross-validation helps compare configurations, but an independent final evaluation remains necessary. Cross-validation documentation.

For your project, write down which decisions each set is allowed to inform. That simple record can prevent accidental use of the final answers during every iteration.

🔍 4. Not every failure is overfitting

Data leakage is a different problem: information that should be unavailable enters training or selection. Choosing features using the entire dataset before splitting can make evaluation overly optimistic. scikit-learn recommends splitting first, learning transformations only on training data and using suitable pipelines. Common pitfalls.

Changing production data can also reduce quality: input formats or distributions may differ from training. Monitor live inputs as well as outcomes. Google's monitoring guide.

A score gap is therefore a reason to investigate, not a complete diagnosis.

💬 5. Ask AI to review the evaluation design

I am evaluating a prediction model.
Here are the data sources, split method, preprocessing,
configuration selection and scores: [insert details].
Identify questions to check about:
1. Duplicate examples across splits.
2. Steps that have seen test data.
3. Information available only after prediction time.
4. Differences between old and production inputs.
Separate existing evidence from hypotheses.
Do not diagnose overfitting from one score alone.

✅ 6. Keep an auditable experiment record

Save data versions, splits, settings and results. Compare changes with a simple baseline and record what changed.

When a result looks unexpectedly good, inspect the data and scoring process before celebrating. When live performance falls, examine the new failing cases. The goal is useful prediction on relevant unseen data, not a perfect score on familiar examples.