1. The forbidden clue
Suppose we predict whether a patient will be admitted using a feature called discharge_summary_length. That feature is created after admission. The model may score beautifully—but the feature will not exist when the prediction is needed.
That is target/future leakage.
2. Leakage through preprocessing
A subtler mistake:
X = scaler.fit_transform(all_data) # leak
X_train, X_test = split(X)The scaler learned test-set statistics. Correct order:
split first
fit preprocessing on train only
apply learned transform to validation/testThe same rule applies to feature selection, imputation, PCA, vocabulary building and many other learned transforms.
3. Entity and time leakage
Randomly splitting medical visits can put the same patient in train and test. Randomly splitting future forecasting data can let later records influence earlier predictions.
The split must imitate deployment.
4. Leakage is dangerous because metrics still look valid
Nothing necessarily crashes. The code runs, accuracy rises, and the experiment may even be reproducible. The question was wrong.
Always ask:
At the exact moment this prediction will be made in production, would every input and every fitted preprocessing statistic already be available?
What to Remember
- Leakage transfers forbidden information into training or model selection.
- Split before fitting learned preprocessing.
- Respect time, users, groups and deployment boundaries.
- Suspiciously excellent results deserve investigation.
Next → Class Imbalance: even without leakage, a metric can mislead when one class dominates the dataset.