A model that scores 0.93 in a notebook and 0.68 in production is rarely a modelling problem. Most of the time, something went wrong before the first call to fit(): in the cleaning steps that feel like admin work and get rushed.

Cleaning is where the quiet damage happens. Nothing errors out, the notebook runs top to bottom, and the validation numbers look excellent. Here are five traps worth checking on your current dataset, and what to do instead.

1. Filling Missing Values Without Asking What the Gap Means

The default move is to fill numeric columns with the mean or median and categorical ones with the most frequent value. It works often enough to become a habit, which is exactly the problem. A missing value is frequently information: a blank income field on a loan application, an empty "last contact date" for a customer who was never contacted, a missing serial number on a returned unit.

Filling a tenure column with 0 is worse than filling it with the median. Zero months of tenure is not a neutral guess; it is a specific and usually false claim about that customer. The model will learn from the fiction you supplied.

Before imputing, study the missingness itself. Count it per column and per row, then check whether the target rate differs between rows that have the value and rows that do not. A rough triage:

• Under 5% missing: impute with a sensible value, and add a binary indicator column anyway. It costs almost nothing.
• 5% to 40% missing: keep the column, keep the indicator, and choose the imputation by column type rather than by habit.
• Very high missingness: check whether missingness alone predicts the target before dropping anything. Sometimes the empty field is the most useful feature you have.

Time series need separate care. Forward-filling is reasonable when a sensor reports on a schedule and the value genuinely persists between readings. It is wrong when the gap means the sensor was offline, because you have just invented measurements.

2. Cleaning the Whole Dataset Before Splitting It

This is the most common source of leakage. You load the full table, impute with the global mean, scale with the global standard deviation, pick your clipping thresholds, collect the list of categories from every row, and only then call train_test_split. Every one of those steps has already seen the test set.

The effect is small on large datasets and brutal on small ones, and it is always in the optimistic direction. The model looks better than it is because the held-out rows quietly shaped the training features.

The fix is mechanical, which is good news:

1. Split first, or wrap the entire process in cross-validation.
2. Fit every transformation on the training fold only: imputer, scaler, encoder, dimensionality reduction, outlier bounds.
3. Apply the fitted transformation to validation and test data without refitting it.

If you work in Python, put each step in a pipeline and fit the pipeline rather than the individual pieces. It removes the temptation to compute "just the means" on the full dataframe. The same rule applies to target encoding and to any resampling for class imbalance: those happen inside the fold, never before the split.

3. Letting the Label or the Future Into the Features

Two versions of one mistake, and both are easy to miss because the code looks reasonable.

Target encoding computed over all rows

Replacing a high-cardinality column such as city or product code with the average target rate for that category is useful. Computing it once across the entire dataset is not: each row's own label contributes to its own feature, so the model can partly memorise the answer. Compute the encoding inside each training fold, apply it to validation, and smooth it so categories with only a handful of rows do not receive extreme values.

Aggregates that reach past the prediction moment

Lifetime spend, total support tickets, number of logins: all useful features, all leaky if the window extends beyond the timestamp of the prediction. If you are predicting whether a customer churned in March, every aggregate must be built from data available at the end of February. Rolling windows and cumulative sums deserve a second look for the same reason.

For every feature, ask one question: would this exact value have been available at the moment the prediction had to be made?

4. Splitting Rows When You Should Split Groups

A random row-level split assumes rows are independent. Often they are not. The same customer appears in dozens of rows, the same device generates hundreds of sessions, the same document was uploaded twice with slightly different formatting. A random split scatters near-identical records across training and validation, and the model looks smart because it has effectively seen the answers.

Before you split:

• Drop exact duplicates, then hunt for near-duplicates using a normalised key such as a lowercased email or a trimmed address.
• Check whether any duplicated rows carry different labels. That is a data collection problem, and it needs a documented rule rather than a silent fix inside the model.
• Split by entity, not by row, and group on customer, device or session so no group spans both sides. Grouped cross-validation does this for you.

5. Treating Every Outlier and Typo as Noise

Trimming extremes feels responsible. Sometimes it is: a person aged 412, a transaction of negative ten million. But extremes often carry the signal you care about. Fraud is an outlier. A bulk order is an outlier. A pressure spike in a sensor log is the event you wanted to detect. Clipping those values teaches the model that the most interesting cases look ordinary.

The more damaging cleaning failures are usually not outliers at all. They are inconsistencies that survive into training:

• The same category written four ways: "NY", "N.Y.", "ny", "New York ".
• Dates parsed with two different formats, quietly turning half the column into missing values.
• One column holding both kilograms and pounds because two sources were merged.
• Leading and trailing whitespace creating phantom categories and inflating cardinality.

Profile before you fix. Print the minimum, maximum, unique count and most frequent values for every column, then normalise casing and whitespace, parse dates with an explicit format, standardise units, and validate ranges against what you know about the domain. Handle outliers per model: a linear model with a long tail may benefit from clipping or a log transform, while tree-based models usually tolerate raw extremes well enough.

A Short Checklist Before You Train

1. Split first, then clean inside the fold.
2. Add missing indicators and choose imputation per column, not by habit.
3. Confirm every feature would exist at prediction time.
4. Deduplicate, and split by group rather than by row.
5. Normalise units, casing and date formats, and write down what you changed.
6. Keep one holdout you never open until the very end.

None of this is glamorous, and none of it takes long once it becomes routine. Keep a cleaning log: one line per decision, written as you make it. When validation and production disagree, that log is the first place to look, and it is usually where the answer is hiding.

Photo: freestocks-photos / Pixabay