Most models fail long before the first training run. Not because the architecture was wrong, but because two years of labels meant slightly different things, or the target column had been quietly copied from the outcome, or the only person who understood the data had left. Training is the cheap part. This checklist covers what is worth doing before you call fit.

1. Start With the Decision, Not the Dataset

Write one sentence: "When this model returns a risk above the threshold, [person or system] will [do something specific] within [timeframe]." If you cannot fill in the blanks, you are doing research, which is fine — just label it that way and skip the production checklist below.

A churn model that feeds a weekly retention call list is a project. A churn model that feeds a dashboard nobody opens is a hobby. The decision determines the threshold, the metric, and how much error is tolerable, so pin it down first.

2. Fix Your Success Metric Before You See Any Results

Choose the metric and the target number in advance. Otherwise you will quietly redefine success around whatever the first run produces, and every result afterward will look acceptable.

Then write down both error costs. A fraud model that blocks one legitimate card in twenty is unusable even at high accuracy, because the false positives land on real customers. A screening model that misses cases is unusable for the opposite reason. Naming that asymmetry early tells you where to set the threshold and which metric to argue about later.

3. Interrogate the Data Before You Model It

Missing values are information

A blank income field on a loan application is rarely random. Check whether missingness correlates with the target, then decide: drop the row, impute, or encode "missing" as its own category. Whatever you choose, do it inside the training pipeline. Imputing across the full dataset before splitting leaks information from the test set into training.

Hunt for leakage

Leakage is the most common cause of a first result that looks too good.

• A feature populated only after the outcome occurs, such as a closure date or a final invoice amount.
• Identifiers that sort with the target, like an auto-incrementing customer number assigned at signup.
• Target encoding, scaling, or feature selection computed across the whole dataset before the split.
• Duplicate rows appearing in both training and test sets.

Sort features by correlation with the target and read the top ten by hand. If one looks implausibly predictive, it is probably describing the past rather than predicting the future.

Check the boring things too

• Row counts per month — did logging break in March?
• Class balance, and how it has shifted over time.
• Ranges and units. An age of 999 is not an age.
• Whether every feature can actually be computed at prediction time, within your latency budget.

4. Split Before You Explore

Split first, explore second. Once you have looked at the test set, it stops being a test set.

For anything with a time dimension, split by time — train on the past, evaluate on the future — because a random split lets the model learn from tomorrow and predict yesterday. When rows share an entity, split by that entity. Multiple visits from the same patient, orders from the same customer, readings from the same sensor: none of those should straddle the boundary. Keep one holdout set that you touch once, at the end.

5. Build the Dumbest Possible Baseline

Before any gradient boosting, train four things: always predict the majority class; predict from a single strong feature; fit a plain logistic or linear regression; and, if a human process already exists, record its performance too.

The baseline gives you the number that matters. Beating logistic regression by half a point is not automatically worth the deployment cost, the monitoring, and the person who has to maintain the pipeline. Sometimes the honest answer is that the simple model is good enough, and that is a useful result to deliver early rather than late.

6. Confirm the Problem Framing Matches the Metric

Classification, regression, ranking, and time-to-event problems need different metrics, and the wrong pairing hides real failures. Accuracy on a 2% positive class tells you almost nothing; precision at the top k is what the reviewer actually experiences. If the output is a probability that feeds a downstream decision, check calibration, not just ranking. If the model ranks, examine the top of the list rather than the average across all rows.

One more pass: ask what the metric rewards. A model optimized for clicks learns to produce whatever gets clicked, which is not always the same as what helps the user. Framing errors are expensive to repair later, because the data, the labels, and the evaluation all have to change with them.

7. Make the Run Reproducible and Cheap to Kill

A result you cannot reproduce tomorrow is not a result. Seed your random operations, pin library versions, and log which version of the data you trained on — a hash or a snapshot identifier is enough. Run the whole pipeline on a 1% sample first; it catches shape mismatches, broken joins, and missing columns in minutes instead of hours. Set a time budget and a stopping rule before you start, so a run that is going nowhere can be cancelled without debate.

The 30-Minute Version

If you have a new dataset and an afternoon, do this in order:

1. Write the decision sentence and the metric, with a target number.
2. Check row counts, class balance, ranges, and duplicates.
3. Split by time or by entity, and set the holdout aside.
4. Read the ten features most correlated with the target.
5. Train a majority-class baseline and a logistic regression.
6. Smoke-test the full pipeline on a small sample.
7. Record the data version, the seed, and the baseline score in one place.

None of it is glamorous, and none of it takes long. But every item on the list is cheaper than retraining, re-labeling, or explaining to a stakeholder why the impressive number from last week disappeared. And if your data includes personal or regulated information, check the handling requirements with whoever owns compliance before you copy anything around.