M40: Baselines & leakage traps

CSCI 1109 — Practical Data Science

Frank Rudzicz

Table of contents

• Conceptual scaffold
• Worked example 1
• Worked example 2
• Summary

Learning outcomes

By the end of this module, you should be able to:

1. Define and justify baseline models for classification and regression tasks.
2. Compare a baseline to a more complex ML model using appropriate metrics.
3. Explain what data leakage is, and identify common leakage patterns in code and study design.
4. Refactor simple ML workflows, including with \(k\)-fold cross-validation.
5. Diagnose suspiciously high performance and propose sanity checks and safer alternatives.

Motivation: why baselines & leakage matter

In practical data science, two naïve failure modes (among others):

1. No baseline
   • You build a fancy model…
   • It’s only marginally better (or even worse) than:
     • “Always predict the majority class”
     • “Predict the historical average”
   • But you’ll never know because you didn’t compare it with anything.
   • Result: wasted complexity; misleading sense of progress.
2. Hidden leakage
   • Your model scores 95% accuracy (or something impressive) on test data.
   • Stakeholders are thrilled; paper gets accepted; dashboard is shipped, et c
   • Months later, in production: performance collapses.
   • Root cause: your model was evaluated sloppily, with information in the test set

Warning

Good ML ≠ high accuracy on some split.
Good ML = beats a strong baseline without cheating.

Conceptual scaffold

Baselines

Baseline n.: A simple, transparent prediction rule/model that sets the minimum performance that any serious model must beat. ~~~ Very simple (and kind of bad) baselines:

• Classification
  • Majority class: always predict the most frequent label.
  • Stratified random: sample labels according to observed class proportions.
• Regression
  • Mean baseline: predict the training set mean.
  • Median baseline: predict the training set median.
  • Naïve time-series: “tomorrow = today” (or last observed value). ~~~ Properties we like:
• Easy to explain to non-technical stakeholders.
• Often robust to overfitting.
• Gives us a sanity check:
  • If model ≈ baseline → maybe not worth deploying.
  • If model ≪ baseline → something is wrong.

Data leakage

Data leakage n.: Any situation where your model gets information during training that would not be available at prediction time in the real world. ~~~ Common leakage types:

• Train–test contamination
  • Computing statistics (e.g., mean, scaler) on the full dataset, then splitting.
  • Having the source of multiple data points in both train and test sets.
• Target leakage
  • Features that are direct or near-direct functions of the label.
    • e.g., has_defaulted_last_month when predicting will_default_this_month?.
    • e.g., including a hours_to_readmission feature when predicting readmit?.
• Temporal leakage
  • Using future information for predictions about the past or present.
    • e.g., using “purchases in next 30 days” to predict churn at day 0.

Warning

Leakage makes models look unrealistically good in notebooks, and dangerously bad in the real world.

Clean vs leaky workflow

We’ll compare a clean evaluation workflow to a leaky one.

• Top: all preprocessing is learned only on the training data.
• Bottom: preprocessing is fit on the full dataset first, so the test set “leaks” into training.

Why a single train/test split is fragile

• A single split is just one random way to carve up the data.
• Change the random split, and:
  • Some “easy” (or “hard”) points might move from train → test or vice versa.
  • The reported test performance can jump around.
• With limited data, this variability can be quite large.
• So a single train/test split gives us one noisy estimate of generalization.

💡 Idea: reuse data more systematically

We want an evaluation procedure that:

• Uses every example for training (most of the time).
• Uses every example for testing (at least once).
• Is systematic and repeatable, not ad-hoc.

This motivates \(k\)-fold cross-validation.

\(k\)-fold cross-validation

• Split the dataset into \(k\) folds (partions) of roughly equal size.
• For each fold in turn:
  • Treat that fold as a validation set.
  • Train the model on the other \(k−1\) folds.
  • Evaluate performance on the held-out fold.
• At the end, average the \(k\) performance numbers.

Conceptually:

1. Shuffle data (optional but common).
2. Split into \(k\) folds.
3. Repeat: train on \(k−1\) folds, evaluate on the remaining \(1\) fold.
4. Average the scores.

Fold 1 Fold 2 Fold 3 Fold 4 Fold 5

Each run: one fold is validation (red), others are training (grey):

Run 1: Fold 1 Fold 2 Fold 3 Fold 4 Fold 5 ➡️ Metric 1

Run 2: Fold 1 Fold 2 Fold 3 Fold 4 Fold 5 ➡️ Metric 2

Run 3: Fold 1 Fold 2 Fold 3 Fold 4 Fold 5 ➡️ Metric 3

Run 4: Fold 1 Fold 2 Fold 3 Fold 4 Fold 5 ➡️ Metric 4

Run 5: Fold 1 Fold 2 Fold 3 Fold 4 Fold 5 ➡️ Metric 5

Worked example 1

Classification baselines in practice

We’ll simulate a simple loan default prediction task:

• Feature: a numeric risk score summarizing a customer’s past payment history.
• Label: whether the customer defaults next month (defaulted vs not_defaulted).
• We’ll compare:
  1. Majority-class baseline, and
  2. A simple k-NN classifier.

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn.dummy import DummyClassifier
from sklearn.metrics import accuracy_score, ConfusionMatrixDisplay
import matplotlib.pyplot as plt

rng = np.random.default_rng(1109)

n = 600
# Synthetic "risk score" feature on a 0–100 scale
risk_score = rng.normal(loc=50, scale=15, size=n).clip(0, 100)

# True probability of default depends on risk score
# (higher risk_score → more likely to default)
p_default = 0.1 + 0.007 * risk_score   # roughly 10%–80%
p_default = np.clip(p_default, 0.05, 0.95)

defaulted = rng.binomial(1, p_default)

df = pd.DataFrame({
    "risk_score": risk_score,
    "defaulted": defaulted
})

X = df[["risk_score"]]
y = df["defaulted"]

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=1109, stratify=y
)

# 1. Baseline: always predict majority class
dummy = DummyClassifier(strategy="most_frequent")
dummy.fit(X_train, y_train)
y_dummy = dummy.predict(X_test)
acc_dummy = accuracy_score(y_test, y_dummy)

# 2. k-NN model
knn = KNeighborsClassifier(n_neighbors=15)
knn.fit(X_train, y_train)
y_knn = knn.predict(X_test)
acc_knn = accuracy_score(y_test, y_knn)

acc_dummy, acc_knn

(0.5388888888888889, 0.5388888888888889)

Baseline (majority class) accuracy: 0.539
k-NN model accuracy:               0.539

Reflection: reading the example

Questions to ask yourself:

• ✏️ Does k-NN beat the baseline by a meaningful margin?
  • If not, is the feature weak? Is the model is mis-specified?
• ✏️ What kind of mistakes does each model make?
  • Baseline always:
    • Gets the majority class right,
    • Misses the minority-class cases.
  • k-NN:
    • Helps recall minority cases, but
    • Misclassifies more majority cases.

Decision context matters

• If missing people who are about to default is costly, accuracy alone may be misleading.

• We might care more about recall for the “will default” group.

• Baselines give you a floor; decision context tells you what is expected.

Worked example 2

A simple leakage trap

We’ll deliberately construct a leaky feature:

• Start from the same loan-default dataset.
• Add a suspicious feature:
  • label_copy = exact copy of the true label.
• Watch how performance changes — and why this is cheating.

from sklearn.linear_model import LogisticRegression

# Make a copy with a leaky feature
X_leaky = X.copy()
X_leaky["label_copy"] = y   # <- pure target leakage

X_tr_leaky, X_te_leaky, y_tr_leaky, y_te_leaky = train_test_split(
    X_leaky, y, test_size=0.3, random_state=1109, stratify=y
)

log_reg_leaky = LogisticRegression(max_iter=1000)
log_reg_leaky.fit(X_tr_leaky, y_tr_leaky)
y_pred_leaky = log_reg_leaky.predict(X_te_leaky)

acc_leaky = accuracy_score(y_te_leaky, y_pred_leaky)
acc_leaky

1.0

Leaky logistic regression accuracy: 1.000

🤷 What happened?

• The model has a feature that is essentially the answer key.
• Perfect performance on the held-out test set.
• But in production, we would never know ahead of time whether the customer defaulted next month.

Note

This is an exaggerated example, but many real leaks are just subtle versions of this pattern.

Leakage in preprocessing: scalers & encoders

Even without an obvious “label copy”, we can leak in more subtle ways.

Bad pattern (leaky):

from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split

scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)   # fit on ALL data ❌

X_train, X_test, y_train, y_test = train_test_split(
    X_scaled, y, test_size=0.3, random_state=0
)

• The scaler has “seen” the entire dataset, including test rows.
• Model performance on test data is slightly optimistic.

Good pattern (clean):

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=0
)

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)   # fit on train ✅
X_test_scaled  = scaler.transform(X_test)        # transform test

Even better: use a Pipeline.

Pipelines as leakage protection

Using 🔗 sklearn.pipeline.Pipeline:

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=1109, stratify=y
)

pipe = Pipeline([
    ("scale", StandardScaler()),
    ("clf",   LogisticRegression(max_iter=1000))
])

pipe.fit(X_train, y_train)          # scaler fits only on train
y_pred = pipe.predict(X_test)       # clean transformation on test

Benefits:

• All preprocessing steps are tied to the model.
• When combined with cross-validation, each fold:
  • Fits transformers on its training portion only.
• Harder to accidentally reuse information from test folds.

Warning

Pipelines don’t guarantee no leakage (there is no ‘silver bullet’ in ML), but they remove some of the most common sources.

Summary

Evaluation & quality checks: making metrics honest

When reading model performance for a decision:

1. Check the baseline
   • Is your metric at least better than:
     • Majority-class (classification),
     • Mean/median baseline (regression),
     • Naïve “yesterday = today” baseline (time-series)?
2. Check the split
   • Was the split appropriate?
     • Random vs time-based (for temporal data),
     • Stratified for imbalanced classes.
   • Were preprocessing steps fit inside the train portion?
3. Check the metrics
   • Do they match the decision stakes?
     • Accuracy vs recall vs precision vs AUC.
   • For imbalanced problems, be suspicious of high accuracy.
4. Check for red flags
   • Performance is “too perfect” (e.g., 99–100%).
   • Training and test scores are both extremely high with little gap.
   • Features that obviously contain future info or direct label proxies.

Tip

Treat every surprisingly good result as an invitation to hunt for leakage before you celebrate.

Ethics & risks of ignoring baselines & leakage

Ethical (and practical) risks:

• Over-promising impact
  • A leaky model can persuade stakeholders to roll out policies that don’t actually work — e.g., interventions targeting the wrong people.
• Unfair allocation of resources
  • If leakage depends on certain subgroups (e.g., certain clinics or schools), model errors may systematically harm already disadvantaged groups.
• Erosion of trust
  • When performance collapses in production, people stop trusting all data-driven tools, including well-designed ones.

🛠️ Simple mitigations (at this course level):

• Always report baseline performance side-by-side with model performance.
  • Usually, using multiple metrics, multiple datasets, multiple model ‘versions’
• Document potential leakage routes:
  • Which features might be proxies for the label?
  • Are there any time-related or process-related leaks?
• May prefer simpler, transparent models that beat baselines fairly over complex black boxes with suspiciously high scores.

Curiosity: baselines & leakage in the wild

These ideas show up everywhere in real DS/ML practice:

• 🔗Kaggle competitions
  • Top teams often start with very strong baselines and carefully avoid leaks, especially in time-series and click-through datasets.
• Academic & industry papers
  • Papers are sometimes retracted or challenged because leakage was discovered years later (e.g., medical imaging models “learning” hospital device artifacts) (Kapoor and Narayanan 2023).
• Product ML teams
  • “Does this beat our current rule-based system?” is always asked.
  • A baseline might be:
    • An existing human process,
    • A simple decision tree,
    • Or even “random plus thresholding”.

Tip

As you build more advanced models later in the course, remember: robust baselines + leak-free pipelines are part of being a responsible data scientist.

Quick-checks

1. Which of the following is a reasonable classification baseline?

   A. A model that memorizes the training set.
   B. A model that predicts the majority class for all cases.
   C. A model that uses future labels as features.
   D. A model that randomizes predictions with 50/50 chance.

---

2. What is the most common cause of train/test contamination?

   A. Using a random seed in train_test_split.
   B. Fitting scalers or encoders on the full dataset before splitting.
   C. Training on 80% instead of 70% of the data.
   D. Using a GPU instead of a CPU.

---

3. You’re predicting churn next month. Which feature is most suspicious for leakage?

   A. Number of support tickets last month.
   B. Account age in days.
   C. Whether the account was closed next month.
   D. Total amount billed so far.

---

4. Your model’s test accuracy is 0.97, while the majority-class baseline is 0.94. Which is the BEST first step?

   A. Declare victory and deploy immediately.
   B. Check for leakage and confirm that preprocessing happens inside the split.
   C. Ignore the baseline, since accuracy is very high.
   D. Add more hyperparameters to increase accuracy even further.

References

Kapoor, Sayash, and Arvind Narayanan. 2023. “Leakage and the Reproducibility Crisis in Machine-Learning-Based Science.” Patterns 4 (9): 100804. https://doi.org/10.1016/j.patter.2023.100804.