M13: Train/validation/test splits & cross-validation

CSCI 3151 — Foundations of Machine Learning

Frank Rudzicz

Table of contents

• Conceptual scaffold
• Math lens
• Worked example 1 — basic splits
• Data leakage
• Worked example 2 — cross-validation
• Variants of splitting & CV

Why this module?

• Metrics from M12 (accuracy, precision/recall, ROC–AUC, etc.) are only useful if they are computed on data the model has never seen.
• Many real-world failures are not about bad models, but about bad evaluation:
• This module is about building a trustworthy evaluation pipeline:
  • Train / validation / test splits.
  • \(k\)-fold cross-validation.
  • Matching splits to a decision context.

Learning outcomes

By the end of this module, you should be able to:

• Explain and justify the roles of training, validation, and test sets.
• Design appropriate split strategies for a given decision problem.
• Implement train/validation/test splits and k-fold cross-validation in Python without leakage.
• Interpret cross-validation results (mean + variability) and use them for model selection.
• Diagnose leakage and overfitting in evaluation pipelines and propose fixes.

Note

Our goal: to ensure, to the extent possible, that what we say about our models is true.

Conceptual scaffold

Train, validation, and test: roles

• Training set
  • Used to fit model parameters (e.g., weights of logistic regression).
• Validation set
  • Used for model selection and hyperparameter tuning.
  • You can iterate here: features, regularization strengths, thresholds, architectures.
• Test set
  • Used once at the end to estimate generalization performance.
  • Must not influence any modelling choices if you want an honest estimate.

Doing the splits

• Split into three partitions. By convention”
  1. ~60–70%: Train.
  2. ~15–20%: Validation.
  3. ~15–20%: Test.
• Feedback loop:
  • Train → Validate → tweak → Train …
  • Only when satisfied: Train+Val → Test.

%%{init: {
  "theme": "neutral",
  "fontsize": "12px",
  "flowchart": { "curve": "basis" }
}}%%
flowchart TD

classDef bigNode   fill:#ffffff,stroke:#333,stroke-width:1.5px,font-size:18px,padding:18px;
classDef smallNode fill:#ffffff,stroke:#333,stroke-width:1.5px,font-size:12px,padding:8px;
classDef noteNode  fill:#ffffff,stroke-dasharray:3 3,stroke:#555,font-size:12px;

subgraph Splits[ ]
  direction TB
  T["TRAIN (~60–70%<br>of data)"]:::bigNode
  V["VALIDATION<br>(~15–20%)"]:::smallNode
  Te["TEST<br>(~15–20%)"]:::smallNode
end

T  -->|train models| V
V  -->|tune &<br>try again| T
V  -->|final chosen<br> model| Te

Note["[Note] Never tune on TEST"]:::noteNode
Te -.-> Note

❌ Anti-example — abusing the test set

A team trains a fraud detector and reports 99.7% accuracy on the test set 🏆.

😮 Later you discover:

1. They repeatedly tried different model architectures and hyperparameters.
2. After each try, they evaluated on the same test set.
3. They picked the configuration that looked best on that test set and reported that number.

😟 Problems:

• The test set is now effectively part of the model selection loop.
• Reported performance is optimistically biased and will not replicate on fresh data.
• In production, the fraud team may over‑trust the model and miss important frauds.

Why a single train/test split is fragile

• A single split is just one random way to carve up the data.
• Change the random split, and:
  • Some “easy” (or “hard”) points might move from train → test or vice versa.
  • The reported test performance can jump around.
• With limited data, this variability can be quite large.
• So a single train/test split gives us one noisy estimate of generalization.

💡 Idea: reuse data more systematically

We want an evaluation procedure that:

• Uses every example for training (most of the time).
• Uses every example for testing (at least once).
• Is systematic and repeatable, not ad-hoc.

This motivates \(k\)-fold cross-validation.

\(k\)-fold cross-validation

• Split the dataset into \(k\) folds (partions) of roughly equal size.
• For each fold in turn:
  • Treat that fold as a validation set.
  • Train the model on the other \(k−1\) folds.
  • Evaluate performance on the held-out fold.
• At the end, average the \(k\) performance numbers.

Conceptually:

1. Shuffle data (optional but common).
2. Split into \(k\) folds.
3. Repeat: train on \(k−1\) folds, evaluate on the remaining \(1\) fold.
4. Average the scores.

Fold 1 Fold 2 Fold 3 Fold 4 Fold 5

Each run: one fold is validation (red), others are training (grey):

Run 1: Fold 1 Fold 2 Fold 3 Fold 4 Fold 5 ➡️ Metric 1

Run 2: Fold 1 Fold 2 Fold 3 Fold 4 Fold 5 ➡️ Metric 2

Run 3: Fold 1 Fold 2 Fold 3 Fold 4 Fold 5 ➡️ Metric 3

Run 4: Fold 1 Fold 2 Fold 3 Fold 4 Fold 5 ➡️ Metric 4

Run 5: Fold 1 Fold 2 Fold 3 Fold 4 Fold 5 ➡️ Metric 5

\(k\)-fold cross-validation — step-by-step

Given dataset indices \(\{1,\dots,n\}\):

1. (Optional) Shuffle the data.
2. Partition indices into \(k\) folds \(V_1,\dots,V_k\) of similar size.
3. For each fold \(j = 1,\dots,k\):
   • Let \(V_j\) be the validation indices.
   • Let \(S_{-j}\) be all indices not in \(V_j\) (the training set for this run).
   • Train model parameters \(\hat{\theta}^{(-j)}\) on \(S_{-j}\).
   • Evaluate a metric (e.g., accuracy, AUC, loss) on \(V_j\).
4. Collect the \(k\) metric values and compute:
   • The mean (our main performance estimate).
   • Optionally the standard deviation (how variable the estimate is).

What do we get from \(k\)-fold CV?

From \(k\)-fold cross-validation we get:

• A mean metric over folds (e.g., mean accuracy, mean loss).
• A standard deviation over folds.

These tell us:

• How well the model tends to generalize (mean).
• How consistent that performance is across different data splits (std).

If the standard deviation is \(\downarrow\) small:

• Our reported mean is stable: different folds give similar results.
• We can be more confident that we’re not just “lucky” with one split.

If the standard deviation is \(\uparrow\) large:

• Performance is sensitive to which examples are in train vs validation.
• We should be more cautious in trusting a single number.

🔬 Fold-wise scores enable basic significance tests

Because cross-validation gives us one score per fold, we can:

• Compare two models (A and B) on the same folds.
• For each fold \(k\), compute the difference: \[d_k = \text{score}_k(A) - \text{score}_k(B).\]
• Look at the mean and standard deviation of these differences.

If the differences are consistently positive (and not too noisy),
we have some evidence that A is genuinely better than B,
not just on one lucky split.

In practice, people might run a small 🔗 paired \(t\)-test on the \(d_k\)’s.
We won’t go into the details, but this is why the standard deviation across folds matters.

Key message: CV scores are not just a single number — they form a small dataset that we can reason about statistically.

✏️ Quick check

Suppose we do 5-fold cross-validation on a dataset with 10,000 examples.

• How many models do we train in total?
• For a given data point:
  • How many times is it used in training?
  • How many times is it used in validation?
• If we re-shuffle and build new folds, can the CV estimate change?
  Why or why not?

Math lens

Generalization risk vs empirical risk

• Let \((x_i, y_i)_{i=1}^n\) be IID samples from some unknown distribution \(\mathcal{P}\).

• Model \(f_\theta\) (parameters \(\theta\)), loss \(\ell(y, f_\theta(x))\).

• True risk (generalization error): \[R(\theta) = \mathbb{E}_{(x,y) \sim \mathcal{P}}[\ell(y, f_\theta(x))]\]

• Empirical risk on a dataset \(S\) of size \(|S|\): \[\hat{R}_S(\theta) = \frac{1}{|S|} \sum_{(x_i, y_i) \in S} \ell(y_i, f_\theta(x_i))\]

• Key ideas:

  • Training minimizes \(\hat{R}_{\text{train}}(\theta)\).
  • We actually care about \(R(\theta)\) on future data.
  • A held-out test set gives a Monte Carlo estimate of \(R(\theta)\): \[\hat{R}_{\text{test}}(\theta) = \hat{R}_{S_{\text{test}}}(\theta).\]

Cross-validation estimator (stretch)

• When data are limited, a single train/test split is high variance.

• \(k\)-fold cross-validation reuses data more efficiently.

• Partition indices into \(K\) disjoint folds \(V_1,\dots,V_K\).

• For each fold \(k\):

  • Train on \(S_{-k}\) (all but \(V_k\)).
  • Evaluate on \(V_k\).

• Cross-validation estimate of risk: \[\hat{R}_{\text{CV}}(\theta) = \frac{1}{K} \sum_{k=1}^K \frac{1}{|V_k|} \sum_{i \in V_k} \ell(y_i, f^{(-k)}_\theta(x_i))\]

• Choosing \(K\):

  • Larger \(K\) (e.g., \(K = n\), leave-one-out):
    • Train on almost all data each time → low bias.
    • But folds are very similar → estimates are highly correlated, higher variance, more compute.
  • Smaller \(K\) (e.g., \(K = 5\) or \(10\)):
    • Slightly more bias (less data per training run).
    • Often lower variance, and much cheaper computationally.

• In practice, \(K = 5\) or \(10\) are common compromises.

Worked example 1 — basic splits

Synthetic fraud dataset

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split, StratifiedKFold, cross_val_score
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, recall_score, classification_report

import plotly.express as px

rng = np.random.default_rng(3151)

X, y = make_classification(
    n_samples=1500,
    n_features=6,
    n_informative=4,
    n_redundant=1,
    n_clusters_per_class=2,
    weights=[0.96, 0.04],  # 4% fraud
    flip_y=0.01,
    random_state=42,
)

feature_names = [
    "amount",
    "days_since_last_tx",
    "card_age_months",
    "merchant_risk_score",
    "hour_of_day",
    "device_trust_score",
]

df = pd.DataFrame(X, columns=feature_names)
df["is_fraud"] = y

df.head()

	amount	days_since_last_tx	card_age_months	merchant_risk_score	hour_of_day	device_trust_score	is_fraud
0	-0.234373	1.004242	-0.386695	-0.862743	-0.032917	-0.140448	0
1	0.918796	-0.262210	1.302603	2.873578	-3.560945	-0.354622	0
2	0.526696	0.524000	-1.071553	0.197359	0.052271	-1.531038	0
3	0.232798	0.205568	0.196695	0.799942	-0.488831	-0.099855	0
4	0.630113	-1.031392	0.017419	-0.549819	-1.360607	-1.313415	0

Stratified train/test split

X = df[feature_names].values
y = df["is_fraud"].values

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.2,
    stratify=y,       # keep class proportions similar
    random_state=42,
)

print("Train size:", X_train.shape[0])
print("Test size: ", X_test.shape[0])

print("Train fraud rate: {:.3f}".format(y_train.mean()))
print("Test fraud rate:  {:.3f}".format(y_test.mean()))

Train size: 1200
Test size:  300
Train fraud rate: 0.042
Test fraud rate:  0.043

Baseline model & metrics

baseline_clf = Pipeline([
    ("scaler", StandardScaler()),
    ("logreg", LogisticRegression(max_iter=1000, random_state=42)),
])

baseline_clf.fit(X_train, y_train)

y_train_pred = baseline_clf.predict(X_train)
y_test_pred = baseline_clf.predict(X_test)

print("Train accuracy: {:.3f}".format(accuracy_score(y_train, y_train_pred)))
print("Test  accuracy: {:.3f}".format(accuracy_score(y_test, y_test_pred)))
print("Test  recall:   {:.3f}".format(recall_score(y_test, y_test_pred)))

Train accuracy: 0.959
Test  accuracy: 0.960
Test  recall:   0.077

# Plot class balance in train vs test
split_labels = ["train"] * len(y_train) + ["test"] * len(y_test)
y_all = np.concatenate([y_train, y_test])

balance_df = pd.DataFrame({
    "split": split_labels,
    "is_fraud": y_all,
})

class_counts = (
    balance_df
    .groupby(["split", "is_fraud"])
    .size()
    .reset_index(name="count")
)

pivot = class_counts.pivot(index="split", columns="is_fraud", values="count")
pivot = pivot.div(pivot.sum(axis=1), axis=0)

pivot.plot(kind="bar", stacked=True)
plt.ylabel("Proportion")
plt.title("Class balance by split")
plt.legend(title="is_fraud", loc="best")
plt.tight_layout()
plt.show()

• Fraud team cares more about recall on fraud than raw accuracy.
• Our split and metrics must reflect that.

How much does a random split matter?

• Smaller test sets → more variation in test recall.
• A single number from one random split is a noisy estimate of generalization.
• Cross-validation: see a distribution of scores rather than one lucky split.

Data leakage

What is data leakage?

• Data leakage: information from outside the training data for that model leaks into the model-fitting process.

Common ways this happens:

1. Preprocessing on the full dataset before splitting.
2. Using test labels during feature engineering.
3. Temporal leakage (using future info to predict the past).
4. Group leakage (same user or patient in train and test).

Consequences:

• Over‑optimistic metrics that fail to replicate.
• Misleading model comparisons.
• In high‑stakes settings, harm to stakeholders.

❌️ Anti-example — scaling before splitting

from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)   # ❌ fits on ALL data, including future test rows

X_train_bad, X_test_bad, y_train_bad, y_test_bad = train_test_split(
    X_scaled, y, test_size=0.2, stratify=y, random_state=42
)

• Test rows influence the scaling parameters.
• Metrics on X_test_bad are too optimistic.

❌️ Bad demo: scaling pipelines

# BAD: scaling on full dataset before splitting
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split

scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)  # ❌ uses all rows (train + future test)

X_train_bad, X_test_bad, y_train_bad, y_test_bad = train_test_split(
    X_scaled, y, test_size=0.2, stratify=y, random_state=0
)

• ❌ StandardScaler sees future test rows.
• ❌ Any metric on X_test_bad is optimistically biased.
• ❌ This pattern completely breaks if you plug it into cross-validation.

✅️ Demo: scaling pipelines

# GOOD: scaling inside a pipeline
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

good_clf = Pipeline([
    ("scaler", StandardScaler()),
    ("logreg", LogisticRegression(max_iter=1000, random_state=0)),
])

good_clf.fit(X_train, y_train)  # ✔ scaler fit only on training data
y_test_pred = good_clf.predict(X_test)

• ✔ Preprocessing is inside the training loop.
• ✔ Safe with cross-validation: scaler refit on each training fold.
• ✔ Test set truly acts as unseen data.

Worked example 2 — cross-validation

Tuning regularization with \(k\)-fold CV

In LogisticRegression, the hyperparameter C controls how strong the L2 regularization is:

• Smaller C ⟶ stronger regularization
  (weights are shrunk more, decision boundary smoother, less overfitting).
• Larger C ⟶ weaker regularization
  (weights can grow, boundary can be more wiggly, more risk of overfitting).

We don’t guess C by eye. Instead, we use \(k\)-fold cross-validation on the training set:

1. Pick a grid of candidate values, e.g. C ∈ {0.01, 0.1, 1, 10, 100}.
2. For each C, run \(k\)-fold CV and compute the mean validation score (and optionally its std).
3. Choose the C with the best mean performance (subject to being reasonably stable).

# First: split off a test set
X_train_val, X_test, y_train_val, y_test = train_test_split(
    X,
    y,
    test_size=0.2,
    stratify=y,
    random_state=42,
)

def make_model(C):
    return Pipeline([
        ("scaler", StandardScaler()),
        ("logreg", LogisticRegression(C=C, max_iter=1000, random_state=42)),
    ])

skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

Cs = [0.01, 0.1, 1.0, 10.0]
rows = []

for C in Cs:
    model = make_model(C)
    scores = cross_val_score(
        model,
        X_train_val,
        y_train_val,
        cv=skf,
        scoring="recall",  # emphasize catching fraud
        n_jobs=None,
    )
    rows.append({"C": C, "mean_recall": scores.mean(), "std_recall": scores.std()})

cv_df = pd.DataFrame(rows).sort_values("C")
cv_df

	C	mean_recall	std_recall
0	0.01	0.00	0.000000
1	0.10	0.00	0.000000
2	1.00	0.04	0.080000
3	10.00	0.08	0.074833

plt.errorbar(cv_df["C"], cv_df["mean_recall"], yerr=cv_df["std_recall"], fmt="-o")
plt.xscale("log")
plt.xlabel("C (log scale)")
plt.ylabel("Mean CV recall")
plt.title("Stratified 5-fold CV for fraud detection")
plt.tight_layout()
plt.show()

# Pick best C by mean CV recall
best_idx = cv_df["mean_recall"].idxmax()
best_C = cv_df.loc[best_idx, "C"]
best_C

np.float64(10.0)

# Train final model on all train_val data and evaluate once on the test set
best_model = make_model(best_C)
best_model.fit(X_train_val, y_train_val)

y_test_pred = best_model.predict(X_test)
print(classification_report(y_test, y_test_pred, digits=3))

              precision    recall  f1-score   support

           0      0.963     1.000     0.981       287
           1      1.000     0.154     0.267        13

    accuracy                          0.963       300
   macro avg      0.982     0.577     0.624       300
weighted avg      0.965     0.963     0.950       300

Cross-validation as experiment: choosing \(K\)

• Smaller \(K\) → bigger test folds, smaller train folds → slightly higher bias, lower variance.
  • (👀 We’ll see these concepts in greater detail in the next module)
• Larger \(K\) → more training data, smaller test folds → lower bias, higher variance.
• 5–10 folds are a pragmatic compromise.

Variants of splitting & CV

Leave-one-out (LOO) cross-validation

• Special case of \(k\)-fold where \(k = n\) (number of examples).
• For each example:
  • Train on all the other examples.
  • Validate on this one example.

👍️ Pros:

• Uses almost all data for training each time → low bias.
• No extra randomness from which examples land in which fold.

👎️ Cons:

• Computationally expensive: train \(n\) models.
• Validation sets are size 1 → estimates for each run are very noisy.
• Folds are highly similar → overall estimate can still have high variance.

In practice:

• Nice theoretically, and sometimes used on very small datasets.
• For larger \(n\), we usually prefer k = 5 or 10 as a compromise.

Connect back to bias–variance and “choice of k” slide.

Leave-one-source-out / grouped CV

Sometimes our examples are not independent:

• Multiple rows can come from the same source:
  • Same patient, same speaker, same document, same user, same hospital…

If we randomly split rows:

• The model may see some data from a source in training, and other data from the same source in validation.
• This can overestimate performance (data leakage).

Leave-one-source-out CV:

• Define a group label (e.g., patient ID, speaker ID).
• Each fold leaves out all examples from one group (or a few groups).
• Train on the other groups, validate on the held-out group(s).

This asks a different question:

“How well do we do on a new source we have never seen before?”

You can call this “leave-one-speaker-out”, “leave-one-patient-out”, etc., depending on your running example.

Beyond random splits

• Stratified: maintain label proportions (default for imbalanced classification).
• Group-aware: keep all rows from a patient / user / device together.
• Time-aware / rolling: train on past, validate on future. Never shuffle across time.

Question to always ask:

“How will this model actually be used?”

Mimic that usage pattern in your split / CV strategy.

Ethics & risks

• Leaky or unrealistic evaluation can:
  • Convince stakeholders to deploy a model that underperforms in production.
  • Hide systematic failures on minority groups or rare but important cases.
• Good practices:
  • Document your splitting strategy, seeds, and rationale.
  • Report subgroup metrics on the held-out test set.
  • When possible, replicate on a truly independent dataset (e.g., next year’s data).

Notes on practice

• In Kaggle/tabular competitions, CV strategy is often more important than model choice.
• Modern research struggles with:
  • Validation under distribution shift.
  • Designing CV schemes for structured data (graphs, text, time-series).

Splits & CV are not “preprocessing boilerplate” — they are part of your core scientific design.

Quick checks

QC1 — roles of splits

You train a model on a training set and tune hyperparameters using a validation set. When should you evaluate on the test set?

A. After every hyperparameter change, to track progress.
B. Only once, after choosing the final model.
C. Never; the validation set is enough.
D. After each epoch during training.

QC2 — leakage or not?

Which of the following definitely introduces data leakage?

A. Scaling features using only the training set mean and variance.
B. Selecting hyperparameters based on cross-validation performance.
C. Computing feature importances on the full dataset (train + test) and then retraining a model using only the top features.
D. Using stratified splits to preserve class proportions.

QC3 — variance of CV

You run 5-fold CV and get recall scores: 0.80, 0.82, 0.79, 0.81, 0.81.

Which is most reasonable?

A. The model generalizes perfectly; deploy without a test set.
B. The CV estimate of recall is around 0.8 with low variance.
C. You must increase \(K\) to 10; 5-fold CV is invalid.
D. The model is certainly overfitting.

QC4 — choosing a split strategy

You have patient-level data where each patient contributes multiple visits. You want performance on new patients.

Which split strategy is best?

A. Randomly split rows into train and test, ignoring patient IDs.
B. Stratify by label only.
C. Group by patient ID so each patient appears in exactly one split.
D. Use leave-one-out CV over rows.