We choose\(\theta\) (or hyperparameters) that keep validation error small.
The test error is our best guess at how the model will perform on new, real-world data.
Worked example 1
Classification with a simple split
🥅 Goal: predict whether a household’s daily water use will exceed a local conservation target (e.g., 150 liters per person per day).
Why this might matter:
City planners want to identify high-usage days so they can plan infrastructure and conservation campaigns.
Households might opt in to receive reminders or tips when they are likely to exceed the target.
The focus is on supporting sustainable resource use, not blaming individual households.
Assume we have a small synthetic dataset with:
temp_c – average outdoor temperature that day (°C),
household_size – number of people in the household,
has_garden – 1 if the household has a garden/lawn, 0 otherwise,
is_weekend – 1 if the day is Saturday/Sunday, 0 otherwise,
above_target – 1 if daily water use exceeded the conservation threshold, 0 otherwise.
Code
import numpy as npimport pandas as pdrng = np.random.default_rng(1109)n =128temp_c = rng.integers(10, 35, size=n) # cooler to very warm dayshousehold_size = rng.integers(1, 6, size=n) # 1–5 peoplehas_garden = rng.integers(0, 2, size=n) # 0 or 1is_weekend = rng.integers(0, 2, size=n) # 0 or 1# A simple rule-of-thumb model for daily water (liters), plus noisedaily_liters = ( (80+20* household_size) +3* (temp_c -20) +# hotter days → more use25* has_garden +15* is_weekend + rng.normal(0, 15, size=n) # random noise (float))target_liters =150above_target = (daily_liters > target_liters).astype(int)df = pd.DataFrame({"temp_c": temp_c,"household_size": household_size,"has_garden": has_garden,"is_weekend": is_weekend,"above_target": above_target,})df.head()
temp_c
household_size
has_garden
is_weekend
above_target
0
24
4
1
0
1
1
20
4
0
1
1
2
30
1
0
1
0
3
15
4
1
0
1
4
27
1
0
0
0
Train/validation/test split in code
We’ll create an ~60% / 20% / 20% split and keep class balance using stratified splits (our dataset is large enough for that here).
Code
from sklearn.model_selection import train_test_splitfeatures = ["temp_c", "household_size", "has_garden", "is_weekend"]target ="above_target"X = df[features]y = df[target]# First: train_temp vs test (stratify to preserve 0/1 proportions)X_train, X_temp, y_train, y_temp = train_test_split( X, y, test_size=0.2, random_state=1109, stratify=y,)# Second: val vs test from the temporary set (also stratified)X_val, X_test, y_val, y_test = train_test_split( X_temp, y_temp, test_size=0.5, random_state=1109, stratify=y_temp,)len(X_train), len(X_val), len(X_test)
(102, 13, 13)
We use stratify=y to keep class proportions balanced in each split.
With 128 rows and reasonably balanced classes, each split still has at least a few examples of both 0 and 1, so stratification is safe.
On very tiny datasets, stratifying multiple times can fail; on realistic tabular data, this pattern is common.
Train accuracy: 0.9019607843137255
Val accuracy: 0.9230769230769231
Test accuracy: 0.8461538461538461
✏️ Think
If train accuracy ≫ val/test accuracy, what might that suggest about overfitting to this particular sample of households?
If val accuracy is higher for k=5 than k=3, which should we prefer (and why)?
What other features might improve this model (e.g., building type, recent rainfall), and how might we accidentally introduce data leakage if we’re not careful about which information is available at prediction time?
Design choices & subtle pitfalls
Key design decisions even in this tiny example:
Random state & reproducibility
Setting random_state helps you and your future self reproduce results.
Stratified splits
For classification, use stratify=y so each split has similar class balances.
Feature scaling
For distance-based models (like k-NN), differing feature scales matter.
In practice, we’d use a Pipeline with StandardScaler before k-NN.
Data leakage
Don’t compute features using future information.
Don’t fit scalers or imputers on the whole dataset; fit them inside a pipeline using only the training data.
Worked example 2
Regression on a familiar dataset
Let’s predict tip percentage from the tips dataset (you used it earlier).
Goal: model tip_pct = tip / total_bill from features like:
punishes large errors more; can be useful when big mistakes are very bad.
In practice
Always ask: “What is this model for? Which mistakes hurt most?”
Tie your metric choices to the stakeholder scenario.
❌ Anti-example: data leakage in splitting
Imagine we’re predicting whether a student passes the final exam:
Features include:
midterm_mark
assignment_average
final_mark 👈 (this is the label itself!)
🤪 Suppose we accidentally:
Use final_mark to compute a feature like overall_course_average, then
Predict “pass_final” using that feature.
🤷 What went wrong?
The model has direct access to the thing we want to predict (or something computed with it).
Train and test performance will look amazing.
In reality, the model is useless before the final exam happens.
Rule of thumb: your features should use only information available at prediction time.
Ethics & risks in ML evaluation
Even in “toy” educational examples, evaluation choices matter:
Biased splits
If the train set under-represents certain groups (e.g., part-time students), the model may perform poorly for them.
Time-based splits matter: training on old cohorts may not generalize to newer ones.
Hiding uncertainty
Reporting a single accuracy number without context hides variance.
Use confidence intervals, replicate splits, and clear caveats where possible.
Misuse of models
Using a soft educational risk model to make hard decisions (e.g., remove students from a program) without human oversight can be harmful.
Share limitations and failure modes in your write-ups (this is part of responsible data science).
Mitigations (at 1109 level):
Stratified splits; fairness checks by subgroup.
Clear documentation of data sources, time windows, and what the model can/cannot do.
In assignments, explicit prompts to reflect on limitations.
Quick checks
Which of the following is a classification problem?
A. Predicting tomorrow’s temperature in °C.
B. Predicting whether a transaction is fraudulent (yes/no).
C. Predicting the number of minutes a student will study.
D. Predicting the total number of visitors next week.
Why do we keep a separate test set?
A. To have more data to train on.
B. To tune hyperparameters like k in k-NN.
C. To get an unbiased estimate of performance on new data.
D. To debug code more easily.
You train a model and see:
Train accuracy: 0.99
Validation accuracy: 0.78
Test accuracy: 0.77
Which is the most likely explanation?
A. The model is underfitting.
B. The model is overfitting the training data.
C. The dataset is too large.
D. The train/val/test split is impossible.
Which situation is an example of data leakage?
A. Using only 80% of the data for training.
B. Fitting a scaler (mean/variance) on the full dataset before splitting.
C. Using a random seed in train_test_split.
D. Stratifying the split by class label.
