M16 - Scaling, encoding, binning, feature creation

CSCI 1109 — Practical Data Science

Frank Rudzicz

Assumptions

• You have already seen basic pandas (DataFrame, Series, indexing) and simple cleaning (missing values, duplicates).
• We do not assume prior experience with scikit-learn; all scaling/encoding is done with pandas + NumPy.
• We briefly mention train/test splits to avoid data leakage, but full ML models come later in the course.
• Tiny, synthetic datasets stand in for realistic cases (e.g., credit scoring) so that you can run everything easily.

Outcomes

By the end of this module you should be able to:

• Diagnose when mismatched scales and encodings will break or mislead a model.
• Implement simple scaling, one-hot encoding, and binning operations in pandas without leaking test information.
• Create at least one new feature that captures useful structure about the data-generating process (DGP).
• Explain, in plain language, how your feature choices affect distances, decision boundaries, and error costs.
• Identify ethically risky features (e.g., proxies for protected attributes) and describe at least one mitigation.

Before models: deciding what “similar” means

Data science models do not see the world directly.
They see numbers arranged in space.

• Before any model runs, we decide:
  • which features exist
  • how large differences are
  • what counts as “close” or “far”
  • which details matter — and which do not

These choices are called feature representations.

In practice, feature choices can matter more than the choice of model itself.

• In this module, we focus on four representation decisions:
  • scaling
  • encoding
  • binning
  • feature creation

1. Why preprocessing matters

A motivating scenario: small credit union

You’re helping a small credit union decide which loan applications to flag for manual review.

• Data (tabular):
  • age (years)
  • monthly_income (CAD)
  • debt_to_income (ratio 0–1)
  • employment_status (categorical: student / part-time / full-time / unemployed)
  • city (Halifax / Truro / Sydney)
  • defaulted (1 if they defaulted on a past loan, else 0)
• Decision:
  • Flag high-risk applications for extra human review.
• Costs of errors:
  • False negative (miss a risky borrower) → financial loss.
  • False positive (over-flag safe borrower) → customers annoyed, slower service.

Now: we don’t build the classifier; we shape the data so future models can learn robustly and fairly.

Conceptual scaffold

We will focus on four ideas:

1. Scaling: putting numeric features on comparable ranges.
2. Encoding: turning categories (text) into numbers in a meaningful way.
3. Binning: turning continuous variables into ordered categories (e.g., age groups).
4. Feature creation (engineering): constructing new variables that better reflect the DGP.

You can think of this as designing a coordinate system in which the model will later make decisions.

Pipeline diagram

flowchart LR
    A["Raw data<br/>CSV / JSON"] --> B["Cleaning<br/>(missingness, duplicates)"]
    B --> C["Preprocessing<br/>(scaling, encoding, binning)"]
    C --> D["Model training<br/>& evaluation"]
    D --> E["Decision<br/>(ship, hold out, escalate)"]

❌ Anti-example: Encoding goes wrong

Imagine you encode employment_status as:

• \(\langle \text{student, part-time, full-time, unemployed} \rangle = \langle 0,1,2,3\rangle\)

What’s wrong?

• The numbers pretend there is a single ordered line from “student” → “unemployed”.
• Distance between “part-time” (1) and “full-time” (2) is the same as between “full-time” (2) and “unemployed” (3), which is nonsense.
• If a model uses Euclidean distance, “unemployed” ends up furthest from “student” by construction, even if reality may be different.

🙂 Better: one-hot encoding (separate 0/1 indicator per category). We’ll implement this later.

2. Math lens

Scaling: standardization (z-scores)

Intuition:

Measure “how many standard deviations away from typical” a value is.

Formal definition for a feature \(x\):

• Mean: \(\mu = \frac{1}{n} \sum_{i=1}^n x_i\)
• Standard deviation: \(\sigma = \sqrt{\frac{1}{n} \sum_{i=1}^n (x_i - \mu)^2}\)
• 👉 Standardized value: \[ z_i = \frac{x_i - \mu}{\sigma} \]

Tiny numeric example (monthly income in $1,000s):

person	raw income	standardized \(z\) (approx)
A	2.0	-0.7
B	3.0	0.0
C	4.0	0.7

🔎 Interpretation: B is “typical”, A is 0.7 standard deviations below typical, C is 0.7 above.

Note

Scaling your data with the z-score is like fitting a Normal curve to your data, then (1) moving it to have 0 mean and (2) stretching it to have standard deviation of 1.

Scaling: min–max (0–1) scaling

Intuition:

Stretch or compress feature values so the minimum becomes 0 and the maximum becomes 1.

Formula for feature \(x\):

\[ x_i^{(scaled)} = \frac{x_i - x_{\min}}{x_{\max} - x_{\min}} \]

Use cases:

• For algorithms that assume all features are in similar numeric ranges (e.g., distance-based).
• For visualizations where you want features to be directly comparable.

Risk: outliers can make the min–max range huge; consider clipping or robust variants.

Encoding: one-hot

Intuition:

Represent each category as its own yes/no column.

Example: employment_status in {student, part-time, full-time}.

We create:

• emp_student (1 if student, else 0)
• emp_part_time
• emp_full_time

For a part-time worker: \([0, 1, 0]\).

This avoids fake numeric ordering and lets models learn arbitrary patterns across categories.

Precision?

• Suppose you measure something precisely:

  • income in dollars
  • age in years
  • distance to the nearest store

• Why would you ever:

  • round it?
  • bucket it?
  • bin it?

• Why would a data scientist throw away precision?

• Before we answer, pause and consider:

  • Is more precision always more information?
  • Can precision amplify noise?
  • Can coarser features sometimes be more meaningful?

• In the next section, we’ll see why binning is not a mistake — but a trade-off.

Binning: continuous \(\mapsto\) categories

Intuition:

Sometimes, the exact value (e.g., age 41 vs 42) doesn’t matter; a range does (e.g., “middle-aged”).

Example: age → bins

• 18–24 → young
• 25–44 → mid
• 45+ → senior

We assign each person to a bin, and the model learns separate behaviours for each group.

Risks:

• Bins too wide → lose signal.
• Bins too narrow → overfit, sparse data.
• Bins based on test labels → leakage (we must not do that).

3. Worked example 1 — Scaling income & debt ratio

Tiny credit dataset

We’ll use a tiny synthetic dataset of past loans.

import pandas as pd
from io import StringIO

csv = """loan_id,age,monthly_income,debt_to_income,defaulted
1,22,1400,0.55,0
2,35,3200,0.40,0
3,41,5100,0.25,0
4,29,2100,0.65,1
5,60,2800,0.70,1
6,50,7200,0.30,0
7,33,2600,0.45,0
8,45,4100,0.80,1
9,27,1900,0.35,0
10,39,3500,0.60,1
"""

df = pd.read_csv(StringIO(csv))
df

	loan_id	age	monthly_income	debt_to_income	defaulted
0	1	22	1400	0.55	0
1	2	35	3200	0.40	0
2	3	41	5100	0.25	0
3	4	29	2100	0.65	1
4	5	60	2800	0.70	1
5	6	50	7200	0.30	0
6	7	33	2600	0.45	0
7	8	45	4100	0.80	1
8	9	27	1900	0.35	0
9	10	39	3500	0.60	1

Units:

• age: years
• monthly_income: CAD
• debt_to_income: ratio in [0, 1]
• defaulted: 1 if they defaulted (binary outcome)

Inspect raw scales

df.describe(include="all")

	loan_id	age	monthly_income	debt_to_income	defaulted
count	10.00000	10.000000	10.000000	10.000000	10.000000
mean	5.50000	38.100000	3390.000000	0.505000	0.400000
std	3.02765	11.445038	1725.913607	0.183258	0.516398
min	1.00000	22.000000	1400.000000	0.250000	0.000000
25%	3.25000	30.000000	2225.000000	0.362500	0.000000
50%	5.50000	37.000000	3000.000000	0.500000	0.000000
75%	7.75000	44.000000	3950.000000	0.637500	1.000000
max	10.00000	60.000000	7200.000000	0.800000	1.000000

Things to notice:

• monthly_income is in thousands of dollars; debt_to_income is 0–1.
• A 1-unit change in monthly_income vs debt_to_income does not mean the same thing.
• Distance-based methods would mostly “see” differences in income, not debt ratio.

Let’s visualize:

import matplotlib.pyplot as plt

fig, axes = plt.subplots(1, 2, figsize=(8, 3))

axes[0].hist(df["monthly_income"], bins=5)
axes[0].set_title("Monthly income (CAD)")

axes[1].hist(df["debt_to_income"], bins=5)
axes[1].set_title("Debt-to-income ratio")

plt.tight_layout()

Standardize numeric features (train-only)

Pretend rows 1–7 are our training set and 8–10 are a tiny test set.

train = df.iloc[:7].copy()
test = df.iloc[7:].copy()

num_cols = ["age", "monthly_income", "debt_to_income"]

train_means = train[num_cols].mean()
train_stds = train[num_cols].std(ddof=0)  # population std for simplicity

train_means, train_stds

(age                 38.571429
 monthly_income    3485.714286
 debt_to_income       0.471429
 dtype: float64,
 age                 11.986387
 monthly_income    1851.970820
 debt_to_income       0.157791
 dtype: float64)

Now scale using only training stats:

train_scaled = train.copy()
test_scaled = test.copy()

train_scaled[num_cols] = (train[num_cols] - train_means) / train_stds
test_scaled[num_cols]  = (test[num_cols]  - train_means) / train_stds

train_scaled[num_cols].describe()

	age	monthly_income	debt_to_income
count	7.000000e+00	7.000000e+00	7.000000e+00
mean	1.506731e-16	-2.379049e-17	9.912706e-17
std	1.080123e+00	1.080123e+00	1.080123e+00
min	-1.382521e+00	-1.126213e+00	-1.403304e+00
25%	-6.316690e-01	-6.132463e-01	-7.695538e-01
50%	-2.979571e-01	-3.702619e-01	-1.358036e-01
75%	5.780367e-01	3.586912e-01	8.148217e-01
max	1.787742e+00	2.005585e+00	1.448572e+00

Quick sanity checks:

• Are the training means of the scaled features close to 0?
• Are the training standard deviations close to 1?

train_scaled[num_cols].mean().round(5), train_scaled[num_cols].std(ddof=0).round(5)

(age               0.0
 monthly_income   -0.0
 debt_to_income    0.0
 dtype: float64,
 age               1.0
 monthly_income    1.0
 debt_to_income    1.0
 dtype: float64)

Visualizing before vs after scaling

fig, axes = plt.subplots(1, 2, figsize=(9, 3))

axes[0].scatter(train["monthly_income"], train["debt_to_income"])
axes[0].set_xlabel("Monthly income (CAD)")
axes[0].set_ylabel("Debt-to-income")
axes[0].set_title("Raw scale")

axes[1].scatter(train_scaled["monthly_income"], train_scaled["debt_to_income"])
axes[1].set_xlabel("Scaled income")
axes[1].set_ylabel("Scaled debt-to-income")
axes[1].set_title("Standardized scale")

plt.tight_layout()

Interpretation:

• On the left, income dominates the horizontal scale.
• On the right, both axes are roughly [-2, 2], so distance-based algorithms would treat income and debt_ratio more symmetrically.

4. Worked example 2 — Encoding & binning

Tiny categorical dataset

csv2 = """customer_id,age,employment_status,city,monthly_income,defaulted
101,22,student,Halifax,1400,0
102,35,full-time,Halifax,3200,0
103,41,full-time,Sydney,5100,0
104,29,part-time,Truro,2100,1
105,60,retired,Halifax,2800,1
106,50,full-time,Sydney,7200,0
107,33,part-time,Halifax,2600,0
108,45,unemployed,Truro,4100,1
"""

df_cat = pd.read_csv(StringIO(csv2))
df_cat

	customer_id	age	employment_status	city	monthly_income	defaulted
0	101	22	student	Halifax	1400	0
1	102	35	full-time	Halifax	3200	0
2	103	41	full-time	Sydney	5100	0
3	104	29	part-time	Truro	2100	1
4	105	60	retired	Halifax	2800	1
5	106	50	full-time	Sydney	7200	0
6	107	33	part-time	Halifax	2600	0
7	108	45	unemployed	Truro	4100	1

Naïve ordinal encoding (anti-pattern)

• Naïve ordinal encoding assigns arbitrary integers to categories.

naive_map = {
    "student": 0,
    "part-time": 1,
    "full-time": 2,
    "unemployed": 3,
    "retired": 4,
}

df_naive = df_cat.copy()
df_naive["employment_code"] = df_naive["employment_status"].map(naive_map)
df_naive[["employment_status", "employment_code"]]

	employment_status	employment_code
0	student	0
1	full-time	2
2	full-time	2
3	part-time	1
4	retired	4
5	full-time	2
6	part-time	1
7	unemployed	3

Warning

• This silently tells the model that:
  • the categories are ordered along a single line, and
    
  • the “distance” between each pair (0→1, 1→2, 2→3) is equally meaningful.
• For most real categories, neither of these assumptions is true.
  • Distance-based or linear models will learn patterns from a fake numerical structure that we invented, not from the underlying reality.

One-hot encoding with pandas

from IPython.display import display 
df_encoded = pd.get_dummies(
    df_cat,
    columns=["employment_status", "city"],
    prefix=["emp", "city"],
    drop_first=False  # keep all columns for clarity
)

def _color_blocks(col):
    if col.name.startswith("emp_"):
        return ["background-color: #e6f2ff"] * len(col)  # employment_status
    if col.name.startswith("city_"):
        return ["background-color: #e8f8e8"] * len(col)  # city
    return [""] * len(col)

styled = df_encoded.head().style.apply(_color_blocks, axis=0)

display(styled)  # <- key bit

	customer_id	age	monthly_income	defaulted	emp_full-time	emp_part-time	emp_retired	emp_student	emp_unemployed	city_Halifax	city_Sydney	city_Truro
0	101	22	1400	0	0	0	0	1	0	1	0	0
1	102	35	3200	0	1	0	0	0	0	1	0	0
2	103	41	5100	0	1	0	0	0	0	0	1	0
3	104	29	2100	1	0	1	0	0	0	0	0	1
4	105	60	2800	1	0	0	1	0	0	1	0	0

Now each category is its own 0/1 feature, and no fake ordering is introduced.

Binning age and inspecting default rate

We’ll bin age into groups and look at default rates.

bins = [18, 30, 45, 80]
labels = ["young", "mid", "senior"]

df_binned = df_cat.copy()
df_binned["age_group"] = pd.cut(df_binned["age"], bins=bins, labels=labels, right=False)

df_binned[["age", "age_group", "defaulted"]]

	age	age_group	defaulted
0	22	young	0
1	35	mid	0
2	41	mid	0
3	29	young	1
4	60	senior	1
5	50	senior	0
6	33	mid	0
7	45	senior	1

Now compute default rate by age group:

group_default = (
    df_binned
    .groupby("age_group")["defaulted"]
    .mean()
    .rename("default_rate")
    .reset_index()
)
group_default

	age_group	default_rate
0	young	0.500000
1	mid	0.000000
2	senior	0.666667

Plot it:

plt.figure(figsize=(4, 3))
plt.bar(group_default["age_group"].astype(str), group_default["default_rate"])
plt.ylabel("Default rate")
plt.xlabel("Age group")
plt.title("Default rate by age group")
plt.tight_layout()

🤔 Interpretation:

• Even with tiny data, we can see whether certain age ranges appear riskier.
• In real data, we’d need more rows and proper uncertainty quantification before acting.

5. Ethics nudge: risks & mitigations

Evaluation aligned to decisions

Even at the preprocessing stage, we should think about which metrics will matter later.

🏦 For the credit union scenario:

• ⚠️ Main risk: approving too many risky loans → false negatives (missed defaulters).
• Metric candidates:
  • Recall for the positive class (defaulted = 1): fraction of defaulters we catch.
  • Precision for defaulters: of those we flag, how many really default?
  • Balanced accuracy: handles class imbalance better than raw accuracy.

🧹 Preprocessing choices change:

• How easily a simple model can separate risky vs safe borrowers.
• How stable those decisions are across subgroups (e.g., cities, employment types).

🧑‍🔬 Example severe test you might run later:

• Train the same model with and without scaling/encoding on the same train/test split.
• Compare recall on defaulters and check whether improvements are consistent across subgroups.

⚠️ Risks

• Proxy variables: certain encodings (e.g., postal code, employer name) can act as proxies for protected traits (race, income, disability).
• Over-granular bins: tiny groups (e.g., very narrow age bins) can make individuals more identifiable or lead to unstable decisions for small subpopulations.

🧑‍🔧 Mitigations (starter list)

• Keep a data sheet: document each feature, its source, and why it exists.
• Perform subgroup checks: compare error rates by group (e.g., city, employment type) once models are built.
• Avoid features whose only role is to encode historical discrimination unless you have a clear remediation strategy.

7. Quick checks

Try these on your own; answers are in the comments below.

1. You are using a distance-based method later (like k-NN). Which combination is most appropriate?
   • 1. Raw income in $, raw age in years.
   • 2. Income scaled to 0–1, age scaled to 0–1.
   • 3. One-hot encoding for income, raw age.
   • 4. All of the above are equally good.
2. Which encoding is most at risk of introducing a fake ordering among categories?
   • 1. One-hot encoding
   • 2. Ordinal encoding with integers 0, 1, 2, …
   • 3. Binning a numeric variable
   • 4. None of them
3. You decide bins for monthly_income after seeing which ranges have the highest observed default rate on the test set. Which threat have you introduced?
   • 1. Data leakage
   • 2. Class imbalance
   • 3. Overfitting in the model, but not preprocessing
   • 4. No threat; this is fine
4. In the credit union scenario, which error is generally more costly and which metric is more aligned?
   • 1. False positives; optimize precision on non-defaulters
   • 2. False negatives; optimize recall on defaulters
   • 3. Both errors are equal; accuracy is enough
   • 4. None of the above