M13 — Missingness, Duplicates, and Outliers

CSCI 1109 — Practical Data Science

Frank Rudzicz

Why this module matters

Many data science failures happen before anyone trains a model.

  • In the real world, our data is rarely “clean”. Instead, we get:
    • Holes where values should be (missingness)
    • Weird, impossible numbers (noise & recording errors)
    • “Clone” rows from systems that sync badly (duplicates)
    • Extreme points that might be mistakes… or the most important cases (outliers)
  • If you ignore these, your model will happily learn patterns in:
    • typos,
    • broken sensors,
    • and half-recorded events.

This module is about learning to notice and question those problems before they quietly wreck your conclusions.

High-level data quality

Some useful lenses:

  • Accuracy – does the value reflect reality?
  • Precision – how finely is it measured?
  • Completeness – which values are simply not there?
  • Consistency – are formats / units / states aligned?
  • Timeliness – is the data current enough?
  • Believability – do we trust who and how it was recorded?
  • Interpretability – can humans understand it?

Five flavours of dirty data

You will see these constantly:

  1. Inconsistent types
  2. Inconsistent values
  3. Missing data
  4. Outliers
  5. Noise and recording errors (including wrong labels)

We’ll anchor everything on a small, fictional dataset.

Campus Wellness Check-ins

Students can do quick wellness check-ins at kiosks (residences, gym). Clinic staff enter some records later. Two different systems get merged weekly.

Each row = one check-in event.

Variables:

  • checkin_id (string; sometimes missing from one system)
  • student_hash (pseudonymous ID, stable per student)
  • timestamp (when the check-in happened)
  • location ∈ {ResidenceA, ResidenceB, Gym, Clinic}
  • sleep_hours_last_night (float)
  • stress_1to10 (int; 1=low, 10=high)
  • temp_c (float; should be Celsius)
  • resting_hr (int; beats per minute)
  • notes (free text, often messy)

Tiny sample of the data

Code 
import pandas as pd
import numpy as np

rows = [
  {"checkin_id":"A-0001","student_hash":"S_19f","timestamp":"2025-01-11 09:02","location":"ResidenceA",
   "sleep_hours_last_night":7.5,"stress_1to10":3,"temp_c":36.8,"resting_hr":62,"notes":""},
  {"checkin_id":"A-0002","student_hash":"S_19f","timestamp":"2025-01-12 09:01","location":"ResidenceA",
   "sleep_hours_last_night":np.nan,"stress_1to10":-1,"temp_c":36.7,"resting_hr":61,"notes":"skipped sleep/stress"},
  {"checkin_id":"B-1148","student_hash":"S_9aa","timestamp":"2025-01-12 17:40","location":"Gym",
   "sleep_hours_last_night":6.0,"stress_1to10":6,"temp_c":98.2,"resting_hr":72,"notes":"temp entered in F?"},
  {"checkin_id":"C-0104","student_hash":"S_0c2","timestamp":"2025-01-13 12:15","location":"Clinic",
   "sleep_hours_last_night":5.0,"stress_1to10":9,"temp_c":37.9,"resting_hr":240,"notes":"device glitch?"},
  {"checkin_id":None,"student_hash":"S_0c2","timestamp":"2025-01-13 12:16","location":"Clinic",
   "sleep_hours_last_night":5.0,"stress_1to10":9,"temp_c":37.9,"resting_hr":120,"notes":"manual entry (no id)"},
  {"checkin_id":"B-1150","student_hash":"S_7d1","timestamp":"2025-01-14 08:58","location":"ResidenceB",
   "sleep_hours_last_night":26.0,"stress_1to10":2,"temp_c":36.5,"resting_hr":58,"notes":"impossible sleep"},
  {"checkin_id":"A-0008","student_hash":"S_5b8","timestamp":"2025-01-14 09:05","location":"ResidenceA",
   "sleep_hours_last_night":8.0,"stress_1to10":0,"temp_c":36.6,"resting_hr":0,"notes":"sensor dropout"},
  {"checkin_id":"B-1151","student_hash":"S_9aa","timestamp":"2025-01-12 17:40","location":"Gym",
   "sleep_hours_last_night":6.0,"stress_1to10":6,"temp_c":98.2,"resting_hr":72,"notes":"duplicate from merge"}
]
df_tiny = pd.DataFrame(rows)
df_tiny
checkin_id student_hash timestamp location sleep_hours_last_night stress_1to10 temp_c resting_hr notes
0 A-0001 S_19f 2025-01-11 09:02 ResidenceA 7.5 3 36.8 62
1 A-0002 S_19f 2025-01-12 09:01 ResidenceA NaN -1 36.7 61 skipped sleep/stress
2 B-1148 S_9aa 2025-01-12 17:40 Gym 6.0 6 98.2 72 temp entered in F?
3 C-0104 S_0c2 2025-01-13 12:15 Clinic 5.0 9 37.9 240 device glitch?
4 None S_0c2 2025-01-13 12:16 Clinic 5.0 9 37.9 120 manual entry (no id)
5 B-1150 S_7d1 2025-01-14 08:58 ResidenceB 26.0 2 36.5 58 impossible sleep
6 A-0008 S_5b8 2025-01-14 09:05 ResidenceA 8.0 0 36.6 0 sensor dropout
7 B-1151 S_9aa 2025-01-12 17:40 Gym 6.0 6 98.2 72 duplicate from merge

Spot the problems

From that tiny table we already see:

  • Missing fields
    • sleep_hours_last_night = NaN
    • checkin_id = None
  • Off-script “fake” values
    • stress_1to10 = -1, stress_1to10 = 0 used as “missing”
  • Unit mistakes
    • temp_c = 98.2 is probably Fahrenheit (an archaic form of measurement)
  • Sensor glitches
    • resting_hr = 0, 240
  • Duplicates
    • Same student & timestamp from the weekly merge

Examples like these are common.

Missingness

Missing data: the easy cases

“Easy” missingness is visible in the table:

  • Explicit nulls (NaN, None, empty strings)
  • Impossible placeholders used for missing:
    • age = -1, age = 999
    • stress_1to10 = -1 in our dataset
    • "unknown", "N/A" etc

These are perfect targets for easy automations we’ll see shortly.

Missing data: harder to see

Sometimes missingness is not obvious:

  • A person is dropped from the database entirely
    (how would you even know?)
  • One system always leaves checkin_id blank, another always fills it
  • Staff sometimes skip certain questions under pressure
  • Different hospitals / apps log different subsets of fields

Visible values can encode invisible decisions about what was or was not recorded.

Three ways missingness happens

Classic categories:

  • MCAR – Missing Completely At Random
    e.g., kiosk offline randomly; some rows lost independent of values.
  • MAR – Missing At Random (given other observed variables)
    e.g., clinic staff always fill all fields; gym kiosks skip temp_c.
  • MNAR – Missing Not At Random
    e.g., people with highest stress skip the stress_1to10 field.

In practice, you often guess which regime you are closest to, based on domain knowledge.

When missingness is actually information

Sometimes “no value recorded” is the value. 😮

  • In EHRs, whether a lab was ever ordered can be more predictive than the lab value itself

    • Example: patients with no cholesterol test often have lower cardiovascular risk, because the clinician never felt worried enough to order it. (Groenwold 2020)

  • Patterns where labs stop being ordered after a normal result can encode disease severity and clinical reassurance. These “informatively missing” labs have been used to stratify COVID-19 inpatients across multiple hospital systems. (Tan et al. 2023)

  • For many common labs, missingness is tied to disease burden: sicker patients get more labs (less missingness), while healthier patients often have more gaps in their records. (Li et al. 2021)

  • In prediction models built from EHRs, one strategy is to keep missingness as a feature:

  • add binary “missing indicator” variables that mark when a predictor is absent. Simulation work shows that, when missingness is informative, indicators can sometimes improve or at least not harm model performance. (Ehrig et al. 2025; Wells et al. 2013)

Take-home

Don’t always rush to “fill in the blanks.” For some variables, “not measured” is itself a signal about clinical judgment, workflow, and patient state — but that signal can change once we start using it in deployed models.

Strategies for missing data (overview)

What you can do:

  1. Drop rows or columns
  2. Regenerate missing values manually (lookup, call, review logs)
  3. Fill in missing values automatically (imputation)

Every strategy adds assumptions:

  • which patterns you consider “OK to lose”
  • which values you think are “close enough”
  • how you change the joint distribution of the data

Dropping data

Two common heuristics:

  • Drop rows that are missing the target label
    (for supervised learning)
  • Drop rows/columns with too many missing fields

Pros:

  • Very easy
  • No invented values

Cons:

  • Can introduce bias
    (you may systematically drop the hardest or most important cases)
  • You lose information the model might still use (other columns)

Simple automatic fills

Easy options:

  • Replace missing values with a constant (e.g., "unknown")
  • Replace with mean or median of that feature
  • Replace with class-conditional mean
    (e.g., mean HR for that location or age group)

Pros:

  • Very easy with libraries (e.g., SimpleImputer in scikit-learn)
  • Keeps dataset rectangular and simple

Cons:

  • Shrinks variance; can create unrealistic “blobs”
  • "unknown" may accidentally become a new category
  • Can distort relationships between variables

Regression vs interpolation (for filling gaps)

Two ways to “draw a curve through the data”:

Regression

  • Fit a single function through all data at once
    (often a line or a smooth curve)
  • Does not have to hit every data point
  • Great when there is substantial noise

Interpolation

  • Construct a function that passes through the observed points
  • Often built locally (piecewise) between neighbours
  • Good when data are clean and gaps are small

Both can be used to estimate missing values in time series or smooth functions.

Visualizing interpolation vs regression

Code 
import numpy as np
import matplotlib.pyplot as plt

x = np.array([0, 1, 2, 3, 4, 5], dtype=float)
y = np.array([0.0, 5.2, 2.9, 2.1, 3.9, 5.0])

xx = np.linspace(0, 5, 200)

# Simple linear regression
coeffs = np.polyfit(x, y, 1)
reg_line = np.polyval(coeffs, xx)

# Simple cubic interpolation
interp_coeffs = np.polyfit(x, y, 6)
interp_curve = np.polyval(interp_coeffs, xx)

plt.figure(figsize=(6,4))
plt.scatter(x, y)
plt.plot(xx, reg_line, label="regression (global line)")
plt.plot(xx, interp_curve, linestyle="--", label="interpolation (local fit)")
plt.xlabel("x")
plt.ylabel("y")
plt.legend()
plt.tight_layout()

Imputation with neighbours (k-NN idea)

Idea: use other similar rows to guess the missing value.

Steps:

  1. For a row with a missing value in dimension (d), look only at the other dimensions.
  2. Find the (k) most similar rows (nearest neighbours).
  3. Replace the missing value with e.g., the mean (for numeric) or mode (for categorical) among neighbours.

In our dataset:

  • If sleep_hours_last_night is missing but HR and location and stress are similar to other rows, we can borrow their typical value.

k-NN imputation in code (scikit-learn)

Code 
from sklearn.impute import KNNImputer
import numpy as np
import pandas as pd

toy = pd.DataFrame(
    [[ 3, 12,  6, 0.01],
     [ 1, 18, np.nan, 0.01],
     [-18,230, 18, 0.34],
     [ -8,512, 94, 0.45],
     [ 4, 16, 12, 0.02]],
    columns=["x1","x2","x3","x4"]
)

imputer = KNNImputer(n_neighbors=2, weights="uniform")
imputed = imputer.fit_transform(toy)
pd.DataFrame(imputed, columns=toy.columns)
x1 x2 x3 x4
0 3.0 12.0 6.0 0.01
1 1.0 18.0 9.0 0.01
2 -18.0 230.0 18.0 0.34
3 -8.0 512.0 94.0 0.45
4 4.0 16.0 12.0 0.02

The missing entry in x3 is replaced with the average of the two closest rows (in the space of x1, x2, x4).

Outliers

Outliers: what are they?

Intuitively:

An outlier is a datapoint that differs strongly from most others.

Common causes:

  • Measurement error (e.g., HR = 0, 240)
  • Unit errors (98.2°F in a “Celsius” column)
  • Rare but real events (post-workout HR of 160)
  • New phenomena (a truly unusual case)

There is no single perfect definition.

For “nice” distributions, a common rule of thumb is: more than 3 standard deviations from the mean.

Outliers on a Gaussian curve

For a Normal distribution:

  • ~68% of data lies within 1σ of the mean
  • ~95% within 2σ
  • ~99.7% within 3σ

Anything beyond ±3σ is very rare (~0.3% total):

Code 
import numpy as np
import matplotlib.pyplot as plt
from math import exp, pi, sqrt

x = np.linspace(-4,4,400)
y = (1/np.sqrt(2*pi)) * np.exp(-0.5 * x**2)

plt.figure(figsize=(6,4))
plt.plot(x,y)
plt.axvline(-3, linestyle="--")
plt.axvline(3, linestyle="--")
plt.text(0, max(y)*0.9, "Most data", ha="center")
plt.xlabel("z-score")
plt.ylabel("density")
plt.title("Outliers on a Gaussian curve (> 3σ from the mean)")
plt.tight_layout()

Points out in the thin tails are candidates for outliers — but context still matters.

Outliers via clustering (preview)

We can also spot outliers with clustering (later module):

  • Imagine grouping points into clusters.
  • Points that are:
    • far from every cluster centre, or
    • stuck in very tiny clusters
      are candidates for outliers.

This works even when the data are not nicely Normal, but depends on the clustering algorithm and distance measures.

Code 
import numpy as np
import matplotlib.pyplot as plt
from sklearn.cluster import KMeans

np.random.seed(1109)

# Three compact clusters
cluster1 = np.random.normal(loc=[0, 0], scale=0.5, size=(80, 2))
cluster2 = np.random.normal(loc=[3, 3], scale=0.5, size=(80, 2))
cluster3 = np.random.normal(loc=[0, 3], scale=0.5, size=(80, 2))
X = np.vstack([cluster1, cluster2, cluster3])

# A few obvious outliers
outliers = np.array([[5, 5], [4, -1], [-2, 4]])
X_all = np.vstack([X, outliers])

# Fit k-means with 3 clusters
kmeans = KMeans(n_clusters=3, n_init="auto", random_state=1109)
labels = kmeans.fit_predict(X_all)

plt.figure(figsize=(6, 4))
# Plot all points coloured by cluster
plt.scatter(X_all[:, 0], X_all[:, 1], c=labels, alpha=0.7)

# Highlight the injected outliers with big hollow markers
plt.scatter(outliers[:, 0], outliers[:, 1],
            s=150, facecolors="none", edgecolors="black", linewidths=2)

plt.xlabel("feature 1")
plt.ylabel("feature 2")
plt.title("Clusters with a few obvious outliers")
plt.tight_layout()

🕵️‍♀️ Handling outliers: choices

Common strategies:

  • Investigate & fix actual errors
    (e.g., convert Fahrenheit to Celsius)
  • Treat as missing when clearly wrong
    (e.g., HR = 0 → missing HR)
  • Clip extreme values to a reasonable range
  • Transform (e.g., log transform for heavy-tailed data)
  • Use robust models that downweight outliers
  • Model them separately
    (e.g., anomaly-detection task)

Deleting outliers blindly can erase exactly the events you want to discover.

Interactive demo: HR outliers over time

We’ll use a slightly larger synthetic sample of resting heart rate from our campus dataset and see how changing the “outlier rule” changes which points are flagged.

Code 
import numpy as np
import pandas as pd
import plotly.graph_objects as go

np.random.seed(13)

n = 240
timestamps = pd.date_range("2025-01-01", periods=n, freq="2H")
student_hash = np.random.choice([f"S_{i:03d}" for i in range(25)], size=n, replace=True)
location = np.random.choice(["ResidenceA","ResidenceB","Gym","Clinic"], size=n, p=[0.35,0.25,0.30,0.10])

# Baseline HR
hr = np.clip(np.random.normal(68, 10, size=n).round().astype(int), 35, 120)

# Inject glitches and extremes
glitch_idx = np.random.choice(np.arange(n), size=8, replace=False)
hr[glitch_idx[:3]] = [0, 240, 220]     # impossible / sensor error
hr[glitch_idx[3:6]] = [32, 31, 130]   # edge cases
for i in glitch_idx[6:]:
    if location[i] == "Gym":
        hr[i] = np.random.randint(130, 175)  # high but plausible after workout

df = pd.DataFrame({
    "timestamp": timestamps,
    "student_hash": student_hash,
    "location": location,
    "resting_hr": hr
})

lower = 35
thresholds = list(range(120, 201, 10))

def make_trace(th):
    flagged = (df["resting_hr"] < lower) | (df["resting_hr"] > th)
    keep = df[~flagged]
    out = df[flagged]
    return (
        go.Scatter(
            x=keep["timestamp"], y=keep["resting_hr"],
            mode="markers", name="kept",
            hovertemplate="time=%{x}<br>HR=%{y}<extra></extra>",
        ),
        go.Scatter(
            x=out["timestamp"], y=out["resting_hr"],
            mode="markers", name="flagged",
            hovertemplate="time=%{x}<br>HR=%{y}<br><b>FLAGGED</b><extra></extra>",
        )
    )

t0 = thresholds[0]
keep0, flag0 = make_trace(t0)

fig = go.Figure(data=[keep0, flag0])

frames = []
for th in thresholds:
    keep, flag = make_trace(th)
    frames.append(go.Frame(data=[keep, flag], name=str(th)))

steps = []
for th in thresholds:
    steps.append({
        "method": "animate",
        "label": f"max={th}",
        "args": [[str(th)], {"mode": "immediate",
                             "frame": {"duration": 0},
                             "transition": {"duration": 0}}],
    })

fig.frames = frames

fig.update_layout(
    title="Flagging outliers in resting heart rate (HR)",
    xaxis_title="timestamp",
    yaxis_title="resting_hr",
    legend_title="",
    updatemenus=[{
        "type": "buttons",
        "direction": "left",
        "x": 0.0,
        "y": 1.15,
        "buttons": [{
            "label": "Reset",
            "method": "animate",
            "args": [[str(t0)], {"mode": "immediate",
                                 "frame": {"duration": 0},
                                 "transition": {"duration": 0}}],
        }],
    }],
    sliders=[{
        "active": 0,
        "currentvalue": {"prefix": "Outlier rule: HR > "},
        "pad": {"t": 30},
        "steps": steps
    }]
)

fig

Duplicates

Duplicates: when are two rows “the same”?

  • A duplicate is not just “two identical lines in a CSV”.

  • Typical scenarios:

    • The same student checks in through two systems
      • two records that really describe the same event.
    • A sync or merge script runs twice.
    • IDs are missing in one system but present in another.

We need an operational definition of “same event”.

Duplicate rules for Wellness Check-ins

One reasonable rule:

Two rows describe the same check-in if:

  • student_hash is the same, and
  • timestamps are within ±2 minutes, and
  • location is the same.

In practice:

  • you might also compare vitals (HR, temp)
  • or require text similarity in the notes field

These rules can be implemented with:

  • df.sort_values(...)
  • rolling windows / grouping
  • string similarity for notes (later in the course)

Easy automations for dirty data (pandas)

Very common quick wins:

  • Missingness
  • Off-script values
    • df.replace({"stress_1to10": {-1: np.nan}})
  • Duplicates
    • df.duplicated(subset=["student_hash","timestamp","location"])
    • df.drop_duplicates(...)
  • Range checks
    • Boolean masks to flag impossible HR, temperature, sleep

Automation finds candidates; we still need to decide what to keep, fix, drop, or model.

Noise

Other kinds of noise you’ll meet

So far, we’ve focused on missingness, duplicates, and outliers.

In the wild, you’ll also see:

  • Sensor noise
    Slight jitter in heart rate, GPS coordinates, accelerometers…

  • Workflow noise
    Values entered late, overwritten, or only filled when something is wrong.

  • Text noise
    Spelling mistakes, emojis, “lol”, URLs, copy–pasted signatures.

  • Label noise
    Two clinicians (or two crowdworkers) disagree about the “correct” label.

We won’t go deep into these in M13.
Think of this as your mental checklist: “Do I trust how this was recorded?”

Example: noisy sensor readings

Even when nothing dramatic happens, sensors produce jittery data.

Code 
import numpy as np
import matplotlib.pyplot as plt

np.random.seed(1109)

t = np.linspace(0, 10, 200)
true_hr = 70 + 5*np.sin(0.6*t)          # smooth underlying trend
noise = np.random.normal(0, 3, size=t.shape)
measured_hr = true_hr + noise           # noisy measurements

# Simple moving average smoothing
window = 7
kernel = np.ones(window) / window
smoothed = np.convolve(measured_hr, kernel, mode="same")

plt.figure(figsize=(6, 4))
plt.plot(t, true_hr, linestyle="--", label="underlying signal")
plt.plot(t, measured_hr, marker="o", linewidth=0, markersize=3, label="noisy readings")
plt.plot(t, smoothed, label="simple smoothing")
plt.xlabel("time")
plt.ylabel("heart rate (bpm)")
plt.title("Noisy sensor readings vs smoothed signal")
plt.legend()
plt.tight_layout()

Example: noise from human workflows

Even simple forms collect messy, inconsistent answers.

Code 
import pandas as pd
import numpy as np

data = [
    ["S_001", "yes",   8,   "08:05", "feeling ok"],
    ["S_002", "Yes",   7,   " 8:5 ", "ok"],
    ["S_003", "y",     9,   "8:05am","Tired..."],
    ["S_004", "no",    "-", "08:07", ""],
    ["S_005", "N/A",   6,   "8:05",  "late to class"],
    ["S_006", "",      5,   "08:5",  "ok"],
]

df_form = pd.DataFrame(data, columns=[
    "student_id",
    "had_breakfast",
    "sleep_hours_last_night",
    "checkin_time",
    "notes"
])
df_form
student_id had_breakfast sleep_hours_last_night checkin_time notes
0 S_001 yes 8 08:05 feeling ok
1 S_002 Yes 7 8:5 ok
2 S_003 y 9 8:05am Tired...
3 S_004 no - 08:07
4 S_005 N/A 6 8:05 late to class
5 S_006 5 08:5 ok

Summary

Habits to build

In this module we saw:

  • Missingness is everywhere and has structure
    • Sometimes it is even useful signal.
    • How you fill gaps encodes assumptions.
  • Noise affects both measurements and labels.
    • Noisy labels → multiple annotators and agreement measures.
  • Outliers can be wrong, rare, or the most interesting part.
    • Flag, inspect, and decide — don’t just delete.
  • Duplicates require a definition of “same entity, same event”.

Before modeling, always ask:

  1. What’s missing, and why might it be missing?
  2. Where could there be duplicates?
  3. Which values look impossible or suspicious?
  4. What cleaning choices might change my conclusions?

Later modules will build on this: scaling, encoding, binning, and more advanced feature engineering.

References

Ehrig, Molly, Garrett S. Bullock, Xiaoyan Iris Leng, Nicholas M. Pajewski, and Jaime Lynn Speiser. 2025. “Imputation and Missing Indicators for Handling Missing Longitudinal Data: Data Simulation Analysis Based on Electronic Health Record Data.” JMIR Medical Informatics 13: e64354. https://doi.org/10.2196/64354.
Groenwold, Rolf H. H. 2020. “Informative Missingness in Electronic Health Record Systems: The Curse of Knowing.” Diagnostic and Prognostic Research 4 (8): 8. https://doi.org/10.1186/s41512-020-00077-0.
Li, Jiang, Xiaowei S. Yan, Durgesh Chaudhary, Venkatesh Avula, Satish Mudiganti, Hannah Husby, et al. 2021. “Imputation of Missing Values for Electronic Health Record Laboratory Data.” Npj Digital Medicine 4: 147. https://doi.org/10.1038/s41746-021-00518-0.
Tan, Amelia L. M., Emily J. Getzen, Meghan R. Hutch, Zachary H. Strasser, et al. 2023. “Informative Missingness: What Can We Learn from Patterns in Missing Laboratory Data in the Electronic Health Record?” Journal of Biomedical Informatics 139: 104306. https://doi.org/10.1016/j.jbi.2023.104306.
Wells, Brian J., Kevin M. Chagin, Amy S. Nowacki, and Michael W. Kattan. 2013. “Strategies for Handling Missing Data in Electronic Health Record Derived Data.” EGEMS (Washington, DC) 1 (3): 1035. https://doi.org/10.13063/2327-9214.1035.