M22 - Descriptive statistics & distributions
CSCI 1109 — Practical Data Science
🔎 Math lens: median (robust centre)
Intuition: line everyone up from smallest to largest; the middle person is the median.
- Median cares about order, not extreme distances.
- So it is robust to outliers.
Code
xs mean: 22.666666666666668 . median: 22.0
ys mean: 47.333333333333336 . median: 22.0Mean jumps when 26→100.
Median stays at 22.
🔎 Math lens: quantiles & IQR
Quantiles: split sorted data into parts containing specified proportions of the data. E.g.,
- the 25th percentile (quartile 1 = Q1) is the value at or below which about 25% of the observations fall;
- the 50th percentile is the median;
- the 75th percentile (quartile 3 = Q3) has about 75% of observations at or below it.
The inter-quartile range (IQR) is: \[ \mathrm{IQR} = Q_3 - Q_1 \]
Intuition: the width of the middle 50% of data.
Code
(np.float64(12.75), np.float64(22.0), np.float64(9.25))IQR ignores the extreme tails → good for a robust spread.
Code
import numpy as np
import matplotlib.pyplot as plt
# Toy data: sorted so students can see positions
xs = np.array([10, 12, 13, 15, 20, 21, 25, 30], dtype=float)
q1 = np.quantile(xs, 0.25)
q2 = np.quantile(xs, 0.50)
q3 = np.quantile(xs, 0.75)
iqr = q3 - q1
fig, ax = plt.subplots(figsize=(7, 1.8))
# Number line
ax.hlines(0, xs.min()-2, xs.max()+2, colors="lightgray")
# All data points
ax.scatter(xs, np.zeros_like(xs), s=60, zorder=3)
# Shade middle 50%
ax.axvspan(q1, q3, color="#ffe699", alpha=0.7, zorder=1)
# Vertical lines at Q1, median, Q3
for x, label in [(q1, "Q1"), (q2, "median"), (q3, "Q3")]:
ax.vlines(x, -0.2, 0.2, colors="black", linestyles="solid")
ax.text(x, 0.35, label, ha="center", va="bottom", fontsize=10)
# IQR bracket and label
ax.annotate(
"",
xy=(q1, -0.35),
xytext=(q3, -0.35),
arrowprops=dict(arrowstyle="<->", linewidth=1.5),
)
ax.text((q1+q3)/2, -0.55, f"IQR = Q3 - Q1 = {iqr:.1f}", ha="center", va="top", fontsize=10)
ax.set_ylim(-0.8, 0.8)
ax.set_yticks([])
ax.set_xlabel("value")
ax.set_xticks(xs)
ax.grid(axis="x", alpha=0.2)
plt.tight_layout()
plt.show()
🏥 “Average wait is 30 minutes…”
“Our average emergency wait time is 30 minutes. We’re doing great.”
Two hospitals have the same reported average wait.
Would you rather go to Hospital A or Hospital B?
We’ll simulate both and see how the same mean can hide very different risks.
Code
def make_hospital_waits(n=500):
# Hospital A: roughly normal around 30 minutes
a = np.random.normal(loc=30, scale=7, size=n)
a = np.clip(a, 5, None) # no negative waits
# Hospital B: mixture — many quick, some very long waits
fast = np.random.exponential(scale=10, size=int(0.7 * n)) + 5
slow = np.random.exponential(scale=40, size=int(0.3 * n)) + 30
b = np.concatenate([fast, slow])
return pd.DataFrame({
"wait_min": np.concatenate([a, b]),
"hospital": ["A"] * len(a) + ["B"] * len(b)
})
waits = make_hospital_waits()
waits.groupby("hospital")["wait_min"].agg(["mean", "median", "std"]).round(1)| mean | median | std | |
|---|---|---|---|
| hospital | |||
| A | 30.6 | 30.4 | 7.4 |
| B | 30.1 | 17.7 | 30.0 |
Same mean, different distributions
Code
fig, axes = plt.subplots(1, 2, figsize=(10, 4), sharey=True)
for ax, name in zip(axes, ["A", "B"]):
subset = waits[waits["hospital"] == name]["wait_min"]
ax.hist(subset, bins=25)
ax.axvline(subset.mean(), linestyle="--", label="mean")
ax.axvline(subset.median(), linestyle=":", label="median")
ax.set_title(f"Hospital {name}")
ax.set_xlabel("wait (min)")
ax.grid(alpha=0.2)
axes[0].set_ylabel("count")
axes[0].legend()
plt.tight_layout()
plt.show()
- 🏥 Hospital A: most patients near 30 minutes.
- 🏥 Hospital B: many quick visits, but a long tail of patients waiting far longer.
The same average encodes very different patient experiences.
Conditional probability
Idea: We often want the probability of an event A, given that another event B has occurred.
Notation: \(P(A \mid B)\) 👉 “probability of A given B”.
Definition (when \(P(B) > 0\)):
\[ P(A \mid B) = \frac{P(A \cap B)}{P(B)} \]
Read this as: Among the outcomes where B happens, what fraction also have A?
Example
- \(A\): “student passes the exam”
- \(B\): “student attended at least 80% of classes”
Then \(P(A \mid B)\) is “the probability a student passes given they attended regularly”.
Unions, intersections, independence
Bayes’ rule: algebra and picture
- We often know:
- prior beliefs like \(P(X)\),
- how likely data are under a hypothesis, like \(P(Y \mid X)\).
- Bayes’ rule lets us flip the conditioning:
\[ P(X \mid Y) = \frac{P(X, Y)}{P(Y)} = \frac{P(X)\,P(Y \mid X)}{P(Y)}. \]
- This is the core tool for going from model → data to data → updated belief.
Bayes’ rule: doing the numbers
We have \(P(\color{red}{S} \mid \cdot)\). We actually want \(P(\color{orange}{C} \mid \color{red}{S})\) and \(P(\color{blue}{M} \mid \color{red}{S})\).
Using Bayes:
\[ P(\color{orange}{C} \mid \color{red}{S}) = \frac{P(\color{orange}{C})\,P(\color{red}{S} \mid \color{orange}{C})}{P(\color{red}{S})}, \qquad P(\color{blue}{M} \mid \color{red}{S}) = \frac{P(\color{blue}{M})\,P(\color{red}{S} \mid \color{blue}{M})}{P(\color{red}{S})}. \]
We can work with proportionality:
\[ P(\color{orange}{C} \mid \color{red}{S}) \propto P(\color{orange}{C}) P(\color{red}{S} \mid \color{orange}{C}) = 0.1 \times 0.8 = 0.08, \] \[ P(\color{blue}{M} \mid \color{red}{S}) \propto P(\color{blue}{M}) P(\color{red}{S} \mid \color{blue}{M}) = 0.9 \times 0.2 = 0.18. \]
Normalizing:
\[ P(\color{orange}{C} \mid \color{red}{S}) = \frac{0.08}{0.08 + 0.18} \approx 0.31, \quad P(\color{blue}{M} \mid \color{red}{S}) = \frac{0.18}{0.08 + 0.18} \approx 0.69. \]
Even though shyness is more common among CS PhDs than MBAs,
the person you met is still more likely to be an MBA — because there are so many more MBAs overall.
