🥅 Objective: discover structure in \(\color{blue}{x}\) alone, e.g.:
Clustering: partition indices \(\{1, \ldots, n\}\) into groups \(S_1, \dots, S_k\) so that points in the same group are more “similar” than points from different groups.
Density models \(p_\theta(\color{blue}{x})\): how likely are different regions of feature space?
Tip
Think of supervised as “predict a label” and unsupervised as “explain or compress the data, or find patterns”.
🔬 Definitions (2/2)
Self-supervised learning
🛢️ Data: still only \(\{\color{blue}{x_i}\}\).
But we create pseudo-labels from the data itself:
Next token prediction in text.
Masked-word prediction (BERT-style).
Did we rotate this image (Y/N)?
We then run supervised learning on this synthetic task.
Semi-supervised learning
🛢️ Data:
A small labeled set \(\mathcal{D}_L = \{(\color{blue}{x_i}, \color{red}{y_i})\}_{i=1}^{n_L}\) and
A larger unlabeled set \(\mathcal{D}_U = \{\color{blue}{x_j}\}_{j=1}^{n_U}\)
Objective: use both\(\mathcal{D}_L\) and \(\mathcal{D}_U\) to learn a predictor for \(\color{red}{Y}\).
Typical assumptions:
Cluster assumption: points in the same cluster tend to share labels.
Smoothness assumption: nearby points in a high-density region should have similar labels.
Each colour is one cluster: a group of points the algorithm judged similar.
🧐 we never told it any \(\color{red}{y}\) labels — clustering is based only on the geometry of \(\color{blue}{X}\).
A popular algorithm that behaves like this is k-means — we’ll define and analyze it carefully in a later module.
❌ Anti-example: unused labels
You collect:
\(n=100k\) customer records with features.
For 500 of them, you also have future revenue (a \(\mathbb{R}\)-valued target).
You cluster all 100k records and call it “semi-supervised regression”.
Warning
What’s wrong?
You never used the labels in your objective → your training is unsupervised.
You might evaluate clusters using revenue, but that’s post hoc.
Calling this “semi-supervised regression” hides the fact that you made no effort to align the clusters with revenue.
Boundary lesson:
What makes something supervised/semi-supervised is how labels enter the objective, not whether labels exist somewhere in your database.
🔬 Supervised risk
Suppose:
Binary classification, \(y \in \{0,1\}\).
Loss: 0–1 loss \(\ell(\hat y, y) = \mathbb{1}[\hat y \neq y]\).
How many people we annoy unnecessarily (precision / false positive rate).
Accuracy treats all mistakes the same.
🤔 Always ask:
“If a model makes this type of mistake, who gets hurt and how badly?”
Possible evaluation choices:
For supervised subscription prediction:
Precision, recall, F1.
ROC / PR curves.
Calibration curves (are probabilities honest?).
For unsupervised clustering:
If no labels: cohesion/separation metrics, stability across runs.
If we do have labels (for teaching): adjusted Rand index, mutual information.
Ethics, fairness, and robustness
Fairness:
If labels (e.g., “will subscribe”) encode historical bias, supervised/semi-supervised models inherit that bias.
Clusters can indirectly encode protected attributes (e.g., geography, prior access).
Privacy:
Self-supervised objectives on logs or text can memorize sensitive information if not regularized / anonymized.
Robustness & shift:
Semi-supervised learning assumes unlabeled data come from the same distribution as labeled data.
If that’s false (e.g., new user demographic), pseudo-labeling can amplify mistakes.
Note
Evidence you should demand:
Performance + error analysis by subgroup (e.g., different student populations).
Tests of robustness under distribution shift (different time periods, campuses).
Documentation of what data was used for self-/semi-supervised pretraining.
🤔 Curiosity hook — how is GPT “self-supervised”?
Modern large language models are mostly trained with self-supervision:
Data: massive corpora of text (no human labels).
Task: predict the next token given previous tokens.
Objective: minimize cross-entropy (a supervised loss!) on this synthetic task.
Why it matters for us:
The paradigm is self-supervised, but under the hood it’s just supervised learning on an automatically-generated target.
Semi-supervised and fine-tuning steps later add small labeled datasets on top.
We’ll come back to this.
Question to keep in mind
“If my pretext task is weird, what representations will my model learn, and which downstream tasks will that help or hurt?”
Quick check (pause and think)
Try these before looking at the answers.
You have millions of unlabeled images and 10k image–label pairs for cats vs. dogs.
You pretrain a model to colourize grayscale images and then fine-tune it on the 10k labels. Q1: Which paradigms are used at the pretraining and fine-tuning stages?
You cluster patient trajectories and then give doctors a dashboard showing cluster summaries, without ever using outcomes. Q2: What learning setting is this? Is there any hidden “supervision”?
You have labeled spam/not-spam emails but accidentally include test-set emails in your training set. Q3: Which assumption is being violated?
You run k-means with \(k=10\) on the student dataset and treat cluster ID as the “true” label. Q4: Name one serious risk of this practice.
