Self-supervised learning turns unlabeled data into useful representations by asking a model to recognize what two transformed views share. Its best-known methods can look like a collection of unrelated recipes. This work shows that many of those recipes follow from one objective: maximize mutual information between the views.

unlabeled input▦
cropxA
colorxB
→
encoderzA
encoderzB
maximizeI(zA; zB)shared information

Start from mutual information

Two augmented views come from the same underlying input. A useful encoder should preserve their shared semantic information while discarding incidental changes such as cropping, color jitter, or noise. The paper begins with the Donsker-Varadhan lower bound, which separates this goal into a joint term and a marginal term.

Donsker-Varadhan lower bound
I(ZA; ZB) ≥ Ep(zA,zB)[T] − log Ep(zA)p(zB)[eT]
Joint termAlign representations of matching viewsMarginal termKeep the representation space diverse

The first term rewards agreement between two views of the same sample. The second compares independently paired representations and prevents a trivial solution in which every input maps to the same point. Once these forces are separated, two valid ways to optimize them become visible.

Two optimization paradigms

01
Self-Distillation MISDMI · alternating optimization
View Aonline encoder fθupdate θ
align
View Btarget encoder gξstop-gradient

Hold one branch fixed while updating the other, then alternate. Stop-gradient implements this block-coordinate structure; predictors or momentum updates act as practical surrogates.

SimSiam · BYOL · MoCo
02
Joint MIJMI · simultaneous optimization
View Ashared encoder fθgradient
align
View Bshared encoder fθgradient

Update both views symmetrically through shared parameters. Negative samples or variance-covariance regularizers approximate the marginal term and preserve feature diversity.

SimCLR · Barlow Twins · VICReg
The paper's SDMI and JMI architecture diagrams. SDMI alternates an E-step and M-step between two encoders using stop-gradients, while JMI performs a single joint update through a shared encoder.
Figure 1 from the paper. SDMI realizes alternating optimization by stopping gradients through one branch at a time. JMI sends gradients through both views in one symmetric step.

This distinction cuts across the usual contrastive/non-contrastive divide. MoCo is contrastive while BYOL is not, yet both belong to SDMI because both alternate an online and target branch. Barlow Twins and SimCLR use very different losses, yet both belong to JMI because gradients update a shared encoder jointly.

MethodTraditional labelMI paradigmKey mechanism
SimSiamNon-contrastiveSDMIStop-gradient + predictor
BYOLNon-contrastiveSDMIMomentum target + predictor
MoCoContrastiveSDMIMomentum encoder + negatives
SimCLRContrastiveJMIShared encoder + negatives
Barlow TwinsNon-contrastiveJMICross-correlation regularizer
VICRegNon-contrastiveJMIVariance + covariance penalties

Why representations do not collapse

If both augmented views are simply encouraged to match, the network can minimize the loss by mapping every image to the same representation. The framework explains the architectural components that prevent this collapse: each one restores or approximates the marginal force in the mutual-information objective.

Two CIFAR-10 ablation plots from the paper. Removing the DV marginal term or the SimSiam predictor causes accuracy to collapse, while adding predictors or using the explicit cosine-DV objective restores accuracy close to the baseline.
Figure 4 from the paper. Left: removing the DV marginal term collapses both SDMI encoders, while predictors recover performance. Right: SimSiam without a predictor collapses, but replacing its loss with the explicit cosine-DV objective restores learning without the predictor.

The paper tests this explanation directly with SimSiam. Removing its predictor causes collapse. Replacing the original cosine objective with the explicit cosine-DV objective restores learning without a predictor: the SimSiam-SDMI variant reaches 88.43 ± 0.78% CIFAR-10 linear-probe accuracy, close to the 89.68 ± 0.35% baseline reported in the ablation.

What the experiments show

The authors instantiate minimal SDMI and JMI prototypes and evaluate them on CIFAR-10, CIFAR-100, TinyImageNet, ImageNet-100, and a controlled five-cluster Gaussian dataset. Three different estimators - cosine-DV, InfoNCE, and JSD - show the same broad pattern of mutual information increasing during training.

Six plots of estimated mutual information during CIFAR-10 training. The top row compares SDMI-based methods and the bottom row compares JMI-based methods under cosine-DV, InfoNCE, and JSD estimators.
Figure 2 from the paper. All three estimators show mutual information growing during CIFAR-10 training. SDMI methods fluctuate more in early epochs; JMI methods generally rise more smoothly.

The prototypes are competitive with established methods, but the most informative result is conceptual: the amount of mutual information and the usefulness of a representation are not identical. MI measures preserved information, while a linear probe also depends on how that information is arranged geometrically. The optimization pathway therefore matters as much as the quantity being maximized.

A design map for future methods

Choose the update structureAlternating branches?

Use the SDMI lens: stop-gradient, target updates, and predictor dynamics.

orShared symmetric encoder?

Use the JMI lens: direct MI estimation, negatives, or statistical regularization.

thenShape useful geometry

Preserve information while arranging features for downstream tasks.

The result is a practical theoretical framework: begin with the information objective, choose an optimization paradigm, and treat familiar architectural devices as principled mechanisms rather than isolated tricks.

Read the research