M30: Feedforward Neural Networks
CSCI 3151 — Foundations of Machine Learning
Biological neurons
A biological neuron (very roughly):
- Receives signals on dendrites.
- Integrates them in the cell body (soma).
- If total input passes a threshold, fires an action potential down the axon to other neurons.
- Learning happens in the adjustment of neurotransmitters that connect our neurons
As a classifier (binary case)
- Interpret \(x\) as a score for class \(1\) vs class \(0\).
- With a step activation, we predict \[ \hat{y} = \begin{cases} 1 & \text{if } x \ge 0\\[4pt] 0 & \text{otherwise.} \end{cases} \]
- This is not differentiable
- Geometrically in \(\mathbb{R}^M\):
- A perceptron is a linear classifier.
- The decision boundary is the hyperplane \(\{\mathbf{a} : \mathbf{w}^\top \mathbf{a} + w_0 = 0\}\).
Activation functions
For differentiable training (gradient-based learning), smooth alternatives became popular:
- Sigmoid (logistic): \[ g(x) = \sigma(x) = \frac{1}{1 + e^{-x}} \]
- Hyperbolic tangent: \[ g(x) = \tanh(x) \] which squashes to \((-1, 1)\) instead of \((0, 1)\).
Rectified linear units (ReLUs) and softplus
Later, rectified linear units (ReLUs) (Glorot, Bordes, and Bengio 2011) became the default for many deep nets:
ReLU: \[ g(x) = \max (0, x) \]
Its smooth cousin, softplus: \[ g(x) = \log\left(1+e^x\right) \]
… and the XOR limitation
But there is a fundamental limitation (Minsky and Papert 1969):
- A single perceptron is a linear classifier.
- It cannot represent decision boundaries that are not linearly separable.
Classic counterexample: XOR
- Inputs: \((a_1, a_2) \in \{0, 1\}^2\).
- Labels: \(y = 1\) iff exactly one of \(a_1, a_2\) is \(1\).
- No straight line in the \((a_1, a_2)\) plane can perfectly separate the \(y=1\) and \(y=0\) points.
This sets up the next step:
- To solve XOR (and many real-world problems), we need hidden layers and compositions of nonlinearities.
- That is the jump from single-layer perceptrons to multilayer neural networks.
Feedforward Neural Networks: Big Picture
🧠 Mental picture
- Inputs: feature vector \(x\)
- 🔑 Hidden layers:
- compute intermediate representations
- Output layer:
- returns logits / probabilities / predictions
Feedforward = information flows one way; no loops, no recurrence
🌙 Example 1: Moons — Linear vs MLP
We’ll revisit the classic two-moons dataset:
- 2D features, binary labels
- Nonlinear boundary needed for good performance
Code
import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split
X, y = make_moons(n_samples=500, noise=0.25, random_state=3151)
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.3, random_state=3151, stratify=y
)
fig, ax = plt.subplots()
scatter = ax.scatter(X_train[:, 0], X_train[:, 1], c=y_train, alpha=0.7)
ax.set_xlabel("x1")
ax.set_ylabel("x2")
ax.set_title("Two-moons training data")
plt.tight_layout()
🌙 Moons: Logistic Regression Baseline
Code
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score
logreg = LogisticRegression()
logreg.fit(X_train, y_train)
y_train_pred = logreg.predict(X_train)
y_test_pred = logreg.predict(X_test)
print("Logistic regression accuracy:")
print(f" train: {accuracy_score(y_train, y_train_pred):.3f}")
print(f" test: {accuracy_score(y_test, y_test_pred):.3f}")
# Plot decision boundary
xx, yy = np.meshgrid(
np.linspace(X[:, 0].min() - 0.5, X[:, 0].max() + 0.5, 200),
np.linspace(X[:, 1].min() - 0.5, X[:, 1].max() + 0.5, 200),
)
grid = np.c_[xx.ravel(), yy.ravel()]
probs = logreg.predict_proba(grid)[:, 1].reshape(xx.shape)
fig, ax = plt.subplots()
cs = ax.contourf(xx, yy, probs, levels=20, alpha=0.6)
scatter = ax.scatter(X_train[:, 0], X_train[:, 1], c=y_train, edgecolor="k", s=20)
ax.set_title("Logistic regression decision boundary")
ax.set_xlabel("x1")
ax.set_ylabel("x2")
plt.tight_layout()Logistic regression accuracy:
train: 0.860
test: 0.847
- Expect a roughly straight-ish separating boundary.
- Likely underfits the curved moons.
💯 Digits: Data & Baseline
Code
from sklearn.datasets import load_digits
digits = load_digits()
X = digits.data # shape (n_samples, 64)
y = digits.target # labels 0–9
print("Digits shape:", X.shape)
# Quick visual of a few digits
fig, axes = plt.subplots(2, 5, figsize=(6, 3))
for i, ax in enumerate(axes.ravel()):
ax.imshow(digits.images[i], cmap="gray")
ax.set_title(int(y[i]))
ax.axis("off")
plt.tight_layout()Digits shape: (1797, 64)
Confusion Matrix on Digits
Note
- Look for:
- Systematic confusions (e.g., 3 vs 5)
- Classes that are much worse than others
