Editing a language model is not only about teaching it a new fact. The system must also decide when that fact applies. A reliable editor should recognize paraphrases of the original request, yet leave a nearby-looking but unrelated prompt untouched. Projector Editor Networks for Model Editing (PENME) learns a representation space designed specifically to make that distinction.

Example from the paperThe model receives one factual edit about Pittsburgh.
Edit

“The twin city of Pittsburgh is …”

Apply the edit

“Pittsburgh is a twin city of …”

same meaning, different order
Do not apply

“The twin city of Portsmouth is …”

similar words, different fact

A useful scope is semantic rather than lexical: the paraphrase should retrieve Pittsburgh’s edit, while the Portsmouth question should follow the original model.

The hidden decision behind every edit

Weight-preserving editors store new information outside the model and use a scoping mechanism to decide when to retrieve it. A common scope is a radius in representation space. If a new prompt falls close enough to an edit key, the system activates the edited response.

The trouble is that language-model representations can be dominated by shared words. “The twin city of Portsmouth is” may sit closer to the Pittsburgh edit than the genuine paraphrase “Pittsburgh is a twin city of.” No single radius solves this geometry: a small radius misses paraphrases, while a large radius captures unrelated neighbours.

Three diagrams of edit scoping. A small radius excludes both the paraphrase and neighbour; a large radius includes both; PENME's learned space places the paraphrase inside the scope and the unrelated lexical neighbour outside.
Figure 1 from the paper. Left: a narrow threshold preserves locality but misses the paraphrase. Middle: a wide threshold generalizes but misfires on Portsmouth. Right: the learned projection changes the geometry so both goals can be met.

In the paper’s CounterFact analysis, 58% of edits are closer to an unrelated lexical neighbour than to their paraphrase in at least one examined representation setting. This is not simply a bad threshold; it is a poorly organized space for the decision the editor needs to make.

Learn the space before choosing the radius

PENME adds a compact two-layer projection network beside a feed-forward block of the frozen language model. It maps internal activations into a new space trained with three relationships:

+Edit ↔ paraphrase

Pull together prompts that express the same edited fact.

−Edit ↔ neighbour

Push away prompts that share words but refer to a different fact.

−Edit ↔ other edits

Separate stored edits so one memory entry does not trigger another.

The contrastive objective uses squared distance for positive pairs and a margin penalty for negative pairs:

Contrastive projection loss
ℒ(a,b) = ½(1−t)‖a−b‖²2 + ½t[max(0, m−‖a−b‖2)]²

Use t = 0 for an edit-paraphrase pair and t = 1 for an edit-neighbour or inter-edit pair. The learned map makes semantic equivalence close and lexical coincidence insufficient.

PENME architecture. A projection network maps the feed-forward activation into a new space. A key-value memory stores projected edit keys, thresholds, and outputs, and retrieves an edit only when the projected prompt falls inside the appropriate scope.
Figure 2 from the paper. Projected activations become keys in a memory. Each value stores the edited output and a learned scope threshold. At inference, the nearest valid key determines whether playback occurs.

Retrieval becomes an explicit rule

For a prompt pt, PENME projects an internal activation to ap, finds the nearest edit key, and applies the edit only when the distance lies inside that key’s threshold:

Scoped edit retrieval
(k*,v*) = arg min(kᵢ,vᵢ) ‖ap−ki‖2

Retrieve the associated edit only if ‖ap−k*‖2 < δ*. Otherwise, the prompt passes through the original model unchanged.

This design preserves the base model’s weights. Edits can be inserted or removed through the memory, while the projector supplies a reusable semantic coordinate system for deciding their scope.

Measuring lexical misfires directly

The authors form triplets containing an edit, a paraphrase, and a highly overlapping but unrelated neighbour. They then measure how often the neighbour is closer to the edit than the paraphrase—a direct failure condition for distance-based scoping.

Bar charts measuring lexical bias. The left chart shows the percentage of prompts where unrelated neighbours are closer than paraphrases across layers of T5-small, GPT2-XL, and Llama-2-7B. The right chart compares original and PENME-projected representations.
Figures 3 and 4 from the paper. Lexical misordering appears across model layers. After projection at layer 2, the measured failure rate falls from 46.8% to 6.4% for T5-small, from 10.0% to 2.8% for GPT2-XL, and from 11.0% to 0% for Llama-2-7B.

The projector is evaluated on held-out triplets, so the reduction reflects learned structure rather than memorization of the test prompts. The result is especially striking for Llama-2-7B: every tested paraphrase becomes closer to its edit than the lexical neighbour.

Editing must satisfy three goals together

The paper evaluates editors using three complementary metrics:

MetricQuestionFailure it catches
Edit successDoes the direct edit return the new answer?The fact was not updated
GeneralizationDo paraphrases retrieve the same edit?The scope is too narrow
LocalityDo unrelated neighbours remain unchanged?The scope is too broad

PENME is tested on CounterFact and zsRE with T5-small, GPT2-XL, and Llama-2-7B. It achieves perfect direct edit success in every reported PENME setting and maintains balanced locality and paraphrase generalization. On zsRE, the combined score reaches 0.951 for T5-small, 0.966 for GPT2-XL, and 0.984 for Llama-2-7B.

That balance distinguishes PENME from methods that optimize one side of the trade-off. A very strict radius can produce excellent locality but almost no paraphrase coverage; a permissive scope can generalize while corrupting unrelated answers. PENME improves the geometry first, making the threshold less responsible for compensating for lexical bias.

Trustworthy editing begins with trustworthy scope

Model editing is often described as changing what a model knows. PENME shows that an equally important problem is controlling where the change propagates. By separating meaning from word overlap, the editor can update a fact, recognize genuine reformulations, and protect nearby knowledge—all without altering the underlying language model.

Read the research