Language models are injective, so a KV-cache is not a summary — it's the prompt, in another basis

Every individual piece of a transformer is lossy. LayerNorm throws away a per-token scale and shift. Softmax attention collapses many key-value pairs into one weighted average. Low-rank projections shrink dimensions on purpose. It would be reasonable to conclude that the hidden state a transformer produces for a prompt is a compressed, irreversible summary of that prompt — the way a hash or a JPEG is.

Nikolaou, Mencattini, Crisostomi, Santilli, Panagakis, and Rodolà prove that conclusion is wrong. Decoder-only transformer language models, as a whole, are almost surely injective: two different prompts essentially never produce the same last-token hidden state. Not "usually don't" — provably, with probability one, for any model trained by gradient descent for any finite number of steps. And they don't stop at the proof. SipIt is an algorithm that exploits injectivity to reconstruct a prompt exactly from its hidden states, in time linear in the prompt's length, with 100% accuracy in their tests. The paper is ten months old, already accepted at ICLR 2026, and has picked up roughly 30 citations (7 flagged "influential") in that time — fast uptake for a theory paper.

The result has an immediate consequence worth sitting with: a hidden state, or a cache built from one, is not a lossy fingerprint of what a user typed. It is that text, in a different basis, and it can be read back out.

Non-injective parts, injective whole

The paper's move is to stop looking at individual components and instead look at the whole map. Write $f: \mathcal{V}^{\le K} \times \mathbb{R}^p \to \mathbb{R}^d$ for the model: a vocabulary $\mathcal{V}$, a context bound $K$, parameters $\theta \in \mathbb{R}^p$, and $r(s;\theta)$ for the last-token hidden state of prompt $s$. The claim is that for any two distinct prompts $s \ne s'$,

The argument runs through real-analyticity. Embedding lookups, affine projections, softmax, LayerNorm with $\varepsilon > 0$, and every real-analytic activation in common use (GELU, SiLU, SwiGLU, GeGLU) are all real-analytic functions of their inputs and parameters. Real-analytic functions are closed under addition, multiplication, division away from poles, and composition — so the entire network, prompt fixed, is a real-analytic function of $\theta$.

That matters because of a classical dichotomy. Fix two prompts $s \ne s'$ and define $h(\theta) = \lVert r(s;\theta) - r(s';\theta) \rVert^2$. Because $h$ is real-analytic, exactly one of two things is true: either $h \equiv 0$ everywhere, or the zero set $\{\theta : h(\theta) = 0\}$ has Lebesgue measure zero — a thin, lower-dimensional slice of parameter space, not a region with any volume. The paper rules out $h \equiv 0$ by hand: it exhibits one concrete $\theta$ where $s$ and $s'$ provably map to different states (for instance, freeze the network down to embeddings plus positions and point at two distinct rows). So the collision set for that pair is measure zero.

That picture is the whole argument. Any initializer with a continuous density — Gaussian, uniform, Xavier — places exactly zero probability mass on a measure-zero set, so at initialization the odds of landing on the collision curve are zero. Training doesn't change that: a gradient step $\varphi(\theta) = \theta - \eta \nabla L(\theta)$ is itself real-analytic, so its Jacobian determinant $\det D\varphi(\theta)$ is real-analytic and not identically zero, which makes $\{\theta : \det D\varphi = 0\}$ measure zero too. Away from that set, the inverse function theorem says $\varphi$ is a local diffeomorphism, and a diffeomorphism cannot squash a positive-volume region down onto a lower-dimensional set. Push an absolutely-continuous parameter distribution through enough of these steps — full-batch, mini-batch, or even adversarially chosen batches — and it stays absolutely continuous. Induct over any finite training horizon $T$ and injectivity holds with probability one after training, not only at init. The same argument extends to any finite set of prompts being pairwise distinct simultaneously, not just one pair at a time.

The theorem is explicit about how a collision would have to happen: two vocabulary items given exactly identical embedding rows, or two positional encodings set exactly equal by hand while everything else is tuned to suppress positional signal. Both are measure-zero, hand-engineered pathologies — never reached by continuous initialization plus gradient descent, but not physically impossible if someone builds them on purpose. That is exactly what "hand-engineered" does in the picture above.

What "almost surely" does and doesn't buy you

This is worth being precise about, because it is the obvious objection. "Probability one" is a statement about a distribution over parameters — the set of continuous initializers, pushed through finitely many gradient steps. It is not a certificate stamped on any one specific, already-trained, already-quantized checkpoint sitting on a GPU. A model built by deliberately engineering a collision (identical embedding rows, say) would sit exactly on the measure-zero set and would not be covered by the "almost surely" — the theorem says that model is vanishingly unlikely to arise by chance, not that no such model can exist.

The empirical side is where finite precision gets tested directly. Table 2 measures the minimum pairwise distance at the final layer under FP4, INT8, and FP32 for three models — for Llama-3.1-8B: 2.281 (FP4) · 6.597 (INT8) · 1.274 (FP32). Quantization didn't shrink the separation margin in these tests; if anything the coarser formats measured larger minimum distances. That's reassuring, but it's a different computational object than the real-analytic map the theorem is stated over, checked empirically rather than derived from the proof.

Zero collisions, at a scale that matters

expand

A separate run — 100,000 prompts sampled from Wikipedia, C4, The Pile, and GitHub Python, roughly 5 billion pairwise comparisons — measured the minimum pairwise distance across four different models at three depths (layer 1, the middle layer, the last layer), against a collision threshold of $10^{-6}$:

Zero collisions, and the separation grows by roughly two to three orders of magnitude from the first layer to the last — consistent with the boxplot above. The closest pairs the authors found anywhere, on manual inspection, were near-duplicate code and documentation snippets differing only by a trailing newline token — still far above the threshold.

Then the authors went looking on purpose. They took the ten closest prompts in their sample and appended every vocabulary token as a one-token continuation to each, an exhaustive collision hunt rather than a random sample: over 343 billion prompt pairs per model. Zero collisions, on both GPT-2 Small and Gemma3-1B. That is the number in the abstract, and it is worth being precise about its scope: it is the most exhaustive test in the paper, and it ran on two of the smaller models tested. The bigger models — Phi-4 (14B) and Llama-3.1-70B — were checked under the sampled protocol above (Table 3), not the exhaustive one; injectivity at 70B+ scale rests on the same theorem plus a smaller, sampled empirical check, not the 343-billion-pair stress test.

expand

Two more things the tests deliberately probed, both reported candidly rather than cherry-picked: separation does not shrink as prompts get longer (above), and — a genuinely counter-intuitive result the authors report without softening it — inverting random, out-of-distribution token sequences is faster than inverting natural language (146s vs. 107s mean, GPT-2, 100-token prompts). Their read: natural-language hidden states sit on a more structured, clustered manifold, which is flatter and worse-conditioned for the gradient-guided search described next; OOD states are more dispersed, giving sharper gradients to follow.

One distinction worth making explicit, because it's easy to blur: injectivity is a claim about the hidden state, not about what a model eventually says. Two prompts can produce the identical next-token answer — "the sum is 12", a translation landing on the same word, a completion ending in "dog" — while their hidden states remain measurably distinct underneath. The paper stress-tests exactly this: translation pairs, arithmetic pairs, and ten thousand different Wikipedia prefixes all forced to the same fixed suffix and the same output token all still show real, measurable separation at the hidden-state level, growing with depth just like everything else. Collapsing to the same output is common and expected; collapsing to the same internal state is what the theorem rules out.

SipIt: turning the proof into an algorithm

Injectivity is a static fact about the map. SipIt (Sequential Inverse Prompt via ITerative updates) is what you get when you notice the map is also causal: the hidden state at position $t$ depends only on the prefix already fixed and the token at $t$. That means the one-step map $v_j \mapsto h_t(\pi \oplus v_j)$, for a fixed correct prefix $\pi$ and candidate $v_j$ ranging over the vocabulary, is itself almost-surely injective — the same argument, run one position at a time. So the algorithm doesn't need to solve the whole sequence at once:

for t = 1..T:
  for each candidate v_j (policy: gradient-guided, or random):
    if the candidate's predicted hidden state matches h_t within tolerance ε:
      append v_j to the reconstructed prefix; move to position t+1

Correctness (Theorem 3.1): this recovers the true sequence with probability 1, in at most $T \cdot |\mathcal{V}|$ candidate checks in the worst case — linear in the prompt length $T$ for a vocabulary of fixed size, which is the "linear time" the abstract promises. Robustness (Theorem 3.2): it still recovers the exact sequence under bounded perturbation of the observed state, as long as the perturbation stays under half the minimum pairwise distance among candidate continuations at that step — which is exactly the separation margin measured above, and exactly why that margin growing with depth matters practically, not just theoretically.

In practice SipIt doesn't try candidates in vocabulary order. It uses a gradient-guided policy — clip the gradient norm to 1, periodically re-project the running estimate back to the nearest true token embedding every 50 proposals — rather than the brute-force random order its own ablation uses as a baseline. And it is explicit about its threat model: it assumes an attacker who already holds the full per-position hidden-state sequence at some layer $\ell$ — the paper's own examples are "a leaked KV-cache, a shared-inference pipeline, or an API exposing intermediate activations." Recovering a prompt from only the final embedding is asserted to be theoretically possible under the same theorem, but no efficient algorithm for it is demonstrated — that's left as future work. Everything below is about the case SipIt actually solves: someone already has the hidden states.

Also worth naming: Thomas et al. (2025), the paper's own "most closely related" citation, recovers prompts from hidden states with a similar sequential structure but without an injectivity guarantee behind it — so it has to score close to the entire vocabulary at each step before committing to a token. SipIt's early exit, explored in under a quarter of one percent of the vocabulary in these experiments, is what the guarantee buys on top of the same basic idea.

How fast, and how much of the vocabulary

On 100 prompts (90% real sentences, 10% random tokens), 20 tokens each, GPT-2 Small:

HardPrompts — the standard gradient-based approximate prompt-search baseline, adapted by the authors from its original vision-language objective to a text-only one — never lands on the exact sequence: it optimizes toward a prompt, not the prompt. Brute-force random search gets there eventually, at over two minutes an average token. SipIt matches brute force's accuracy at roughly 1/140th the time, purely by trying candidates in a smarter order.

That gap holds up against real vocabularies, not just GPT-2's ~50K tokens. Under FP4 quantization, 50 prompts, 10 tokens each:

The unquantized appendix ablation lands within noise of the same numbers (0.21% and 0.22% explored respectively) — quantizing the model barely moves how much of the vocabulary SipIt has to touch. Every measurement here is a single NVIDIA A100-SXM (64GB), no custom kernels for SipIt itself — the ~28-second figure is an unoptimized, single-GPU number, not a lower bound on how fast this can go.

The consequence: a KV-cache is the prompt, in another basis

Put the two halves together. The hidden state is provably (almost surely) a lossless encoding of the prompt that produced it, and there is a linear-time algorithm that decodes it back exactly, needing only a sliver of the vocabulary and no training data of its own. That means the sentence "the model doesn't store your prompt, it just computes with it" is not true in the way people mean it. The computation is the storage. A hidden state is not a fingerprint or a hash of the input — it's the input, run through an invertible function.

This is precisely what a KV-cache is built from. Every serving stack that skips recomputing attention for tokens it has already seen — prefix caching in vLLM and SGLang, multi-tenant inference sharing a cache across requests, cache offload from GPU to CPU DRAM or disk when memory is tight, activation logging for debugging or evals — is, under this paper's result, holding recoverable prompt text, not an opaque compressed artifact. One nuance worth being exact about: SipIt's proven target is the residual-stream hidden state $r(s;\theta)$ itself, not the $K$ and $V$ tensors a serving stack actually caches — those are per-head linear projections of that hidden state, generically lower-dimensional per head. But stacked across every layer and every head, a full KV-cache is a far higher-dimensional linear image of exactly the same per-position hidden-state sequence the paper's own threat model names as its motivating example: "a leaked KV-cache, a shared-inference pipeline, or an API exposing intermediate activations." The paper doesn't run SipIt against raw $K$/$V$ tensors — it inverts hidden states directly — so read "the cache is invertible" as the natural extension the authors themselves point at, not a number they measured.

A concrete, current example: Kimi K3's reinforcement-learning infrastructure writes idle KV prefixes out to an external CPU DRAM pool between rollouts, so paused sandboxes stay cheap. Under this paper's result, that pool isn't holding compressed activations — it's holding recoverable prompt text, sitting outside the GPU's usual trust boundary, on a different piece of hardware entirely. That's not a criticism specific to K3 — it's the same design every prefix-cache and cache-offload system makes for the same performance reasons — it's just a live instance to point at.

And compression doesn't obviously buy you out of this: quantizing the cache, as TurboQuant does for entirely different (memory and throughput) reasons, doesn't collapse distinct prompts into a shared entry either — Table 2 above shows minimum pairwise distances at FP4/INT8 holding up or growing relative to FP32. Shrinking the cache for efficiency and erasing what's recoverable from it are different problems, and solving the first doesn't solve the second.

There's a regulatory angle here too, which the paper raises directly. The Hamburg Data Protection Commissioner argued in 2024 that a model's trained parameters aren't personal data, because training folds the data into abstract, non-retrievable representations. The authors' point is narrower and, on their result, correct as far as it goes: that argument is about parameters at rest, not about inference-time hidden states, which this paper shows are lossless, recoverable encodings of whatever a specific user typed, right now. Any system that stores or transmits those states — including as a cache — is storing something closer to the original text than "abstract representation" suggests.

(If you've read the Jacobian lens piece: that method also reads information out of a hidden state, but by linearizing the model around a corpus average — an approximation. SipIt's guarantee is exact, because it has an injectivity theorem underneath it instead of a linear approximation.)

How new is this, and what to weigh

The paper is genuinely recent — submitted October 2025 — but it isn't a fringe preprint sitting uncited. It's accepted at ICLR 2026, and by the time of writing has around 30 citations, 7 of them flagged "influential" by Semantic Scholar, which is a fast citation trajectory for a ten-month-old theory paper. Code (SIPIT) is public.

None of that undermines the core claims — billions of comparisons across multiple independent experimental setups, a working algorithm with two proven theorems behind it, and honest reporting of the results that don't flatter the paper (OOD prompts inverting faster than natural language; the biggest models tested under the less exhaustive protocol). It's the right amount of scrutiny for a result this consequential, not a reason to discount it.

The take

The intuition that hidden states are lossy comes from staring at individual layers — LayerNorm, softmax, low-rank projections — each of which really is lossy on its own. The paper's point is that lossiness doesn't compose the way that intuition assumes: the full map from prompt to last-token state is, almost surely, injective, and an algorithm exists that inverts it exactly, in linear time, using a sliver of the vocabulary. The finite-precision and hand-engineered caveats are real, and worth stating precisely rather than waving away — but they don't touch the core result, which is that a hidden state is not a summary of a prompt. It's the prompt. Anything built to store, cache, offload, or ship hidden states around — for speed, for multi-tenancy, for debugging — is, whether it says so or not, in the business of storing exact user text.

Built on Language Models are Injective and Hence Invertible (Giorgos Nikolaou, Tommaso Mencattini, Donato Crisostomi, Andrea Santilli, Yannis Panagakis, Emanuele Rodolà; EPFL / Sapienza University of Rome / University of Athens / Archimedes, Athena RC; accepted ICLR 2026), and its SipIt code release. Figures are the paper's own Figures 3 and 9, reproduced for commentary. Tables and numbers are the authors' except where marked as this site's own illustrative simplification (the SipIt walker's compressed vocabulary); interactive diagrams are mine.