FrontiersMind
Get in touch
Blog

Grouped Value Attention

Store the values. Rebuild the keys. The decode cache comes out about half of grouped-query attention, and the benchmarks stay with it.

Diagram comparing multi-head attention, grouped-query attention, multi-head latent attention, and grouped value attention by what stays in the KV cache
What each attention variant keeps in the KV cache at decode time.

So here's the thing about running large language models at scale: the attention cache is quietly eating your memory budget alive.

Every time a transformer generates a token, it needs to look back at everything it's seen so far. To avoid recomputing all of that from scratch, we store what are called keys and values - the KV cache. Sounds fine until you realize this thing grows linearly with your context length. Long conversation? Big cache. Thousands of users? Multiply that by however many concurrent sessions you're running. It adds up fast.

The standard fix people reach for is grouped-query attention, or GQA. Instead of storing a separate key and value for every attention head, you share them across groups. Llama 2 uses it. Most modern models do. It works. But here's what bugged us: you're still storing both a key and a value at every step. Two streams, always.

What if you only needed one?

The idea

The Core Idea

Here's the insight that kicked this off: keys and values aren't really independent. They both come from the same underlying token. Values carry the actual content that gets aggregated into the output. Keys are what you use to decide how much to attend to each position.

So what if, instead of caching keys, you just... reconstructed them from the values you already have?

That's GVA - Grouped Value Attention. You cache only the grouped values. Then, at inference time, you reconstruct a key for each query head using a small learned linear map: \(K = VM\). Here \(M\) is that projection.

And here's the neat part - because that map is fixed at inference time, you can absorb it into the query. You never have to materialize the key tensor at all.

The math works out to this: instead of \(q \cdot k\), you're doing \((q M^{\top}) \cdot v\). Same result. The content-key cache is gone.

The figure above shows how this compares to existing approaches.

The identity

The Math, Actually Explained

Here's why this works.
At decode time, the score between query head \(h\) and cached position \(j\) is normally \(q \cdot k^{\top}\). With GVA the reconstructed key is the cached value read through \(M_h\), and the product expands exactly:

\[ q_h k_{j,h}^{\top} = q_h \bigl(v_{j,g(h)} M_h\bigr)^{\top} = \bigl(q_h M_h^{\top}\bigr)\, v_{j,g(h)}^{\top}. \]

This is exact - no approximation. \(M_h\) is fixed at inference, so it folds into the query.
Define \(\tilde{q}_h = q_h M_h^{\top}\), compute it once per token, and attention reads only the value cache.
No content-key tensor is materialized.

\[ \tilde{q}_h = q_h M_h^{\top}, \qquad \tilde{q}_h \, v_{j,g(h)}^{\top} = q_h k_{j,h}^{\top}. \]

In the decode loop you transform the query once, then score \(\tilde{q} \cdot v\) directly. The intended persistent cache per layer is the grouped values plus one shared positional key of width \(d_r\):

\[ N_{\mathrm{GQA}} = 2\, T\, G\, d_h, \qquad N_{\mathrm{GVA}} = T\, G\, d_h + T\, d_r, \]
\[ \frac{N_{\mathrm{GVA}}}{N_{\mathrm{GQA}}} = \frac{1}{2} + \frac{d_r}{2\, G\, d_h}. \]

The first term is the grouped value cache. The second is that short positional key — \(d_r\) scalars per token, shared across heads, not \(G\) of them. Position is the part that does not fold into \(M\). For our configuration, \(d_r = 16\), \(G = 4\), \(d_h = 64\):

\[ \frac{1}{2} + \frac{16}{2 \cdot 4 \cdot 64} = \frac{1}{2} + \frac{16}{512} = 0.53125. \]

The ratio is about 0.53, so the cache is about 47% smaller than matched GQA. At \(d_r = 24\) the same grouping lands near 0.55, about 45% smaller.

Position

But Wait: What About Positional Encoding?

This is where it gets tricky. Standard RoPE (the positional encoding most modern models use) breaks the absorption trick. The rotation between a query and a cached key depends on their positions, so you can't fold it into a single transformed query that works across all cached positions.

The fix: borrow the decoupled RoPE design from DeepSeek's MLA. Split each head into two slices - an unrotated content slice (which absorbs cleanly) and a short rotated positional slice that gets cached separately and shared across heads. You keep position information without blowing up your cache.

That shared positional key only adds \(d_r\) scalars per token - not \(G \times d_r\) - so it's cheap.

The measurements

Does It Actually Work?

We trained 350M-parameter models on 30B tokens of FineWeb-Edu and compared GVA against GQA and MLA across five benchmarks: HellaSwag, WinoGrande, OpenBookQA, and both ARC splits.

The short answer: yeah, pretty much.

GVA with decoupled RoPE (\(d_r = 16\)) hits 44.35 average accuracy vs GQA's 44.36. That's a gap of 0.01 points — within seed variation noise. MLA sits at 43.88. Meanwhile, the intended cache is 45–47% smaller than GQA.

Zero-shot accuracy 350M parameters, 30B FineWeb-Edu tokens. Each cell is the mean of three seeds. Average is the unweighted mean of the five tasks. Scroll the table sideways for the other tasks.
Accuracy in percent from Table 2 of the Grouped Value Attention paper. The starred row is GVA with decoupled RoPE at dr = 16.
Method HellaSwag WinoGrande OBQA ARC-E ARC-C Avg.
GQA (baseline) 43.41 52.48 33.40 63.42 29.09 44.36
MLA 43.20 51.61 34.60 62.87 27.13 43.88
GVA baseline 42.12 52.96 34.80 61.95 27.73 43.91
GVA, scale-matched + Q-norm 42.05 53.51 34.40 62.75 29.35 44.41
GVA + DRoPE dr=16 ✦ 42.69 53.35 33.60 63.81 28.32 44.35
GVA + DRoPE dr=24 42.94 53.12 33.20 63.69 28.50 44.29

Bold is the best score in that column. ✦ is the serving configuration: decoupled RoPE with \(d_r = 16\). Numbers are Table 2 of arXiv:2609.13285.

One thing we learned the hard way: initialization matters more than you'd expect. If you just default-initialize that reconstruction map \(M\), your keys start way below queries in scale — and you spend the first chunk of training just digging out of that hole. The fix is to initialize \(M\) so keys and queries start at the same RMS. Draw each entry from a zero-mean Gaussian with that match built in, where \(d_{\mathrm{in}} = d_h\) is the value width and \(\sigma_Q\) is measured after any query normalization:

\[ M_{ij} \sim \mathcal{N}\!\left(0,\; \frac{\sigma_Q^{2}}{\sigma_V^{2}\, d_{\mathrm{in}}}\right). \]

The same rule applies per slice. The fan-in of \(M\) stays the value width, and the shared positional projection is initialized so \(k^{\mathrm{rope}}\) starts at the RMS of \(q^{\mathrm{rope}}\). After that, GVA tracks GQA's loss curve pretty cleanly.

What's next

We're Open-Sourcing the Kernels Soon

We've built custom decode kernels for GVA and are currently running end-to-end inference evaluations. Fused decode throughput, peak serving memory, batch capacity - the full picture. Once that's done, everything goes public. Open-source release is coming soon.

The Takeaway

GVA isn't trying to beat GQA on quality. It matches it. The win is cache — roughly half the footprint for the same downstream accuracy, by leaning on the relationship between keys and values that was always there, just unused.

If you're serving long contexts at scale, that 45–47% reduction in cache reads is the number that matters.

Read the full paper: Grouped Value Attention: Efficient KV Caching via On-Demand Key Reconstruction — arXiv:2609.13285, 2026.