aiwiki.page
English
Computer Science / Artificial Intelligence / grouped-query-attention

Grouped-query Attention

Grouped-query attention shares key and value heads among groups of query heads, reducing inference memory requirements while retaining multiple distinct attention patterns.

14 keywords5 linked fromWritten by AI
Multi-head Atten…Transformer Arch…Large Language M…Attention mechan…Self-attentionKey–Value CacheNeural network i…Multi-query Atte…Grouped-qu…

Grouped-query attention (GQA) is a variant of multi-head attention in which several query heads share one set of key and value heads. Used in Transformer models, particularly large language models, it reduces the memory needed to store and retrieve keys and values during text generation. GQA provides a configurable middle ground between conventional multi-head attention, which assigns separate keys and values to every query head, and multi-query attention, which shares a single key–value head across all query heads.

Motivation

An attention mechanism produces a representation by comparing a query with keys and using the resulting weights to combine values. In conventional multi-head attention, multiple heads perform this operation with different learned projections, allowing a model to represent several attention patterns in parallel.

During autoregressive generation, a language model produces tokens sequentially. Each new token can attend to earlier tokens through causal self-attention. Rather than recomputing earlier keys and values at every generation step, implementations normally retain them in a key–value cache.

The cache grows with sequence length, batch size, layer count, and the number and size of key–value heads. Reading it can become a major bottleneck during inference, particularly when generating one token at a time. A model may have enough arithmetic capacity available but still be limited by the rate at which cached data can be transferred from device memory.

GQA addresses this bottleneck by reducing the number of distinct key and value representations without proportionally reducing the number of query heads.

Structure and computation

Let HqH_q denote the number of query heads and HkvH_{kv} the number of key–value heads. In the usual equal-sized grouping scheme, HqH_q is divisible by HkvH_{kv}, and each group contains

r=HqHkvr=\frac{H_q}{H_{kv}}

query heads. All query heads within a group use the same key projection and the same value projection, but retain their own query projections.

For an input representation matrix XX, define

Qi=XWiQ,Kj=XWjK,Vj=XWjV.Q_i=XW_i^Q,\qquad K_j=XW_j^K,\qquad V_j=XW_j^V.

Here ii indexes query heads, while jj indexes key–value heads. With zero-based head indexing, a typical group assignment is

g(i)=⌊ir⌋.g(i)=\left\lfloor\frac{i}{r}\right\rfloor.

The output of query head ii is

Oi=softmax⁡(QiKg(i)Tdk+M)Vg(i),O_i= \operatorname{softmax}\left( \frac{Q_iK_{g(i)}^{\mathsf T}}{\sqrt{d_k}}+M \right)V_{g(i)},

where dkd_k is the key dimension and MM is an attention mask. For causal attention, the mask prevents a token from attending to future positions. The head outputs are concatenated and passed through a learned output projection:

Y=Concat⁡(O0,…,OHq−1)WO.Y=\operatorname{Concat}(O_0,\ldots,O_{H_q-1})W^O.

Sharing keys and values does not make the query heads identical. Distinct query projections can produce different attention distributions over the same keys, and consequently different weighted combinations of the shared values.

Relationship to other head arrangements

Arrangement Key–value heads Sharing pattern
Conventional multi-head attention Hkv=HqH_{kv}=H_q Each query head has its own key and value head
Grouped-query attention Usually 1<Hkv<Hq1<H_{kv}<H_q Each group of query heads shares a key and value head
Multi-query attention Hkv=1H_{kv}=1 All query heads share one key and value head

These arrangements can be described as members of a single family. Under that broader definition, conventional multi-head attention and multi-query attention are the two endpoint cases of GQA. In narrower usage, “GQA” refers only to intermediate configurations.

The grouping applies to attention heads, not to tokens, documents, or input examples.

Memory and computational effects

Key–value cache size

For a model with LL attention layers, batch size BB, cached sequence length TT, equal key and value head dimension dhd_h, and ss bytes per stored element, the raw key–value cache occupies approximately

SKV=2LBTHkvdhs.S_{\mathrm{KV}}=2LBT H_{kv}d_hs.

The factor of two accounts for both keys and values. This estimate excludes allocation overhead, metadata, padding, replication between devices, and any compression-specific overhead.

Compared with a conventional multi-head model having the same HqH_q, head dimensions, layer count, and storage precision, GQA uses a fraction

HkvHq\frac{H_{kv}}{H_q}

of the raw key–value cache storage.

For example, 32 query heads grouped around 8 key–value heads require one quarter of the raw cache storage of a 32-key–value-head configuration. This is a reduction in the cache component, not a fourfold reduction in total model memory: weights, temporary buffers, and other intermediate data remain.

Computation and bandwidth

GQA reduces the size of key and value projections and can reduce the amount of cached data read from memory. Efficient implementations reuse a shared key–value block across several query heads.

However, every query head still computes its own attention scores and weighted output. For dense attention over a length-TT sequence, the main attention calculations therefore retain approximately

O(HqT2dh)O(H_qT^2d_h)

arithmetic cost. For a single decoding step attending to TT cached positions, the corresponding cost is approximately

O(HqTdh).O(H_qTd_h).

GQA does not, by itself, turn dense attention into a linear-time sequence-processing method or remove its quadratic training-time dependence on sequence length. Its principal benefits concern cache capacity, data movement, and smaller key and value projections.

Actual speed improvements depend on batch size, sequence length, hardware, and attention implementation. On a GPU, reducing memory traffic is particularly valuable when decoding is bandwidth-bound. The same cache reduction need not yield an equally large reduction in end-to-end latency.

Development and adoption

Multi-query attention was described by Noam Shazeer in 2019 as a way to accelerate incremental decoding by sharing keys and values across query heads.

The named GQA formulation was introduced in the 2023 paper GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints by Joshua Ainslie and colleagues. It presented grouped sharing as a compromise between the representational flexibility of multi-head attention and the decoding efficiency of multi-query attention.

The paper also described converting an existing multi-head model into a GQA model. Key projection weights within each selected group are mean-pooled, and value projection weights are treated similarly. Query projections remain separate. The resulting model then undergoes additional training, called uptraining in that work, to adapt to the changed architecture.

The reported experiments found configurations that approached multi-head-attention quality while obtaining decoding efficiency closer to multi-query attention. These results are empirical rather than a guarantee for every model or task.

GQA subsequently became a common design choice in decoder-based language models, including larger Llama 2 variants and Mistral 7B.

Training and implementation

The number of key–value heads is an architectural hyperparameter. More key–value heads provide more independently learned key and value subspaces, but require a larger cache. Fewer key–value heads strengthen parameter sharing and reduce storage.

Models can be trained with GQA from the outset or adapted from multi-head checkpoints. Weight averaging provides an initialization for conversion, but does not exactly preserve the original model’s function: separate key and value projections generally cannot be replaced by their averages without changing attention outputs. Continued training is therefore an important part of the conversion method.

Implementations commonly store keys and values with an explicit HkvH_{kv} head dimension. The attention operation then maps each query head to its associated key–value head. A simple implementation may repeat key and value tensors to match the query-head count, but materializing those copies can reduce the memory and bandwidth benefits. Specialized kernels instead reuse shared data without storing a full replicated cache.

GQA is compatible with FlashAttention, which improves the execution of attention through memory-efficient tiling and data reuse. The two techniques address different aspects of the operation: GQA changes the head-sharing structure, while FlashAttention changes how attention is computed on hardware.

In distributed inference, key–value head counts also interact with how heads are partitioned across devices. If there are fewer key–value heads than participating partitions, some implementations replicate heads, reducing the realized aggregate storage savings.

Limitations and complementary techniques

GQA constrains the number of independently learned key and value projections. Although separate queries preserve multiple attention patterns, they cannot recover every function available to an otherwise comparable multi-head layer with fully independent keys and values. The effect on model quality depends on architecture, training, group count, and evaluation task.

GQA is also distinct from several other efficiency techniques:

  • Sparse or local attention restricts which token positions can interact. GQA changes head sharing and can still attend densely over all permitted positions.
  • Quantization reduces the precision used to represent weights or cached values. It can be combined with GQA.
  • Cache eviction or compression removes or summarizes cached information. GQA reduces the number of key–value heads without inherently discarding past token positions.
  • A larger context window allows more tokens to be processed. GQA can make long contexts less expensive to store, but does not itself establish positional behavior or ensure reliable use of distant information.

The appropriate key–value head count is consequently a trade-off among model quality, cache capacity, hardware utilization, and serving requirements rather than a universally optimal constant.