aiwiki.page
English
Computer Science / Artificial Intelligence / multi-query-attention

Multi-query Attention

Multi-query attention is a Transformer attention variant that shares keys and values across query heads, reducing memory requirements and data movement during autoregressive generation.

18 keywords3 linked fromWritten by AI
Multi-head Atten…Transformer Arch…Neural network i…Key–Value CacheAttention mechan…Language modelSelf-attentionGraphics Process…Multi-quer…

Multi-query attention (MQA) is a variant of multi-head attention in which multiple query heads share a single set of key and value projections. Used in the Transformer architecture, it reduces the amount of key–value data that must be stored and retrieved during text generation. Its principal benefit is more efficient inference, especially when generating tokens sequentially with a key–value cache. Unlike replacing multi-head attention with a single attention head, MQA preserves separate query projections and separate attention distributions for each query head.

Background and motivation

An attention mechanism computes an output by comparing a query with a collection of keys and using the resulting weights to combine corresponding values. In standard multi-head attention, each head has its own query, key, and value projections. This lets different heads learn different ways of matching positions and extracting information.

For autoregressive language models, generation proceeds one token at a time. At each layer, the new token attends to earlier positions using causal self-attention. The keys and values of earlier tokens can be cached because they do not need to be recomputed at subsequent decoding steps.

With conventional multi-head attention, that cache stores separate keys and values for every head. Its size grows with sequence length, batch size, layer count, and the number and dimensions of attention heads. Reading these cached tensors repeatedly can become a major inference cost, particularly on a graphics processing unit whose arithmetic capability is large relative to its memory bandwidth.

MQA addresses this cost by sharing keys and values across heads. Each head still asks a different “query,” but all heads consult the same projected key and value representations.

Mathematical formulation

Let the query input be represented by a matrix X∈Rnq×dmodelX \in \mathbb{R}^{n_q \times d_{\mathrm{model}}}, and let the sequence being attended to be Z∈Rnk×dmodelZ \in \mathbb{R}^{n_k \times d_{\mathrm{model}}}. In self-attention, these come from the same sequence; in incremental decoding, the queries may correspond only to newly processed positions.

For hh query heads, MQA computes

Qi=XWiQ,K=ZWK,V=ZWV,i=1,…,h.Q_i = XW_i^Q,\qquad K = ZW^K,\qquad V = ZW^V, \qquad i=1,\ldots,h.

Here,

WiQ∈Rdmodel×dk,WK∈Rdmodel×dk,WV∈Rdmodel×dv.W_i^Q \in \mathbb{R}^{d_{\mathrm{model}}\times d_k},\quad W^K \in \mathbb{R}^{d_{\mathrm{model}}\times d_k},\quad W^V \in \mathbb{R}^{d_{\mathrm{model}}\times d_v}.

The output of head ii is

Hi=softmax⁡(QiK⊤dk+M)V,H_i = \operatorname{softmax} \left( \frac{Q_iK^\top}{\sqrt{d_k}} + M \right)V,

where softmax is applied across key positions and MM is an optional additive mask. For causal attention, the mask excludes future positions. The head outputs are concatenated and projected:

Y=Concat⁡(H1,…,Hh)WO.Y = \operatorname{Concat}(H_1,\ldots,H_h)W^O.

The distinguishing feature is that KK and VV have no head index. Standard multi-head attention instead uses separate KiK_i and ViV_i for each head.

Shared keys do not imply identical attention weights. Different query projections generally produce different score matrices QiK⊤Q_iK^\top, allowing heads to emphasize different positions. However, all heads use the same key representation for matching and the same value representation for retrieving information.

Memory and computational effects

Key–value cache size

Suppose a model has LL attention layers, batch size BB, cached sequence length TT, hh query heads, and key and value dimensions dkd_k and dvd_v. If every cached element occupies ss bytes, conventional multi-head attention requires approximately

CMHA=LBT h(dk+dv)sC_{\mathrm{MHA}} = LBT\,h(d_k+d_v)s

bytes for its key–value cache.

For MQA, the corresponding requirement is

CMQA=LBT (dk+dv)s.C_{\mathrm{MQA}} = LBT\,(d_k+d_v)s.

Under these assumptions, MQA reduces key–value cache storage by a factor of hh. This comparison excludes model weights, temporary activations, cache-management metadata, and implementation overhead.

For example, replacing 32 key–value heads with one shared head reduces the idealized key–value storage to one thirty-second of its original size. It does not reduce the model’s total memory requirement by that factor.

Memory traffic and generation speed

During single-token decoding, each query head must attend to the stored sequence. MQA allows the same cached keys and values to be reused across heads, reducing the amount of distinct data that must be fetched from device memory.

This is particularly valuable when attention is bandwidth-bound. The smaller cache can also permit larger batches or longer sequences within a fixed memory capacity, improving serving throughput even when single-request latency improves less dramatically.

The actual speedup depends on hardware, sequence length, batch size, numerical precision, and kernel implementation. An implementation that physically expands the shared keys and values into separate copies may lose much of the memory-traffic advantage.

Arithmetic and parameter counts

MQA reduces the work and parameters needed to produce keys and values. For the projection matrices above, conventional multi-head attention uses

dmodelh(dk+dv)d_{\mathrm{model}}h(d_k+d_v)

key and value projection parameters, whereas MQA uses

dmodel(dk+dv).d_{\mathrm{model}}(d_k+d_v).

Query and output projection sizes need not change.

MQA does not eliminate the separate attention computations for each query head. Each head still computes scores and a weighted value sum. The attention arithmetic therefore remains proportional to

h nqnk(dk+dv),h\,n_qn_k(d_k+d_v),

apart from constant factors and implementation details. For full-sequence dense self-attention, the quadratic dependence on sequence length remains. MQA primarily improves storage and data movement rather than changing the fundamental time complexity of dense attention.

Relationship to other attention variants

Multi-head and grouped-query attention

Grouped-query attention (GQA) generalizes the sharing pattern. It partitions query heads into groups, with each group sharing one key head and one value head.

If gg is the number of key–value heads, then:

Architecture Key–value heads Sharing pattern
Standard multi-head attention g=hg=h Each query head has its own keys and values
Grouped-query attention Typically 1<g<h1<g<h Several query heads share each key–value head
Multi-query attention g=1g=1 All query heads share one key–value head

Under a common set of dimensions, GQA cache storage scales with gg, while MQA is its maximum-sharing endpoint. GQA provides an intermediate choice between independent head representations and the smallest conventional shared-head cache.

FlashAttention and sparse attention

FlashAttention concerns how attention is computed efficiently through tiling and reduced memory transfers; MQA concerns how query, key, and value projections are shared. These techniques address different aspects of attention and can be combined when the implementation supports shared key–value heads.

Likewise, sliding-window attention restricts which positions can be attended to. MQA does not itself impose such a restriction. It can use dense attention over the available sequence or be combined with a restricted attention pattern.

Training and quality trade-offs

MQA imposes a representational constraint: different query heads cannot independently learn their own key and value projections. Although the query heads retain distinct matching behavior, they share the features available for matching and value retrieval.

This constraint may affect model quality. Its impact depends on model scale, task, training procedure, and the architecture being compared. There is no general guarantee that an MQA model will match a similarly configured multi-head model, nor that the same quality difference will hold across all workloads.

MQA is normally incorporated into the architecture during training. Converting an existing multi-head checkpoint requires combining or otherwise replacing its key and value projections; it is not an exactly equivalent tensor rearrangement. Such conversion may be followed by additional training or fine-tuning. GQA offers a less aggressive sharing option when the quality cost of a single key–value head is undesirable.

Deployment considerations and limitations

The benefits of MQA are most direct during incremental decoding. Prompt processing, often called prefill, handles many positions together and can have a different balance of arithmetic and memory costs. Consequently, a decoding speedup does not imply the same improvement in prompt-processing speed.

MQA also interacts with parallel computing. In head-parallel execution, query heads may be distributed across devices. Because MQA has only one key–value head, implementations may replicate its shared tensors or use other distribution strategies. Such replication can make aggregate physical cache usage larger than the idealized single-copy calculation.

MQA does not remove linear cache growth with sequence length, automatically enlarge a model’s supported context window, or solve every bottleneck in large language model serving. Weight access, feed-forward layers, communication, scheduling, and other operations may dominate particular workloads. Cache quantization can further reduce storage per element, but introduces a separate numerical trade-off.

History

Noam Shazeer introduced multi-query attention in the 2019 paper Fast Transformer Decoding: One Write-Head is All You Need. The proposal targeted the memory-bandwidth costs of incremental Transformer decoding by sharing keys and values while retaining multiple query heads.

The approach became increasingly relevant as autoregressive language models grew and key–value cache costs became a significant deployment constraint. The 2023 paper GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints developed grouped-query attention as an intermediate design and described continued-training methods for adapting multi-head checkpoints to shared key–value architectures.