{
  "id": "multi-query-attention",
  "title": "Multi-query Attention",
  "lang": "en",
  "source_lang": "en",
  "summary": "Multi-query attention is a Transformer attention variant that shares keys and values across query heads, reducing memory requirements and data movement during autoregressive generation.",
  "category": "computer-science",
  "subcategory": "artificial-intelligence",
  "type": "concept",
  "aliases": [
    "MQA",
    "Multi-query attention"
  ],
  "generated_at": "2026-11-04",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/multi-query-attention/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/multi-query-attention/",
      "json": "https://aiwiki.page/zh/multi-query-attention.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "Multi-query attention (MQA) is a variant of multi-head attention in which multiple query heads share a single set of key and value projections. Used in the Transformer architecture, it reduces the amount of key–value data that must be stored and retrieved during text generation. Its principal benefit is more efficient inference, especially when generating tokens sequentially with a key–value cache. Unlike replacing multi-head attention with a single attention head, MQA preserves separate query projections and separate attention distributions for each query head."
    },
    {
      "heading": "Background and motivation",
      "text": "An attention mechanism computes an output by comparing a query with a collection of keys and using the resulting weights to combine corresponding values. In standard multi-head attention, each head has its own query, key, and value projections. This lets different heads learn different ways of matching positions and extracting information.\n\nFor autoregressive language models, generation proceeds one token at a time. At each layer, the new token attends to earlier positions using causal self-attention. The keys and values of earlier tokens can be cached because they do not need to be recomputed at subsequent decoding steps.\n\nWith conventional multi-head attention, that cache stores separate keys and values for every head. Its size grows with sequence length, batch size, layer count, and the number and dimensions of attention heads. Reading these cached tensors repeatedly can become a major inference cost, particularly on a graphics processing unit whose arithmetic capability is large relative to its memory bandwidth.\n\nMQA addresses this cost by sharing keys and values across heads. Each head still asks a different “query,” but all heads consult the same projected key and value representations."
    },
    {
      "heading": "Mathematical formulation",
      "text": "Let the query input be represented by a matrix \\(X \\in \\mathbb{R}^{n_q \\times d_{\\mathrm{model}}}\\), and let the sequence being attended to be \\(Z \\in \\mathbb{R}^{n_k \\times d_{\\mathrm{model}}}\\). In self-attention, these come from the same sequence; in incremental decoding, the queries may correspond only to newly processed positions.\n\nFor \\(h\\) query heads, MQA computes\n\n\\[Q_i = XW_i^Q,\\qquad K = ZW^K,\\qquad V = ZW^V, \\qquad i=1,\\ldots,h.\\]\n\nHere,\n\n\\[W_i^Q \\in \\mathbb{R}^{d_{\\mathrm{model}}\\times d_k},\\quad W^K \\in \\mathbb{R}^{d_{\\mathrm{model}}\\times d_k},\\quad W^V \\in \\mathbb{R}^{d_{\\mathrm{model}}\\times d_v}.\\]\n\nThe output of head \\(i\\) is\n\n\\[H_i = \\operatorname{softmax} \\left( \\frac{Q_iK^\\top}{\\sqrt{d_k}} + M \\right)V,\\]\n\nwhere softmax is applied across key positions and \\(M\\) is an optional additive mask. For causal attention, the mask excludes future positions. The head outputs are concatenated and projected:\n\n\\[Y = \\operatorname{Concat}(H_1,\\ldots,H_h)W^O.\\]\n\nThe distinguishing feature is that \\(K\\) and \\(V\\) have no head index. Standard multi-head attention instead uses separate \\(K_i\\) and \\(V_i\\) for each head.\n\nShared keys do not imply identical attention weights. Different query projections generally produce different score matrices \\(Q_iK^\\top\\), allowing heads to emphasize different positions. However, all heads use the same key representation for matching and the same value representation for retrieving information."
    },
    {
      "heading": "Memory and computational effects",
      "text": "Key–value cache size\n\nSuppose a model has \\(L\\) attention layers, batch size \\(B\\), cached sequence length \\(T\\), \\(h\\) query heads, and key and value dimensions \\(d_k\\) and \\(d_v\\). If every cached element occupies \\(s\\) bytes, conventional multi-head attention requires approximately\n\n\\[C_{\\mathrm{MHA}} = LBT\\,h(d_k+d_v)s\\]\n\nbytes for its key–value cache.\n\nFor MQA, the corresponding requirement is\n\n\\[C_{\\mathrm{MQA}} = LBT\\,(d_k+d_v)s.\\]\n\nUnder these assumptions, MQA reduces key–value cache storage by a factor of \\(h\\). This comparison excludes model weights, temporary activations, cache-management metadata, and implementation overhead.\n\nFor example, replacing 32 key–value heads with one shared head reduces the idealized key–value storage to one thirty-second of its original size. It does not reduce the model’s total memory requirement by that factor.\n\nMemory traffic and generation speed\n\nDuring single-token decoding, each query head must attend to the stored sequence. MQA allows the same cached keys and values to be reused across heads, reducing the amount of distinct data that must be fetched from device memory.\n\nThis is particularly valuable when attention is bandwidth-bound. The smaller cache can also permit larger batches or longer sequences within a fixed memory capacity, improving serving throughput even when single-request latency improves less dramatically.\n\nThe actual speedup depends on hardware, sequence length, batch size, numerical precision, and kernel implementation. An implementation that physically expands the shared keys and values into separate copies may lose much of the memory-traffic advantage.\n\nArithmetic and parameter counts\n\nMQA reduces the work and parameters needed to produce keys and values. For the projection matrices above, conventional multi-head attention uses\n\n\\[d_{\\mathrm{model}}h(d_k+d_v)\\]\n\nkey and value projection parameters, whereas MQA uses\n\n\\[d_{\\mathrm{model}}(d_k+d_v).\\]\n\nQuery and output projection sizes need not change.\n\nMQA does not eliminate the separate attention computations for each query head. Each head still computes scores and a weighted value sum. The attention arithmetic therefore remains proportional to\n\n\\[h\\,n_qn_k(d_k+d_v),\\]\n\napart from constant factors and implementation details. For full-sequence dense self-attention, the quadratic dependence on sequence length remains. MQA primarily improves storage and data movement rather than changing the fundamental time complexity of dense attention."
    },
    {
      "heading": "Relationship to other attention variants",
      "text": "Multi-head and grouped-query attention\n\nGrouped-query attention (GQA) generalizes the sharing pattern. It partitions query heads into groups, with each group sharing one key head and one value head.\n\nIf \\(g\\) is the number of key–value heads, then:\n\nArchitecture | Key–value heads | Sharing pattern\nStandard multi-head attention | \\(g=h\\) | Each query head has its own keys and values\nGrouped-query attention | Typically \\(1<g<h\\) | Several query heads share each key–value head\nMulti-query attention | \\(g=1\\) | All query heads share one key–value head\n\nUnder a common set of dimensions, GQA cache storage scales with \\(g\\), while MQA is its maximum-sharing endpoint. GQA provides an intermediate choice between independent head representations and the smallest conventional shared-head cache.\n\nFlashAttention and sparse attention\n\nFlashAttention concerns how attention is computed efficiently through tiling and reduced memory transfers; MQA concerns how query, key, and value projections are shared. These techniques address different aspects of attention and can be combined when the implementation supports shared key–value heads.\n\nLikewise, sliding-window attention restricts which positions can be attended to. MQA does not itself impose such a restriction. It can use dense attention over the available sequence or be combined with a restricted attention pattern."
    },
    {
      "heading": "Training and quality trade-offs",
      "text": "MQA imposes a representational constraint: different query heads cannot independently learn their own key and value projections. Although the query heads retain distinct matching behavior, they share the features available for matching and value retrieval.\n\nThis constraint may affect model quality. Its impact depends on model scale, task, training procedure, and the architecture being compared. There is no general guarantee that an MQA model will match a similarly configured multi-head model, nor that the same quality difference will hold across all workloads.\n\nMQA is normally incorporated into the architecture during training. Converting an existing multi-head checkpoint requires combining or otherwise replacing its key and value projections; it is not an exactly equivalent tensor rearrangement. Such conversion may be followed by additional training or fine-tuning. GQA offers a less aggressive sharing option when the quality cost of a single key–value head is undesirable."
    },
    {
      "heading": "Deployment considerations and limitations",
      "text": "The benefits of MQA are most direct during incremental decoding. Prompt processing, often called prefill, handles many positions together and can have a different balance of arithmetic and memory costs. Consequently, a decoding speedup does not imply the same improvement in prompt-processing speed.\n\nMQA also interacts with parallel computing. In head-parallel execution, query heads may be distributed across devices. Because MQA has only one key–value head, implementations may replicate its shared tensors or use other distribution strategies. Such replication can make aggregate physical cache usage larger than the idealized single-copy calculation.\n\nMQA does not remove linear cache growth with sequence length, automatically enlarge a model’s supported context window, or solve every bottleneck in large language model serving. Weight access, feed-forward layers, communication, scheduling, and other operations may dominate particular workloads. Cache quantization can further reduce storage per element, but introduces a separate numerical trade-off."
    },
    {
      "heading": "History",
      "text": "Noam Shazeer introduced multi-query attention in the 2019 paper Fast Transformer Decoding: One Write-Head is All You Need. The proposal targeted the memory-bandwidth costs of incremental Transformer decoding by sharing keys and values while retaining multiple query heads.\n\nThe approach became increasingly relevant as autoregressive language models grew and key–value cache costs became a significant deployment constraint. The 2023 paper GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints developed grouped-query attention as an intermediate design and described continued-training methods for adapting multi-head checkpoints to shared key–value architectures."
    }
  ],
  "links": [
    "multi-head-attention",
    "transformer-architecture",
    "neural-network-inference",
    "kv-cache",
    "attention-mechanism",
    "language-model",
    "self-attention",
    "graphics-processing-unit",
    "matrix",
    "time-complexity",
    "grouped-query-attention",
    "flashattention",
    "sliding-window-attention",
    "fine-tuning",
    "parallel-computing",
    "context-window",
    "large-language-model",
    "neural-network-quantization"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}