{
  "id": "grouped-query-attention",
  "title": "Grouped-query Attention",
  "lang": "en",
  "source_lang": "en",
  "summary": "Grouped-query attention shares key and value heads among groups of query heads, reducing inference memory requirements while retaining multiple distinct attention patterns.",
  "category": "computer-science",
  "subcategory": "artificial-intelligence",
  "type": "concept",
  "aliases": [
    "Grouped-query attention",
    "Grouped query attention",
    "GQA"
  ],
  "generated_at": "2026-11-03",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/grouped-query-attention/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/grouped-query-attention/",
      "json": "https://aiwiki.page/zh/grouped-query-attention.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "Grouped-query attention (GQA) is a variant of multi-head attention in which several query heads share one set of key and value heads. Used in Transformer models, particularly large language models, it reduces the memory needed to store and retrieve keys and values during text generation. GQA provides a configurable middle ground between conventional multi-head attention, which assigns separate keys and values to every query head, and multi-query attention, which shares a single key–value head across all query heads."
    },
    {
      "heading": "Motivation",
      "text": "An attention mechanism produces a representation by comparing a query with keys and using the resulting weights to combine values. In conventional multi-head attention, multiple heads perform this operation with different learned projections, allowing a model to represent several attention patterns in parallel.\n\nDuring autoregressive generation, a language model produces tokens sequentially. Each new token can attend to earlier tokens through causal self-attention. Rather than recomputing earlier keys and values at every generation step, implementations normally retain them in a key–value cache.\n\nThe cache grows with sequence length, batch size, layer count, and the number and size of key–value heads. Reading it can become a major bottleneck during inference, particularly when generating one token at a time. A model may have enough arithmetic capacity available but still be limited by the rate at which cached data can be transferred from device memory.\n\nGQA addresses this bottleneck by reducing the number of distinct key and value representations without proportionally reducing the number of query heads."
    },
    {
      "heading": "Structure and computation",
      "text": "Let \\(H_q\\) denote the number of query heads and \\(H_{kv}\\) the number of key–value heads. In the usual equal-sized grouping scheme, \\(H_q\\) is divisible by \\(H_{kv}\\), and each group contains\n\n\\[r=\\frac{H_q}{H_{kv}}\\]\n\nquery heads. All query heads within a group use the same key projection and the same value projection, but retain their own query projections.\n\nFor an input representation matrix \\(X\\), define\n\n\\[Q_i=XW_i^Q,\\qquad K_j=XW_j^K,\\qquad V_j=XW_j^V.\\]\n\nHere \\(i\\) indexes query heads, while \\(j\\) indexes key–value heads. With zero-based head indexing, a typical group assignment is\n\n\\[g(i)=\\left\\lfloor\\frac{i}{r}\\right\\rfloor.\\]\n\nThe output of query head \\(i\\) is\n\n\\[O_i= \\operatorname{softmax}\\left( \\frac{Q_iK_{g(i)}^{\\mathsf T}}{\\sqrt{d_k}}+M \\right)V_{g(i)},\\]\n\nwhere \\(d_k\\) is the key dimension and \\(M\\) is an attention mask. For causal attention, the mask prevents a token from attending to future positions. The head outputs are concatenated and passed through a learned output projection:\n\n\\[Y=\\operatorname{Concat}(O_0,\\ldots,O_{H_q-1})W^O.\\]\n\nSharing keys and values does not make the query heads identical. Distinct query projections can produce different attention distributions over the same keys, and consequently different weighted combinations of the shared values.\n\nRelationship to other head arrangements\n\nArrangement | Key–value heads | Sharing pattern\nConventional multi-head attention | \\(H_{kv}=H_q\\) | Each query head has its own key and value head\nGrouped-query attention | Usually \\(1<H_{kv}<H_q\\) | Each group of query heads shares a key and value head\nMulti-query attention | \\(H_{kv}=1\\) | All query heads share one key and value head\n\nThese arrangements can be described as members of a single family. Under that broader definition, conventional multi-head attention and multi-query attention are the two endpoint cases of GQA. In narrower usage, “GQA” refers only to intermediate configurations.\n\nThe grouping applies to attention heads, not to tokens, documents, or input examples."
    },
    {
      "heading": "Memory and computational effects",
      "text": "Key–value cache size\n\nFor a model with \\(L\\) attention layers, batch size \\(B\\), cached sequence length \\(T\\), equal key and value head dimension \\(d_h\\), and \\(s\\) bytes per stored element, the raw key–value cache occupies approximately\n\n\\[S_{\\mathrm{KV}}=2LBT H_{kv}d_hs.\\]\n\nThe factor of two accounts for both keys and values. This estimate excludes allocation overhead, metadata, padding, replication between devices, and any compression-specific overhead.\n\nCompared with a conventional multi-head model having the same \\(H_q\\), head dimensions, layer count, and storage precision, GQA uses a fraction\n\n\\[\\frac{H_{kv}}{H_q}\\]\n\nof the raw key–value cache storage.\n\nFor example, 32 query heads grouped around 8 key–value heads require one quarter of the raw cache storage of a 32-key–value-head configuration. This is a reduction in the cache component, not a fourfold reduction in total model memory: weights, temporary buffers, and other intermediate data remain.\n\nComputation and bandwidth\n\nGQA reduces the size of key and value projections and can reduce the amount of cached data read from memory. Efficient implementations reuse a shared key–value block across several query heads.\n\nHowever, every query head still computes its own attention scores and weighted output. For dense attention over a length-\\(T\\) sequence, the main attention calculations therefore retain approximately\n\n\\[O(H_qT^2d_h)\\]\n\narithmetic cost. For a single decoding step attending to \\(T\\) cached positions, the corresponding cost is approximately\n\n\\[O(H_qTd_h).\\]\n\nGQA does not, by itself, turn dense attention into a linear-time sequence-processing method or remove its quadratic training-time dependence on sequence length. Its principal benefits concern cache capacity, data movement, and smaller key and value projections.\n\nActual speed improvements depend on batch size, sequence length, hardware, and attention implementation. On a GPU, reducing memory traffic is particularly valuable when decoding is bandwidth-bound. The same cache reduction need not yield an equally large reduction in end-to-end latency."
    },
    {
      "heading": "Development and adoption",
      "text": "Multi-query attention was described by Noam Shazeer in 2019 as a way to accelerate incremental decoding by sharing keys and values across query heads.\n\nThe named GQA formulation was introduced in the 2023 paper GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints by Joshua Ainslie and colleagues. It presented grouped sharing as a compromise between the representational flexibility of multi-head attention and the decoding efficiency of multi-query attention.\n\nThe paper also described converting an existing multi-head model into a GQA model. Key projection weights within each selected group are mean-pooled, and value projection weights are treated similarly. Query projections remain separate. The resulting model then undergoes additional training, called uptraining in that work, to adapt to the changed architecture.\n\nThe reported experiments found configurations that approached multi-head-attention quality while obtaining decoding efficiency closer to multi-query attention. These results are empirical rather than a guarantee for every model or task.\n\nGQA subsequently became a common design choice in decoder-based language models, including larger Llama 2 variants and Mistral 7B."
    },
    {
      "heading": "Training and implementation",
      "text": "The number of key–value heads is an architectural hyperparameter. More key–value heads provide more independently learned key and value subspaces, but require a larger cache. Fewer key–value heads strengthen parameter sharing and reduce storage.\n\nModels can be trained with GQA from the outset or adapted from multi-head checkpoints. Weight averaging provides an initialization for conversion, but does not exactly preserve the original model’s function: separate key and value projections generally cannot be replaced by their averages without changing attention outputs. Continued training is therefore an important part of the conversion method.\n\nImplementations commonly store keys and values with an explicit \\(H_{kv}\\) head dimension. The attention operation then maps each query head to its associated key–value head. A simple implementation may repeat key and value tensors to match the query-head count, but materializing those copies can reduce the memory and bandwidth benefits. Specialized kernels instead reuse shared data without storing a full replicated cache.\n\nGQA is compatible with FlashAttention, which improves the execution of attention through memory-efficient tiling and data reuse. The two techniques address different aspects of the operation: GQA changes the head-sharing structure, while FlashAttention changes how attention is computed on hardware.\n\nIn distributed inference, key–value head counts also interact with how heads are partitioned across devices. If there are fewer key–value heads than participating partitions, some implementations replicate heads, reducing the realized aggregate storage savings."
    },
    {
      "heading": "Limitations and complementary techniques",
      "text": "GQA constrains the number of independently learned key and value projections. Although separate queries preserve multiple attention patterns, they cannot recover every function available to an otherwise comparable multi-head layer with fully independent keys and values. The effect on model quality depends on architecture, training, group count, and evaluation task.\n\nGQA is also distinct from several other efficiency techniques:\n\n- Sparse or local attention restricts which token positions can interact. GQA changes head sharing and can still attend densely over all permitted positions.\n- Quantization reduces the precision used to represent weights or cached values. It can be combined with GQA.\n- Cache eviction or compression removes or summarizes cached information. GQA reduces the number of key–value heads without inherently discarding past token positions.\n- A larger context window allows more tokens to be processed. GQA can make long contexts less expensive to store, but does not itself establish positional behavior or ensure reliable use of distant information.\n\nThe appropriate key–value head count is consequently a trade-off among model quality, cache capacity, hardware utilization, and serving requirements rather than a universally optimal constant."
    }
  ],
  "links": [
    "multi-head-attention",
    "transformer-architecture",
    "large-language-model",
    "attention-mechanism",
    "self-attention",
    "kv-cache",
    "neural-network-inference",
    "multi-query-attention",
    "graphics-processing-unit",
    "hyperparameter",
    "flashattention",
    "distributed-computing",
    "neural-network-quantization",
    "context-window"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}