{
  "id": "sliding-window-attention",
  "title": "Sliding-Window Attention",
  "lang": "en",
  "source_lang": "en",
  "summary": "Sliding-window attention restricts each token’s attention to a nearby range of positions, reducing computation and memory requirements while allowing information to propagate across layers.",
  "category": "computer-science",
  "subcategory": "artificial-intelligence",
  "type": "concept",
  "aliases": [
    "Sliding window attention",
    "Windowed attention",
    "Local window attention"
  ],
  "generated_at": "2026-11-03",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/sliding-window-attention/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/sliding-window-attention/",
      "json": "https://aiwiki.page/zh/sliding-window-attention.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "Sliding-window attention is a form of self-attention in which each position in a sequence attends only to a limited neighborhood, rather than to every position. Used in the Transformer architecture, it reduces the number of attention interactions from quadratic in sequence length to linear when the window size is fixed. Its defining trade-off is between efficient processing and restricted direct access to distant information: stacking layers broadens the effective receptive field, but does not reproduce unrestricted attention."
    },
    {
      "heading": "Mechanism",
      "text": "In standard attention, each input position produces a query, a key, and a value. A query is compared with eligible keys, and the resulting scores determine a weighted combination of their associated values.\n\nSliding-window attention changes which keys are eligible. For a bidirectional sequence, position \\(i\\) might attend to positions within a radius \\(r\\):\n\n\\[\\mathcal{N}(i)=\\{j:1\\leq j\\leq n,\\ |i-j|\\leq r\\},\\]\n\nwhere \\(n\\) is the sequence length. Away from sequence boundaries, this neighborhood contains \\(2r+1\\) positions.\n\nFor causal language modeling, attention must exclude future positions. If \\(W\\) denotes the maximum number of visible positions, including the current position, the neighborhood is\n\n\\[\\mathcal{N}(i)=\\{j:\\max(1,i-W+1)\\leq j\\leq i\\}.\\]\n\nFor one attention head, the output is\n\n\\[o_i= \\sum_{j\\in\\mathcal{N}(i)} \\frac{\\exp(q_i^\\top k_j/\\sqrt{d_k})} {\\sum_{t\\in\\mathcal{N}(i)} \\exp(q_i^\\top k_t/\\sqrt{d_k})} v_j,\\]\n\nwhere \\(q_i\\), \\(k_j\\), and \\(v_j\\) are query, key, and value vectors, and \\(d_k\\) is the query and key dimension. Positional biases or other score modifications may also be included.\n\nThe same restriction can be expressed as an attention mask whose entries are zero for permitted interactions and negative infinity elsewhere. However, applying such a mask after calculating a full attention matrix does not itself deliver the computational savings of a genuinely local implementation.\n\nWindow-size conventions differ across papers and software libraries. A “window of 512” may mean 512 visible positions, 512 preceding positions plus the current one, or a radius of 512 on either side. Comparisons require checking the exact neighborhood definition."
    },
    {
      "heading": "Computational and memory costs",
      "text": "For a sequence of length \\(n\\), full attention calculates approximately \\(n^2\\) query–key interactions per head. Sliding-window attention with at most \\(W\\) visible positions calculates at most \\(nW\\).\n\nIf query, key, and value dimensions are all of order \\(d\\), the principal attention operations have the following time complexity:\n\nAttention pattern | Score computation and value aggregation\nFull attention | \\(O(n^2d)\\)\nSliding-window attention | \\(O(nWd)\\)\n\nThis is linear in \\(n\\) only when \\(W\\) is held constant. If the window grows proportionally with sequence length, the asymptotic cost becomes quadratic again. These expressions exclude projection layers and other Transformer components, which retain their own costs.\n\nA conventional implementation that stores attention scores uses \\(O(nW)\\) score memory rather than \\(O(n^2)\\). This is not a universal statement about total memory: model parameters, activations, temporary buffers, and implementation choices also matter. FlashAttention demonstrates that even full attention can avoid storing the entire score matrix in high-bandwidth memory without changing its attention pattern.\n\nPractical speedups depend on hardware utilization. Dense matrix operations are highly optimized on graphics processing units, whereas local operations may incur indexing, padding, or data-movement overhead. Theoretical reductions in arithmetic therefore do not translate automatically into proportional reductions in execution time."
    },
    {
      "heading": "Receptive field across layers",
      "text": "A local attention layer cannot directly retrieve a value outside its window. Several layers can nevertheless transmit information over longer distances because each layer processes representations that already contain information gathered by previous layers.\n\nFor causal attention with \\(W\\) positions per window, the representation at position \\(i\\) after \\(L\\) local-attention layers can, in principle, depend on positions as early as\n\n\\[i-L(W-1),\\]\n\nsubject to the beginning of the sequence. Its maximum receptive-field span is therefore\n\n\\[1+L(W-1).\\]\n\nThis assumes a stack of the specified local layers without additional long-range connections. For bidirectional attention with radius \\(r\\), the maximum radius grows to \\(Lr\\).\n\nThese bounds describe possible information flow, not guaranteed effective recall. Distant information must pass through intermediate representations and may be weakened, transformed, or omitted. Being inside the theoretical receptive field is not equivalent to being directly accessible through an attention lookup.\n\nThe local window should consequently be distinguished from the model’s context window. The former specifies permitted interactions within a layer; the latter describes the input or history supported under a model’s architecture and operating configuration. Neither alone establishes reliable use of every distant detail."
    },
    {
      "heading": "Autoregressive inference and caching",
      "text": "During inference, an autoregressive language model commonly stores previous keys and values in a key–value cache. Full causal attention requires this cache to grow with the retained history.\n\nIn a layer using strictly causal sliding-window attention, keys and values that can no longer enter any future query’s window may be discarded. A rolling or circular buffer can retain only the recent positions required by that layer. For fixed head dimensions and head counts, persistent cache storage can thus remain \\(O(W)\\) per layer rather than growing with the total generated sequence length.\n\nSeveral qualifications matter:\n\n- Every layer’s attention pattern must be considered. A model combining local and full-attention layers still needs longer caches for the full-attention layers.\n- Cache eviction and positional coordinates are different operations. Reusing a buffer slot does not imply resetting the token’s position. This distinction matters for methods such as rotary position embeddings.\n- Prompt processing requires correct scheduling. When processing several prompt positions together, keys needed by earlier queries in that block must remain available until those queries have been evaluated.\n- Discarding a key does not erase all influence of its token. Information may already have propagated into retained representations, although this is not equivalent to retaining the original key and value.\n\nGrouped-query attention provides a separate, compatible optimization: it reduces the number of key and value heads, while sliding-window attention limits the positions those heads retain or access."
    },
    {
      "heading": "Variants and related patterns",
      "text": "Sliding-window attention belongs to the broader family of sparse attention, in which only selected query–key pairs interact.\n\nLocal and global attention\n\nSome architectures combine local windows with a small set of globally connected positions. Global positions can provide routes for information to travel between distant regions. Longformer, introduced in 2020 for long-document processing, combines sliding-window local attention with task-dependent global attention.\n\nWith a fixed number of global positions and a fixed local window, the interaction count remains linear in sequence length. If the number of global positions grows substantially, its cost must also be included.\n\nDilated windows\n\nA dilated pattern samples positions at intervals rather than attending to every nearby position. It can span a wider range with a similar number of interactions, but omits some immediate neighbors. Dilation and window size can vary across heads or layers in multi-head attention.\n\nBlock and shifted-window attention\n\nIn a true sliding window, neighboring queries generally have overlapping but different neighborhoods. In fixed-block attention, all positions in a block attend within that block. These patterns are not identical: a block boundary can prevent interaction between adjacent positions.\n\nThe Swin Transformer, introduced in 2021 for computer vision, alternates window partitions to enable communication across previous window boundaries. Its shifted-window approach is related to local attention, but differs from assigning every position its own overlapping neighborhood.\n\nHybrid local and full attention\n\nA model may alternate local layers with full-attention layers, or assign different patterns to different heads. These designs retain some direct long-range communication while reducing work elsewhere.\n\nSliding-window attention also appeared in the Mistral 7B model released in 2023, demonstrating its use in a decoder-only large language model alongside cache-management and attention-head optimizations."
    },
    {
      "heading": "Applications and limitations",
      "text": "Sliding-window attention is useful when nearby relationships dominate or unrestricted attention is too expensive. Applications include long-document natural language processing, streaming generation, and local processing of image patches. In each case, locality acts as an inductive bias: the architecture prioritizes nearby interactions.\n\nIts principal limitation is the absence of direct access to distant positions. Exact retrieval of an earlier identifier, comparison of widely separated passages, or reasoning across remote document sections may require long-range connections or another mechanism. Increasing depth expands the theoretical receptive field but adds computation and does not guarantee preservation of the needed information.\n\nWindow size is therefore an architectural hyperparameter. Smaller windows reduce attention work and cache requirements but restrict communication; larger windows increase direct access at greater cost. Assessment should separate processing speed, memory consumption, and task performance rather than treating supported sequence length as evidence of effective long-range understanding.\n\nReplacing full attention with local attention in an already trained model also changes the function it computes. Compatibility cannot be assumed merely because tensor shapes remain unchanged; suitable training or fine-tuning and evaluation may be necessary."
    }
  ],
  "links": [
    "self-attention",
    "transformer-architecture",
    "attention-mechanism",
    "time-complexity",
    "flashattention",
    "graphics-processing-unit",
    "context-window",
    "neural-network-inference",
    "language-model",
    "kv-cache",
    "rotary-position-embedding",
    "grouped-query-attention",
    "multi-head-attention",
    "computer-vision",
    "large-language-model",
    "natural-language-processing",
    "inductive-bias",
    "hyperparameter",
    "fine-tuning"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}