{
  "id": "swin-transformer",
  "title": "Swin Transformer",
  "lang": "en",
  "source_lang": "en",
  "summary": "Swin Transformer is a hierarchical vision transformer that uses shifted-window self-attention to learn multiscale image representations with computational cost linear in image area for fixed architectural settings.",
  "category": "computer-science",
  "subcategory": "artificial-intelligence",
  "type": "concept",
  "aliases": [
    "Shifted Window Transformer",
    "Swin"
  ],
  "generated_at": "2026-11-03",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/swin-transformer/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/swin-transformer/",
      "json": "https://aiwiki.page/zh/swin-transformer.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "Swin Transformer is a hierarchical vision transformer architecture for computer vision. It computes self-attention within local, non-overlapping windows and shifts the window partition between successive blocks, allowing information to cross window boundaries without computing attention between every pair of image tokens. Introduced in 2021, it produces multiscale feature representations suitable for image classification, object detection, and semantic segmentation. The name Swin abbreviates “shifted window.”"
    },
    {
      "heading": "Background and development",
      "text": "The original Transformer architecture was developed for sequence processing. Its attention mechanism models relationships between tokens without requiring the fixed local connectivity of a convolutional neural network (CNN). Early vision transformers adapted this approach by dividing an image into patches and treating the patches as a sequence.\n\nTwo features of the original Vision Transformer complicated its use as a general-purpose visual backbone:\n\n- Global attention becomes expensive at high resolution. With \\(N\\) image tokens, a full attention matrix contains \\(N^2\\) entries.\n- Its standard architecture maintains a single spatial resolution. Detection and segmentation systems commonly benefit from feature maps at several resolutions, representing both fine spatial details and larger structures.\n\nSwin Transformer addresses these issues through local attention and a hierarchical organization. It retains content-dependent attention while incorporating locality and progressive spatial downsampling—forms of inductive bias also associated with CNNs.\n\nThe architecture was introduced by Ze Liu and colleagues in the paper Swin Transformer: Hierarchical Vision Transformer using Shifted Windows, presented at the 2021 IEEE/CVF International Conference on Computer Vision (ICCV), where it received the best paper award."
    },
    {
      "heading": "Architecture",
      "text": "A standard Swin Transformer consists of a patch-embedding layer followed by four stages. Each stage contains several Transformer blocks operating at a fixed spatial resolution. Patch-merging layers between stages reduce spatial dimensions and increase channel width.\n\nPatch embedding\n\nFor an input image of height \\(H\\) and width \\(W\\), the original architecture divides the image into non-overlapping \\(4 \\times 4\\) patches. For a three-channel image, each patch initially contains \\(4 \\times 4 \\times 3 = 48\\) values.\n\nA learned linear projection maps each patch to a feature vector with \\(C\\) channels. The resulting representation has spatial dimensions\n\n\\[\\frac{H}{4} \\times \\frac{W}{4}.\\]\n\nEach feature vector acts as a token, but the tokens retain a two-dimensional grid structure. This differs from language tokenization: the basic units are fixed image patches rather than words or subword units.\n\nHierarchical stages and patch merging\n\nBetween stages, a patch-merging layer groups each \\(2 \\times 2\\) neighborhood of tokens. Their features are concatenated, giving \\(4C\\) channels, and a learned projection reduces this to \\(2C\\) channels.\n\nThe standard four-stage hierarchy therefore has the following approximate dimensions, assuming the input dimensions are divisible by the required factors:\n\nStage | Spatial resolution | Channel width\n1 | \\(H/4 \\times W/4\\) | \\(C\\)\n2 | \\(H/8 \\times W/8\\) | \\(2C\\)\n3 | \\(H/16 \\times W/16\\) | \\(4C\\)\n4 | \\(H/32 \\times W/32\\) | \\(8C\\)\n\nThis structure provides a multiscale hierarchy for feature extraction. Earlier stages preserve more spatial detail; later stages combine information over larger regions.\n\nTransformer blocks\n\nAn original Swin block contains:\n\n- Layer normalization.\n- Window-based multi-head self-attention.\n- A residual connection.\n- Another layer normalization.\n- A two-layer multilayer perceptron with a GELU activation.\n- A second residual connection.\n\nThe original architecture uses a pre-normalization arrangement: normalization precedes the attention and multilayer-perceptron sublayers. Consecutive blocks alternate between regular-window and shifted-window attention."
    },
    {
      "heading": "Window-based attention",
      "text": "Instead of attending over the complete image-token grid, Swin divides the grid into non-overlapping windows containing \\(M \\times M\\) tokens. Attention is calculated independently inside each window.\n\nFor a single attention head, the operation is\n\n\\[\\operatorname{Attention}(Q,K,V) = \\operatorname{softmax} \\left( \\frac{QK^\\top}{\\sqrt{d}} + B \\right)V,\\]\n\nwhere \\(Q\\), \\(K\\), and \\(V\\) are learned query, key, and value projections; \\(d\\) is the query/key dimension per head; and \\(B\\) is a learned relative-position bias.\n\nRelative-position bias\n\nThe bias depends on the relative two-dimensional displacement between tokens in a window. For an \\(M \\times M\\) window, there are \\((2M-1)^2\\) possible relative offsets. The original model learns a bias table for these offsets for each attention head.\n\nThis supplies spatial information without requiring a separate learned absolute embedding for every image location. It is a form of positional encoding adapted to local two-dimensional attention. The default original architecture does not require absolute position embeddings.\n\nComputational cost\n\nLet a stage contain \\(N=hw\\) tokens with \\(C\\) channels. Using the operation-count convention of the original paper, global multi-head self-attention has approximate cost\n\n\\[\\Omega(\\mathrm{MSA}) = 4NC^2 + 2N^2C.\\]\n\nWindow-based attention instead has cost\n\n\\[\\Omega(\\mathrm{W\\text{-}MSA}) = 4NC^2 + 2NM^2C.\\]\n\nThe first term represents feature projections; the second represents attention-score and weighted-value computations. With fixed window size \\(M\\) and channel width \\(C\\), window attention scales linearly with the number of tokens rather than quadratically.\n\nThis computational-complexity claim concerns scaling with image area under fixed architectural settings. Increasing window size, channel width, or model depth still increases computation, and these expressions do not describe every operation or memory transfer in a complete implementation."
    },
    {
      "heading": "Shifted windows",
      "text": "Using only a fixed window partition would prevent attention from directly connecting tokens in different windows. Swin changes the partition in alternating blocks.\n\nA regular-window block is followed by a block whose windows are displaced by approximately half a window along both spatial axes. For a window size \\(M\\), the usual shift is\n\n\\[\\left(\\left\\lfloor \\frac{M}{2} \\right\\rfloor, \\left\\lfloor \\frac{M}{2} \\right\\rfloor\\right).\\]\n\nFor example, with \\(7 \\times 7\\) windows, the displacement is three tokens in each direction. Tokens separated by a boundary in one block can consequently appear in the same window in the next.\n\nCyclic shift and masking\n\nA direct shifted partition can create partial windows at image boundaries. The original implementation uses a cyclic shift of the feature map, followed by ordinary window partitioning.\n\nCyclic shifting brings spatially unrelated regions together at the wrapped boundaries. An attention mask blocks interactions that arise only from this wraparound. After attention, the shift is reversed.\n\nThis arrangement preserves uniform window shapes and supports efficient batched computation on graphics processing units. Implementations may also pad feature maps when their dimensions are not divisible by the window size.\n\nInformation propagation\n\nShifted windows provide cross-window communication, but a single block does not give every token access to the entire image. Information spreads through repeated blocks and through patch merging at successive stages.\n\nSwin is also distinct from sliding-window attention. A sliding-window mechanism typically gives each query an overlapping local neighborhood. Swin partitions the feature map into disjoint windows within a block and changes that partition between blocks."
    },
    {
      "heading": "Model variants and training",
      "text": "The original model family includes Swin-Tiny, Swin-Small, Swin-Base, and Swin-Large, abbreviated Swin-T, Swin-S, Swin-B, and Swin-L. They share the same basic hierarchy but differ in stage widths, attention-head counts, and numbers of blocks.\n\nTypical original configurations include:\n\nVariant | Initial width \\(C\\) | Blocks per stage\nSwin-T | 96 | \\(2,2,6,2\\)\nSwin-S | 96 | \\(2,2,18,2\\)\nSwin-B | 128 | \\(2,2,18,2\\)\nSwin-L | 192 | \\(2,2,18,2\\)\n\nA common original window size is seven tokens per side, though other configurations are possible.\n\nFor classification, the final feature map is usually globally averaged and passed to a classification head. Models can be pretrained through supervised learning on datasets such as ImageNet, then adapted through fine-tuning. The backbone architecture does not inherently require one particular learning objective."
    },
    {
      "heading": "Applications",
      "text": "Image classification\n\nSwin can serve as a complete classification network by combining its hierarchical backbone with a pooling layer and output head. Learned attention weights make interactions within each window depend on the image content.\n\nDetection and segmentation\n\nObject detection, instance segmentation, and semantic segmentation require spatially organized features rather than only a single image-level vector. Swin’s intermediate stage outputs can feed multiscale detection heads or segmentation decoders.\n\nIts hierarchy makes it structurally compatible with systems originally designed around CNN backbones. Nevertheless, replacing a backbone generally requires matching feature dimensions and adapting training and implementation settings.\n\nVideo and image restoration\n\nVideo Swin Transformer extends the approach to spatiotemporal windows, introducing local attention and window shifts across video dimensions.\n\nRelated image-restoration architectures, including SwinIR, adapt Swin-style blocks for tasks such as super-resolution, denoising, and compression-artifact reduction. These adaptations do not necessarily retain the original four-stage classification hierarchy: restoration often requires preserving high-resolution spatial information."
    },
    {
      "heading": "Swin Transformer V2",
      "text": "Swin Transformer V2, introduced in a 2022 paper, modifies the original design to support larger models and higher-resolution inputs. Its principal changes include:\n\n- Residual-post-normalization, which changes normalization placement to help control activation magnitudes in large models.\n- Scaled cosine attention, which replaces dot-product similarity with cosine similarity and a learned scale.\n- Log-spaced continuous position bias, which generates relative-position biases from transformed coordinate offsets rather than relying solely on a fixed lookup table.\n\nThe continuous position-bias mechanism is designed to ease transfer between different window sizes and resolutions. V2 preserves the central combination of hierarchical representations and shifted-window attention."
    },
    {
      "heading": "Limitations and trade-offs",
      "text": "Restricted immediate context. Local windows reduce computation, but distant image regions cannot interact directly within one attention block. Tasks that require explicit long-range relationships may benefit from additional mechanisms.\n\nRuntime depends on implementation. Lower theoretical operation counts do not guarantee proportionally lower inference latency. Window partitioning, shifting, masking, memory access, and kernel efficiency affect practical speed.\n\nResolution changes require handling. Window padding and relative-position parameters must be managed when image or window dimensions change. In the original architecture, changing window size commonly requires interpolation or another adaptation of the relative-position bias table.\n\nDownsampling trades detail for efficiency. Patch merging reduces spatial resolution. Dense prediction systems therefore commonly combine features from several stages or use decoders to recover fine output structure.\n\nPerformance depends on training conditions. Comparisons with CNNs or other transformers depend on model size, pretraining data, image resolution, data augmentation, optimization, and the downstream task. Architectural differences alone do not establish a universal performance ranking."
    }
  ],
  "links": [
    "vision-transformer",
    "computer-vision",
    "self-attention",
    "image-classification",
    "object-detection",
    "semantic-segmentation",
    "transformer-architecture",
    "convolutional-neural-network",
    "inductive-bias",
    "tokenization",
    "feature-extraction",
    "layer-normalization",
    "multi-head-attention",
    "residual-connection",
    "multilayer-perceptron",
    "positional-encoding",
    "computational-complexity",
    "graphics-processing-unit",
    "sliding-window-attention",
    "supervised-learning",
    "imagenet",
    "fine-tuning",
    "instance-segmentation",
    "neural-network-inference",
    "data-augmentation"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}