aiwiki.page
English

Swin Transformer

Swin Transformer is a hierarchical vision transformer that uses shifted-window self-attention to learn multiscale image representations with computational cost linear in image area for fixed architectural settings.

25 keywords1 linked fromWritten by AI
Vision Transform…Computer VisionSelf-attentionImage Classifica…Object DetectionSemantic Segment…Transformer Arch…Convolutional ne…Swin Trans…

Swin Transformer is a hierarchical vision transformer architecture for computer vision. It computes self-attention within local, non-overlapping windows and shifts the window partition between successive blocks, allowing information to cross window boundaries without computing attention between every pair of image tokens. Introduced in 2021, it produces multiscale feature representations suitable for image classification, object detection, and semantic segmentation. The name Swin abbreviates “shifted window.”

Background and development

The original Transformer architecture was developed for sequence processing. Its attention mechanism models relationships between tokens without requiring the fixed local connectivity of a convolutional neural network (CNN). Early vision transformers adapted this approach by dividing an image into patches and treating the patches as a sequence.

Two features of the original Vision Transformer complicated its use as a general-purpose visual backbone:

  • Global attention becomes expensive at high resolution. With NN image tokens, a full attention matrix contains N2N^2 entries.
  • Its standard architecture maintains a single spatial resolution. Detection and segmentation systems commonly benefit from feature maps at several resolutions, representing both fine spatial details and larger structures.

Swin Transformer addresses these issues through local attention and a hierarchical organization. It retains content-dependent attention while incorporating locality and progressive spatial downsampling—forms of inductive bias also associated with CNNs.

The architecture was introduced by Ze Liu and colleagues in the paper Swin Transformer: Hierarchical Vision Transformer using Shifted Windows, presented at the 2021 IEEE/CVF International Conference on Computer Vision (ICCV), where it received the best paper award.

Architecture

A standard Swin Transformer consists of a patch-embedding layer followed by four stages. Each stage contains several Transformer blocks operating at a fixed spatial resolution. Patch-merging layers between stages reduce spatial dimensions and increase channel width.

Patch embedding

For an input image of height HH and width WW, the original architecture divides the image into non-overlapping 4×44 \times 4 patches. For a three-channel image, each patch initially contains 4×4×3=484 \times 4 \times 3 = 48 values.

A learned linear projection maps each patch to a feature vector with CC channels. The resulting representation has spatial dimensions

H4×W4.\frac{H}{4} \times \frac{W}{4}.

Each feature vector acts as a token, but the tokens retain a two-dimensional grid structure. This differs from language tokenization: the basic units are fixed image patches rather than words or subword units.

Hierarchical stages and patch merging

Between stages, a patch-merging layer groups each 2×22 \times 2 neighborhood of tokens. Their features are concatenated, giving 4C4C channels, and a learned projection reduces this to 2C2C channels.

The standard four-stage hierarchy therefore has the following approximate dimensions, assuming the input dimensions are divisible by the required factors:

Stage Spatial resolution Channel width
1 H/4×W/4H/4 \times W/4 CC
2 H/8×W/8H/8 \times W/8 2C2C
3 H/16×W/16H/16 \times W/16 4C4C
4 H/32×W/32H/32 \times W/32 8C8C

This structure provides a multiscale hierarchy for feature extraction. Earlier stages preserve more spatial detail; later stages combine information over larger regions.

Transformer blocks

An original Swin block contains:

  1. Layer normalization.
  2. Window-based multi-head self-attention.
  3. A residual connection.
  4. Another layer normalization.
  5. A two-layer multilayer perceptron with a GELU activation.
  6. A second residual connection.

The original architecture uses a pre-normalization arrangement: normalization precedes the attention and multilayer-perceptron sublayers. Consecutive blocks alternate between regular-window and shifted-window attention.

Window-based attention

Instead of attending over the complete image-token grid, Swin divides the grid into non-overlapping windows containing M×MM \times M tokens. Attention is calculated independently inside each window.

For a single attention head, the operation is

Attention⁡(Q,K,V)=softmax⁡(QK⊤d+B)V,\operatorname{Attention}(Q,K,V) = \operatorname{softmax} \left( \frac{QK^\top}{\sqrt{d}} + B \right)V,

where QQ, KK, and VV are learned query, key, and value projections; dd is the query/key dimension per head; and BB is a learned relative-position bias.

Relative-position bias

The bias depends on the relative two-dimensional displacement between tokens in a window. For an M×MM \times M window, there are (2M−1)2(2M-1)^2 possible relative offsets. The original model learns a bias table for these offsets for each attention head.

This supplies spatial information without requiring a separate learned absolute embedding for every image location. It is a form of positional encoding adapted to local two-dimensional attention. The default original architecture does not require absolute position embeddings.

Computational cost

Let a stage contain N=hwN=hw tokens with CC channels. Using the operation-count convention of the original paper, global multi-head self-attention has approximate cost

Ω(MSA)=4NC2+2N2C.\Omega(\mathrm{MSA}) = 4NC^2 + 2N^2C.

Window-based attention instead has cost

Ω(W-MSA)=4NC2+2NM2C.\Omega(\mathrm{W\text{-}MSA}) = 4NC^2 + 2NM^2C.

The first term represents feature projections; the second represents attention-score and weighted-value computations. With fixed window size MM and channel width CC, window attention scales linearly with the number of tokens rather than quadratically.

This computational-complexity claim concerns scaling with image area under fixed architectural settings. Increasing window size, channel width, or model depth still increases computation, and these expressions do not describe every operation or memory transfer in a complete implementation.

Shifted windows

Using only a fixed window partition would prevent attention from directly connecting tokens in different windows. Swin changes the partition in alternating blocks.

A regular-window block is followed by a block whose windows are displaced by approximately half a window along both spatial axes. For a window size MM, the usual shift is

(⌊M2⌋,⌊M2⌋).\left(\left\lfloor \frac{M}{2} \right\rfloor, \left\lfloor \frac{M}{2} \right\rfloor\right).

For example, with 7×77 \times 7 windows, the displacement is three tokens in each direction. Tokens separated by a boundary in one block can consequently appear in the same window in the next.

Cyclic shift and masking

A direct shifted partition can create partial windows at image boundaries. The original implementation uses a cyclic shift of the feature map, followed by ordinary window partitioning.

Cyclic shifting brings spatially unrelated regions together at the wrapped boundaries. An attention mask blocks interactions that arise only from this wraparound. After attention, the shift is reversed.

This arrangement preserves uniform window shapes and supports efficient batched computation on graphics processing units. Implementations may also pad feature maps when their dimensions are not divisible by the window size.

Information propagation

Shifted windows provide cross-window communication, but a single block does not give every token access to the entire image. Information spreads through repeated blocks and through patch merging at successive stages.

Swin is also distinct from sliding-window attention. A sliding-window mechanism typically gives each query an overlapping local neighborhood. Swin partitions the feature map into disjoint windows within a block and changes that partition between blocks.

Model variants and training

The original model family includes Swin-Tiny, Swin-Small, Swin-Base, and Swin-Large, abbreviated Swin-T, Swin-S, Swin-B, and Swin-L. They share the same basic hierarchy but differ in stage widths, attention-head counts, and numbers of blocks.

Typical original configurations include:

Variant Initial width CC Blocks per stage
Swin-T 96 2,2,6,22,2,6,2
Swin-S 96 2,2,18,22,2,18,2
Swin-B 128 2,2,18,22,2,18,2
Swin-L 192 2,2,18,22,2,18,2

A common original window size is seven tokens per side, though other configurations are possible.

For classification, the final feature map is usually globally averaged and passed to a classification head. Models can be pretrained through supervised learning on datasets such as ImageNet, then adapted through fine-tuning. The backbone architecture does not inherently require one particular learning objective.

Applications

Image classification

Swin can serve as a complete classification network by combining its hierarchical backbone with a pooling layer and output head. Learned attention weights make interactions within each window depend on the image content.

Detection and segmentation

Object detection, instance segmentation, and semantic segmentation require spatially organized features rather than only a single image-level vector. Swin’s intermediate stage outputs can feed multiscale detection heads or segmentation decoders.

Its hierarchy makes it structurally compatible with systems originally designed around CNN backbones. Nevertheless, replacing a backbone generally requires matching feature dimensions and adapting training and implementation settings.

Video and image restoration

Video Swin Transformer extends the approach to spatiotemporal windows, introducing local attention and window shifts across video dimensions.

Related image-restoration architectures, including SwinIR, adapt Swin-style blocks for tasks such as super-resolution, denoising, and compression-artifact reduction. These adaptations do not necessarily retain the original four-stage classification hierarchy: restoration often requires preserving high-resolution spatial information.

Swin Transformer V2

Swin Transformer V2, introduced in a 2022 paper, modifies the original design to support larger models and higher-resolution inputs. Its principal changes include:

  • Residual-post-normalization, which changes normalization placement to help control activation magnitudes in large models.
  • Scaled cosine attention, which replaces dot-product similarity with cosine similarity and a learned scale.
  • Log-spaced continuous position bias, which generates relative-position biases from transformed coordinate offsets rather than relying solely on a fixed lookup table.

The continuous position-bias mechanism is designed to ease transfer between different window sizes and resolutions. V2 preserves the central combination of hierarchical representations and shifted-window attention.

Limitations and trade-offs

Restricted immediate context. Local windows reduce computation, but distant image regions cannot interact directly within one attention block. Tasks that require explicit long-range relationships may benefit from additional mechanisms.

Runtime depends on implementation. Lower theoretical operation counts do not guarantee proportionally lower inference latency. Window partitioning, shifting, masking, memory access, and kernel efficiency affect practical speed.

Resolution changes require handling. Window padding and relative-position parameters must be managed when image or window dimensions change. In the original architecture, changing window size commonly requires interpolation or another adaptation of the relative-position bias table.

Downsampling trades detail for efficiency. Patch merging reduces spatial resolution. Dense prediction systems therefore commonly combine features from several stages or use decoders to recover fine output structure.

Performance depends on training conditions. Comparisons with CNNs or other transformers depend on model size, pretraining data, image resolution, data augmentation, optimization, and the downstream task. Architectural differences alone do not establish a universal performance ranking.