Swin Transformer is a hierarchical vision transformer architecture for computer vision. It computes self-attention within local, non-overlapping windows and shifts the window partition between successive blocks, allowing information to cross window boundaries without computing attention between every pair of image tokens. Introduced in 2021, it produces multiscale feature representations suitable for image classification, object detection, and semantic segmentation. The name Swin abbreviates “shifted window.”
Background and development
The original Transformer architecture was developed for sequence processing. Its attention mechanism models relationships between tokens without requiring the fixed local connectivity of a convolutional neural network (CNN). Early vision transformers adapted this approach by dividing an image into patches and treating the patches as a sequence.
Two features of the original Vision Transformer complicated its use as a general-purpose visual backbone:
- Global attention becomes expensive at high resolution. With image tokens, a full attention matrix contains entries.
- Its standard architecture maintains a single spatial resolution. Detection and segmentation systems commonly benefit from feature maps at several resolutions, representing both fine spatial details and larger structures.
Swin Transformer addresses these issues through local attention and a hierarchical organization. It retains content-dependent attention while incorporating locality and progressive spatial downsampling—forms of inductive bias also associated with CNNs.
The architecture was introduced by Ze Liu and colleagues in the paper Swin Transformer: Hierarchical Vision Transformer using Shifted Windows, presented at the 2021 IEEE/CVF International Conference on Computer Vision (ICCV), where it received the best paper award.
Architecture
A standard Swin Transformer consists of a patch-embedding layer followed by four stages. Each stage contains several Transformer blocks operating at a fixed spatial resolution. Patch-merging layers between stages reduce spatial dimensions and increase channel width.
Patch embedding
For an input image of height and width , the original architecture divides the image into non-overlapping patches. For a three-channel image, each patch initially contains values.
A learned linear projection maps each patch to a feature vector with channels. The resulting representation has spatial dimensions
Each feature vector acts as a token, but the tokens retain a two-dimensional grid structure. This differs from language tokenization: the basic units are fixed image patches rather than words or subword units.
Hierarchical stages and patch merging
Between stages, a patch-merging layer groups each neighborhood of tokens. Their features are concatenated, giving channels, and a learned projection reduces this to channels.
The standard four-stage hierarchy therefore has the following approximate dimensions, assuming the input dimensions are divisible by the required factors:
| Stage | Spatial resolution | Channel width |
|---|---|---|
| 1 | ||
| 2 | ||
| 3 | ||
| 4 |
This structure provides a multiscale hierarchy for feature extraction. Earlier stages preserve more spatial detail; later stages combine information over larger regions.
Transformer blocks
An original Swin block contains:
- Layer normalization.
- Window-based multi-head self-attention.
- A residual connection.
- Another layer normalization.
- A two-layer multilayer perceptron with a GELU activation.
- A second residual connection.
The original architecture uses a pre-normalization arrangement: normalization precedes the attention and multilayer-perceptron sublayers. Consecutive blocks alternate between regular-window and shifted-window attention.
Window-based attention
Instead of attending over the complete image-token grid, Swin divides the grid into non-overlapping windows containing tokens. Attention is calculated independently inside each window.
For a single attention head, the operation is
where , , and are learned query, key, and value projections; is the query/key dimension per head; and is a learned relative-position bias.
Relative-position bias
The bias depends on the relative two-dimensional displacement between tokens in a window. For an window, there are possible relative offsets. The original model learns a bias table for these offsets for each attention head.
This supplies spatial information without requiring a separate learned absolute embedding for every image location. It is a form of positional encoding adapted to local two-dimensional attention. The default original architecture does not require absolute position embeddings.
Computational cost
Let a stage contain tokens with channels. Using the operation-count convention of the original paper, global multi-head self-attention has approximate cost
Window-based attention instead has cost
The first term represents feature projections; the second represents attention-score and weighted-value computations. With fixed window size and channel width , window attention scales linearly with the number of tokens rather than quadratically.
This computational-complexity claim concerns scaling with image area under fixed architectural settings. Increasing window size, channel width, or model depth still increases computation, and these expressions do not describe every operation or memory transfer in a complete implementation.
Shifted windows
Using only a fixed window partition would prevent attention from directly connecting tokens in different windows. Swin changes the partition in alternating blocks.
A regular-window block is followed by a block whose windows are displaced by approximately half a window along both spatial axes. For a window size , the usual shift is
For example, with windows, the displacement is three tokens in each direction. Tokens separated by a boundary in one block can consequently appear in the same window in the next.
Cyclic shift and masking
A direct shifted partition can create partial windows at image boundaries. The original implementation uses a cyclic shift of the feature map, followed by ordinary window partitioning.
Cyclic shifting brings spatially unrelated regions together at the wrapped boundaries. An attention mask blocks interactions that arise only from this wraparound. After attention, the shift is reversed.
This arrangement preserves uniform window shapes and supports efficient batched computation on graphics processing units. Implementations may also pad feature maps when their dimensions are not divisible by the window size.
Information propagation
Shifted windows provide cross-window communication, but a single block does not give every token access to the entire image. Information spreads through repeated blocks and through patch merging at successive stages.
Swin is also distinct from sliding-window attention. A sliding-window mechanism typically gives each query an overlapping local neighborhood. Swin partitions the feature map into disjoint windows within a block and changes that partition between blocks.
Model variants and training
The original model family includes Swin-Tiny, Swin-Small, Swin-Base, and Swin-Large, abbreviated Swin-T, Swin-S, Swin-B, and Swin-L. They share the same basic hierarchy but differ in stage widths, attention-head counts, and numbers of blocks.
Typical original configurations include:
| Variant | Initial width | Blocks per stage |
|---|---|---|
| Swin-T | 96 | |
| Swin-S | 96 | |
| Swin-B | 128 | |
| Swin-L | 192 |
A common original window size is seven tokens per side, though other configurations are possible.
For classification, the final feature map is usually globally averaged and passed to a classification head. Models can be pretrained through supervised learning on datasets such as ImageNet, then adapted through fine-tuning. The backbone architecture does not inherently require one particular learning objective.
Applications
Image classification
Swin can serve as a complete classification network by combining its hierarchical backbone with a pooling layer and output head. Learned attention weights make interactions within each window depend on the image content.
Detection and segmentation
Object detection, instance segmentation, and semantic segmentation require spatially organized features rather than only a single image-level vector. Swin’s intermediate stage outputs can feed multiscale detection heads or segmentation decoders.
Its hierarchy makes it structurally compatible with systems originally designed around CNN backbones. Nevertheless, replacing a backbone generally requires matching feature dimensions and adapting training and implementation settings.
Video and image restoration
Video Swin Transformer extends the approach to spatiotemporal windows, introducing local attention and window shifts across video dimensions.
Related image-restoration architectures, including SwinIR, adapt Swin-style blocks for tasks such as super-resolution, denoising, and compression-artifact reduction. These adaptations do not necessarily retain the original four-stage classification hierarchy: restoration often requires preserving high-resolution spatial information.
Swin Transformer V2
Swin Transformer V2, introduced in a 2022 paper, modifies the original design to support larger models and higher-resolution inputs. Its principal changes include:
- Residual-post-normalization, which changes normalization placement to help control activation magnitudes in large models.
- Scaled cosine attention, which replaces dot-product similarity with cosine similarity and a learned scale.
- Log-spaced continuous position bias, which generates relative-position biases from transformed coordinate offsets rather than relying solely on a fixed lookup table.
The continuous position-bias mechanism is designed to ease transfer between different window sizes and resolutions. V2 preserves the central combination of hierarchical representations and shifted-window attention.
Limitations and trade-offs
Restricted immediate context. Local windows reduce computation, but distant image regions cannot interact directly within one attention block. Tasks that require explicit long-range relationships may benefit from additional mechanisms.
Runtime depends on implementation. Lower theoretical operation counts do not guarantee proportionally lower inference latency. Window partitioning, shifting, masking, memory access, and kernel efficiency affect practical speed.
Resolution changes require handling. Window padding and relative-position parameters must be managed when image or window dimensions change. In the original architecture, changing window size commonly requires interpolation or another adaptation of the relative-position bias table.
Downsampling trades detail for efficiency. Patch merging reduces spatial resolution. Dense prediction systems therefore commonly combine features from several stages or use decoders to recover fine output structure.
Performance depends on training conditions. Comparisons with CNNs or other transformers depend on model size, pretraining data, image resolution, data augmentation, optimization, and the downstream task. Architectural differences alone do not establish a universal performance ranking.