A fully convolutional network (FCN) is a convolutional neural network that processes spatially arranged features without relying on fully connected layers tied to a fixed input size. It can produce a spatial grid of predictions rather than a single prediction for an entire input. FCNs are especially associated with semantic segmentation, in which each image location receives a category label. The term describes both a general architectural principle and, more specifically, the influential segmentation models introduced by Jonathan Long, Evan Shelhamer, and Trevor Darrell in work first released in 2014 and presented at the 2015 Conference on Computer Vision and Pattern Recognition.
Architectural principle
A conventional network for image classification often reduces an image to a fixed-length representation and then uses fully connected layers to assign an image-level category. Flattening a spatial feature map into a vector makes the following fully connected layer depend on the map’s dimensions.
An FCN instead retains spatial organization through its prediction pathway. Convolutional filters apply the same learned weights at different positions, allowing a single network to process inputs with different spatial dimensions. Its output dimensions generally vary with those of the input.
“Fully convolutional” does not mean that every operation must literally be a convolution. Such networks commonly include pooling, activation functions, normalization, interpolation, and feature-merging operations. The defining distinction is the absence of fixed-size, flattened fully connected stages in the spatial prediction pathway.
For a two-dimensional input, a dense predictor can be represented as
where and are input height and width, is the number of input channels, is the number of output channels, and denotes learned parameters. For segmentation, the output channels commonly correspond to semantic classes. The output need not initially have the same resolution as the input.
The same principle extends to one-dimensional sequences and three-dimensional volumes; it is not restricted to ordinary two-dimensional images.
Convolution and spatial prediction
A convolutional layer combines information from a local neighborhood across input channels. In the cross-correlation convention commonly called convolution in deep learning, its output can be written as
Here, contains learned filter weights and is a bias for output channel . Stride, padding, and dilation determine how the input is sampled and how output dimensions change.
Repeated layers allow predictions to depend on increasingly large parts of the input. The region capable of influencing a particular output is its receptive field. Early layers typically capture relatively local patterns, while deeper layers combine information over broader regions.
A convolution is particularly useful for dense prediction. It mixes channels independently at each spatial position without combining neighboring positions. Applied to a feature map, it can produce a vector of class scores at every location using shared parameters.
Converting fully connected layers
An image classifier can often be converted into a fully convolutional predictor by reinterpreting its fully connected layers as convolutions.
Suppose a fully connected layer receives a flattened feature map of shape and produces outputs. Its weights can be rearranged into convolutional filters, each spanning positions and all channels. On a feature map of exactly the original dimensions, this produces the same numerical result, subject to equivalent surrounding operations. On a larger feature map, the filters produce outputs at multiple spatial positions.
Subsequent fully connected layers can commonly be converted into convolutions. This permits a classifier to evaluate overlapping spatial regions while sharing intermediate computations, rather than independently processing many cropped patches.
Resolution reduction and recovery
Dense prediction requires both semantic understanding and accurate localization. These goals create a central architectural trade-off.
Pooling and strided convolution reduce spatial resolution, making computation less expensive and allowing deeper features to incorporate broader context. However, coarse features can lose information about small objects, thin structures, and precise boundaries.
Output stride
The output stride is the spacing, measured in input coordinates, between adjacent positions in an output feature map. A network with output stride 32 produces a coarse prediction grid with roughly one position for every 32 input positions along each dimension, although exact dimensions depend on padding and rounding.
Upsampling enlarges this grid, but enlargement alone cannot reliably reconstruct spatial information discarded during downsampling.
Upsampling methods
Common ways to recover a finer prediction grid include:
- Interpolation, such as bilinear interpolation, which uses a fixed rule to resize feature maps or class scores.
- Transposed convolution, a learnable operation that maps a lower-resolution grid to a higher-resolution grid. It is not generally the mathematical inverse of convolution.
- Multistage decoding, which progressively increases resolution and refines features at successive scales.
The original FCN work used upsampling layers that could be initialized to perform bilinear interpolation and subsequently learned during training.
Combining coarse and fine features
A skip connection provides a pathway between nonadjacent parts of a network. In segmentation architectures, such connections often combine deep, semantically informative features with shallower, higher-resolution features.
The original FCN models fused class-score maps by addition. Other architectures concatenate feature maps and then learn how to combine them. These fusion operations differ, but both can help retain spatial detail that is difficult to recover from the deepest features alone. Segmentation skip connections need not follow the identity-addition design associated with residual networks.
The original FCN segmentation models
Long, Shelhamer, and Darrell demonstrated that classification networks could be transformed into dense predictors and trained end to end for semantic segmentation. Their approach combined convolutional conversion, upsampling, and fusion of information from different depths.
The best-known variants were based on the VGG-16 classification architecture:
| Variant | Principal prediction sources | Main distinction |
|---|---|---|
| FCN-32s | Deep features with output stride 32 | Upsamples a coarse score map to input resolution |
| FCN-16s | Deep scores fused with scores derived from the pool4 feature map | Adds finer information at stride 16 |
| FCN-8s | Further fusion with scores derived from the pool3 feature map | Adds finer information at stride 8 |
The suffixes refer to the stride of the fused prediction representation before final upsampling, not to the number of layers or output classes.
A significant aspect of this work was the use of transfer learning: classification networks trained on large datasets, including ImageNet, supplied useful initial representations. Fine-tuning then adapted those representations to pixelwise prediction.
The work did not introduce convolutional dense prediction in its broadest sense; earlier neural networks had already produced spatially arranged outputs. Its importance lay in establishing a practical and influential framework for adapting deep image classifiers to end-to-end semantic segmentation.
Training and evaluation
FCNs for semantic segmentation are commonly trained through supervised learning, using images paired with pixelwise labels. The network produces class scores, which are converted into class probabilities. A common loss function is pixelwise cross-entropy:
where is the input image, is the target class at position , and is the set of positions with valid labels. Unlabeled or ignored positions can be excluded from this set.
Gradients propagate through the prediction and upsampling pathways by backpropagation. Because parameters are shared spatially, errors at many locations contribute to updates of the same filters.
Important training considerations include:
- Class imbalance: large background regions can dominate the loss while rare classes contribute relatively little.
- Annotation precision: uncertain boundaries or inconsistent labels can limit attainable accuracy.
- Spatial augmentation: crops, flips, and geometric transformations must remain aligned between images and labels.
- Memory use: high-resolution feature maps can restrict batch size or crop size, especially when training on a GPU.
Segmentation quality is commonly evaluated with intersection over union, often averaged across classes. Pixel accuracy is also used, but can conceal poor performance on small classes when a dominant class occupies most of an image.
Applications and related architectures
FCNs are used wherever predictions should retain a spatial relationship to the input. Applications include scene labeling, aerial-image mapping, document layout analysis, depth estimation, and optical flow. Their outputs need not be categorical: dense regression networks can predict continuous quantities at each location.
They are closely associated with the encoder–decoder architecture. An encoder extracts increasingly abstract features, while a decoder reconstructs a spatially detailed output. However, not every FCN has a symmetric encoder–decoder structure.
U-Net is a prominent fully convolutional architecture with a contracting pathway, an expanding pathway, and connections that concatenate corresponding feature maps. It is distinct from the original FCN-8s design, despite sharing the broader principle of combining semantic abstraction with spatial detail.
Other architectures use dilated convolution to enlarge receptive fields without an equivalent increase in downsampling. Some also incorporate attention mechanisms or transformer-based components. Such hybrid systems can remain dense predictors, but whether they are called fully convolutional depends on their actual operations rather than on the shape of their outputs alone.
Limitations and implementation constraints
Variable-sized inputs are not completely unrestricted. A network may require minimum dimensions or dimensions compatible with successive downsampling stages. Padding, cropping, and feature alignment may be needed when combining different resolutions. Processing differently sized images together in a batch also commonly requires padding or resizing.
Spatial sharing does not guarantee exact translation equivariance. Convolution supports predictable responses to spatial shifts under appropriate conditions, but stride, pooling, finite boundaries, and padding can break exact correspondence. Translation equivariance is also different from translation invariance: a dense prediction should usually move when its corresponding object moves.
Local operations can limit contextual reasoning. A theoretical receptive field may be large, yet information from distant regions may still have limited influence on a prediction. Ambiguous objects and scene-wide relationships can require additional context mechanisms.
Upsampling can introduce artifacts. Some transposed-convolution configurations produce uneven overlap and checkerboard patterns. Interpolation-based methods avoid that particular mechanism but can still yield blurred or poorly localized boundaries.
Semantic segmentation does not inherently separate instances. An FCN assigning one category to every location may label two touching objects identically without distinguishing them as separate objects. Instance segmentation requires additional outputs, grouping mechanisms, or task-specific architecture.
Efficiency depends on resolution. Sharing computations across overlapping regions is more efficient than independently evaluating each patch, but high-resolution dense feature maps can still require substantial computation and memory.