{
  "id": "residual-neural-network",
  "title": "Residual Neural Network",
  "lang": "en",
  "source_lang": "en",
  "summary": "A residual neural network learns additive corrections to intermediate representations, enabling effective optimization of deep architectures through shortcut connections.",
  "category": "computer-science",
  "subcategory": "artificial-intelligence",
  "type": "concept",
  "aliases": [
    "ResNet",
    "residual network",
    "deep residual network"
  ],
  "generated_at": "2026-11-03",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/residual-neural-network/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/residual-neural-network/",
      "json": "https://aiwiki.page/zh/residual-neural-network.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "A residual neural network is an artificial neural network in which groups of layers learn an additive modification to their input rather than an entirely new representation. Its defining operation is a shortcut connection: the input to a block is added to the output of a trainable transformation. In its simplest form, a block computes \\(y=x+F(x)\\), where \\(F\\) is called the residual function. The term ResNet refers both to this general architectural principle and, more narrowly, to the influential family of convolutional neural networks introduced by Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun in 2015.\n\nResidual networks were a major development in deep learning because they made networks with dozens or hundreds of trainable layers practical to optimize. Their importance lies not merely in increasing depth, but in changing how information and gradients move through a network. Residual connections became standard components of image-recognition systems and subsequently appeared in language models, speech systems, generative models, and other architectures. Their benefits depend on the surrounding design: shortcut connections facilitate training but do not automatically guarantee stable gradients, accurate predictions, or efficient computation."
    },
    {
      "heading": "Historical development and the degradation problem",
      "text": "Before residual networks, increasingly deep convolutional models had produced substantial advances in computer vision. AlexNet, which achieved a decisive result in the 2012 ImageNet Large Scale Visual Recognition Challenge, demonstrated the value of training a large convolutional network on graphics processors. Subsequent architectures, including VGG and GoogLeNet, explored greater depth and more elaborate arrangements of convolutional operations. These developments established depth as a useful source of representational capacity, but also exposed difficulties in optimization.\n\nOne difficulty was the vanishing-gradient problem, in which repeated multiplication by small derivatives weakens the learning signal reaching early layers. Initialization methods and normalization techniques partly addressed this issue. Nevertheless, experiments revealed another phenomenon: a deeper conventional network could have higher training error than a shallower counterpart. This was termed the degradation problem. Because the deterioration occurred on the training set, it could not simply be attributed to overfitting, which concerns a mismatch between training performance and performance on unseen examples.\n\nThe apparent paradox was that a deeper network should, in principle, be able to reproduce a shallower network by making its additional layers implement identity mappings. If an extra group of layers could simply copy its input, inserting that group would not need to worsen the represented solution. In practice, ordinary parameterizations did not make such copying easy for the optimizer to discover. Residual learning addressed this mismatch between representational possibility and optimization accessibility: instead of asking a stack of layers to learn an identity mapping directly, it supplied that mapping explicitly and trained the stack to learn a correction.\n\nHe and his colleagues described the approach in “Deep Residual Learning for Image Recognition,” released in December 2015 and presented at the 2016 Conference on Computer Vision and Pattern Recognition. Their experiments included 152-layer ImageNet models and substantially deeper networks on CIFAR-10. An ensemble of residual networks achieved a 3.57% top-five error rate on the ImageNet competition test set and formed the basis of the team’s first-place classification result in 2015. That figure was an ensemble competition result, not the error rate of every individual ResNet.\n\nShortcut connections themselves had precedents, and highway networks introduced learned gates controlling information flow in 2015. ResNet’s distinctive contribution was a simple additive residual formulation with identity shortcuts that worked effectively at large scale. The subsequent 2016 paper “Identity Mappings in Deep Residual Networks” examined information propagation more closely and introduced a pre-activation block design that further improved the training of very deep networks."
    },
    {
      "heading": "Residual mappings and shortcut connections",
      "text": "Let \\(x_l\\) denote the representation entering residual block \\(l\\). A general residual unit can be written as\n\n\\[x_{l+1}=f_l\\!\\left(h_l(x_l)+F_l(x_l;\\theta_l)\\right),\\]\n\nwhere \\(F_l\\) is the trainable residual branch, \\(\\theta_l\\) contains its parameters, \\(h_l\\) is the shortcut transformation, and \\(f_l\\) is an operation applied after the addition. In an identity shortcut, \\(h_l(x_l)=x_l\\). In the original ResNet blocks, \\(f_l\\) is generally a rectified linear unit, or ReLU. Some later architectures omit an activation immediately after the addition.\n\nThe residual interpretation follows from expressing a desired mapping \\(H(x)\\) as\n\n\\[H(x)=x+F(x), \\qquad F(x)=H(x)-x.\\]\n\nThis is a reparameterization rather than an assumption that the desired mapping is literally a small correction everywhere. The learned residual can be large and nonlinear. The practical advantage is that the identity mapping has a particularly simple representation: the residual branch can produce zero. Whether that state is readily reached depends on normalization, biases, initialization, and activation placement.\n\nElementwise addition requires compatible tensor shapes. If a block changes spatial resolution or channel count, its shortcut cannot simply copy the input unchanged. A common solution is a learned projection,\n\n\\[y=W_s*x+F(x;\\theta),\\]\n\nwhere \\(W_s*x\\) is typically a \\(1\\times1\\) convolution, possibly with stride greater than one. The original work also considered shortcuts using subsampling and zero-padding without additional learned parameters. Projection shortcuts are widely used at the boundaries between network stages.\n\nResidual addition differs fundamentally from concatenation. Addition combines corresponding coordinates and preserves the output channel count, whereas concatenation places representations alongside one another and increases the number of channels unless followed by a reduction. Both are forms of shortcut connectivity, but they produce different architectures and computational requirements. Similarly, an encoder-to-decoder connection that restores spatial detail is not necessarily a residual connection: it may concatenate features from widely separated stages rather than add a correction to a block’s own input.\n\nThe word “residual” describes the block’s internal parameterization, not necessarily an error measured against a target label. In supervised learning, the network may be trained using a classification loss function even though none of its residual branches explicitly predicts classification errors. This distinguishes residual networks from methods such as gradient boosting, where successive learners are commonly fitted to quantities derived from the current prediction error."
    },
    {
      "heading": "Canonical ResNet architectures",
      "text": "The original ImageNet family includes ResNet-18, ResNet-34, ResNet-50, ResNet-101, and ResNet-152. Their names count convolutional and final fully connected layers under the original convention; normalization layers, activation functions, and shortcut additions are not counted separately. Consequently, the stated depth is not the total number of operations in the computational graph. Later models carrying a ResNet name may also differ in their input stem, stride placement, normalization, or classification head.\n\nResNet-18 and ResNet-34 use a basic block consisting of two \\(3\\times3\\) convolutions. In the original post-activation design, the first convolution is followed by batch normalization and ReLU; the second is followed by batch normalization, shortcut addition, and a final ReLU. Their four main stages contain respectively \\([2,2,2,2]\\) and \\([3,4,6,3]\\) blocks. Stage widths are commonly 64, 128, 256, and 512 channels. At transitions between stages, spatial dimensions decrease while channel counts increase.\n\nResNet-50, ResNet-101, and ResNet-152 use bottleneck blocks with a \\(1\\times1\\), \\(3\\times3\\), \\(1\\times1\\) sequence. The first convolution reduces the channel dimension, the middle convolution performs spatial processing, and the last expands the representation. In the standard ImageNet design, the output channel count is four times the bottleneck’s internal width. The four stages contain \\([3,4,6,3]\\), \\([3,4,23,3]\\), and \\([3,8,36,3]\\) bottleneck blocks, respectively.\n\nThe bottleneck structure permits substantial depth without requiring every spatial convolution to operate at the full output width. For example, a \\(3\\times3\\) convolution mapping 256 channels to 256 channels contains \\(9\\times256^2\\) kernel weights. Performing its spatial processing at 64 channels instead uses \\(9\\times64^2\\) weights for that middle convolution, although the surrounding \\(1\\times1\\) convolutions add their own costs. The design exchanges a more structured sequence of transformations for lower spatial-convolution cost.\n\nThe original ImageNet stem uses a \\(7\\times7\\) convolution with stride 2, followed by normalization, ReLU, and max pooling. After the four residual stages, global average pooling produces one value per channel, and a fully connected layer predicts class scores. For a \\(224\\times224\\) input, the final feature map commonly has spatial dimensions \\(7\\times7\\). CIFAR versions use a different, smaller-image design and should not be interpreted as scaled-down copies with identical stage layouts. These distinctions matter when comparing published depths, reproducing results, or transferring pretrained weights between implementations."
    },
    {
      "heading": "Gradient propagation and optimization behavior",
      "text": "Residual connections alter the derivatives encountered by backpropagation. For an identity-addition block without a post-addition nonlinearity,\n\n\\[x_{l+1}=x_l+F_l(x_l),\\]\n\nthe block’s Jacobian is\n\n\\[\\frac{\\partial x_{l+1}}{\\partial x_l} =I+J_{F_l}(x_l).\\]\n\nThe identity matrix contributes a direct derivative term independently of the residual branch’s internal sequence of convolutions and nonlinearities. This is the mathematical basis for the claim that shortcuts provide a more direct route for gradients through a deep network.\n\nAcross several such blocks, the forward representation can be expanded as\n\n\\[x_L=x_l+\\sum_{i=l}^{L-1}F_i(x_i).\\]\n\nIts derivative contains an explicit identity contribution,\n\n\\[\\frac{\\partial x_L}{\\partial x_l} =I+\\frac{\\partial}{\\partial x_l} \\sum_{i=l}^{L-1}F_i(x_i).\\]\n\nThis expression does not mean that the other terms are independent, nor does it ensure that the overall gradient remains well conditioned. The residual transformations depend recursively on earlier states. Their derivatives may amplify, suppress, or cancel components of the direct contribution. Nevertheless, the additive parameterization changes the optimization problem substantially compared with a plain stack of transformations.\n\nThe original post-activation block has an additional derivative from the nonlinearity after addition. A ReLU can set negative components to zero, so its shortcut is not an unrestricted identity route across the entire block. The pre-activation design moves normalization and activation before the weight layers and leaves the addition free of an immediate activation. This arrangement makes the identity path more explicit and was particularly useful in experiments with networks hundreds or more layers deep.\n\nThe advantage is therefore more precise than saying that residual networks “solve vanishing gradients.” They reduce important barriers to optimization, but training still depends on initialization, normalization, residual magnitude, and the learning rate. If residual branches repeatedly produce large perturbations, activations and gradients can become unstable. Conversely, branch parameters initialized so that the residual output is initially small can make a deep model begin near an identity-like computation, allowing departures from that state to develop during training.\n\nResidual networks also illustrate a distinction between expressivity and trainability. A plain network may be capable of representing a useful function, yet an available optimization procedure may fail to find suitable parameters. Residual parameterization changes the routes through parameter space without necessarily changing the intended input-output task. Its empirical success is therefore evidence about the interaction of architecture and optimization, not a proof that deeper functions are always inherently better predictors."
    },
    {
      "heading": "Training practices and reproducibility",
      "text": "A residual architecture specifies connectivity, but a trained ResNet is also the product of a complete optimization and data-processing procedure. The original ImageNet experiments used stochastic gradient descent with momentum, weight decay, batch normalization, and a scheduled learning rate. They trained with a minibatch size of 256 and an initial learning rate of 0.1, reducing that rate during training. These settings were part of a particular experimental regime rather than universal requirements for residual learning.\n\nBatch normalization played an important role in the original design. During training it normalizes activations using minibatch statistics and learns scale and offset parameters. At inference, conventional implementations use accumulated estimates of the statistics. This creates a distinction between training and evaluation modes. A model evaluated with inappropriate statistics, or inadvertently left in training mode, can behave differently even when its convolutional weights have not changed. Small batches, nonindependent samples, and shifts in the input distribution can complicate normalization behavior.\n\nInitialization determines the scale of signals before learning. Variance-preserving initialization designed for rectifiers helps avoid rapid growth or decay in activation magnitudes. A later practice initializes the scale parameter of the last normalization layer in each residual branch to zero, making the branch initially produce zero under the usual parameter settings. This encourages the network to start with blocks close to their shortcut transformations. Other methods use carefully chosen scaling to train residual networks without normalization.\n\nThe data pipeline can be as influential as architectural changes. Random crops and horizontal flips are conventional forms of data augmentation for ImageNet classification. Later training recipes have incorporated stronger augmentation, label smoothing, mixup, alternative learning-rate schedules, and longer training. As a result, an accuracy reported for “ResNet-50” does not identify one immutable performance level. Comparisons must distinguish changes to the architecture from changes to the training procedure, resolution, data, or evaluation method.\n\nReproducibility also requires specifying the exact implementation. A widely used ResNet-50 variation places downsampling stride in the \\(3\\times3\\) convolution rather than the first \\(1\\times1\\) convolution of a bottleneck. Differences of this kind may preserve the broad block layout while changing accuracy and computation. Frameworks such as PyTorch and TensorFlow provide established implementations, but checkpoint names alone do not fully document preprocessing and architecture.\n\nFor transfer learning, a pretrained backbone may be frozen, partially updated, or fully subjected to fine-tuning. Whether normalization statistics are updated is a separate experimental choice. Reliable reports identify the pretrained data, input normalization, trainable layers, optimizer, training duration, and evaluation split. These details are necessary to distinguish an architectural effect from a stronger training or transfer procedure."
    },
    {
      "heading": "Major architectural variants",
      "text": "The success of ResNet led to a family of architectures that retain additive shortcuts while changing what each residual branch computes. Pre-activation ResNets, introduced in 2016, place batch normalization and ReLU before convolutional weight layers. Their aim is not simply to rearrange operations for implementation convenience, but to preserve a cleaner identity route through successive additions. This design helped separate the effects of shortcut structure from the effects of nonlinearities placed directly on the main information path.\n\nWide Residual Networks, introduced by Sergey Zagoruyko and Nikos Komodakis in 2016, challenged the assumption that increasing depth should be the principal way to improve residual models. They increased channel widths and used fewer layers than some very deep alternatives. The results showed that depth, width, parameter count, and training efficiency must be considered together. A shallower wide network may use parallel hardware more effectively, although wider intermediate activations can also increase memory demands.\n\nResNeXt, introduced by Saining Xie and colleagues, uses aggregated transformations implemented through grouped convolutions. Its defining design dimension is cardinality, the number of parallel transformation groups. A block can be expressed conceptually as\n\n\\[y=x+\\sum_{i=1}^{C}T_i(x),\\]\n\nwhere \\(C\\) is the cardinality. In practice, grouped convolution provides an efficient implementation of much of this aggregation. ResNeXt demonstrated that increasing the number of transformation groups could be an effective alternative to increasing only depth or width.\n\nSqueeze-and-excitation networks add learned channel-wise modulation. A typical module globally pools spatial features, uses a small network to generate channel weights, and rescales the feature channels before the residual addition. This mechanism gives the residual branch input-dependent channel emphasis. Other variants alter the input stem, downsampling operation, convolutional kernel sizes, or internal activation functions. Such changes can produce meaningful improvements while preserving the overall residual-stage organization.\n\nStochastic depth, introduced in 2016, randomly bypasses selected residual branches during training. The shortcut remains active, so the network can still propagate representations when a branch is omitted. At inference, the full architecture is used with the corresponding scaling convention. Related methods are often called drop-path regularization, although terminology and sampling details vary. This is structurally different from ordinary dropout, which usually suppresses individual activations rather than entire residual transformations.\n\nNot every architecture with shortcuts is a ResNet variant in the narrow sense. DenseNet, for example, concatenates earlier representations rather than combining them by residual addition. Highway networks use learned gates. Modern convolutional architectures may preserve residual blocks while extensively redesigning their internal operations. The broader category is residual architecture; the narrower ResNet family denotes particular block and stage designs descended from the original models."
    },
    {
      "heading": "Mathematical interpretations and their limits",
      "text": "Residual networks have motivated several complementary theoretical interpretations. None is a complete explanation of their practical behavior, but each highlights a property obscured by treating them as ordinary layer stacks. One influential perspective connects residual blocks to discrete dynamical systems. If a block is written as\n\n\\[x_{l+1}=x_l+h\\,f_l(x_l),\\]\n\nits form resembles a forward Euler step for an ordinary differential equation,\n\n\\[\\frac{dx(t)}{dt}=f(x(t),t),\\]\n\nwith depth corresponding to discretized time and \\(h\\) to a step size.\n\nThis connection helped motivate neural ordinary differential equations, introduced in 2018, which parameterize a continuous-time derivative and use a numerical solver to obtain the output state. Ordinary ResNets, however, are not automatically numerical approximations to a fixed continuous system. Their layers may have independently learned parameters, change channel dimensions, downsample spatially, and contain operations that do not fit a smooth constant-dimensional flow. The continuous-time interpretation is most direct for suitably structured blocks with controlled step sizes.\n\nA related stability perspective examines how perturbations evolve through products of Jacobians \\(I+J_{F_l}\\). If the residual Jacobians are appropriately bounded or scaled, transformations may remain closer to identity than those in an unconstrained deep stack. This can improve conditioning, but eigenvalues, singular values, and interactions between successive blocks all matter. Merely placing an identity matrix in every factor does not establish that the product is stable. Analytical conclusions therefore depend on assumptions about branch norms, initialization, width, or data.\n\nAnother interpretation describes residual networks as collections of paths of different lengths. In a simplified linear block, choosing either the identity term or the learned term in products of expressions such as \\(I+W_l\\) produces many path terms. This offers intuition for why shallow routes can coexist with deep computation and why removing some blocks need not destroy all functionality. It does not make a nonlinear ResNet equivalent to an ordinary ensemble of separately trained models: the paths share parameters and intermediate states.\n\nResidual learning can also be viewed as iterative refinement in a representation-learning system. Each block updates a current state rather than replacing it through an entirely separate mapping. This resembles the update structure of many numerical and optimization algorithms, but resemblance alone does not identify what objective an individual block minimizes. Standard ResNet blocks are learned jointly from a network-level loss; they are not generally guaranteed to perform descent on an explicit per-block energy.\n\nThese interpretations address different questions—continuous-depth limits, signal stability, path structure, and state refinement. Their assumptions should remain separate from empirical findings. A theorem for a linear residual network or an infinite-width limit does not directly establish the behavior of a finite convolutional model trained with normalization and augmentation."
    },
    {
      "heading": "Uses in visual recognition and image processing",
      "text": "The most direct use of ResNet is image classification. A convolutional backbone transforms pixels into increasingly abstract spatial features, global pooling aggregates those features, and a classifier assigns scores to categories. Pretraining on ImageNet made ResNet backbones widely available and facilitated comparisons across research projects. The architecture’s regular stage structure, established implementations, and range of model sizes helped it become a standard experimental baseline.\n\nIn object detection, a ResNet commonly supplies features to a separate localization and classification system. Faster R-CNN, for example, can use residual features for region proposals and region-based predictions. Feature Pyramid Networks combine features across backbone stages to provide representations at several spatial scales. This is useful because small and large objects require different balances between spatial detail and semantic abstraction. A residual backbone by itself is not a complete detector: proposal generation, prediction heads, matching rules, and postprocessing remain separate components.\n\nFor semantic segmentation, predictions must retain or reconstruct spatial organization rather than collapse the image into a single category vector. ResNet backbones are therefore adapted by changing output stride, using dilated convolutions, or connecting them to decoder modules. DeepLab systems have used residual backbones with atrous convolution, while Mask R-CNN combines a residual backbone and feature pyramid with heads for object classes, bounding boxes, and instance masks. These systems illustrate how the same feature extractor can support distinct output structures.\n\nResidual learning is also useful in low-level image processing, including denoising, super-resolution, and restoration. Here there are two related but distinct ideas. Internal residual blocks improve the optimization of the network, while a global input-output shortcut may let the model predict the difference between a degraded image and a desired output. A denoising system can estimate noise and subtract it, whereas a super-resolution model can add learned high-frequency detail to an upsampled input. Neither application requires the exact ImageNet classification architecture.\n\nResidual backbones have also been used in self-supervised learning, including influential contrastive systems such as MoCo and SimCLR. In these settings, the network learns visual representations without ordinary category labels for every training example. A projection head may be used during pretraining and then discarded for downstream evaluation. The effectiveness of the resulting representation depends on the learning objective, augmentation, training data, and evaluation procedure, not solely on residual connectivity.\n\nAcross these applications, the term backbone identifies the network’s role as a reusable feature extractor. Classification accuracy is informative but does not fully determine detection, segmentation, or restoration quality. Spatial resolution, intermediate feature availability, computational cost, and compatibility with task-specific heads can be equally important."
    },
    {
      "heading": "Residual connections beyond convolutional networks",
      "text": "Residual connections are not intrinsically tied to images or convolutions. The same additive pattern can connect dense layers, attention modules, temporal convolutions, or transformations on graph-structured data. The essential requirement is that the shortcut and residual outputs be compatible at the point of addition. As a result, residual connectivity became a general design principle even where the original ResNet stage structure is absent.\n\nThe Transformer architecture, introduced in 2017, uses residual connections around its attention and feed-forward sublayers. A simplified update has the form\n\n\\[x'=x+\\operatorname{Attention}(\\operatorname{Norm}(x)),\\]\n\nfollowed by another residual update using a feed-forward network. The expression shown is a pre-normalization arrangement; the original Transformer instead applied layer normalization after each residual addition. Both arrangements combine a persistent representation with learned contextual updates, but their gradient behavior differs.\n\nIn large language models, the sequence of hidden representations is often described as a residual stream. Attention and feed-forward modules read from that stream and write additive updates into it. This terminology is useful for architectural analysis, but the stream’s coordinates should not be assumed to correspond to uniquely separable human-interpretable concepts. The learned representation can distribute information across many coordinates and layers. Residual connections support the computation without by themselves explaining its semantic organization.\n\nResidual designs also occur in speech recognition and audio generation. WaveNet, introduced in 2016, used dilated causal convolutions together with residual and skip connections. The residual connections supported propagation through its stack, while additional skip outputs contributed to the prediction pathway. In networks for graph-structured data, residual updates can preserve access to earlier node representations and facilitate deeper message passing. They may reduce some optimization difficulties but do not automatically eliminate oversmoothing or information bottlenecks associated with repeated graph aggregation.\n\nModern diffusion models frequently use residual blocks inside U-Net-style denoisers. These blocks can incorporate conditioning signals, such as an embedding of the diffusion timestep. Their local residual additions coexist with longer encoder-decoder shortcuts, which often use concatenation. Convolutional U-Nets and diffusion Transformers therefore both employ residual reasoning, but with different surrounding operations and information pathways.\n\nThe broad adoption of residual connections should not erase architectural distinctions. A Transformer is not a convolutional ResNet merely because both add learned updates to a shortcut. Their inductive biases, computation costs, normalization practices, and treatment of spatial or sequential structure differ substantially. What transfers between them is an architectural principle: preserve a direct state pathway while learning transformations that update it."
    },
    {
      "heading": "Computational cost, evaluation, and limitations",
      "text": "An identity shortcut has no learned parameters, but an entire residual network is not computationally free to deepen. Each branch adds convolutions or other transformations, intermediate activations, and normalization operations. A conventional ImageNet ResNet-50 contains about 25.6 million parameters, whereas ResNet-152 contains about 60.2 million. Exact counts depend on the classification head and implementation. At \\(224\\times224\\) input resolution, their convolutional and fully connected work is commonly reported at roughly 4 and 11 billion multiply-accumulate operations, respectively.\n\nOperation-count conventions require care. Some reports count one multiplication and its associated addition as a single operation; others count them as two floating-point operations. Results labeled “FLOPs” can therefore differ by approximately a factor of two even before architectural differences are considered. Parameter count, arithmetic count, latency, and energy consumption are also distinct quantities. A model with fewer arithmetic operations may run more slowly if its operations map poorly to the available hardware.\n\nDuring training, memory use depends heavily on stored activations, not only on parameter count. Depth, channel width, input resolution, and batch size all influence this cost. Activation checkpointing reduces stored intermediates by recomputing portions of the forward pass during backpropagation. During inference, batch-normalization parameters and statistics can often be folded into preceding convolutions, reducing separate operations without changing the intended evaluation-mode function.\n\nDeployment on a graphics processing unit, processor, or specialized accelerator also depends on tensor layouts, kernel fusion, memory bandwidth, and supported numerical formats. Quantization can reduce storage and arithmetic costs, but residual addition requires compatible treatment of the two branches’ numerical scales. Projection shortcuts introduce additional computation, and narrow branches do not always produce proportional speedups because hardware utilization and data movement impose their own limits.\n\nResidual connectivity does not guarantee generalization. Very deep networks can overfit, and the original research reported that a 1,202-layer CIFAR-10 model generalized worse than a 110-layer model despite reaching lower training error. Residual models can also fail under distribution shift or respond incorrectly to an adversarial example. Shortcut connections address an architectural and optimization problem, not a general solution to unreliable predictions.\n\nMeaningful evaluation therefore reports more than a model name and a single accuracy figure. Relevant details include training data, augmentation, input resolution, parameter count, arithmetic convention, hardware, latency measurement conditions, and performance on an independent test set. For pretrained models, overlap between pretraining and evaluation data can also affect interpretation. ResNet remains a useful reference architecture precisely because its broad structure is well understood, but comparisons still require control over the many experimental choices surrounding that structure."
    }
  ],
  "links": [
    "artificial-neural-network",
    "convolutional-neural-network",
    "deep-learning",
    "computer-vision",
    "alexnet",
    "imagenet-large-scale-visual-recognition-challenge",
    "vanishing-gradient-problem",
    "overfitting",
    "rectified-linear-unit",
    "supervised-learning",
    "loss-function",
    "gradient-boosting",
    "computational-graph",
    "batch-normalization",
    "backpropagation",
    "learning-rate",
    "data-augmentation",
    "pytorch",
    "tensorflow",
    "transfer-learning",
    "fine-tuning",
    "dropout",
    "ensemble-learning",
    "representation-learning",
    "image-classification",
    "imagenet",
    "object-detection",
    "semantic-segmentation",
    "image-processing",
    "self-supervised-learning",
    "transformer-architecture",
    "layer-normalization",
    "large-language-model",
    "speech-recognition",
    "diffusion-model",
    "neural-network-inference",
    "graphics-processing-unit",
    "neural-network-quantization",
    "generalization",
    "distribution-shift",
    "adversarial-example",
    "test-set"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}