Leaky ReLU is an activation function used in artificial neural networks that passes positive inputs through unchanged and multiplies negative inputs by a small positive constant. It modifies the rectified linear unit (ReLU) by replacing its flat negative branch with a sloping line. This allows negative inputs to produce nonzero outputs and transmit gradients during training, reducing a mechanism by which ordinary ReLU units can become inactive.
Definition
For a real-valued input , Leaky ReLU is usually defined as
where is a fixed hyperparameter, conventionally satisfying . The parameter is called the negative slope or leak coefficient. A frequently used value is , although there is no universally optimal setting.
For this conventional parameter range, an equivalent expression is
For example, with , an input of produces , whereas an input of produces . Ordinary ReLU would produce zero for the latter input.
When applied to a vector or tensor, the function usually operates independently on each element. It does not mix information between channels or features.
The parameter choices and recover ordinary ReLU and the identity function, respectively. The latter removes the activation’s nonlinearity.
Mathematical properties
Leaky ReLU is continuous and piecewise linear. For , it is strictly increasing, convex, and unbounded in both directions. Its derivative away from zero is
Unless , the function is not differentiable at zero. Software implementations select a convention for the backward calculation at that point; this convention is not a classical derivative.
Leaky ReLU is positively homogeneous:
Like ordinary ReLU, it therefore preserves scaling by nonnegative factors. Unlike ReLU, it does not map an entire negative half-line to one value. For , its scalar mapping is invertible, although a neural network containing Leaky ReLU need not be invertible.
Gradient flow and inactive units
Neural networks commonly use backpropagation to calculate how a loss function changes with model parameters. If a unit has preactivation , the local gradient relationship is
For ordinary ReLU, when . If a unit produces negative preactivations for every relevant training example, gradients through that activation vanish, preventing its incoming parameters from receiving updates through that path. This is commonly called the dying ReLU problem.
Leaky ReLU replaces the zero local multiplier with . Negative-side activations can therefore contribute gradients, allowing an otherwise inactive unit to change during optimization.
This does not guarantee that every unit receives a useful update. The upstream gradient may itself be zero, and a small still attenuates the gradient. Across many layers, repeated multiplication by small slopes can contribute to the vanishing gradient problem. Leaky ReLU addresses one source of blocked gradients, not all difficulties in training deep networks.
Development and applications
The leaky rectifier was described by Andrew L. Maas, Awni Y. Hannun, and Andrew Y. Ng in their 2013 work on rectifier nonlinearities for neural-network acoustic models. It became a standard member of the family of rectifier activations used in deep learning.
Leaky ReLU can serve as a hidden-layer activation in multilayer perceptrons and convolutional neural networks. It has also been used in discriminators in generative adversarial networks, where retaining a negative-side gradient provides an alternative to a hard cutoff.
Its computational cost is low: each element requires a sign-dependent choice between the input and a scaled input. For a fixed slope, it introduces no trainable parameters.
Libraries including PyTorch and TensorFlow provide implementations. Parameter names and default slopes may differ between interfaces, so specifying only “Leaky ReLU” does not completely determine a model’s numerical behavior.
Related activation functions
Several related functions change the treatment of negative inputs:
- Ordinary ReLU outputs zero for every negative input. It creates exact zero activations but can block gradients on the negative branch.
- Parametric ReLU (PReLU) uses the same basic piecewise formula but learns the negative slope during training. The slope may be shared across units or assigned separately to channels.
- Randomized Leaky ReLU samples the negative slope from a specified interval during training. Some implementations use a fixed representative slope during inference.
- Exponential linear unit (ELU) uses an exponential negative branch. Unlike Leaky ReLU, its negative output approaches a finite lower bound.
These distinctions affect gradient propagation, activation distributions, computational requirements, and whether additional parameters must be learned.
Limitations and design trade-offs
Leaky ReLU exchanges ReLU’s exact negative-side zeros for continued sensitivity to negative inputs. Consequently, it does not produce the same activation sparsity: negative inputs generally yield small negative outputs rather than zeros.
The negative slope controls a trade-off. A very small value closely approximates ReLU but permits only weak negative-side gradients. A value approaching one makes the function nearly linear, weakening the nonlinearity supplied by the activation. Its effectiveness depends on the architecture, data, optimization settings, and interactions with components such as batch normalization and residual connections.
Leaky ReLU does not guarantee zero-centered activations, stable activation variance, faster convergence, or improved predictive accuracy. Allowing negative outputs does not by itself make their mean zero. It also remains nondifferentiable at the origin.
Initialization schemes can account for its negative slope. Under the simplifying assumptions used in variance-preserving initialization, a common generalized rectifier scaling is
where is the number of inputs to a unit. This compensates approximately for the activation’s effect on signal magnitude; it does not ensure stable training in arbitrary architectures.