RMSProp is an iterative optimization algorithm used to minimize differentiable objective functions, particularly when training artificial neural networks. It adjusts the update scale separately for each parameter by maintaining an exponentially weighted average of recent squared gradients. Dividing each gradient component by the square root of this average reduces updates along coordinates with consistently large gradients and relatively increases them along coordinates with small gradients. RMSProp belongs to the family of adaptive stochastic gradient methods used in machine learning.
Background and motivation
Training a model commonly involves minimizing a loss function with respect to a vector of parameters. Ordinary stochastic gradient descent updates those parameters using gradients estimated from individual examples or small batches of training data. A single global learning rate controls the magnitude of every parameter update.
This uniform scaling can be inefficient when different coordinates have very different gradient magnitudes. A step size suitable for one coordinate may produce excessive oscillation in another, while a step size small enough to avoid those oscillations may make progress elsewhere slow.
AdaGrad addresses this problem by dividing each gradient component by the square root of the accumulated sum of its past squared values. However, this accumulator never decreases. Over long training runs, the resulting effective learning rates can become very small.
RMSProp replaces the cumulative sum with an exponential moving average. Older squared gradients receive progressively less weight, allowing the algorithm to respond to changes in gradient scale rather than being dominated indefinitely by early observations.
The method became widely known through Geoffrey Hinton’s 2012 neural networks course, whose RMSProp lecture material credited Tijmen Tieleman with the approach. It subsequently became a standard optimizer in deep learning software.
Update equations
Let denote the parameter vector before update , and let
be the gradient of the current objective estimate. In mini-batch training, is the loss computed on the current batch. For neural networks, this gradient is typically obtained through backpropagation.
A common uncentered RMSProp formulation is
Here:
- is the running average of squared gradients, usually initialized to zero.
- , with , is the decay coefficient.
- is the global learning rate.
- is a numerical stabilizer.
- , division, and the square root operate element by element.
For coordinate , the update is therefore
The coordinate-specific effective learning rate is
Some implementations instead use in the denominator. These conventions are not numerically equivalent, especially when is small, so an value cannot always be transferred unchanged between implementations.
Interpretation
A local gradient-scale estimate
Expanding the accumulator, with , gives
Thus, a squared gradient from steps earlier has a weight proportional to . The quantity is a useful approximate measure of the averaging timescale, although the average has no sharp cutoff.
A large produces a smoother estimate that reacts slowly to changes. A smaller value responds more quickly but makes the denominator more sensitive to individual batches.
Root mean square rather than variance
In basic RMSProp, estimates an uncentered second moment of the gradient. Its square root approximates a recent root mean square gradient magnitude, which gives the algorithm its name.
This quantity is not generally a variance. For a scalar random gradient ,
Basic RMSProp tracks the first term, not the difference.
Diagonal preconditioning
RMSProp can also be viewed as a diagonal preconditioned gradient method:
where
This changes the scaling of parameter coordinates using gradient history. It does not estimate a full curvature matrix or account directly for interactions between coordinates, so it is not a full second-order optimization method.
Variants
RMSProp with momentum
RMSProp is often combined with momentum, which accumulates a direction of motion across updates. One convention is
where is the momentum coefficient.
Other conventions incorporate the learning rate into the momentum accumulator. Such formulations can behave differently when the learning rate changes over time. Consequently, “RMSProp with momentum” does not fully specify an implementation.
Centered RMSProp
Centered RMSProp additionally maintains an exponential average of the gradients:
It then uses a denominator based on
rather than alone. A representative update is
This estimates gradient variability around a running mean. It requires an additional parameter-sized state vector and differs from ordinary uncentered RMSProp. Numerical implementations must account for roundoff in the subtraction.
Relationship to other optimizers
| Optimizer | Gradient scaling or accumulation | Main distinction |
|---|---|---|
| Stochastic gradient descent | Uses a shared learning rate without coordinate-wise historical normalization | Simple update with little optimizer state |
| AdaGrad | Accumulates all past squared gradients | Its accumulator is nondecreasing |
| RMSProp | Exponentially averages squared gradients | Old gradient magnitudes gradually lose influence |
| [[adam-optimizer | Adam]] | Exponentially averages both gradients and squared gradients, with bias correction |
RMSProp and Adam share the use of a running squared-gradient statistic, but Adam is not simply another name for RMSProp with momentum. Its first-moment estimator, bias corrections, and precise update equations distinguish the algorithms.
Standard RMSProp commonly omits initialization-bias correction. Starting the accumulator at zero therefore affects its early updates. For example, ignoring , the first nonzero coordinate update has magnitude
regardless of that coordinate’s initial gradient magnitude. This illustrates how initialization and decay affect the transient behavior.
Applications and implementation
RMSProp has been used to train feedforward and recurrent neural networks, as well as neural models in reinforcement learning. Its moving average is useful when gradient scales change during optimization, although this does not guarantee successful adaptation to arbitrary changes in the objective.
Implementations are available in major frameworks, including PyTorch and TensorFlow. Their defaults and numerical conventions can differ. A reproducible specification includes the learning rate, decay coefficient, stabilizer and its placement, momentum convention, centered setting, and any regularization or learning-rate schedule.
For parameters, basic RMSProp requires additional optimizer storage and arithmetic per update, excluding gradient computation. Momentum and centering each add another parameter-sized state vector.
The learning rate and decay coefficient remain important hyperparameters. Adaptive coordinate scaling does not remove the need to choose an overall update scale.
Limitations
No universal convergence guarantee. The moving squared-gradient average can decrease as well as increase, so effective coordinate-wise learning rates can grow. Basic RMSProp does not inherit AdaGrad’s monotone accumulator structure. Convergence results require assumptions and may involve particular learning-rate schedules or algorithmic modifications.
Sensitivity to settings and transients. A small stabilizer can permit large effective learning rates when the running squared gradient is tiny. A large stabilizer can dominate the denominator and weaken adaptive scaling. The decay coefficient controls the trade-off between responsiveness and smoothing.
Limited geometric information. Coordinate-wise rescaling cannot directly represent correlations between parameters. Rotating the parameterization can change the algorithm’s behavior substantially.
No automatic solution to unstable gradients. RMSProp may moderate differences in gradient magnitude, but it does not by itself eliminate numerical overflow or the vanishing-gradient problem.
Optimization and generalization are distinct. Faster reduction of training loss does not necessarily imply better generalization on unseen data. Relative performance depends on the model, data, regularization, training budget, and optimizer settings.