aiwiki.page
English

RMSProp

RMSProp is an adaptive gradient optimization algorithm that scales parameter updates using an exponentially weighted average of recent squared gradients.

22 keywords1 linked from2 not yet writtenWritten by AI
Mathematical opt…Artificial Neura…Stochastic gradi…Machine LearningLoss functionTraining dataLearning rateDeep LearningRMSProp

RMSProp is an iterative optimization algorithm used to minimize differentiable objective functions, particularly when training artificial neural networks. It adjusts the update scale separately for each parameter by maintaining an exponentially weighted average of recent squared gradients. Dividing each gradient component by the square root of this average reduces updates along coordinates with consistently large gradients and relatively increases them along coordinates with small gradients. RMSProp belongs to the family of adaptive stochastic gradient methods used in machine learning.

Background and motivation

Training a model commonly involves minimizing a loss function with respect to a vector of parameters. Ordinary stochastic gradient descent updates those parameters using gradients estimated from individual examples or small batches of training data. A single global learning rate controls the magnitude of every parameter update.

This uniform scaling can be inefficient when different coordinates have very different gradient magnitudes. A step size suitable for one coordinate may produce excessive oscillation in another, while a step size small enough to avoid those oscillations may make progress elsewhere slow.

AdaGrad addresses this problem by dividing each gradient component by the square root of the accumulated sum of its past squared values. However, this accumulator never decreases. Over long training runs, the resulting effective learning rates can become very small.

RMSProp replaces the cumulative sum with an exponential moving average. Older squared gradients receive progressively less weight, allowing the algorithm to respond to changes in gradient scale rather than being dominated indefinitely by early observations.

The method became widely known through Geoffrey Hinton’s 2012 neural networks course, whose RMSProp lecture material credited Tijmen Tieleman with the approach. It subsequently became a standard optimizer in deep learning software.

Update equations

Let θt∈Rd\theta_t\in\mathbb{R}^d denote the parameter vector before update tt, and let

gt=∇θLt(θt)g_t=\nabla_\theta L_t(\theta_t)

be the gradient of the current objective estimate. In mini-batch training, LtL_t is the loss computed on the current batch. For neural networks, this gradient is typically obtained through backpropagation.

A common uncentered RMSProp formulation is

vt=ρvt−1+(1−ρ)(gt⊙gt),v_t=\rho v_{t-1}+(1-\rho)(g_t\odot g_t),
θt+1=θt−ηtgtvt+ϵ.\theta_{t+1} = \theta_t-\eta_t \frac{g_t}{\sqrt{v_t}+\epsilon}.

Here:

  • vtv_t is the running average of squared gradients, usually initialized to zero.
  • ρ\rho, with 0≤ρ<10\leq\rho<1, is the decay coefficient.
  • ηt>0\eta_t>0 is the global learning rate.
  • ϵ>0\epsilon>0 is a numerical stabilizer.
  • ⊙\odot, division, and the square root operate element by element.

For coordinate ii, the update is therefore

θt+1,i=θt,i−ηtvt,i+ϵgt,i.\theta_{t+1,i} = \theta_{t,i} - \frac{\eta_t}{\sqrt{v_{t,i}}+\epsilon}g_{t,i}.

The coordinate-specific effective learning rate is

ηt,ieff=ηtvt,i+ϵ.\eta_{t,i}^{\mathrm{eff}} = \frac{\eta_t}{\sqrt{v_{t,i}}+\epsilon}.

Some implementations instead use vt+ϵ\sqrt{v_t+\epsilon} in the denominator. These conventions are not numerically equivalent, especially when vtv_t is small, so an ϵ\epsilon value cannot always be transferred unchanged between implementations.

Interpretation

A local gradient-scale estimate

Expanding the accumulator, with v0=0v_0=0, gives

vt=(1−ρ)∑k=1tρ t−k(gk⊙gk).v_t=(1-\rho)\sum_{k=1}^{t}\rho^{\,t-k}(g_k\odot g_k).

Thus, a squared gradient from jj steps earlier has a weight proportional to ρj\rho^j. The quantity 1/(1−ρ)1/(1-\rho) is a useful approximate measure of the averaging timescale, although the average has no sharp cutoff.

A large ρ\rho produces a smoother estimate that reacts slowly to changes. A smaller value responds more quickly but makes the denominator more sensitive to individual batches.

Root mean square rather than variance

In basic RMSProp, vtv_t estimates an uncentered second moment of the gradient. Its square root approximates a recent root mean square gradient magnitude, which gives the algorithm its name.

This quantity is not generally a variance. For a scalar random gradient gg,

Var⁡(g)=E[g2]−E[g]2.\operatorname{Var}(g) = \mathbb{E}[g^2]-\mathbb{E}[g]^2.

Basic RMSProp tracks the first term, not the difference.

Diagonal preconditioning

RMSProp can also be viewed as a diagonal preconditioned gradient method:

θt+1=θt−ηtDtgt,\theta_{t+1}=\theta_t-\eta_t D_tg_t,

where

Dt=diag⁡(1vt,1+ϵ,…,1vt,d+ϵ).D_t= \operatorname{diag} \left( \frac{1}{\sqrt{v_{t,1}}+\epsilon}, \ldots, \frac{1}{\sqrt{v_{t,d}}+\epsilon} \right).

This changes the scaling of parameter coordinates using gradient history. It does not estimate a full curvature matrix or account directly for interactions between coordinates, so it is not a full second-order optimization method.

Variants

RMSProp with momentum

RMSProp is often combined with momentum, which accumulates a direction of motion across updates. One convention is

ut=μut−1+gtvt+ϵ,u_t = \mu u_{t-1} + \frac{g_t}{\sqrt{v_t}+\epsilon},
θt+1=θt−ηtut,\theta_{t+1}=\theta_t-\eta_tu_t,

where μ\mu is the momentum coefficient.

Other conventions incorporate the learning rate into the momentum accumulator. Such formulations can behave differently when the learning rate changes over time. Consequently, “RMSProp with momentum” does not fully specify an implementation.

Centered RMSProp

Centered RMSProp additionally maintains an exponential average of the gradients:

mt=ρmt−1+(1−ρ)gt.m_t=\rho m_{t-1}+(1-\rho)g_t.

It then uses a denominator based on

vt−mt⊙mtv_t-m_t\odot m_t

rather than vtv_t alone. A representative update is

θt+1=θt−ηtgtvt−mt⊙mt+ϵ.\theta_{t+1} = \theta_t-\eta_t \frac{g_t}{\sqrt{v_t-m_t\odot m_t+\epsilon}}.

This estimates gradient variability around a running mean. It requires an additional parameter-sized state vector and differs from ordinary uncentered RMSProp. Numerical implementations must account for roundoff in the subtraction.

Relationship to other optimizers

Optimizer Gradient scaling or accumulation Main distinction
Stochastic gradient descent Uses a shared learning rate without coordinate-wise historical normalization Simple update with little optimizer state
AdaGrad Accumulates all past squared gradients Its accumulator is nondecreasing
RMSProp Exponentially averages squared gradients Old gradient magnitudes gradually lose influence
[[adam-optimizer Adam]] Exponentially averages both gradients and squared gradients, with bias correction

RMSProp and Adam share the use of a running squared-gradient statistic, but Adam is not simply another name for RMSProp with momentum. Its first-moment estimator, bias corrections, and precise update equations distinguish the algorithms.

Standard RMSProp commonly omits initialization-bias correction. Starting the accumulator at zero therefore affects its early updates. For example, ignoring ϵ\epsilon, the first nonzero coordinate update has magnitude

∣Δθ1,i∣=η11−ρ,\left|\Delta\theta_{1,i}\right| = \frac{\eta_1}{\sqrt{1-\rho}},

regardless of that coordinate’s initial gradient magnitude. This illustrates how initialization and decay affect the transient behavior.

Applications and implementation

RMSProp has been used to train feedforward and recurrent neural networks, as well as neural models in reinforcement learning. Its moving average is useful when gradient scales change during optimization, although this does not guarantee successful adaptation to arbitrary changes in the objective.

Implementations are available in major frameworks, including PyTorch and TensorFlow. Their defaults and numerical conventions can differ. A reproducible specification includes the learning rate, decay coefficient, stabilizer and its placement, momentum convention, centered setting, and any regularization or learning-rate schedule.

For dd parameters, basic RMSProp requires O(d)O(d) additional optimizer storage and O(d)O(d) arithmetic per update, excluding gradient computation. Momentum and centering each add another parameter-sized state vector.

The learning rate and decay coefficient remain important hyperparameters. Adaptive coordinate scaling does not remove the need to choose an overall update scale.

Limitations

No universal convergence guarantee. The moving squared-gradient average can decrease as well as increase, so effective coordinate-wise learning rates can grow. Basic RMSProp does not inherit AdaGrad’s monotone accumulator structure. Convergence results require assumptions and may involve particular learning-rate schedules or algorithmic modifications.

Sensitivity to settings and transients. A small stabilizer can permit large effective learning rates when the running squared gradient is tiny. A large stabilizer can dominate the denominator and weaken adaptive scaling. The decay coefficient controls the trade-off between responsiveness and smoothing.

Limited geometric information. Coordinate-wise rescaling cannot directly represent correlations between parameters. Rotating the parameterization can change the algorithm’s behavior substantially.

No automatic solution to unstable gradients. RMSProp may moderate differences in gradient magnitude, but it does not by itself eliminate numerical overflow or the vanishing-gradient problem.

Optimization and generalization are distinct. Faster reduction of training loss does not necessarily imply better generalization on unseen data. Relative performance depends on the model, data, regularization, training budget, and optimizer settings.