A gated recurrent unit (GRU) is a type of recurrent neural network component that processes sequential data by maintaining a hidden state and updating it through learned gates. Introduced by Kyunghyun Cho and colleagues in 2014, it is designed to learn dependencies across time while reducing some of the training difficulties of conventional recurrent networks. Like long short-term memory (LSTM), a GRU provides a gated mechanism for retaining information, but it uses a single recurrent state rather than separate hidden and cell states.
Architecture and equations
A GRU receives an input vector and the previous hidden state , then produces a new hidden state . Its two principal gates are the update gate, which controls how much of the previous state is retained, and the reset gate, which controls how the previous state contributes to a candidate replacement.
For input dimension and hidden dimension , one common formulation is:
Here:
- is the update gate.
- is the reset gate.
- is the candidate hidden state.
- is the logistic sigmoid.
- is the hyperbolic tangent.
- denotes elementwise multiplication.
The input-weight matrices have shape , the recurrent matrices have shape , and the biases have dimension . These parameters are learned during training and reused at every time step. Sigmoid and hyperbolic tangent functions serve as activation functions.
With the convention above, an update-gate value near one preserves the corresponding component of the previous state, whereas a value near zero replaces it with the candidate. Some descriptions instead define the update gate as the fraction of new information and write
These are complementary gate conventions, not fundamentally different architectures. Statements about whether a “large update gate” retains or replaces information therefore depend on the equations being used.
How the gates control memory
Update gate
The update gate creates an additive path between successive hidden states. Each state component can preserve its previous value, adopt a candidate value, or interpolate between them. This allows different components to learn different effective timescales: some can track rapidly changing features, while others retain information across many inputs.
In a conventional recurrent network, the state is repeatedly transformed through a recurrent matrix and a nonlinear function. Repeating these transformations can cause gradients to shrink, contributing to the vanishing-gradient problem. A GRU's gated additive update provides a path through which information and gradients can persist more readily.
This mechanism does not guarantee perfect long-term memory. The gates depend on the input and previous state, so the full derivative contains additional terms beyond the direct retention path. Gates may also saturate, and gradients may still vanish or grow excessively.
Reset gate
The reset gate controls how much of the previous state enters the candidate computation. In the formulation above, a reset-gate component near zero suppresses the corresponding previous-state component before the recurrent transformation. This allows the network to construct a candidate that depends less on earlier context.
Resetting the candidate computation is not the same as erasing the final hidden state. Even when the reset gate suppresses past information in the candidate, the update gate can still preserve the previous state through its separate retention path.
The two gates therefore perform distinct functions: the reset gate controls the context used to propose new information, while the update gate controls how much of that proposal becomes the current state.
History and relationship to LSTM
The GRU was introduced in the 2014 paper Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation. The work used recurrent networks in an encoder–decoder architecture to learn representations of phrase pairs for machine translation.
Both GRUs and LSTMs belong to the broader family of gated recurrent architectures in deep learning. They were developed to make learning sequential dependencies more effective than in simple recurrent networks.
A standard LSTM has a cell state and a hidden state, together with input, forget, and output gates. A standard GRU has one recurrent state and two gates. Its update gate combines functions related to retaining old information and admitting new information, and it has no separate output gate.
For equal input and hidden dimensions, a conventional GRU usually has fewer parameters than a conventional LSTM. This can reduce computation and storage, but it does not establish that either architecture is universally faster or more accurate. Results depend on the task, model dimensions, training procedure, sequence lengths, and software implementation.
Training and sequence organization
GRUs are typically trained using backpropagation through time: the recurrence is expanded across a sequence, and backpropagation computes gradients of a task-specific loss function with respect to shared parameters. Optimization methods such as the Adam optimizer can then update those parameters.
For long sequences, training may use truncated backpropagation through time. The hidden state can be carried across consecutive sequence segments while the gradient is propagated only through a limited number of steps. This reduces training memory requirements but limits the temporal range over which learning directly assigns credit.
A GRU may be organized in several ways:
- Single-layer: one recurrent layer processes the sequence.
- Stacked: multiple recurrent layers are placed above one another, with each layer receiving the preceding layer's outputs.
- Bidirectional: separate recurrent networks process a sequence in forward and reverse order, and their outputs are combined.
- Encoder–decoder: one network encodes an input sequence, while another generates an output sequence, optionally using an attention mechanism.
A bidirectional GRU uses future as well as past context and therefore cannot directly provide strictly causal outputs before that future context arrives.
The initial hidden state is commonly set to zero, learned as a parameter, or supplied by another model component. The hidden state is not necessarily the final prediction: a separate output layer can map it to classes, numerical values, or a probability distribution over possible outputs.
Implementation variants
The placement of the reset operation differs between important GRU implementations. The candidate equation given earlier applies the reset gate before the recurrent matrix multiplication:
Another common form applies it after the recurrent transformation:
These expressions are generally not equivalent, because elementwise gating does not generally commute with multiplication by a dense matrix. Bias placement also differs.
Libraries such as PyTorch and TensorFlow provide GRU layers, but matching a model across implementations requires checking the exact recurrence, gate ordering, bias layout, and parameter storage conventions. Matching only the layer name and hidden dimension is insufficient to ensure numerical equivalence.
Computational properties
With one bias vector for each of its three affine computations, a GRU layer has
trainable parameters, excluding any output layer. Implementations with separate input and recurrent biases commonly have
For a sequence of length , the principal arithmetic cost of a dense GRU layer is
ignoring batching and lower-order elementwise operations.
A unidirectional GRU needs only its current hidden state to carry information from one step to the next during streaming inference. Its recurrent-state storage is therefore independent of the number of already processed steps. Training requires additional storage for intermediate activations unless recomputation or truncation is used.
Recurrence imposes a sequential dependency: must be available before can be computed. Matrix operations within a step and processing across independent sequences can exploit parallel computing, but ordinary GRUs do not expose the same across-position parallelism during training as a Transformer.
Applications and limitations
GRUs are used in natural language processing, speech recognition, time-series forecasting, sequence classification, and anomaly detection. They are particularly relevant when a model must update a compact state as observations arrive, rather than repeatedly process an entire history.
Their main limitations include:
- A finite-state bottleneck: relevant history must be represented in a fixed-dimensional state unless an additional memory mechanism is provided.
- Imperfect long-range learning: gating helps retain information but does not ensure that arbitrarily distant dependencies can be learned.
- Sequential execution: time-step dependencies can limit hardware utilization and training throughput.
- Sensitivity to model and training choices: hidden-state size, initialization, optimization, and regularization affect performance.
- Limited direct interpretability: gate values describe internal information flow, but they do not by themselves establish why a prediction was made.
Transformers have displaced recurrent architectures in many large-scale language tasks, while GRUs remain useful for compact sequence models and stateful processing. Comparisons with LSTMs, Transformers, or other sequence architectures require task-specific evaluation rather than an assumption that fewer gates or parameters necessarily produce better results.