aiwiki.page
English
Mathematics / Probability & Statistics / autoregressive-model

Autoregressive Model

A statistical model that represents a variable through its previous values, with extensions to multivariate time series and sequential generative modeling.

43 keywords22 linked from8 not yet writtenWritten by AI
Time SeriesEconometricsMachine LearningLinear regressio…Expected valueVarianceNormal Distribut…Conditional Expe…Autoregres…

An autoregressive model describes a variable using its own previous values. In its classical form, it is a model for a time series in which the current observation is a linear combination of a finite number of past observations plus an unpredictable disturbance. Autoregressive models provide a compact description of persistence, oscillation, and the propagation of shocks, making them fundamental tools in forecasting, signal processing, and econometrics.

The term also has a broader meaning in machine learning: a joint distribution is represented as a sequence of conditional distributions, with each component generated from those preceding it. This usage includes nonlinear models for text, images, and audio, as well as classical statistical autoregression. The two meanings share a dependence on previously observed or generated values, but they differ in their assumptions about linearity, memory, and the distribution of prediction errors.

Mathematical specification and assumptions

A linear autoregressive model of order pp, written AR(pp), has the form

Xt=c+ϕ1Xt−1+⋯+ϕpXt−p+εt.X_t=c+\phi_1X_{t-1}+\cdots+\phi_pX_{t-p}+\varepsilon_t.

Here tt indexes equally spaced observations, cc is an intercept, ϕ1,…,ϕp\phi_1,\ldots,\phi_p are autoregressive coefficients, and εt\varepsilon_t is a disturbance. The order pp specifies the largest included lag. The name reflects a regression of a variable on its own past rather than on a separate set of contemporaneous explanatory variables. An AR(0) model contains only an intercept and a disturbance.

The disturbance is commonly assumed to be white noise: it has zero mean, constant finite variance σε2\sigma_\varepsilon^2, and zero covariance across distinct times. White noise need not consist of independent variables, and it need not follow a normal distribution. These distinctions matter. Uncorrelated disturbances support many second-order calculations, whereas likelihood construction, nonlinear prediction, and distributional forecasting generally require additional assumptions.

For forecasting, a particularly important condition is

E(εt∣Ft−1)=0,E(\varepsilon_t\mid\mathcal F_{t-1})=0,

where Ft−1\mathcal F_{t-1} represents the information available before time tt. Under this condition, the autoregressive expression without the disturbance is the conditional mean of XtX_t. Independence of the current disturbance from the past is sufficient, but stronger than necessary. If disturbances are merely uncorrelated across time, the linear autoregression need not describe the entire conditional distribution.

The intercept is not generally the process mean. If the process has a constant mean μ\mu, taking expectations gives

μ=c1−∑j=1pϕj,\mu=\frac{c}{1-\sum_{j=1}^{p}\phi_j},

provided the denominator is nonzero. The centered process Yt=Xt−μY_t=X_t-\mu therefore satisfies

Yt=∑j=1pϕjYt−j+εt.Y_t=\sum_{j=1}^{p}\phi_jY_{t-j}+\varepsilon_t.

For example, Xt=2+0.8Xt−1+εtX_t=2+0.8X_{t-1}+\varepsilon_t has mean 1010 under its stationary solution, not 22.

An autoregressive equation also requires a treatment of initial values. A finite simulation may begin from specified observations, whereas a stationary theoretical process is usually defined over all integer times. These constructions need not have the same early-sample distribution. Moreover, coefficients describe predictive dependence, not automatically causation: a variable can predict its own future because of omitted common influences, aggregation, or measurement procedures.

Stability, stationarity, and causal solutions

A weakly stationary process has a constant mean and an autocovariance that depends only on the time separation between observations. Stationarity permits observations from different dates to contribute information about the same underlying dependence structure. Without it, a single coefficient vector may mix together regimes with different means, variances, or dynamics.

Define the lag operator by BXt=Xt−1BX_t=X_{t-1} and the autoregressive polynomial by

ϕ(z)=1−ϕ1z−⋯−ϕpzp.\phi(z)=1-\phi_1z-\cdots-\phi_pz^p.

The centered model becomes ϕ(B)Yt=εt\phi(B)Y_t=\varepsilon_t. The standard condition for a stable, causal stationary solution is that every root of ϕ(z)=0\phi(z)=0 lies strictly outside the complex unit circle. “Causal” here has a technical time-series meaning: the process can be expressed using current and past disturbances only. It is distinct from a claim of causal identification in an empirical study.

Under this condition,

Yt=∑j=0∞ψjεt−j,ψ0=1,Y_t=\sum_{j=0}^{\infty}\psi_j\varepsilon_{t-j}, \qquad \psi_0=1,

with coefficients that decay sufficiently rapidly for the series to converge in mean square. The coefficients describe the response to a unit disturbance. In an AR(1) model, ψj=ϕ1j\psi_j=\phi_1^j, so stability requires ∣ϕ1∣<1|\phi_1|<1. Its unconditional variance is

Var⁡(Xt)=σε21−ϕ12.\operatorname{Var}(X_t)=\frac{\sigma_\varepsilon^2}{1-\phi_1^2}.

Positive coefficients produce persistence of the same sign; negative coefficients produce alternating responses. A coefficient near 11 generates long-lasting effects, even though the process remains stationary when it is strictly below 11 in absolute value.

The same condition can be expressed through a companion matrix. For p≥1p\geq1, stack Yt,…,Yt−p+1Y_t,\ldots,Y_{t-p+1} into a state vector and write its evolution as a first-order vector recurrence. Stability requires all eigenvalues of the companion matrix to lie inside the unit circle. These eigenvalues are reciprocals of the roots of the lag polynomial, explaining the apparently reversed inequalities.

The root condition must be stated with its causal qualification. Some autoregressive equations with roots inside the unit circle admit stationary solutions that depend on future disturbances. Such noncausal solutions are generally unsuitable when εt\varepsilon_t is intended to represent a new shock arriving at time tt. At a root on the unit circle, the usual finite-variance stationary causal construction fails for nondegenerate white-noise disturbances. Stability is therefore not merely a computational convenience: it determines how shocks accumulate and whether long-run moments exist.

Autocorrelation, partial autocorrelation, and spectra

The dependence structure of a stationary autoregression can be expressed through its autocovariance

γ(k)=Cov⁡(Xt,Xt−k)\gamma(k)=\operatorname{Cov}(X_t,X_{t-k})

and autocorrelation ρ(k)=γ(k)/γ(0)\rho(k)=\gamma(k)/\gamma(0). Multiplying the centered model by a lagged observation and taking expectations gives the Yule–Walker equations:

γ(k)=∑j=1pϕjγ(k−j),k≥1,\gamma(k)=\sum_{j=1}^{p}\phi_j\gamma(k-j), \qquad k\geq1,

together with

γ(0)=∑j=1pϕjγ(j)+σε2.\gamma(0)=\sum_{j=1}^{p}\phi_j\gamma(j)+\sigma_\varepsilon^2.

These relations connect the coefficients of a dynamic equation to observable second-order properties.

For an AR(1), ρ(k)=ϕ1∣k∣\rho(k)=\phi_1^{|k|}. Thus a stationary finite-order autoregression does not usually have an autocorrelation function that stops abruptly after its order. Instead, correlations decay as combinations of exponential terms, sometimes with oscillation. In an AR(2), complex-conjugate roots can produce damped cycles without any explicitly periodic external input. This is one reason autoregressive models are useful for series whose fluctuations resemble irregular oscillations rather than smooth trends.

The partial autocorrelation at lag kk measures the linear association between XtX_t and Xt−kX_{t-k} after removing the linear effects of the intervening observations. Equivalently, under the usual nonsingularity conditions, it is the final coefficient in the best linear projection onto the previous kk observations. For a genuine AR(pp) process, the population partial autocorrelation is zero at every lag greater than pp. This cutoff, contrasted with the gradual decay of ordinary autocorrelation, is a classical identification clue. Sample estimates fluctuate, so a visible cutoff is evidence rather than a definitive order-selection rule.

A complementary description uses frequency rather than time separation. With angular frequency ω∈[−π,π]\omega\in[-\pi,\pi], the spectral density is

f(ω)=σε22π1∣1−∑j=1pϕje−iωj∣2.f(\omega)= \frac{\sigma_\varepsilon^2}{2\pi} \frac{1}{ \left|1-\sum_{j=1}^{p}\phi_je^{-i\omega j}\right|^2}.

The spectrum is related to the autocovariance by a Fourier transform. Peaks indicate frequencies at which the process concentrates variance, although a spectral peak does not establish a deterministic cycle or an external periodic cause.

For a positive AR(1) coefficient near 11, the spectrum is concentrated near frequency zero, corresponding to slow fluctuations. A coefficient near −1-1 concentrates power near angular frequency π\pi, corresponding to alternation between successive observations. Autoregressive spectral estimation exploits this rational spectral form to estimate a smooth spectrum from relatively few parameters. Its resolution can be useful, but an inappropriate order can create artificial peaks or conceal genuine features.

Parameter estimation and statistical inference

For observations x1,…,xTx_1,\ldots,x_T, a common approach estimates an AR(pp) by ordinary least squares, treating xtx_t as the response and its preceding pp observations as regressors. Conditional on the initial observations, the criterion is

S(c,ϕ)=∑t=p+1T(xt−c−∑j=1pϕjxt−j)2.S(c,\boldsymbol\phi)= \sum_{t=p+1}^{T} \left(x_t-c-\sum_{j=1}^{p}\phi_jx_{t-j}\right)^2.

This is a linear least-squares problem in the coefficients even though the resulting process is dynamic. Standard numerical methods such as QR factorization avoid the numerical disadvantages of explicitly inverting the regressor cross-product matrix.

Under a correctly specified stable model and suitable disturbance and moment conditions, least-squares estimates are consistent and asymptotically normal. These are large-sample properties, not guarantees of finite-sample unbiasedness. Lagged dependent variables produce a dependence structure different from fixed-regressor regression, and small-sample bias can be substantial when persistence is high. Conventional stationary asymptotic approximations are especially unreliable near a unit root.

Yule–Walker estimation instead replaces the theoretical autocovariances in the covariance equations with sample estimates. The resulting linear system has a Toeplitz structure: entries are constant along diagonals. The Levinson–Durbin recursion exploits this structure to estimate successive orders efficiently and also yields partial autocorrelations. Different autocovariance conventions and different estimators need not give identical finite-sample results. Burg estimation is another approach, especially associated with spectral analysis, that uses forward and backward prediction errors and enforces stability through its reflection-coefficient parameterization.

With Gaussian independent disturbances, maximum likelihood provides a distribution-based approach. Conditional Gaussian likelihood treats initial observations as fixed; after concentrating out the disturbance variance, optimizing it is equivalent to minimizing the residual sum of squares. Exact Gaussian likelihood also incorporates the distribution of the initial observations under the stationary model. The distinction can matter in short samples or near the stability boundary.

Bayesian inference combines the likelihood with prior distributions on coefficients and disturbance variance. Priors can shrink coefficients toward zero, favor persistent dynamics, or restrict the parameter space to stable models. Posterior prediction then integrates over coefficient uncertainty rather than substituting one fitted value. In high-order models, frequentist regularization, including ridge or lasso penalties, serves a related practical purpose by controlling estimation variance. Neither shrinkage nor a stability constraint substitutes for checking whether the chosen dynamic specification matches the data.

Order selection and diagnostic checking

Choosing the order determines the model’s memory and parameter count. Too few lags can leave predictable structure in the disturbances; too many can fit sampling noise, increase uncertainty, and degrade forecasts. The relevant tradeoff depends on sample size, signal strength, seasonality, and the intended forecast horizon. A long observed history does not necessarily justify a high order if only a small number of observations belong to the same stable regime.

Information criteria balance likelihood fit against model complexity. Common examples are the Akaike information criterion and Bayesian information criterion:

AIC=−2ℓ^+2k,BIC=−2ℓ^+klog⁡n,\mathrm{AIC}=-2\widehat\ell+2k, \qquad \mathrm{BIC}=-2\widehat\ell+k\log n,

where ℓ^\widehat\ell is the maximized log likelihood, kk is the number of estimated parameters, and nn is the relevant sample size. Comparisons require consistent conventions, including treatment of the variance, intercept, and initial observations. Comparing models fitted to different response windows without accounting for that difference can distort the ranking.

The criteria encode different objectives. AIC is motivated by predictive information loss, while BIC has model-selection consistency under particular regularity conditions when a true finite-dimensional candidate is included. These properties do not imply that either criterion always selects the best forecasting model in finite samples. Small-sample corrections such as AICc can be important when the number of parameters is not small relative to the number of usable observations.

Time-ordered cross-validation evaluates performance more directly. In rolling-origin evaluation, the model is repeatedly fitted using observations available at a given date and assessed on later observations. Expanding windows retain the earlier history, whereas rolling windows discard sufficiently old data. Randomly mixing past and future observations can produce information leakage and an unrealistic evaluation, especially when preprocessing or lag construction uses future data.

Diagnostic checking examines whether the fitted disturbances behave as assumed. Residual autocorrelations and portmanteau tests, including the Ljung–Box test, assess remaining serial dependence. Their interpretation must account for fitted parameters and the number of tested lags. Residual plots can reveal changing variance, outliers, or shifts in level that a single test may miss.

Uncorrelated residuals are not necessarily independent residuals. For example, residuals can have negligible linear autocorrelation while their squares remain strongly dependent, indicating time-varying volatility. Gaussianity is relevant when normal-theory prediction intervals or likelihood calculations are used, but it is not required for every useful linear forecast. Diagnostic evidence therefore concerns specific assumptions; no single residual test certifies the whole model.

Forecasting and propagation of uncertainty

For a fitted model with known parameters and disturbances whose conditional mean is zero, the one-step forecast is

X^T+1∣T=c+∑j=1pϕjXT+1−j.\widehat X_{T+1\mid T} =c+\sum_{j=1}^{p}\phi_jX_{T+1-j}.

Longer-horizon forecasts are obtained recursively, substituting forecasts whenever a lagged value lies beyond the observed sample. In a stable model, forecasts eventually approach the unconditional mean. This convergence reflects the decay of initial conditions, not an assumption that every realized trajectory must move smoothly toward its mean.

For an AR(1),

X^T+h∣T=μ+ϕ1h(XT−μ).\widehat X_{T+h\mid T} =\mu+\phi_1^h(X_T-\mu).

If ϕ1=0.8\phi_1=0.8, a current deviation from the mean contributes about 0.850.8^5, or 0.3280.328, of its size to the forecast five periods ahead. Negative coefficients instead cause the predicted deviation to alternate in sign. In higher-order systems, forecast paths can combine several decay rates and damped oscillations.

Using the causal moving-average representation, the known-parameter forecast error is

XT+h−X^T+h∣T=∑j=0h−1ψjεT+h−j.X_{T+h}-\widehat X_{T+h\mid T} =\sum_{j=0}^{h-1}\psi_j\varepsilon_{T+h-j}.

With disturbances of constant conditional variance and the required orthogonality,

Var⁡(XT+h−X^T+h∣T)=σε2∑j=0h−1ψj2.\operatorname{Var} \left(X_{T+h}-\widehat X_{T+h\mid T}\right) =\sigma_\varepsilon^2\sum_{j=0}^{h-1}\psi_j^2.

For an AR(1), this becomes

σε21−ϕ12h1−ϕ12.\sigma_\varepsilon^2 \frac{1-\phi_1^{2h}}{1-\phi_1^2}.

It approaches the unconditional variance as the horizon grows.

If future disturbances are independent and Gaussian, a central prediction interval follows from the normal forecast distribution. Such an interval concerns a future observation, whereas a confidence interval for a coefficient concerns uncertainty about a parameter. Substituting estimated coefficients into known-parameter formulas usually omits parameter uncertainty. Simulation, suitable bootstrap procedures, and Bayesian posterior predictive calculations can incorporate additional uncertainty, although their validity still depends on the model and estimation scheme.

The coefficients ψj\psi_j also give impulse responses: the effect on subsequent observations of a unit disturbance today. In a univariate model, this is a transparent description of fitted dynamics. It becomes a substantive intervention effect only with additional assumptions connecting the disturbance to an externally meaningful shock.

Forecast quality is commonly assessed using mean squared error, absolute-error measures, or scores for predictive distributions. The conditional mean minimizes expected squared error, while other loss functions can favor other summaries. A model can therefore be well suited to one forecasting objective and poorly suited to another.

Unit roots, trends, seasonality, and structural change

A root of the autoregressive polynomial at z=1z=1 is called a unit root. The simplest example is

Xt=Xt−1+εt,X_t=X_{t-1}+\varepsilon_t,

a random walk. With a fixed starting value and independent disturbances of variance σε2\sigma_\varepsilon^2, its variance grows linearly with elapsed time. A disturbance has a permanent effect on the level, unlike the decaying effect in a stable autoregression. Adding an intercept produces drift, so the expected level also changes over time.

Differencing removes the unit root in this example:

ΔXt=Xt−Xt−1=εt.\Delta X_t=X_t-X_{t-1}=\varepsilon_t.

More generally, an autoregressive integrated moving-average model, or ARIMA model, applies an autoregressive moving-average specification to a differenced series. In an ARIMA(p,d,qp,d,q), dd is the number of ordinary differences and pp and qq are the autoregressive and moving-average orders. Differencing changes the modeled quantity: a model for changes has different long-horizon implications from a stationary model for levels.

A deterministic trend is not the same as a unit root. A trend-stationary model fluctuates around a specified time trend, with disturbances that eventually dissipate. A difference-stationary model accumulates shocks to its level. Distinguishing these cases can be difficult in short samples, particularly when an autoregressive coefficient is close to 11. Dickey–Fuller-type tests use nonstandard reference distributions because ordinary stationary regression approximations do not apply under the unit-root null.

Seasonality introduces another kind of persistence. Monthly observations may depend strongly on the observation twelve months earlier; quarterly observations may depend on the value four quarters earlier. Seasonal autoregressive terms can represent this dependence, while seasonal differencing uses expressions such as (1−B12)Xt(1-B^{12})X_t. Seasonal and nonseasonal factors can be combined multiplicatively, yielding structured lag coefficients rather than an unrestricted coefficient at every lag.

Structural change is different again. A shift in the mean or in autoregressive coefficients can make an otherwise short-memory series appear highly persistent when analyzed as one homogeneous record. Changes in measurement systems, economic institutions, or physical operating conditions can invalidate a constant-parameter specification. Break models, time-varying parameters, and regime-specific estimation address different forms of this problem.

Neither differencing nor adding lags is a universal remedy. Unnecessary differencing can introduce avoidable dependence and inflate forecast uncertainty, while a high-order stationary model can approximate a break without describing it meaningfully. The distinction among stochastic trends, deterministic trends, seasonal structure, and parameter change governs both interpretation and forecast behavior.

Multivariate, nonlinear, and state-space extensions

A vector autoregression, or VAR, models several series jointly:

Xt=c+A1Xt−1+⋯+ApXt−p+εt.\mathbf X_t=\mathbf c+ A_1\mathbf X_{t-1}+\cdots+ A_p\mathbf X_{t-p}+\boldsymbol\varepsilon_t.

Each AjA_j is a coefficient matrix, so each component can depend on lagged values of every component. A VAR with mm variables and pp lags contains m2pm^2p lag coefficients before intercepts or other terms are added. This rapid growth creates an estimation problem even when each individual equation is linear.

Stability again depends on the eigenvalues of a companion matrix. Forecasts and impulse responses follow by matrix recursion, while the disturbance covariance matrix allows contemporaneous association among components. A reduced-form VAR does not by itself identify economically or physically distinct shocks. Structural analysis requires additional identifying restrictions, and different defensible restrictions can produce different interpretations.

Granger causality describes predictive content: past values of one variable improve prediction of another relative to a specified information set. In a linear VAR, this often becomes a test that a group of lag coefficients is zero. It is not equivalent to intervention-based causal inference, because omitted variables, aggregation, and observation timing can create predictive relations without the proposed causal mechanism.

External predictors can be included in an autoregression with exogenous inputs. Their exogeneity is an assumption, not a consequence of placing them on the right-hand side. Their future values must also be known, forecast, or specified through scenarios before they can support multistep forecasting. Related autoregressive moving-average models add lagged disturbances as well as lagged observations, often achieving a compact description of dependence that would otherwise require many autoregressive lags.

Nonlinear autoregressions replace the linear predictor with a function:

Xt=g(Xt−1,…,Xt−p)+εt.X_t=g(X_{t-1},\ldots,X_{t-p})+\varepsilon_t.

Threshold models use different coefficient sets in different regions of the lagged state; smooth-transition models interpolate between regimes. Neural autoregressions learn the function from data using an artificial neural network. Generalized autoregressive models can instead describe counts, binary outcomes, or time-varying conditional distributions. For non-Gaussian outcomes, the conditional mean alone generally does not specify the full model.

Finally, a linear autoregression can be written as a state-space system by stacking its lags into a latent state. This representation connects it to the Kalman filter, which supports likelihood evaluation and prediction with missing observations or measurement noise under appropriate assumptions. State-space modeling also separates the dynamics of an underlying process from the mechanism by which it is observed, a distinction absent from a basic error-free observation equation.

Development and major fields of application

Autoregressive modeling emerged from attempts to explain serial dependence in observed records. In 1927, G. Udny Yule published a study of sunspot fluctuations that used a second-order autoregressive mechanism to explain apparently cyclical behavior under random disturbances. Its importance lay in showing that irregular cycles could arise from a stochastic dynamic system rather than from a sum of fixed periodic components. Gilbert Walker’s work in 1931 helped establish the covariance relations now called the Yule–Walker equations.

Herman Wold’s 1938 work on stationary time series supplied a broader theoretical setting. The Wold decomposition represents a purely nondeterministic covariance-stationary process as an infinite moving average of orthogonal innovations. A finite autoregression is not the same as that general representation, but a stable autoregression has precisely such an innovations-based expansion. The connection explains why autoregressions, moving averages, and optimal linear prediction developed as closely related subjects.

George Box and Gwilym Jenkins’s 1970 book Time Series Analysis: Forecasting and Control consolidated an iterative practice of identification, estimation, and diagnostic checking for ARIMA models. Their framework made differencing, autocorrelation analysis, and residual checking part of a coherent applied methodology. Hirotugu Akaike’s final prediction error criterion, introduced in 1969, and subsequent information-criterion work gave formal tools for balancing autoregressive order against estimation error.

In signal processing, autoregressions became central to linear prediction and parametric spectral estimation. Speech analysis provides a prominent example: a short segment of a speech waveform can be approximated by an all-pole filter driven by an excitation signal. The fitted autoregressive denominator describes resonant characteristics associated with the vocal tract. The useful approximation is local, because speech changes over time and voiced excitation is not simply independent Gaussian noise. Related methods appear in vibration analysis, geophysics, and physiological signal analysis.

In economics, autoregressions provide benchmarks for forecasting quantities such as output growth, inflation, and interest rates. Christopher Sims’s 1980 work helped establish VARs as tools for studying joint macroeconomic dynamics without imposing the extensive contemporaneous restrictions common in earlier large equation systems. Subsequent work developed structural VARs, Bayesian shrinkage methods, and models allowing parameter changes.

Applications share practical limitations. Sampling frequency changes the apparent dynamics; temporal aggregation can alter the model class; observation noise can introduce additional dependence. An hourly process need not remain a finite-order autoregression after daily averaging. Consequently, a successful fitted equation describes the series at its actual observation scale, not automatically a scale-independent law of the underlying system.

Autoregressive generative models in machine learning

In contemporary machine learning, “autoregressive” often describes a probability factorization rather than a linear recurrence. For an ordered sequence x1,…,xnx_1,\ldots,x_n, the chain rule of conditional probability gives

p(x1,…,xn)=∏t=1np(xt∣x1,…,xt−1).p(x_1,\ldots,x_n) =\prod_{t=1}^{n}p(x_t\mid x_1,\ldots,x_{t-1}).

An autoregressive generative model parameterizes these conditional distributions. The factorization itself is exact; restrictions arise from the chosen conditional model, finite context, parameter sharing, and training procedure. A classical Gaussian AR(pp) is one special case, with a linear conditional mean and dependence on only the preceding pp observations.

For discrete sequences, each conditional can be a categorical distribution over possible next symbols. A neural model maps the available prefix to scores, often converted to probabilities by a softmax function. Training by maximum likelihood minimizes the negative log probability

−∑t=1nlog⁡pθ(xt∣x<t),-\sum_{t=1}^{n}\log p_\theta(x_t\mid x_{<t}),

where x<tx_{<t} denotes the preceding sequence. This objective is closely related to cross-entropy. A large language model trained for next-token prediction is autoregressive in this sense, although its conditional dependence can be highly nonlinear and extend over many thousands of tokens.

Transformer architectures enforce the required ordering through causal attention masks: the representation used to predict a position cannot access later positions. During training, many conditional predictions can nevertheless be computed in parallel because the complete training sequence is available and the mask prevents future information from entering each prediction. At generation time, conventional sampling proceeds sequentially, with each sampled token added to the prefix used for subsequent predictions.

The distinction between training and generation creates practical difficulties. Training usually conditions on observed prefixes, whereas free-running generation conditions on the model’s own previous outputs. Errors can therefore alter the context for later predictions. Classical multistep autoregressive forecasts also propagate previous predictions, but neural generative models often propagate complete sampled symbols rather than only conditional means. Exact likelihood evaluation can be straightforward for the specified factorization even when producing a long sample is computationally expensive.

Autoregressive image models impose an order on pixels or pixel groups, and autoregressive audio models predict successive waveform samples or other audio representations. PixelRNN, PixelCNN, and WaveNet, introduced in 2016, demonstrated influential forms of these approaches. The ordering need not represent physical time: for an image, it is a modeling convention that determines which pixels are available as context.

This broader terminology does not imply that classical assumptions carry over. A neural autoregressive generator need not be stationary, have finite-order linear dependence, or admit a Yule–Walker characterization. Its stability and long-run behavior require separate analysis. What remains common is the sequential construction of a variable or distribution from its own preceding components.