A latent variable is a variable included in a statistical model but not directly observed in the data being analyzed. Its values, distribution, or relationships with other variables are inferred from observed measurements and assumptions about how those measurements arise. In statistics and machine learning, latent variables can represent underlying attributes, unobserved group membership, evolving system states, or compact representations of complex data. They are features of a model: their usefulness does not by itself establish that they correspond to independently existing entities.
Definition and mathematical formulation
Let denote observed variables, latent variables, and model parameters. A latent-variable model specifies a joint probability distribution, often factored as
Here, describes the distribution of the latent variables, while describes how observations depend on them. The distribution of the observed data is obtained by summing or integrating over the unobserved values. For continuous ,
whereas for discrete ,
This operation is called marginalization. The resulting observed-data distribution provides the basis for estimating model parameters.
Once an observation is available, uncertainty about its associated latent value is described by the conditional distribution
A latent value may be estimated using the mean, mode, or another summary of this distribution. Such an estimate is not a direct measurement, and it generally remains uncertain even when the model parameters are known.
The distinction between latent variables and parameters depends partly on modeling conventions. A latent variable commonly varies across people, observations, groups, or time points, whereas a parameter governs relationships shared across those units. In Bayesian inference, both may receive probability distributions, so their different roles in the model are often more informative than whether they are treated as random.
Latent variables, measurements, and constructs
Observed variables are sometimes called manifest variables or indicators. For example, responses to several examination questions may be modeled as depending on an unobserved proficiency variable. The responses are observed; proficiency is inferred through a measurement model.
Latent variables serve several distinct purposes:
- Measurement: representing an attribute through multiple imperfect indicators.
- Classification: representing unknown membership in a group or population component.
- State estimation: representing a system condition that changes over time but is only indirectly observed.
- Representation: encoding high-dimensional observations using fewer underlying coordinates.
- Dependence modeling: explaining associations among observations through shared unobserved quantities.
These purposes should not be conflated. A latent coordinate used to compress images need not have the same interpretation as a carefully defined psychological construct.
Latent variables also differ from ordinary missing observations. A missing observation is a value that would normally be recorded but is absent from a particular dataset. A latent variable is unobserved by design within the model. Nevertheless, the same computational methods often handle both, and missing-data methods can treat missing entries as additional unknown quantities.
Main types of latent-variable models
Continuous latent variables and factor analysis
Factor analysis models dependence among observed measurements using a smaller number of continuous latent factors. A common linear formulation is
where is a vector of observations, is a mean vector, is a loading matrix, is a vector of latent factors, and represents measurement-specific variation.
If and are uncorrelated, with covariance matrices and , respectively, then
In the standard common-factor model, is usually diagonal, so the common factors account for cross-variable covariance. More general models permit specified residual associations.
Exploratory factor analysis investigates possible loading patterns; confirmatory factor analysis tests a prespecified measurement structure. Both depend on assumptions about dimensionality, errors, and factor relationships.
Factor analysis is related to, but distinct from, principal component analysis. Ordinary principal components are defined as linear combinations of observed variables chosen to explain variance. Factor analysis instead specifies a model involving unobserved factors and measurement-specific errors. Probabilistic formulations of principal component analysis connect the two approaches without making their objectives identical.
Discrete latent variables and mixture models
In a finite mixture model, a latent categorical variable indicates which component generated an observation:
Consequently,
A Gaussian mixture model uses Gaussian component distributions. Posterior component probabilities provide uncertain membership assignments rather than necessarily placing every observation into one component with certainty.
Latent class models apply a related approach, commonly to categorical indicators. Their classes describe population heterogeneity under the fitted model; they need not correspond to sharply separated natural categories.
Latent states in sequences
For time series and other sequential data, a latent variable can represent a state that evolves over time. A common state-space factorization is
A hidden Markov model typically uses discrete latent states. Linear Gaussian state-space models use continuous states and support inference through the Kalman filter and related smoothing algorithms.
Filtering estimates the current state using observations available up to that time. Smoothing estimates states using observations from the entire available sequence, including later measurements.
Measurement and structural models
Structural equation modeling combines measurement relationships between latent variables and indicators with structural relationships among variables. It can represent hypotheses about pathways among underlying attributes while accounting explicitly for measurement error.
Item response theory models the probability of a response as a function of a latent attribute and item characteristics. A basic binary-response model has the form
where controls discrimination and locates the item on the latent scale.
Such models often assume local independence: item responses are independent conditional on the latent attribute. This is an instance of conditional independence, not a claim that the observed responses are independent without conditioning.
Latent representations in machine learning
Latent representations also underpin nonlinear generative models. A variational autoencoder combines a latent-variable generative model with an approximate inference model, often implemented using an artificial neural network.
These models can generate observations by sampling a latent value and then sampling from the corresponding observation distribution. Their latent coordinates may capture useful structure, but individual coordinates are not automatically interpretable or causally meaningful.
Estimation and inference
Two tasks must be distinguished:
- Parameter estimation: learning shared quantities such as loadings, transition probabilities, or component means.
- Latent-variable inference: estimating unobserved states, scores, or memberships for particular observations.
In maximum likelihood estimation, parameters are chosen to maximize the observed-data likelihood. For independent observations,
The integration can make optimization difficult even when the complete-data model is simple.
Expectation–maximization
The expectation–maximization algorithm alternates between:
- an E-step, which computes an expected complete-data log likelihood under the current conditional distribution of the latent variables;
- an M-step, which maximizes that expectation with respect to the parameters.
An exact implementation does not decrease the observed-data likelihood at each iteration, subject to the usual conditions. It does not generally guarantee a global maximum. Initialization, local optima, and boundary solutions can matter substantially.
Bayesian and approximate methods
Bayesian approaches estimate a joint posterior distribution over parameters and latent quantities. When direct calculation is impractical, Markov chain Monte Carlo can approximate this distribution through sampling.
Variational inference instead chooses a tractable approximation to the posterior by optimization. A central identity is
where
Because Kullback–Leibler divergence is nonnegative, is an evidence lower bound on the log evidence. The quality of the approximation depends on the chosen family of distributions and the optimization procedure.
Identification and indeterminacy
Identification concerns whether distinct model specifications can produce the same distribution of observed data. If they can, observations alone cannot distinguish them.
Latent-variable models have several characteristic ambiguities:
- Location and scale: changing the origin or units of a latent attribute can be offset by changes in measurement parameters.
- Sign: reversing a factor and its loadings can leave the observed distribution unchanged.
- Rotation: in an unrestricted factor model, transformed factor coordinates and corresponding transformed loadings can reproduce the same observations.
- Label switching: permuting mixture-component labels leaves the mixture distribution unchanged.
For example, in the one-factor model
replacing by and by , for nonzero , leaves the measurement equation unchanged. A scale constraint is therefore required.
Common identifying restrictions include fixing a latent mean and variance, fixing selected loadings, or imposing a particular measurement structure. Such restrictions establish a convention or exploit substantive assumptions; they do not create additional observations.
Identification of model parameters also does not imply exact recovery of an individual's latent value. With a finite number of noisy indicators, substantial uncertainty about that value may remain.
Interpretation, validation, and limitations
A latent variable can explain observed associations without being their unique explanation. Similar data patterns may arise from different latent structures, residual dependence, nonlinear relationships, or omitted variables.
In particular, a shared latent cause can induce correlation among observations, but correlation alone does not establish that shared cause. In causal inference, unobserved common causes may create confounding. Introducing a latent variable into a statistical model does not by itself identify causal effects or resolve unmeasured confounding.
Model evaluation may examine overall fit, residual associations, predictive performance, stability across samples, and consistency with independently obtained measurements. Posterior predictive checking can assess whether data generated under a fitted Bayesian model reproduce relevant features of the observations.
Important limitations include:
- Assumption dependence: inferred latent structure can change when distributional, error, or independence assumptions change.
- Uncertain dimensionality: the number of factors, states, or classes may not be clearly determined by the available data.
- Measurement comparability: comparisons across groups or time require scrutiny of whether indicators relate to the latent variable in sufficiently comparable ways.
- Computational difficulty: high-dimensional integration, multimodal objectives, and weak identification can make inference unstable.
- Reification: assigning a name to a factor or class does not demonstrate that the named entity exists as a distinct natural object.
Using estimated latent scores in later analyses as if they were observed without error can also understate uncertainty. Joint modeling or methods that propagate score uncertainty distinguish latent quantities from their estimates.
Historical development
An important early foundation was Charles Spearman's 1904 work on common factors in cognitive test performance. Factor analysis subsequently became a major method in psychometrics and multivariate statistics.
During the twentieth century, latent class analysis, item response theory, structural equation modeling, mixture models, and state-space methods developed partly distinct traditions for modeling unobserved quantities. The expectation–maximization algorithm, presented in a general form by Arthur Dempster, Nan Laird, and Donald Rubin in 1977, supplied a widely applicable computational framework for incomplete-data problems, including latent-variable models.
Later developments in graphical modeling, simulation-based inference, and neural generative modeling extended latent-variable methods to increasingly large and complex datasets.