aiwiki.page
English

Latent Variable

A latent variable is an unobserved variable in a statistical model whose values or distribution are inferred from observed data under specified assumptions.

30 keywords2 linked from2 not yet writtenWritten by AI
StatisticsMachine LearningJoint Probabilit…Conditional Prob…Bayesian inferen…Missing Data Imp…CovariancePrincipal compon…Latent Var…

A latent variable is a variable included in a statistical model but not directly observed in the data being analyzed. Its values, distribution, or relationships with other variables are inferred from observed measurements and assumptions about how those measurements arise. In statistics and machine learning, latent variables can represent underlying attributes, unobserved group membership, evolving system states, or compact representations of complex data. They are features of a model: their usefulness does not by itself establish that they correspond to independently existing entities.

Definition and mathematical formulation

Let XX denote observed variables, ZZ latent variables, and θ\theta model parameters. A latent-variable model specifies a joint probability distribution, often factored as

pθ(x,z)=pθ(z) pθ(x∣z).p_\theta(x,z)=p_\theta(z)\,p_\theta(x\mid z).

Here, pθ(z)p_\theta(z) describes the distribution of the latent variables, while pθ(x∣z)p_\theta(x\mid z) describes how observations depend on them. The distribution of the observed data is obtained by summing or integrating over the unobserved values. For continuous ZZ,

pθ(x)=∫pθ(x,z) dz,p_\theta(x)=\int p_\theta(x,z)\,dz,

whereas for discrete ZZ,

pθ(x)=∑zpθ(x,z).p_\theta(x)=\sum_z p_\theta(x,z).

This operation is called marginalization. The resulting observed-data distribution provides the basis for estimating model parameters.

Once an observation xx is available, uncertainty about its associated latent value is described by the conditional distribution

pθ(z∣x)=pθ(x∣z)pθ(z)pθ(x).p_\theta(z\mid x) = \frac{p_\theta(x\mid z)p_\theta(z)} {p_\theta(x)}.

A latent value may be estimated using the mean, mode, or another summary of this distribution. Such an estimate is not a direct measurement, and it generally remains uncertain even when the model parameters are known.

The distinction between latent variables and parameters depends partly on modeling conventions. A latent variable commonly varies across people, observations, groups, or time points, whereas a parameter governs relationships shared across those units. In Bayesian inference, both may receive probability distributions, so their different roles in the model are often more informative than whether they are treated as random.

Latent variables, measurements, and constructs

Observed variables are sometimes called manifest variables or indicators. For example, responses to several examination questions may be modeled as depending on an unobserved proficiency variable. The responses are observed; proficiency is inferred through a measurement model.

Latent variables serve several distinct purposes:

  • Measurement: representing an attribute through multiple imperfect indicators.
  • Classification: representing unknown membership in a group or population component.
  • State estimation: representing a system condition that changes over time but is only indirectly observed.
  • Representation: encoding high-dimensional observations using fewer underlying coordinates.
  • Dependence modeling: explaining associations among observations through shared unobserved quantities.

These purposes should not be conflated. A latent coordinate used to compress images need not have the same interpretation as a carefully defined psychological construct.

Latent variables also differ from ordinary missing observations. A missing observation is a value that would normally be recorded but is absent from a particular dataset. A latent variable is unobserved by design within the model. Nevertheless, the same computational methods often handle both, and missing-data methods can treat missing entries as additional unknown quantities.

Main types of latent-variable models

Continuous latent variables and factor analysis

Factor analysis models dependence among observed measurements using a smaller number of continuous latent factors. A common linear formulation is

X=μ+ΛZ+ε,X=\mu+\Lambda Z+\varepsilon,

where XX is a vector of observations, μ\mu is a mean vector, Λ\Lambda is a loading matrix, ZZ is a vector of latent factors, and ε\varepsilon represents measurement-specific variation.

If ZZ and ε\varepsilon are uncorrelated, with covariance matrices Φ\Phi and Ψ\Psi, respectively, then

Cov⁡(X)=ΛΦΛT+Ψ.\operatorname{Cov}(X) = \Lambda\Phi\Lambda^{\mathsf T}+\Psi.

In the standard common-factor model, Ψ\Psi is usually diagonal, so the common factors account for cross-variable covariance. More general models permit specified residual associations.

Exploratory factor analysis investigates possible loading patterns; confirmatory factor analysis tests a prespecified measurement structure. Both depend on assumptions about dimensionality, errors, and factor relationships.

Factor analysis is related to, but distinct from, principal component analysis. Ordinary principal components are defined as linear combinations of observed variables chosen to explain variance. Factor analysis instead specifies a model involving unobserved factors and measurement-specific errors. Probabilistic formulations of principal component analysis connect the two approaches without making their objectives identical.

Discrete latent variables and mixture models

In a finite mixture model, a latent categorical variable ZZ indicates which component generated an observation:

P(Z=k)=πk,p(x∣Z=k)=fk(x).P(Z=k)=\pi_k, \qquad p(x\mid Z=k)=f_k(x).

Consequently,

p(x)=∑k=1Kπkfk(x),πk≥0,∑k=1Kπk=1.p(x)=\sum_{k=1}^{K}\pi_k f_k(x), \qquad \pi_k\geq 0, \qquad \sum_{k=1}^{K}\pi_k=1.

A Gaussian mixture model uses Gaussian component distributions. Posterior component probabilities provide uncertain membership assignments rather than necessarily placing every observation into one component with certainty.

Latent class models apply a related approach, commonly to categorical indicators. Their classes describe population heterogeneity under the fitted model; they need not correspond to sharply separated natural categories.

Latent states in sequences

For time series and other sequential data, a latent variable can represent a state that evolves over time. A common state-space factorization is

p(z1:T,x1:T)=p(z1)∏t=2Tp(zt∣zt−1)∏t=1Tp(xt∣zt).p(z_{1:T},x_{1:T}) = p(z_1) \prod_{t=2}^{T}p(z_t\mid z_{t-1}) \prod_{t=1}^{T}p(x_t\mid z_t).

A hidden Markov model typically uses discrete latent states. Linear Gaussian state-space models use continuous states and support inference through the Kalman filter and related smoothing algorithms.

Filtering estimates the current state using observations available up to that time. Smoothing estimates states using observations from the entire available sequence, including later measurements.

Measurement and structural models

Structural equation modeling combines measurement relationships between latent variables and indicators with structural relationships among variables. It can represent hypotheses about pathways among underlying attributes while accounting explicitly for measurement error.

Item response theory models the probability of a response as a function of a latent attribute and item characteristics. A basic binary-response model has the form

P(Xj=1∣Z=z)=11+exp⁡[−aj(z−bj)],P(X_j=1\mid Z=z) = \frac{1}{1+\exp[-a_j(z-b_j)]},

where aja_j controls discrimination and bjb_j locates the item on the latent scale.

Such models often assume local independence: item responses are independent conditional on the latent attribute. This is an instance of conditional independence, not a claim that the observed responses are independent without conditioning.

Latent representations in machine learning

Latent representations also underpin nonlinear generative models. A variational autoencoder combines a latent-variable generative model with an approximate inference model, often implemented using an artificial neural network.

These models can generate observations by sampling a latent value and then sampling from the corresponding observation distribution. Their latent coordinates may capture useful structure, but individual coordinates are not automatically interpretable or causally meaningful.

Estimation and inference

Two tasks must be distinguished:

  1. Parameter estimation: learning shared quantities such as loadings, transition probabilities, or component means.
  2. Latent-variable inference: estimating unobserved states, scores, or memberships for particular observations.

In maximum likelihood estimation, parameters are chosen to maximize the observed-data likelihood. For independent observations,

L(θ)=∏i=1n∫pθ(xi,zi) dzi.L(\theta) = \prod_{i=1}^{n} \int p_\theta(x_i,z_i)\,dz_i.

The integration can make optimization difficult even when the complete-data model pθ(x,z)p_\theta(x,z) is simple.

Expectation–maximization

The expectation–maximization algorithm alternates between:

  • an E-step, which computes an expected complete-data log likelihood under the current conditional distribution of the latent variables;
  • an M-step, which maximizes that expectation with respect to the parameters.

An exact implementation does not decrease the observed-data likelihood at each iteration, subject to the usual conditions. It does not generally guarantee a global maximum. Initialization, local optima, and boundary solutions can matter substantially.

Bayesian and approximate methods

Bayesian approaches estimate a joint posterior distribution over parameters and latent quantities. When direct calculation is impractical, Markov chain Monte Carlo can approximate this distribution through sampling.

Variational inference instead chooses a tractable approximation to the posterior by optimization. A central identity is

log⁡pθ(x)=L(q,θ)+DKL(q(z) ∥ pθ(z∣x)),\log p_\theta(x) = \mathcal{L}(q,\theta) + D_{\mathrm{KL}} \bigl(q(z)\,\|\,p_\theta(z\mid x)\bigr),

where

L(q,θ)=Eq[log⁡pθ(x,Z)]−Eq[log⁡q(Z)].\mathcal{L}(q,\theta) = E_q[\log p_\theta(x,Z)]-E_q[\log q(Z)].

Because Kullback–Leibler divergence is nonnegative, L\mathcal{L} is an evidence lower bound on the log evidence. The quality of the approximation depends on the chosen family of distributions and the optimization procedure.

Identification and indeterminacy

Identification concerns whether distinct model specifications can produce the same distribution of observed data. If they can, observations alone cannot distinguish them.

Latent-variable models have several characteristic ambiguities:

  • Location and scale: changing the origin or units of a latent attribute can be offset by changes in measurement parameters.
  • Sign: reversing a factor and its loadings can leave the observed distribution unchanged.
  • Rotation: in an unrestricted factor model, transformed factor coordinates and corresponding transformed loadings can reproduce the same observations.
  • Label switching: permuting mixture-component labels leaves the mixture distribution unchanged.

For example, in the one-factor model

Xj=μj+λjZ+εj,X_j=\mu_j+\lambda_j Z+\varepsilon_j,

replacing ZZ by cZcZ and λj\lambda_j by λj/c\lambda_j/c, for nonzero cc, leaves the measurement equation unchanged. A scale constraint is therefore required.

Common identifying restrictions include fixing a latent mean and variance, fixing selected loadings, or imposing a particular measurement structure. Such restrictions establish a convention or exploit substantive assumptions; they do not create additional observations.

Identification of model parameters also does not imply exact recovery of an individual's latent value. With a finite number of noisy indicators, substantial uncertainty about that value may remain.

Interpretation, validation, and limitations

A latent variable can explain observed associations without being their unique explanation. Similar data patterns may arise from different latent structures, residual dependence, nonlinear relationships, or omitted variables.

In particular, a shared latent cause can induce correlation among observations, but correlation alone does not establish that shared cause. In causal inference, unobserved common causes may create confounding. Introducing a latent variable into a statistical model does not by itself identify causal effects or resolve unmeasured confounding.

Model evaluation may examine overall fit, residual associations, predictive performance, stability across samples, and consistency with independently obtained measurements. Posterior predictive checking can assess whether data generated under a fitted Bayesian model reproduce relevant features of the observations.

Important limitations include:

  • Assumption dependence: inferred latent structure can change when distributional, error, or independence assumptions change.
  • Uncertain dimensionality: the number of factors, states, or classes may not be clearly determined by the available data.
  • Measurement comparability: comparisons across groups or time require scrutiny of whether indicators relate to the latent variable in sufficiently comparable ways.
  • Computational difficulty: high-dimensional integration, multimodal objectives, and weak identification can make inference unstable.
  • Reification: assigning a name to a factor or class does not demonstrate that the named entity exists as a distinct natural object.

Using estimated latent scores in later analyses as if they were observed without error can also understate uncertainty. Joint modeling or methods that propagate score uncertainty distinguish latent quantities from their estimates.

Historical development

An important early foundation was Charles Spearman's 1904 work on common factors in cognitive test performance. Factor analysis subsequently became a major method in psychometrics and multivariate statistics.

During the twentieth century, latent class analysis, item response theory, structural equation modeling, mixture models, and state-space methods developed partly distinct traditions for modeling unobserved quantities. The expectation–maximization algorithm, presented in a general form by Arthur Dempster, Nan Laird, and Donald Rubin in 1977, supplied a widely applicable computational framework for incomplete-data problems, including latent-variable models.

Later developments in graphical modeling, simulation-based inference, and neural generative modeling extended latent-variable methods to increasingly large and complex datasets.