Nonnegative matrix factorization (NMF) is a family of matrix factorization methods that express a matrix with nonnegative entries as the exact or approximate product of two smaller nonnegative matrices. The factors describe latent components and their contributions to individual observations. Because negative coefficients are excluded, reconstruction is additive: components contribute without canceling one another. NMF is widely used in dimensionality reduction, exploratory data analysis, and machine learning, particularly when measurements represent counts, intensities, concentrations, or other naturally nonnegative quantities.
Mathematical formulation
Let be a matrix whose columns represent observations measured across features. For a chosen inner dimension , NMF seeks
The columns of are component profiles, while column of contains the weights used to reconstruct observation :
Thus, each observation is approximated by a nonnegative linear combination of shared components. The component profiles are sometimes called “basis vectors,” although they need not be linearly independent and therefore need not form a basis in the strict linear-algebraic sense.
Typically, . The factorization then represents the data using parameters rather than , although this yields a reduction in parameter count only when .
The general fitting problem is
where measures reconstruction discrepancy, and are optional penalties, and control their strength. The choice of loss function and constraints determines which patterns the factorization emphasizes.
Additive interpretation and geometry
The nonnegativity constraints distinguish NMF from unrestricted low-rank approximations. A component cannot be subtracted to cancel another component. For suitable data, this can produce a parts-based representation: an image may be represented by recurring visual features, a document by weighted topics, or a mixture spectrum by contributions from constituent signatures.
This interpretation is not automatic. Nonnegativity alone does not guarantee sparse factors, localized image features, independent sources, or scientifically meaningful components. Such properties depend on the data, loss, constraints, and optimization procedure.
Geometrically, the columns of generate a cone: all vectors obtainable as nonnegative combinations of those columns. NMF seeks a cone that contains or approximately represents the observations. If additional normalization makes the coefficients in each column of sum to one, the reconstructions instead lie in the convex hull of the component profiles. That normalization introduces a mixture interpretation not present in unconstrained NMF.
Reconstruction losses and statistical models
Squared Euclidean loss
A common objective is
where the Frobenius norm is defined by
This objective penalizes squared absolute reconstruction errors. It corresponds to maximum likelihood estimation under independent additive Gaussian errors with a common variance, subject to a nonnegative factorized mean.
Because errors are squared, large residuals can dominate the fit. Differences in feature scale can therefore substantially change the resulting components.
Generalized Kullback–Leibler divergence
Another widely used objective minimizes the generalized Kullback–Leibler divergence:
Here, is interpreted as zero; a positive observation assigned a zero reconstructed value gives infinite divergence.
For count data modeled as independent Poisson random variables with means , minimizing this divergence is equivalent to maximizing the likelihood, up to terms independent of . Unlike KL divergence between probability distributions, this generalized expression does not require the matrices to sum to one.
Other losses
The Itakura–Saito divergence,
is defined for strictly positive entries in its basic form and is common in audio analysis. Its scalar divergence is unchanged when both arguments are multiplied by the same positive constant, giving it a different sensitivity to signal scale than squared error.
The beta-divergence family includes squared Euclidean, generalized KL, and Itakura–Saito losses as particular cases or limits. Weighted objectives can represent unequal measurement reliability or exclude missing entries from the reconstruction loss.
Optimization algorithms
The joint optimization over and is generally nonconvex. For squared Euclidean and generalized KL losses, however, optimizing one factor while holding the other fixed is a convex optimization problem. This structure motivates alternating algorithms.
Multiplicative updates
For squared Euclidean loss, classical multiplicative updates take the form
Multiplication and division outside ordinary matrix products are entrywise, and denotes matrix transpose. Each step uses the current value of the other factor.
These rules preserve nonnegativity. Under the assumptions used in their derivation, the objective does not increase. Nevertheless, monotonic decrease alone does not establish convergence to a global optimum, or even guarantee satisfactory convergence behavior in every implementation.
Multiplicative updates may converge slowly. They can also exhibit zero locking: an entry initialized at zero remains zero under a purely multiplicative rule, even when increasing it would improve the fit. Zero denominators require careful handling.
Alternating nonnegative least squares
For squared error, fixing gives
which separates into nonnegative least-squares problems for the columns of . Fixing gives corresponding problems for . Active-set, coordinate-descent, and block-coordinate methods can solve these subproblems.
Projected gradient descent is another approach. For example,
where the maximum is entrywise and is a step size. Step-size selection affects convergence and stability.
Initialization matters because different runs can reach different stationary points. Common choices include positive random factors and nonnegative initializations derived from a singular value decomposition. Stopping rules may use objective changes, parameter changes, iteration limits, or violations of the Karush–Kuhn–Tucker conditions.
Rank, nonuniqueness, and identifiability
Ordinary rank and nonnegative rank
For an exact factorization , the smallest possible inner dimension is the nonnegative rank, denoted . It satisfies
The ordinary matrix rank may therefore underestimate the number of components needed for an exact nonnegative representation.
In approximate NMF, is usually a modeling choice rather than an exactly determined property. Larger values allow lower training reconstruction error but can split patterns into redundant components or fit noise. Rank assessment may consider held-out reconstruction, cross-validation, stability across runs, and domain-specific interpretability.
Ambiguities
NMF always has scaling and permutation ambiguities. For any positive diagonal matrix ,
Likewise, permuting the columns of and applying the corresponding permutation to the rows of leaves the product unchanged. Normalizing component profiles removes a scaling convention but does not establish uniqueness.
More substantial ambiguities can also occur: distinct nonnegative component sets may reconstruct the same data equally well. Identifiability requires additional assumptions about the geometry, sparsity, or distribution of the factors.
One important assumption is separability, under which each component occurs as a data column up to scaling. Closely related models use anchor features or pure observations. Such assumptions enable specialized recovery algorithms, but they are stronger than nonnegativity alone.
Regularization and variants
Regularization and structural constraints adapt NMF to particular tasks.
- Sparse NMF: Penalties or explicit constraints encourage selected factor entries to be zero. This can restrict how many components contribute to an observation or how many features characterize a component.
- Smooth NMF: Penalties encourage neighboring coefficients to vary gradually, as in spatial measurements or time series.
- Graph-regularized NMF: A graph-based penalty encourages nearby or related observations to have similar representations.
- Orthogonal NMF: Additional orthogonality constraints can make representations resemble clustering, though nonnegative orthogonality is restrictive.
- Semi-NMF: One factor may contain negative entries while the other remains nonnegative, allowing signed input data.
- Convex NMF: Component profiles are constrained to be combinations of observed data, often convex combinations.
- Online NMF: Factors are updated incrementally as observations arrive, reducing the need to process the entire dataset at once.
Nonnegative tensor factorization extends the additive-component idea to multidimensional arrays represented as tensors, rather than simply reshaping every dataset into a matrix.
Applications
Text analysis. A nonnegative term-by-document matrix can be decomposed into term profiles and document weights. The components often resemble topics, although ordinary NMF does not by itself define a probabilistic topic model.
Audio processing. A magnitude or power spectrogram can be approximated by spectral templates and time-varying activations. This supports source separation and transcription. Reconstructing waveforms also requires phase information or an additional reconstruction procedure; factorizing magnitudes alone does not determine the waveform.
Spectroscopy and chemical mixtures. Nonnegative spectra and concentrations motivate additive models in spectroscopy. Physical interpretation depends on whether the measurement process is adequately approximated by linear mixing and whether the components are identifiable.
Biological data analysis. NMF is used to explore patterns in gene expression and other nonnegative molecular measurements. A component describes statistical structure in the analyzed data; it does not automatically establish a biological mechanism.
Images and recommendation data. NMF can extract recurring image patterns or represent user–item interactions through latent factors. In recommendation settings, unobserved interactions must be distinguished from observed zero values.
Relationship to other decompositions
Principal component analysis usually operates on centered data and uses orthogonal directions that can contain positive and negative entries. NMF typically uses nonnegative, uncentered data and does not require orthogonal components. Adding a constant to make signed data nonnegative changes the modeled structure and is not equivalent to applying NMF to naturally nonnegative measurements.
Truncated SVD gives a globally optimal rank-constrained approximation under squared Frobenius loss, without nonnegativity constraints. NMF generally lacks this closed-form optimum, but its additive representation may better match the meaning of particular measurements.
NMF is also distinct from independent-component methods: nonnegative components are not necessarily statistically independent. Hard clustering assigns an observation to a single cluster, whereas an NMF representation can assign positive weight to several components.
Historical development and limitations
Modern NMF developed from earlier work on nonnegative factor models, mixture analysis, and positive matrix factorization. Pentti Paatero and Unto Tapper described a positive matrix factorization method in 1994 that incorporated measurement uncertainties. Daniel D. Lee and H. Sebastian Seung brought broad attention to NMF through a 1999 study of parts-based representations and presented influential multiplicative-update algorithms in 2001.
The main limitations remain sensitivity to scaling, initialization, component count, and loss specification; possible nonuniqueness; and the absence of a general global-optimality guarantee for standard local algorithms. Missing entries, outliers, and unequal uncertainties require explicit treatment. Component interpretation must also be separated from reconstruction quality: a small numerical error does not by itself demonstrate that the recovered factors correspond to unique physical sources or causal processes.