Multinomial logistic regression is a statistical model for predicting an outcome that belongs to one of several mutually exclusive, unordered categories. It extends binary logistic regression by assigning a probability to every category, with the probabilities constrained to sum to one. The model expresses log-probability ratios between categories as linear functions of predictors, or equivalently transforms category-specific linear scores using the softmax function. It is used for both statistical inference and classification in machine learning.
Model specification
Let be a categorical outcome with possible values, and let contain the predictors. An intercept can be included by defining the augmented predictor vector
For each category , define a score
where is a vector of coefficients. The modeled conditional probability of category is
For finite scores, every probability is strictly between zero and one, and
The categories need not have an ordering. Numbers used to encode them are labels rather than quantitative measurements.
Reference-category parameterization
The unrestricted softmax representation is redundant: adding the same vector to every coefficient vector leaves all probabilities unchanged. This creates a statistical identification issue.
A common solution is to choose category as the reference category and set . The model then becomes
with probabilities
and
With predictors and an intercept, this parameterization has coefficients. Another identification convention is to impose .
Changing the reference category changes the coefficient representation, but not the fitted probabilities of an otherwise equivalent unpenalized model. For , the formulation reduces to ordinary binary logistic regression.
Interpretation
Category probability ratios
For any two categories and ,
Thus, if predictor increases by one unit while the other predictors remain fixed, the ratio is multiplied by
When is the reference category, this multiplier is . It is often called a relative risk ratio in multinomial-logit software. It compares probabilities of two outcome categories, not the probability of category against all remaining categories combined.
Accordingly, its interpretation differs from the category-versus-rest odds ratio
For example, consider travel mode with categories car, bus, and bicycle, using car as the reference. A coefficient of for travel distance in the bus equation means that a one-unit increase in distance multiplies the modeled bus-to-car probability ratio by , holding other predictors fixed. It does not mean that the probability of bus travel increases by 40 percentage points.
Effects on probabilities
A coefficient’s sign does not by itself determine the direction of change in the corresponding category probability. With linear scores,
The probability change depends on all categories’ coefficients and on the current predictor values. A positive coefficient relative to the reference category can therefore coexist with a negative effect on that category’s probability.
Probability-scale effects are often reported as derivatives, average derivatives over observed predictor values, or discrete changes between specified predictor settings. These quantities describe modeled associations; a causal interpretation requires additional assumptions and an appropriate causal inference framework.
Likelihood and estimation
For observations , define
Under conditional independence across observations, the likelihood function is
and the log-likelihood is
Maximum likelihood estimation maximizes this expression. Equivalently, it minimizes the negative log-likelihood,
which is the summed cross-entropy loss for one-hot categorical labels.
The same probability model applies to grouped data. If a group with shared predictors contains counts , its log-likelihood contribution, apart from a parameter-independent multinomial coefficient, is
Optimization
For fixed predictors, the negative log-likelihood is a convex function of the coefficient parameters. In the all-category representation, its gradient for category is
The block of its Hessian matrix is
where equals one when and zero otherwise.
Common fitting methods include Newton’s method, quasi-Newton algorithms, and stochastic gradient descent. Convexity rules out nonglobal local minima, but does not guarantee that a finite, unique unpenalized estimate exists.
Numerical stability
Directly computing exponentials can overflow for large scores. For numerical stability, implementations use the identity
The same subtraction can be used when computing softmax probabilities, without changing their values.
Assumptions and limitations
Functional form
The basic model assumes that each pairwise log-probability ratio is linear in the supplied features. It does not require the original measured variables themselves to have a linear relationship with those ratios: polynomial terms, transformations, and interactions can be supplied as additional features.
Nevertheless, an inadequate feature specification can produce systematic prediction errors. The model does not require normally distributed predictors or equal predictor variances across outcome categories.
Independence of irrelevant alternatives
A characteristic property of multinomial logit is that
The normalization over other categories cancels. In a choice interpretation, holding the two relevant scores fixed, their probability ratio does not depend on the scores of other alternatives. This property is called independence of irrelevant alternatives, or IIA.
IIA can be restrictive when alternatives are close substitutes. In transport choice, adding a second bus service very similar to an existing one may draw disproportionately from the existing bus service rather than proportionately from all modes. A basic multinomial-logit specification may not capture that substitution pattern.
IIA is a restriction on relative choice probabilities, not an assertion that the outcome categories are statistically independent. Mutually exclusive category indicators are necessarily dependent.
Choice models with richer substitution structures include nested logit, mixed logit, and multinomial probit. In a standard random-utility derivation, multinomial-logit probabilities arise when utility disturbances are independent, identically distributed type-I extreme-value variables with a common scale.
Separation and sparse categories
If predictors separate outcome categories sufficiently well, the likelihood can improve as some coefficients grow without bound. Under complete or quasi-complete separation, a finite unpenalized maximum-likelihood estimate may fail to exist.
Rare categories, sparse combinations of predictors, and strong collinearity can also cause large uncertainty or unstable estimates. An optimizer reporting convergence does not, by itself, establish that the model is well identified or that its estimates are reliable.
Dependence and sampling
The ordinary product likelihood treats observations as conditionally independent. Repeated measurements, clustered samples, and survey designs require appropriate changes to the model, estimation procedure, or uncertainty calculation.
Outcome-dependent sampling can alter the relationship between fitted and population probabilities. Consequently, probability predictions from selectively sampled training data need not represent population probabilities without accounting for the sampling scheme.
Regularization and uncertainty
Regularization adds a penalty to the fitting objective. A common form is
where contains slope coefficients and controls shrinkage. Intercepts are often excluded from the penalty. An penalty can instead encourage zero coefficients.
Penalties can stabilize estimation and reduce overfitting. Their effects depend on the precise parameterization: penalizing reference-category coefficients is not generally equivalent to symmetrically penalizing all class coefficient vectors. Changing the reference category without transforming the penalty appropriately can therefore change fitted probabilities.
For an identifiable, finite estimate under standard regularity conditions, inverse information provides an approximate covariance matrix for estimated coefficients. This supports standard errors, confidence intervals, and hypothesis tests. Likelihood-ratio tests compare nested models, while resampling can assess uncertainty for derived quantities such as predicted probabilities.
Bayesian inference instead combines the likelihood with prior distributions over coefficients and obtains posterior uncertainty for parameters and predictions.
Prediction and evaluation
A common classification rule selects the category with highest fitted probability:
This rule minimizes conditional misclassification probability when every incorrect classification has the same cost. Unequal costs can imply a different decision rule.
Evaluation should distinguish hard-label classification from probability prediction. Accuracy and confusion matrices describe predicted labels; log loss evaluates the probability assigned to the observed category. Calibration examines whether predicted probabilities agree with observed outcome frequencies.
Held-out evaluation or cross-validation can assess generalization and select regularization strength. With imbalanced categories, overall accuracy can obscure poor performance for rare outcomes, so category-specific results are informative.
Related models and applications
Multinomial logistic regression belongs to the broader generalized linear model framework, using categorical or multinomial responses and vector-valued logit links. Its principal distinctions from neighboring models are:
- Ordinal logistic regression: uses an outcome ordering and often imposes shared effects across cumulative category boundaries. Multinomial logistic regression does not use such an ordering.
- One-versus-rest logistic regression: fits separate binary models. Their raw probabilities need not sum to one, and their fitting objective generally differs from the joint multinomial likelihood.
- Conditional logit choice models: commonly express utility using alternative-specific attributes and shared coefficients. Terminology overlaps with multinomial logit, but the data organization and coefficient restrictions can differ.
- Neural-network classifiers: a linear score layer followed by softmax is multinomial logistic regression when the input features are fixed. If an artificial neural network also learns the features, the overall model is generally nonlinear and its training objective need not remain convex.
Applications include travel-mode choice, product selection, document classification, and other problems with mutually exclusive nominal outcomes. The model is often used as an interpretable baseline: it produces a coherent probability distribution while making explicit restrictions on log-probability ratios and substitution patterns.