{
  "id": "multinomial-logistic-regression",
  "title": "Multinomial Logistic Regression",
  "lang": "en",
  "source_lang": "en",
  "summary": "Multinomial logistic regression models probabilities for unordered categorical outcomes by expressing category log-odds as linear functions of predictors.",
  "category": "mathematics",
  "subcategory": "probability-statistics",
  "type": "concept",
  "aliases": [
    "Multinomial logit model",
    "Multiclass logistic regression",
    "Softmax regression",
    "Baseline-category logit model",
    "Polytomous logistic regression"
  ],
  "generated_at": "2026-11-05",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/multinomial-logistic-regression/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/multinomial-logistic-regression/",
      "json": "https://aiwiki.page/zh/multinomial-logistic-regression.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "Multinomial logistic regression is a statistical model for predicting an outcome that belongs to one of several mutually exclusive, unordered categories. It extends binary logistic regression by assigning a probability to every category, with the probabilities constrained to sum to one. The model expresses log-probability ratios between categories as linear functions of predictors, or equivalently transforms category-specific linear scores using the softmax function. It is used for both statistical inference and classification in machine learning."
    },
    {
      "heading": "Model specification",
      "text": "Let \\(Y\\) be a categorical outcome with \\(K\\) possible values, and let \\(x\\in\\mathbb{R}^{p}\\) contain the predictors. An intercept can be included by defining the augmented predictor vector\n\n\\[\\tilde{x}=(1,x_1,\\ldots,x_p)^\\top.\\]\n\nFor each category \\(k\\), define a score\n\n\\[\\eta_k(x)=\\theta_k^\\top\\tilde{x},\\]\n\nwhere \\(\\theta_k\\) is a vector of coefficients. The modeled conditional probability of category \\(k\\) is\n\n\\[p_k(x) = P(Y=k\\mid x) = \\frac{\\exp\\{\\eta_k(x)\\}} {\\sum_{j=1}^{K}\\exp\\{\\eta_j(x)\\}}.\\]\n\nFor finite scores, every probability is strictly between zero and one, and\n\n\\[\\sum_{k=1}^{K}p_k(x)=1.\\]\n\nThe categories need not have an ordering. Numbers used to encode them are labels rather than quantitative measurements.\n\nReference-category parameterization\n\nThe unrestricted softmax representation is redundant: adding the same vector \\(a\\) to every coefficient vector leaves all probabilities unchanged. This creates a statistical identification issue.\n\nA common solution is to choose category \\(K\\) as the reference category and set \\(\\theta_K=0\\). The model then becomes\n\n\\[\\log\\frac{p_k(x)}{p_K(x)} = \\theta_k^\\top\\tilde{x}, \\qquad k=1,\\ldots,K-1,\\]\n\nwith probabilities\n\n\\[p_k(x) = \\frac{\\exp(\\theta_k^\\top\\tilde{x})} {1+\\sum_{j=1}^{K-1}\\exp(\\theta_j^\\top\\tilde{x})},\\]\n\nand\n\n\\[p_K(x) = \\frac{1} {1+\\sum_{j=1}^{K-1}\\exp(\\theta_j^\\top\\tilde{x})}.\\]\n\nWith \\(p\\) predictors and an intercept, this parameterization has \\((K-1)(p+1)\\) coefficients. Another identification convention is to impose \\(\\sum_k\\theta_k=0\\).\n\nChanging the reference category changes the coefficient representation, but not the fitted probabilities of an otherwise equivalent unpenalized model. For \\(K=2\\), the formulation reduces to ordinary binary logistic regression."
    },
    {
      "heading": "Interpretation",
      "text": "Category probability ratios\n\nFor any two categories \\(k\\) and \\(l\\),\n\n\\[\\log\\frac{p_k(x)}{p_l(x)} = (\\theta_k-\\theta_l)^\\top\\tilde{x}.\\]\n\nThus, if predictor \\(x_r\\) increases by one unit while the other predictors remain fixed, the ratio \\(p_k/p_l\\) is multiplied by\n\n\\[\\exp(\\theta_{kr}-\\theta_{lr}).\\]\n\nWhen \\(l\\) is the reference category, this multiplier is \\(\\exp(\\theta_{kr})\\). It is often called a relative risk ratio in multinomial-logit software. It compares probabilities of two outcome categories, not the probability of category \\(k\\) against all remaining categories combined.\n\nAccordingly, its interpretation differs from the category-versus-rest odds ratio\n\n\\[\\frac{p_k}{1-p_k}.\\]\n\nFor example, consider travel mode with categories car, bus, and bicycle, using car as the reference. A coefficient of \\(0.4\\) for travel distance in the bus equation means that a one-unit increase in distance multiplies the modeled bus-to-car probability ratio by \\(e^{0.4}\\), holding other predictors fixed. It does not mean that the probability of bus travel increases by 40 percentage points.\n\nEffects on probabilities\n\nA coefficient’s sign does not by itself determine the direction of change in the corresponding category probability. With linear scores,\n\n\\[\\frac{\\partial p_k(x)}{\\partial x_r} = p_k(x) \\left[ \\theta_{kr} - \\sum_{j=1}^{K}p_j(x)\\theta_{jr} \\right].\\]\n\nThe probability change depends on all categories’ coefficients and on the current predictor values. A positive coefficient relative to the reference category can therefore coexist with a negative effect on that category’s probability.\n\nProbability-scale effects are often reported as derivatives, average derivatives over observed predictor values, or discrete changes between specified predictor settings. These quantities describe modeled associations; a causal interpretation requires additional assumptions and an appropriate causal inference framework."
    },
    {
      "heading": "Likelihood and estimation",
      "text": "For observations \\((x_i,y_i)\\), define\n\n\\[y_{ik}= \\begin{cases} 1,&y_i=k,\\\\ 0,&y_i\\ne k. \\end{cases}\\]\n\nUnder conditional independence across observations, the likelihood function is\n\n\\[L(\\theta) = \\prod_{i=1}^{n}\\prod_{k=1}^{K} p_k(x_i)^{y_{ik}},\\]\n\nand the log-likelihood is\n\n\\[\\ell(\\theta) = \\sum_{i=1}^{n}\\sum_{k=1}^{K} y_{ik}\\log p_k(x_i).\\]\n\nMaximum likelihood estimation maximizes this expression. Equivalently, it minimizes the negative log-likelihood,\n\n\\[\\mathcal{L}(\\theta) = -\\sum_{i=1}^{n}\\sum_{k=1}^{K} y_{ik}\\log p_k(x_i),\\]\n\nwhich is the summed cross-entropy loss for one-hot categorical labels.\n\nThe same probability model applies to grouped data. If a group with shared predictors contains counts \\(n_{i1},\\ldots,n_{iK}\\), its log-likelihood contribution, apart from a parameter-independent multinomial coefficient, is\n\n\\[\\sum_{k=1}^{K}n_{ik}\\log p_k(x_i).\\]\n\nOptimization\n\nFor fixed predictors, the negative log-likelihood is a convex function of the coefficient parameters. In the all-category representation, its gradient for category \\(k\\) is\n\n\\[\\frac{\\partial\\mathcal{L}}{\\partial\\theta_k} = \\sum_{i=1}^{n} \\bigl(p_k(x_i)-y_{ik}\\bigr)\\tilde{x}_i.\\]\n\nThe \\((k,l)\\) block of its Hessian matrix is\n\n\\[H_{kl} = \\sum_{i=1}^{n} p_k(x_i)\\bigl(\\delta_{kl}-p_l(x_i)\\bigr) \\tilde{x}_i\\tilde{x}_i^\\top,\\]\n\nwhere \\(\\delta_{kl}\\) equals one when \\(k=l\\) and zero otherwise.\n\nCommon fitting methods include Newton’s method, quasi-Newton algorithms, and stochastic gradient descent. Convexity rules out nonglobal local minima, but does not guarantee that a finite, unique unpenalized estimate exists.\n\nNumerical stability\n\nDirectly computing exponentials can overflow for large scores. For numerical stability, implementations use the identity\n\n\\[\\log\\sum_{j=1}^{K}e^{\\eta_j} = m+\\log\\sum_{j=1}^{K}e^{\\eta_j-m}, \\qquad m=\\max_j\\eta_j.\\]\n\nThe same subtraction can be used when computing softmax probabilities, without changing their values."
    },
    {
      "heading": "Assumptions and limitations",
      "text": "Functional form\n\nThe basic model assumes that each pairwise log-probability ratio is linear in the supplied features. It does not require the original measured variables themselves to have a linear relationship with those ratios: polynomial terms, transformations, and interactions can be supplied as additional features.\n\nNevertheless, an inadequate feature specification can produce systematic prediction errors. The model does not require normally distributed predictors or equal predictor variances across outcome categories.\n\nIndependence of irrelevant alternatives\n\nA characteristic property of multinomial logit is that\n\n\\[\\frac{p_k(x)}{p_l(x)} = \\exp\\{\\eta_k(x)-\\eta_l(x)\\}.\\]\n\nThe normalization over other categories cancels. In a choice interpretation, holding the two relevant scores fixed, their probability ratio does not depend on the scores of other alternatives. This property is called independence of irrelevant alternatives, or IIA.\n\nIIA can be restrictive when alternatives are close substitutes. In transport choice, adding a second bus service very similar to an existing one may draw disproportionately from the existing bus service rather than proportionately from all modes. A basic multinomial-logit specification may not capture that substitution pattern.\n\nIIA is a restriction on relative choice probabilities, not an assertion that the outcome categories are statistically independent. Mutually exclusive category indicators are necessarily dependent.\n\nChoice models with richer substitution structures include nested logit, mixed logit, and multinomial probit. In a standard random-utility derivation, multinomial-logit probabilities arise when utility disturbances are independent, identically distributed type-I extreme-value variables with a common scale.\n\nSeparation and sparse categories\n\nIf predictors separate outcome categories sufficiently well, the likelihood can improve as some coefficients grow without bound. Under complete or quasi-complete separation, a finite unpenalized maximum-likelihood estimate may fail to exist.\n\nRare categories, sparse combinations of predictors, and strong collinearity can also cause large uncertainty or unstable estimates. An optimizer reporting convergence does not, by itself, establish that the model is well identified or that its estimates are reliable.\n\nDependence and sampling\n\nThe ordinary product likelihood treats observations as conditionally independent. Repeated measurements, clustered samples, and survey designs require appropriate changes to the model, estimation procedure, or uncertainty calculation.\n\nOutcome-dependent sampling can alter the relationship between fitted and population probabilities. Consequently, probability predictions from selectively sampled training data need not represent population probabilities without accounting for the sampling scheme."
    },
    {
      "heading": "Regularization and uncertainty",
      "text": "Regularization adds a penalty to the fitting objective. A common form is\n\n\\[\\mathcal{L}_{\\lambda}(\\theta) = \\mathcal{L}(\\theta) + \\lambda\\sum_k\\|\\beta_k\\|_2^2,\\]\n\nwhere \\(\\beta_k\\) contains slope coefficients and \\(\\lambda\\ge 0\\) controls shrinkage. Intercepts are often excluded from the penalty. An \\(L_1\\) penalty can instead encourage zero coefficients.\n\nPenalties can stabilize estimation and reduce overfitting. Their effects depend on the precise parameterization: penalizing reference-category coefficients is not generally equivalent to symmetrically penalizing all class coefficient vectors. Changing the reference category without transforming the penalty appropriately can therefore change fitted probabilities.\n\nFor an identifiable, finite estimate under standard regularity conditions, inverse information provides an approximate covariance matrix for estimated coefficients. This supports standard errors, confidence intervals, and hypothesis tests. Likelihood-ratio tests compare nested models, while resampling can assess uncertainty for derived quantities such as predicted probabilities.\n\nBayesian inference instead combines the likelihood with prior distributions over coefficients and obtains posterior uncertainty for parameters and predictions."
    },
    {
      "heading": "Prediction and evaluation",
      "text": "A common classification rule selects the category with highest fitted probability:\n\n\\[\\hat y(x)=\\arg\\max_k p_k(x).\\]\n\nThis rule minimizes conditional misclassification probability when every incorrect classification has the same cost. Unequal costs can imply a different decision rule.\n\nEvaluation should distinguish hard-label classification from probability prediction. Accuracy and confusion matrices describe predicted labels; log loss evaluates the probability assigned to the observed category. Calibration examines whether predicted probabilities agree with observed outcome frequencies.\n\nHeld-out evaluation or cross-validation can assess generalization and select regularization strength. With imbalanced categories, overall accuracy can obscure poor performance for rare outcomes, so category-specific results are informative."
    },
    {
      "heading": "Related models and applications",
      "text": "Multinomial logistic regression belongs to the broader generalized linear model framework, using categorical or multinomial responses and vector-valued logit links. Its principal distinctions from neighboring models are:\n\n- Ordinal logistic regression: uses an outcome ordering and often imposes shared effects across cumulative category boundaries. Multinomial logistic regression does not use such an ordering.\n- One-versus-rest logistic regression: fits separate binary models. Their raw probabilities need not sum to one, and their fitting objective generally differs from the joint multinomial likelihood.\n- Conditional logit choice models: commonly express utility using alternative-specific attributes and shared coefficients. Terminology overlaps with multinomial logit, but the data organization and coefficient restrictions can differ.\n- Neural-network classifiers: a linear score layer followed by softmax is multinomial logistic regression when the input features are fixed. If an artificial neural network also learns the features, the overall model is generally nonlinear and its training objective need not remain convex.\n\nApplications include travel-mode choice, product selection, document classification, and other problems with mutually exclusive nominal outcomes. The model is often used as an interpretable baseline: it produces a coherent probability distribution while making explicit restrictions on log-probability ratios and substitution patterns."
    }
  ],
  "links": [
    "logistic-regression",
    "softmax-function",
    "machine-learning",
    "conditional-probability",
    "statistical-identification",
    "odds-ratio",
    "causal-inference",
    "likelihood-function",
    "maximum-likelihood-estimation",
    "cross-entropy",
    "convex-function",
    "gradient",
    "hessian-matrix",
    "newtons-method",
    "stochastic-gradient-descent",
    "numerical-stability",
    "regularization",
    "covariance-matrix",
    "standard-error",
    "confidence-interval",
    "statistical-hypothesis-testing",
    "bayesian-inference",
    "cross-validation",
    "generalized-linear-model",
    "artificial-neural-network"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}