{
  "id": "lasso-regression",
  "title": "Lasso Regression",
  "lang": "en",
  "source_lang": "en",
  "summary": "Lasso regression is a regularized linear modeling method that shrinks coefficients and can set some exactly to zero, combining prediction with variable selection.",
  "category": "mathematics",
  "subcategory": "probability-statistics",
  "type": "concept",
  "aliases": [
    "Lasso",
    "LASSO",
    "Least absolute shrinkage and selection operator",
    "L1-penalized least squares"
  ],
  "generated_at": "2026-10-04",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/lasso-regression/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/lasso-regression/",
      "json": "https://aiwiki.page/zh/lasso-regression.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "Lasso regression is a form of linear regression that combines regularization with automatic feature selection. It adds a penalty proportional to the sum of the absolute values of the regression coefficients, shrinking their magnitudes and potentially setting some exactly to zero. The resulting model uses only a subset of the available predictors. Robert Tibshirani introduced the method under the name “least absolute shrinkage and selection operator” in a 1996 paper. (doi.org)"
    },
    {
      "heading": "Mathematical formulation",
      "text": "For \\(n\\) observations and \\(p\\) predictors, let \\(y_i\\) denote the response and \\(x_{ij}\\) the value of predictor \\(j\\) for observation \\(i\\). A common formulation estimates an intercept \\(\\beta_0\\) and coefficients \\(\\beta_j\\) by minimizing the objective function\n\n\\[\\frac{1}{2n}\\sum_{i=1}^{n} \\left(y_i-\\beta_0-\\sum_{j=1}^{p}x_{ij}\\beta_j\\right)^2 +\\lambda\\sum_{j=1}^{p}|\\beta_j|.\\]\n\nThe first term measures squared prediction error; the second is the coefficient vector’s \\(\\ell_1\\) norm, multiplied by a nonnegative tuning parameter \\(\\lambda\\). The intercept is ordinarily excluded from the penalty. Equivalent conventions omit the factor \\(1/n\\), changing the numerical scale of \\(\\lambda\\) but not the underlying method. (scikit-learn.org)\n\nWhen \\(\\lambda=0\\), the objective reduces to ordinary least squares. Positive values penalize large coefficients, while sufficiently large values set all penalized coefficients to zero. An equivalent constrained formulation minimizes squared error subject to\n\n\\[\\sum_{j=1}^{p}|\\beta_j|\\leq t.\\]\n\nHere, a smaller bound \\(t\\) imposes stronger shrinkage. Penalized and constrained formulations correspond through an appropriate relationship between their tuning parameters, rather than a universal numerical conversion. (scikit-learn.org)"
    },
    {
      "heading": "Why coefficients become zero",
      "text": "The absolute-value penalty is convex but has a corner at zero. Consequently, lasso is a convex optimization problem whose optimum can lie exactly on a coordinate axis. Geometrically, the two-dimensional \\(\\ell_1\\) constraint region is diamond-shaped: its corners favor solutions with zero coordinates. By contrast, ridge regression uses a squared \\(\\ell_2\\) penalty, whose smooth geometry ordinarily shrinks coefficients without eliminating them. (homepages.math.uic.edu)\n\nFor centered data with a predictor matrix satisfying \\(X^\\top X/n=I\\), the solution has a particularly simple form:\n\n\\[\\hat\\beta_j= \\operatorname{sign}(z_j)\\max(|z_j|-\\lambda,0), \\qquad z_j=X_j^\\top y/n.\\]\n\nThis operation, called soft-thresholding, subtracts a fixed amount from each coefficient’s absolute magnitude and truncates values below the threshold to zero. With correlated predictors, coefficients interact, so this independent formula no longer applies directly. (homepages.math.uic.edu)"
    },
    {
      "heading": "Statistical role and limitations",
      "text": "Lasso introduces estimation bias through shrinkage, but can reduce variance enough to improve predictive accuracy. This illustrates the bias–variance trade-off: a less flexible fitted model may predict new observations better than an unpenalized model. Its sparse representation also makes large predictor sets easier to inspect, although improved prediction is not guaranteed for every dataset. (homepages.math.uic.edu)\n\nThe method remains applicable when \\(p>n\\), where ordinary least-squares coefficients are not uniquely determined. Lasso coefficients may also be nonunique for some predictor matrices, but all minimizing coefficient vectors produce the same fitted values on the observed design. Suitable general-position conditions ensure uniqueness even when predictors outnumber observations. Thus, high dimensionality alone does not imply a nonunique lasso solution. (arxiv.org)\n\nStrong correlation among predictors can make variable selection unstable: lasso may retain one member of a group while excluding others with similar information. Elastic net combines \\(\\ell_1\\) and squared \\(\\ell_2\\) penalties, retaining sparsity while often handling correlated groups more stably. A coefficient set to zero indicates exclusion from the particular fitted model, not proof that the predictor is universally irrelevant. (scikit-learn.org)"
    },
    {
      "heading": "Scaling and parameter selection",
      "text": "Because the penalty acts on coefficient magnitudes, changing a predictor’s measurement units changes its effective penalization. Feature scaling, commonly centering and standardizing predictors, makes penalties more comparable across variables. Scaling is a modeling choice rather than a mathematical requirement, and its interpretation depends on the features and application. (homepages.math.uic.edu)\n\nThe parameter \\(\\lambda\\) is a hyperparameter commonly selected through cross-validation, using prediction error on held-out folds. Preprocessing parameters must be estimated from each fold’s training data, not from observations used for validation. Otherwise, data leakage can make performance estimates overly optimistic. A separate test set can assess the final modeling procedure without participating in parameter selection. (scikit-learn.org)"
    },
    {
      "heading": "Computation and inference",
      "text": "A widely used algorithm is coordinate descent, which repeatedly optimizes one coefficient while holding the others fixed. For squared-error lasso, each coordinate update uses soft-thresholding of its partial-residual association. Efficient implementations calculate a sequence of solutions across penalty values, reusing neighboring solutions as starting points. This sequence is called a regularization path. (jstatsoft.org)\n\nPrediction and statistical inference require different interpretations. Ordinary confidence intervals and significance tests applied after data-driven selection do not automatically account for that selection. Post-selection inference methods explicitly incorporate the selection event to construct valid inferential statements under specified assumptions. The same absolute-value penalty also extends beyond squared-error regression to generalized linear models, including logistic regression, where a likelihood-based loss replaces squared error. (arxiv.org)"
    }
  ],
  "links": [
    "linear-regression",
    "regularization",
    "feature-selection",
    "objective-function",
    "norm-mathematics",
    "ordinary-least-squares",
    "convex-optimization",
    "ridge-regression",
    "matrix",
    "estimator-bias",
    "variance",
    "bias-variance-tradeoff",
    "correlation",
    "elastic-net",
    "feature-scaling",
    "hyperparameter",
    "cross-validation",
    "training-data",
    "data-leakage",
    "test-set",
    "algorithm",
    "coordinate-descent",
    "confidence-interval",
    "generalized-linear-model",
    "logistic-regression"
  ],
  "sources": [
    {
      "title": "Regression Shrinkage and Selection via the Lasso",
      "url": "https://homepages.math.uic.edu/~lreyzin/papers/lasso.pdf"
    },
    {
      "title": "Lasso — scikit-learn documentation",
      "url": "https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Lasso.html"
    },
    {
      "title": "1. Linear Models — scikit-learn documentation",
      "url": "https://scikit-learn.org/stable/modules/linear_model.html"
    },
    {
      "title": "Regularization Paths for Generalized Linear Models via Coordinate Descent",
      "url": "https://pmc.ncbi.nlm.nih.gov/articles/PMC2929880/"
    },
    {
      "title": "The Lasso Problem and Uniqueness",
      "url": "https://arxiv.org/abs/1206.0313"
    },
    {
      "title": "The Lasso Problem and Uniqueness",
      "url": "https://www.stat.berkeley.edu/~ryantibs/papers/lassounique.pdf"
    },
    {
      "title": "Common pitfalls and recommended practices — scikit-learn documentation",
      "url": "https://scikit-learn.org/stable/common_pitfalls.html?highlight=linearregression"
    },
    {
      "title": "Preprocessing data — scikit-learn documentation",
      "url": "https://scikit-learn.org/stable/modules/preprocessing.html?gdpr=%24%7BGDPR%7D&gdpr_consent=%24%7BGDPR_CONSENT_97%7D"
    },
    {
      "title": "Exact post-selection inference, with application to the lasso",
      "url": "https://arxiv.org/abs/1311.6238"
    }
  ],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}