{
  "id": "curse-of-dimensionality",
  "title": "Curse of dimensionality",
  "lang": "en",
  "source_lang": "en",
  "summary": "The curse of dimensionality describes geometric, statistical, and computational difficulties that arise as the number of dimensions in a problem increases.",
  "category": "mathematics",
  "subcategory": "probability-statistics",
  "type": "concept",
  "aliases": [
    "Curse of dimension",
    "Dimensionality curse"
  ],
  "generated_at": "2026-10-04",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/curse-of-dimensionality/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/curse-of-dimensionality/",
      "json": "https://aiwiki.page/zh/curse-of-dimensionality.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "The curse of dimensionality is a collective name for difficulties that arise when analyzing, learning from, or computing over spaces with many dimensions. In statistics and machine learning, observations become sparse relative to the space they occupy; in numerical computation, maintaining a fixed resolution can require exponentially increasing resources. The expression originated with Richard Bellman in his work on dynamic programming. It describes several related phenomena rather than one universal theorem or a fixed threshold beyond which computation becomes impossible. (cs.cmu.edu)"
    },
    {
      "heading": "Exponential growth and sparse coverage",
      "text": "A basic example is a regular grid in a \\(d\\)-dimensional unit cube. If each coordinate has \\(m\\) grid positions, their Cartesian product contains\n\n\\[N=m^d\\]\n\npoints. Ten positions per coordinate require 100 points in two dimensions, one million in six dimensions, and ten billion in ten dimensions. Thus a modest increase in dimension can make exhaustive tabulation impractical. This is especially important when storing values for every possible state in a discretized optimization problem. (cs.cmu.edu)\n\nThe same scaling affects sampling. Suppose training data follow a uniform distribution on \\([0,1]^d\\). An axis-aligned subcube with side length \\(r\\), entirely inside the unit cube, occupies a fraction \\(r^d\\) of its volume. Among \\(n\\) observations, the expected number inside it is therefore \\(nr^d\\). Maintaining an expected count of \\(k\\) requires \\(n=k/r^d\\), which grows exponentially with \\(d\\) when \\(r<1\\) is fixed. (stat.cmu.edu)\n\nConversely, a subcube containing a fraction \\(q\\) of the population has side length \\(q^{1/d}\\). For \\(q=0.01\\), this is approximately 0.10 in two dimensions, 0.63 in ten dimensions, and 0.955 in one hundred dimensions. A neighborhood containing a small fraction of observations can consequently span almost the entire range of every coordinate. These calculations depend on the uniform-distribution assumption, but illustrate why “local” estimation becomes difficult. (courses.cs.cornell.edu)"
    },
    {
      "heading": "High-dimensional geometry",
      "text": "High-dimensional Euclidean space also exhibits distance concentration. Under suitable assumptions, such as independent, comparably scaled coordinates, squared Euclidean distance is a sum of many contributions. Its absolute magnitude typically increases with dimension, while its relative fluctuations diminish. Distances can therefore become less effective at distinguishing nearby observations from typical observations. This does not mean that all distances become exactly equal, or that every dataset loses meaningful neighborhoods. (courses.cs.cornell.edu)\n\nBoundary effects provide another illustration. The fraction of a unit cube lying at least \\(\\epsilon\\) from every face is\n\n\\[(1-2\\epsilon)^d,\\qquad 0<\\epsilon<\\tfrac12.\\]\n\nFor fixed \\(\\epsilon\\), this tends to zero as dimension increases: almost all volume lies near at least one boundary face. This statement concerns distance to the nearest face, not necessarily distance to a corner. It follows directly from the volume of the smaller cube remaining after removing a boundary strip along each coordinate. (stat.cmu.edu)"
    },
    {
      "heading": "Statistical learning",
      "text": "The k-nearest neighbors algorithm predicts from nearby observations. Sparse coverage forces it to use increasingly broad neighborhoods unless sample size increases substantially. Similarly, density estimation and local regression must balance small neighborhoods with too few observations against large neighborhoods that smooth over genuine variation. This is an expression of the bias–variance trade-off: stronger smoothing reduces variance but can increase estimation bias. (stat.cmu.edu)\n\nUnder standard smoothness assumptions, a local regression estimator with bandwidth \\(h\\) can have squared bias of order \\(h^4\\) and variance of order \\(1/(nh^d)\\). Balancing these terms gives a mean squared error rate of order\n\n\\[n^{-4/(d+4)}.\\]\n\nThe exponent becomes smaller as dimension increases. This is a representative nonparametric result, not a performance formula for every learning algorithm. (stat.cmu.edu)\n\nDimensionality-related difficulty is also distinct from overfitting. An unrestricted model can fit accidental patterns in a limited sample, but sparse coverage can impair prediction even without elaborate model fitting. Generalization depends on sample size, noise, model assumptions, and the structure of the data—not simply the number of recorded features. (stat.cmu.edu)"
    },
    {
      "heading": "Computational consequences",
      "text": "In dynamic programming, a table representing a value function over a discretized state space can grow exponentially with the number of state variables. The Bellman equation supplies a recursive relationship, but recursion alone does not eliminate the cost of representing all states. (cs.cmu.edu)\n\nNearest-neighbor search faces a different computational manifestation. Tree-based indexes can efficiently exclude large regions in low dimensions, but their pruning advantage often declines as dimension increases. Search may approach the cost of checking every observation. Performance also depends on sample size, the distance metric, and the dataset’s underlying structure; high dimension does not impose one universal computational complexity bound on all methods. (scikit-learn.org)"
    },
    {
      "heading": "Structural assumptions and mitigation",
      "text": "The ambient number of coordinates need not equal the data’s effective dimension. Observations may lie near a lower-dimensional linear subspace or manifold. Dimensionality reduction, including principal component analysis and manifold learning, seeks representations that exploit this structure. Feature selection instead retains a subset of the original variables. Their usefulness depends on whether the reduced representation preserves information relevant to the task. (scikit-learn.org)\n\nRegularization restricts model flexibility rather than necessarily reducing the number of inputs. For example, lasso regression can set coefficients to zero, whereas ridge penalties shrink coefficients without generally eliminating them. Such restrictions trade unrestricted approximation capacity for more stable estimation. (scikit-learn.org)\n\nSome numerical methods also avoid exhaustive coverage. Monte Carlo integration has a root-mean-square error proportional to \\(n^{-1/2}\\) for independent samples and finite integrand variance. However, its variance and evaluation cost may themselves depend on dimension, so a dimension-independent exponent does not guarantee dimension-independent computational effort. (arxiv.org)"
    }
  ],
  "links": [
    "statistics",
    "machine-learning",
    "richard-bellman",
    "dynamic-programming",
    "cartesian-product",
    "training-data",
    "uniform-distribution",
    "expected-value",
    "euclidean-space",
    "euclidean-distance",
    "k-nearest-neighbors-algorithm",
    "density-estimation",
    "bias-variance-tradeoff",
    "variance",
    "estimator-bias",
    "mean-squared-error",
    "overfitting",
    "generalization",
    "value-function",
    "bellman-equation",
    "computational-complexity",
    "linear-subspace",
    "manifold",
    "dimensionality-reduction",
    "principal-component-analysis",
    "manifold-learning",
    "feature-selection",
    "regularization",
    "lasso-regression"
  ],
  "sources": [
    {
      "title": "Dynamic Programming",
      "url": "https://www.cs.cmu.edu/~cga/dynopt/DP-167645-part-2.pdf"
    },
    {
      "title": "ADAfaEPoV",
      "url": "https://www.stat.cmu.edu/~ryantibs/advmethods/shalizi/ch09.pdf"
    },
    {
      "title": "k-nearest neighbors / Curse of Dimensionality",
      "url": "https://courses.cs.cornell.edu/cs4780/2024sp/lectures/lecturenote02_kNN.html"
    },
    {
      "title": "Waste, fraud and abuse: Sources of failure in applied statistical learning projects",
      "url": "https://www.stat.cmu.edu/~cshalizi/dm/22/lectures/26/lecture-26.html"
    },
    {
      "title": "6. Nearest Neighbors — scikit-learn documentation",
      "url": "https://scikit-learn.org/stable/modules/neighbors.html"
    },
    {
      "title": "Is the k-NN classifier in high dimensions affected by the curse of dimensionality?",
      "url": "https://arxiv.org/abs/1110.4347"
    },
    {
      "title": "1. Linear Models — scikit-learn documentation",
      "url": "https://scikit-learn.org/stable/modules/linear_model.html"
    },
    {
      "title": "Approximate and integrate: Variance reduction in Monte Carlo integration via function approximation",
      "url": "https://arxiv.org/abs/1806.05492"
    },
    {
      "title": "A Note on Monte Carlo Integration in High Dimensions",
      "url": "https://arxiv.org/abs/2206.09036"
    }
  ],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}