{
  "id": "no-free-lunch-theorem",
  "title": "No-Free-Lunch Theorem",
  "lang": "en",
  "source_lang": "en",
  "summary": "No-free-lunch theorems show that, under specified averages over all possible problems, no learning or optimization algorithm universally outperforms its competitors.",
  "category": "mathematics",
  "subcategory": "applied",
  "type": "concept",
  "aliases": [
    "No Free Lunch Theorem",
    "NFL theorem",
    "No-free-lunch theorems"
  ],
  "generated_at": "2026-11-06",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/no-free-lunch-theorem/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/no-free-lunch-theorem/",
      "json": "https://aiwiki.page/zh/no-free-lunch-theorem.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "The no-free-lunch theorem is the name given to a family of mathematical results concerning the limits of universally effective algorithms, particularly in mathematical optimization and machine learning. Their central message is conditional: when performance is averaged over a sufficiently unrestricted collection of problems using a specified symmetric weighting, advantages on some problems are offset by disadvantages on others. The theorems do not imply that all algorithms perform equally on practical tasks. They show that superior performance depends on how an algorithm’s assumptions match the structure of the problems it encounters."
    },
    {
      "heading": "Background and development",
      "text": "The phrase “no free lunch” expresses the idea that an advantage requires something in return. In these theorems, that “something” is typically information or an assumption about the problem class: which functions are plausible, which patterns persist beyond observed data, or which regions of a search space are promising.\n\nDavid H. Wolpert developed no-free-lunch results for supervised learning in work published in 1996. Wolpert and William G. Macready presented influential optimization results in their 1997 paper No Free Lunch Theorems for Optimization. Related results differ in their assumptions, averaging procedures, and definitions of performance, so the name does not designate a single theorem applicable without qualification.\n\nThe learning results concern what observations can establish about unseen cases when target relationships are unrestricted. The optimization results concern what evaluations of an unknown objective function reveal about its values at unexamined locations. Both depend on a common principle: without restrictions connecting observed and unobserved values, observed success need not predict future success."
    },
    {
      "heading": "The finite optimization theorem",
      "text": "Setting\n\nA standard formulation considers:\n\n- A finite, nonempty search space \\(X\\), with \\(|X|=N\\).\n- A finite, nonempty set of possible objective values \\(Y\\).\n- The collection \\(Y^X\\) of all functions \\(f:X\\to Y\\).\n- An algorithm that queries points in \\(X\\), choosing each point using the previously queried points and observed values.\n- A fixed evaluation budget \\(m\\leq N\\), with no point queried more than once.\n\nThe objective function is selected from the uniform distribution over \\(Y^X\\). Under this distribution, each point’s value is independently and uniformly selected from \\(Y\\).\n\nFor an algorithm \\(A\\), let\n\n\\[V_A^m(f)=\\bigl(f(x_1),\\ldots,f(x_m)\\bigr)\\]\n\ndenote the sequence of objective values it observes. Its choices of \\(x_2,\\ldots,x_m\\) may depend on earlier observations.\n\nFor any two deterministic non-revisiting algorithms \\(A\\) and \\(B\\), and any sequence \\(v\\in Y^m\\),\n\n\\[\\Pr_f\\!\\left(V_A^m(f)=v\\right) = \\Pr_f\\!\\left(V_B^m(f)=v\\right) = |Y|^{-m}.\\]\n\nThus every such algorithm has the same distribution of observed value sequences when averaged uniformly over all objective functions.\n\nEquality of performance\n\nSuppose performance is measured by a real-valued function \\(C\\) of the observed values. Examples include the best value found after \\(m\\) evaluations or whether a specified value threshold has been reached. Then\n\n\\[\\frac{1}{|Y|^N} \\sum_{f\\in Y^X}C\\!\\left(V_A^m(f)\\right) = \\frac{1}{|Y|^N} \\sum_{f\\in Y^X}C\\!\\left(V_B^m(f)\\right).\\]\n\nThis is an equality of average performance under the stated problem distribution, not an equality on each individual function. Equality of the full sequence distributions is stronger than equality of a single expected value: it establishes equality for every performance statistic determined solely by that sequence.\n\nRandomized algorithms can also be included when their internal randomness is independent of the objective function and every realized query sequence obeys the non-revisiting condition. The equality follows by averaging over their random choices.\n\nWhy the theorem holds\n\nAfter an algorithm observes \\(m\\) distinct input-output pairs, exactly\n\n\\[|Y|^{N-m}\\]\n\nfunctions in \\(Y^X\\) remain consistent with those observations. Every unqueried point still has each possible value with equal probability, regardless of how that point was selected.\n\nAdaptation therefore cannot exploit information about unqueried values, because the assumed distribution contains no relationship between those values and the observations. A sophisticated search rule and a predetermined ordering encounter the same value-sequence distribution under this averaging scheme.\n\nThis argument does not say that adaptation is generally useless. It says that adaptation has nothing to exploit when the model makes all unobserved values independent of the observed ones."
    },
    {
      "heading": "No-free-lunch results for learning",
      "text": "In supervised learning, an algorithm uses training data to construct predictions for cases not included in the observed sample. The central issue is whether the sample provides information about those unseen cases.\n\nConsider a finite input space \\(X\\) and binary labels \\(Y=\\{0,1\\}\\). Fix a set of observed inputs \\(S\\subset X\\), together with their labels, and place a uniform distribution over all target functions \\(f:X\\to\\{0,1\\}\\) consistent with those observations.\n\nFor any unseen input \\(x\\notin S\\),\n\n\\[\\Pr\\!\\left(f(x)=0\\mid f|_S\\right) = \\Pr\\!\\left(f(x)=1\\mid f|_S\\right) = \\frac12.\\]\n\nAny prediction rule based only on the observed sample therefore has expected misclassification probability \\(1/2\\) at that input. With \\(K\\) equally likely labels, expected error under the zero-one loss function is \\(1-1/K\\).\n\nThis is a basic finite learning no-free-lunch result. Other formulations compare learning procedures through averages over target functions, samples, or distributions. Their conclusions depend on precisely which quantities are held fixed and which are averaged.\n\nInductive bias\n\nLearning becomes possible when observations constrain plausible behavior on unseen inputs. The assumptions that enable this extrapolation are called inductive bias.\n\nExamples include assumptions that:\n\n- Nearby inputs tend to have similar outputs.\n- A target function is well approximated by a linear relationship.\n- Relevant structure is shared across different positions in an image.\n- Simpler or smoother models are more plausible than arbitrary alternatives.\n\nSuch assumptions can be embodied in a model class, a prior distribution, or a regularization penalty. For example, linear regression restricts attention to a particular functional form, while a smoothness assumption restricts how rapidly outputs can change.\n\nNo-free-lunch results do not establish which bias is appropriate. They explain why some restriction or nonuniform weighting of possibilities is needed to justify predictions beyond the observations."
    },
    {
      "heading": "Restricted problem classes and permutation symmetry",
      "text": "Uniform averaging over all functions is sufficient for the standard optimization theorem, but it is not the only setting in which an equality can hold.\n\nA central extension concerns classes of functions that are closed under permutations of the search space. A class \\(F\\subseteq Y^X\\) has this property if, for every \\(f\\in F\\) and every permutation \\(\\pi:X\\to X\\),\n\n\\[f\\circ\\pi\\in F.\\]\n\nUniform averaging over such a class preserves the relevant no-free-lunch equality for non-revisiting search. Arbitrarily relabeling search locations does not change the class, so location-based structure cannot systematically favor an algorithm.\n\nMore generally, a probability distribution over functions that is invariant under every permutation of the search space provides a corresponding symmetry. This need not be uniform over all functions.\n\nMany useful problem classes lack this symmetry. Permuting values among locations can destroy continuity, locality, geometric relationships, or convexity. These are precisely the kinds of properties that specialized algorithms exploit.\n\nFailure of a no-free-lunch assumption, however, does not itself identify a superior algorithm. It merely removes the particular equality guaranteed by that theorem."
    },
    {
      "heading": "Practical significance",
      "text": "Optimization methods exploit structure\n\nMethods such as gradient descent and Newton’s method use local derivative information. Their convergence results depend on assumptions such as smoothness, curvature conditions, and suitable initialization.\n\nLikewise, guarantees in convex optimization rely on a restricted problem class in which local and global properties are connected. Such guarantees do not conflict with no-free-lunch theorems: the restrictions exclude the arbitrary functions over which the standard equality is established.\n\nIn discrete optimization, a representation may expose useful neighborhood relationships or decompositions. Search methods can benefit when those relationships correlate with objective values, but that correlation is additional problem structure rather than a consequence of unrestricted search.\n\nModel selection and evaluation\n\nA learning algorithm’s performance must be assessed relative to a task distribution, a data-generating process, and a performance criterion. Cross-validation estimates predictive performance under assumptions about how the available data relate to future cases. It does not establish superiority over every possible target function or future distribution.\n\nSimilarly, benchmark results describe performance on the tested collection and protocol. They need not extend to arbitrarily different tasks. No-free-lunch results clarify this limitation without making benchmark comparisons meaningless: a benchmark can be informative when its tasks represent the intended application.\n\nPrior information changes the comparison\n\nIn Bayesian inference, a nonuniform prior gives some hypotheses greater weight than others. Algorithms can then differ in expected performance because the averaging distribution is no longer indifferent to all target relationships.\n\nThe same principle applies outside Bayesian methods. A restricted hypothesis class, a geometric representation, or an assumption about data dependence changes the relevant problem family. Algorithmic advantage is evaluated relative to those choices."
    },
    {
      "heading": "Limitations and common misinterpretations",
      "text": "“All algorithms are equally good.”\nThe theorem establishes equality only under its specified averaging distribution and admissibility conditions. Algorithms can differ greatly on individual problems and structured problem families.\n\n“No algorithm can outperform random search.”\nOn suitable problem classes, algorithms routinely outperform uninformed search. In the standard finite formulation, uniform random search without replacement shares the same all-functions average as other non-revisiting algorithms; this does not make it equally effective on every practical distribution.\n\n“Uniform averaging is assumption-free.”\nIt is a substantive modeling choice. Uniformity over all finite functions makes unobserved values independent of observed ones and removes the relationships that practical algorithms are designed to exploit.\n\n“The finite result automatically applies to continuous domains.”\nThere is no direct counterpart of “choose uniformly from all functions” for arbitrary infinite domains. Continuous-domain results require their own function spaces, probability measures, and definitions of information and performance.\n\n“Revisiting points never matters.”\nThe non-revisiting condition is important to the standard optimization equality. Repeated evaluations of a deterministic function can spend a query budget without revealing a new value. In noisy settings, repeated measurements may instead provide information about uncertainty; that is a different model.\n\n“No-free-lunch theorems rule out general-purpose methods.”\nA method can be useful across a broad collection of related tasks without being best over all mathematically possible problems. Broad applicability is compatible with shared structure in that collection.\n\n“The theorem is a computational-complexity lower bound.”\nThe standard result concerns information and performance averaged over problem instances, not a direct bound on running time. Algorithms with the same objective-value statistics may still differ in memory use, computational overhead, or implementation cost."
    }
  ],
  "links": [
    "algorithm",
    "mathematical-optimization",
    "machine-learning",
    "objective-function",
    "mathematical-function",
    "uniform-distribution",
    "expected-value",
    "training-data",
    "loss-function",
    "inductive-bias",
    "prior-distribution",
    "regularization",
    "linear-regression",
    "permutation",
    "probability-distribution",
    "convex-function",
    "gradient-descent",
    "newtons-method",
    "convex-optimization",
    "cross-validation",
    "bayesian-inference"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}