Bayesian optimization is a class of methods for finding the best value of an expensive-to-evaluate objective function by constructing a probabilistic model of that function and using the model to select informative evaluations. It is especially useful when a function has no convenient analytical expression, its derivatives are unavailable, or each measurement requires substantial computation or a physical experiment. Typical applications include tuning machine-learning systems, designing materials, and calibrating engineering simulations.
The defining feature is the combination of uncertainty-aware prediction with sequential decision-making. After each evaluation, a model is updated through Bayesian inference, and an acquisition rule determines where to evaluate next. The rule balances exploiting locations that appear promising against exploring locations where uncertainty might conceal a better result. Bayesian optimization therefore concerns not only predicting an unknown function but also deciding which observations are worth purchasing.
Its effectiveness depends on the relationship between the evaluation budget, the structure of the search space, and the reliability of the probabilistic model. The classical setting involves a small or moderate number of continuous variables and costly scalar observations, but extensions accommodate noise, constraints, discrete choices, parallel experiments, multiple objectives, and evaluations of varying fidelity.
Decision-theoretic formulation and historical development
Consider maximization over a feasible set :
The function is unknown except through observations. At iteration , the optimizer selects and receives an observation such as
where represents measurement or simulation noise. The accumulated data are . Minimization is equivalent after replacing by . The challenge is to choose a useful sequence of evaluations without spending most of the available budget learning irrelevant parts of the domain.
The Bayesian formulation places a prior distribution over possible functions and updates it to a posterior distribution after observations. A decision rule then assigns a value to possible experiments. In principle, one could solve a finite-horizon problem that accounts for every possible future observation and decision. This is usually computationally prohibitive: an experiment changes both the best available solution and the information used to choose subsequent experiments. Most practical methods therefore use one-step utilities or tractable approximations to longer-term planning.
These ideas developed at the intersection of experimental design, decision theory, and mathematical optimization. Harold Kushner’s 1964 work used stochastic-process models to guide optimization of unknown functions. Jonas Mockus and collaborators developed Bayesian approaches to global optimization during the 1970s, including improvement-based criteria. Donald Jones, Matthias Schonlau, and William Welch’s 1998 Efficient Global Optimization algorithm helped establish the modern combination of a Gaussian-process surrogate and expected improvement for expensive deterministic computer experiments.
The approach differs from conventional numerical optimization in what it treats as scarce. A gradient-descent method may use many inexpensive evaluations and derivative calculations. Bayesian optimization can justify considerable computation between evaluations if one physical experiment takes hours or one simulation consumes substantial computing resources. Conversely, its overhead can make it unattractive when the original function is cheap.
It also differs from learning an accurate global emulator. An emulator may be judged by prediction error across the entire domain, whereas an optimizer primarily needs to identify excellent decisions. A model can be imperfect away from the optimum yet support effective optimization. Conversely, low average prediction error does not guarantee that a model correctly identifies a narrow, high-performing region.
Gaussian-process surrogate models
The classical surrogate is a Gaussian process, a probability distribution over functions whose values at any finite collection of inputs have a joint multivariate normal distribution. It is specified by a mean function and a covariance kernel :
The kernel expresses assumptions about smoothness, characteristic length scales, and relationships between inputs. Nearby points generally have strongly correlated values, although “nearby” is defined by the kernel rather than necessarily by ordinary distance.
For independent Gaussian observation noise with variance , let be the matrix with entries , and let contain the covariances between and the observed inputs. Conditional on fixed kernel parameters, the posterior mean and latent-function variance are
Here is the vector of prior means at the observed inputs. These expressions distinguish uncertainty about the underlying function from the variability of a future noisy observation. Predictive observation variance includes an additional noise term.
Common kernels include squared-exponential and Matérn kernels. The squared-exponential kernel encodes very smooth functions; Matérn kernels permit controlled degrees of roughness. Automatic relevance determination assigns separate length scales to input coordinates. A large length scale means that variation along that coordinate is expected to be gradual, while a small one permits rapid variation. Such parameters can suggest which variables matter, but they do not by themselves establish causal importance.
Kernel parameters are often estimated by maximizing the marginal likelihood. This balances data fit against the complexity of the covariance structure. A more fully Bayesian treatment integrates over their posterior distribution, using methods such as Markov chain Monte Carlo. Integrating parameters propagates uncertainty that plug-in estimates ignore, although it increases computational cost. With very few observations, both the prior and the parameter-estimation procedure can substantially influence subsequent experiments.
Gaussian-process calculations normally use a Cholesky decomposition rather than explicitly forming an inverse. Standard dense fitting requires roughly cubic time and quadratic storage in the number of observations. A small diagonal “jitter” term can improve numerical conditioning, but it should not be confused with a scientifically justified noise model. Sparse approximations and structured calculations extend the practical range, sometimes at the cost of altering uncertainty estimates precisely where acquisition decisions are sensitive.
Alternative surrogates and structured search spaces
Gaussian processes are not essential to Bayesian optimization. The broader requirement is a predictive representation that supports useful decisions under uncertainty. Different models may be preferable when the domain contains categorical variables, conditional parameters, discontinuities, or large amounts of historical data. The quality of uncertainty estimates matters at least as much as ordinary predictive accuracy, because an acquisition rule actively seeks regions where its model predicts either high performance or useful uncertainty.
A random forest can represent nonlinear interactions and mixtures of numerical and categorical variables without requiring a single smooth covariance geometry. Variation among trees can be used to construct uncertainty estimates, though this variation is not automatically a calibrated Bayesian posterior. Sequential Model-based Algorithm Configuration, or SMAC, became an influential example of optimization using random-forest surrogates for algorithm configuration. Its setting includes conditional parameters: a variable may be meaningful only when a particular algorithmic component is selected.
The tree-structured Parzen estimator takes another route. Instead of directly modeling a distribution over objective values conditional on inputs, it estimates input densities associated with better and worse observations. If a threshold divides low losses from high losses, these densities can be written as
Under the method’s assumptions, an expected-improvement criterion can be related to a density ratio involving and . This construction is particularly convenient for hierarchical configuration spaces. It is not equivalent to Gaussian-process regression, and its uncertainty representation should not be interpreted as a Gaussian-process posterior.
Other surrogates include Bayesian linear models, ensembles, and neural networks with approximate Bayesian inference. A linear model over carefully chosen features can be efficient and interpretable, but its flexibility depends on the features. Neural representations can capture complex structure and transfer information between related tasks, while uncertainty calibration remains a central difficulty. An ensemble’s disagreement may reflect initialization or training instability rather than all relevant uncertainty about the objective.
Representing the domain is often as consequential as choosing the model family. Integer settings must respect discreteness, and categorical labels generally should not be assigned an arbitrary numerical ordering. Conditional variables require representations that distinguish “inactive” from meaningful numerical values. Specialized kernels can compare graphs, sequences, or molecular structures, but then acquisition optimization must also operate over those objects. A powerful surrogate is insufficient if its proposed candidates cannot be constructed, synthesized, or evaluated.
Acquisition functions and the exploration–exploitation trade-off
An acquisition function assigns a score to a candidate experiment using the current posterior. Unlike the expensive objective, it is usually cheap enough to evaluate repeatedly. The next point is often chosen as
Different acquisitions encode different notions of experimental value. They need not produce the same balance between immediate improvement and information gathering, even when they use identical surrogate predictions.
Expected improvement is a classical choice. For noiseless maximization, let denote the best observed value and define
If the posterior value at is normal with mean and standard deviation , its expected improvement is
where and are the standard normal cumulative distribution and density functions. When , the appropriate expression is the deterministic positive improvement. The two terms reflect the contribution of a favorable predicted mean and the possibility of improvement arising from uncertainty.
Probability of improvement instead evaluates the chance of exceeding a threshold. It can favor a tiny, highly probable gain over a larger but less certain gain because it does not account for improvement magnitude. Upper confidence bound methods use a score such as
The exploration parameter controls the uncertainty bonus. Its theoretically justified schedule depends on assumptions about the function, domain, and noise; a fixed practical setting does not automatically inherit a published regret guarantee.
Thompson sampling draws a function from the posterior and evaluates a maximizer of that sampled function. Across repeated draws, uncertainty induces exploration. Information-based methods take a more explicit learning objective. Entropy search and predictive entropy search seek information about the optimizer, while max-value entropy search focuses on uncertainty about the optimal value. Their calculations use quantities related to mutual information and often require approximation.
The knowledge gradient values an experiment by the expected increase in the best posterior mean after observing its outcome. This is particularly meaningful when the final recommendation need not be an already evaluated point. Unlike a simple improvement rule, it can value an observation that clarifies a different candidate’s quality through model correlations.
These acquisitions are generally approximations to a sequential decision problem, not universally optimal policies. Their relative performance depends on noise, budget, prior assumptions, and numerical implementation. An acquisition may be theoretically attractive yet perform poorly if its internal optimization misses useful regions or its Monte Carlo estimates are too unstable.
The sequential algorithm and its numerical implementation
A practical run begins with a defined domain, an evaluation budget, an observation protocol, and an initial design. Initial observations prevent the first model from relying entirely on prior assumptions. Random designs, Latin hypercube sampling, and Sobol sequences are common choices. Space-filling designs can cover a bounded domain more evenly than a small independent random sample, although no universal initial-design size suits every problem.
The basic loop fits or updates the surrogate, constructs an acquisition function, optimizes it, evaluates the selected candidate, and appends the result to the data. Input rescaling is often important: a parameter measured in thousands should not dominate one measured in fractions merely because of its units. Positive scale parameters, such as learning rates, are frequently represented logarithmically. Transformations change the geometry implied by the model and therefore embody substantive assumptions rather than being only cosmetic preprocessing.
Acquisition optimization is itself often a nonconvex problem. Because evaluations are cheap relative to the original experiment, implementations may combine a large candidate screen with local searches from multiple starting points. When the acquisition is differentiable, automatic differentiation can supply gradients. Mixed or discrete domains may instead use enumeration, local combinatorial moves, evolutionary search, or relaxations followed by projection. Rounding a continuous optimum to an integer is convenient but can repeatedly propose the same point or misrepresent the best discrete choice.
Monte Carlo acquisition functions introduce an additional numerical layer. Their expectations can be approximated using shared base samples so that the approximate acquisition varies smoothly as a candidate moves. Independent simulation noise at every candidate can make acquisition optimization unnecessarily difficult. Numerical underflow is another issue: expected improvement may be extremely small over most of the domain, motivating stable logarithmic formulations rather than direct evaluation of vanishing quantities.
Bookkeeping is part of the algorithm. Completed, failed, and still-running evaluations must be distinguished. A failed simulation may indicate an infeasible region, a numerical instability, or an infrastructure problem; assigning all failures an arbitrary objective penalty can distort the surrogate. Recorded metadata such as evaluation duration, random seed, and experimental batch can help explain variation or support richer models.
Stopping may be determined by a fixed number of evaluations, elapsed time, or exhausted resources. Model-based criteria, such as a small predicted improvement, are less definitive because they inherit model misspecification and acquisition-optimization errors. The final recommendation must also be specified: possibilities include the best observed point, the maximizer of the posterior mean, or a candidate validated through additional measurements. These choices can differ substantially in noisy settings.
Noisy observations, unknown constraints, and safe exploration
Noise changes both learning and the definition of improvement. If the objective is observed with error, the largest recorded value may simply be an unusually favorable fluctuation. Substituting this value into noiseless expected improvement can create an unrealistic threshold and misdirect the search. Noisy improvement methods instead integrate over uncertainty in the latent values at previously evaluated points and candidate points.
Repeated measurements can help distinguish true performance differences from observation noise. Their value depends on the noise level, the remaining budget, and the availability of informative alternatives. In a noisy experiment, revisiting a point need not be redundant: it can reduce uncertainty about the final recommendation or clarify the shape of the objective nearby. Heteroscedastic models allow noise variance to depend on the input, as when some operating regimes produce much more variable results than others.
Unknown constraints require additional models. Suppose feasibility is defined by
When objective and constraint predictions are modeled independently, a common heuristic multiplies expected improvement by the posterior probability of feasibility:
The factorization into separate constraint probabilities requires an additional independence assumption. Correlated constraints, or correlations between feasibility and performance, may require joint modeling rather than a simple product.
The treatment of constraints also depends on how experiments are conducted. In a coupled evaluation, the objective and all constraints are observed together. In a decoupled setting, different tests reveal different quantities at different costs. A cheap stability test might screen a configuration before an expensive performance experiment. The acquisition problem then includes choosing which quantity to observe, not merely which input to evaluate.
Constrained optimization should be distinguished from safe optimization. Ordinary constrained Bayesian optimization may deliberately evaluate a point that could be infeasible in order to learn where the boundary lies. Safe methods restrict experiments to a region certified, under explicit assumptions, to satisfy safety conditions with high probability. SafeOpt, introduced in 2015, is a prominent Gaussian-process-based formulation. Its guarantees depend on valid uncertainty bounds and assumptions that permit expansion from an initial safe set.
Safety is therefore conditional on the mathematical and measurement model. A narrow predictive interval from a misspecified model does not establish physical safety. Some feasible regions may also be unreachable without crossing uncertified territory. Under such restrictions, the attainable target can be the best safely reachable point rather than the unrestricted global optimum.
Parallel evaluations, multiple fidelities, and multiple objectives
When several experiments can run simultaneously, selecting a batch requires accounting for dependence among their outcomes. Choosing the top several points under a single-point acquisition can produce redundant experiments concentrated in one small region. Batch expected improvement instead values the improvement produced by the batch as a whole:
Its joint expectation reflects posterior correlations. Optimizing it is more demanding because the decision includes every coordinate of every batch member.
Asynchronous optimization selects a new experiment whenever a worker becomes available. Results from pending experiments are unknown but relevant. “Fantasizing” approximates this situation by drawing possible pending outcomes, conditioning on them, and averaging candidate utilities. Other methods use posterior samples or local penalization. Evaluation duration can influence the resulting dataset: if some regions finish quickly, an asynchronous scheduler may accumulate disproportionate information there unless timing is explicitly considered.
Multi-fidelity optimization uses cheaper approximations to the target evaluation. Examples include coarser simulation meshes, smaller datasets, shorter training runs, or simplified physical models. A decision is then a pair , where specifies fidelity. The optimizer must judge both how promising is and whether the information gained at fidelity justifies its cost.
Low-fidelity observations are useful only insofar as they inform the high-fidelity target. A cheap simulator can be systematically biased, and rankings may reverse between short and long training runs. Models may represent fidelities through correlated outputs or by making fidelity an additional input. Cost-aware acquisition rules evaluate information or expected improvement relative to resource use, though a simple division by cost is not a universal solution to a finite-budget planning problem.
In multi-objective optimization, there is usually no single best point. The aim is often to identify a set of trade-offs satisfying Pareto optimality: no objective can be improved without worsening another. Expected hypervolume improvement measures the expected expansion of the objective-space region dominated by the current nondominated solutions, relative to a reference point. Its behavior depends on that reference point and the scaling of the objectives.
Scalarization provides an alternative by combining objectives into a single utility or repeatedly optimizing different combinations. Weighted sums are simple but may fail to recover portions of a nonconvex Pareto frontier. More elaborate scalarizations or direct set-based acquisitions address different trade-off structures. Across parallel, multi-fidelity, and multi-objective settings, the central task remains selecting experiments according to their expected contribution to a clearly defined final decision.
High-dimensional optimization and model structure
Classical Bayesian optimization becomes difficult as the number of variables increases. This reflects both statistical and computational problems. A sparse dataset provides little coverage of a high-dimensional domain, kernel parameters become harder to identify, and acquisition optimization becomes more challenging. The curse of dimensionality is not removed by a probabilistic surrogate; successful high-dimensional methods normally exploit additional structure.
One possibility is low effective dimensionality: the objective depends strongly on only a few combinations of variables. Random-embedding methods search in a lower-dimensional space whose image intersects useful regions of the original domain. Their effectiveness depends on the relationship between the embedding, the active directions, and domain boundaries. A low-dimensional search can be efficient when its assumptions hold, but it can exclude good solutions if important variation lies outside the represented subspace.
Additive models assume that the objective decomposes into simpler components, for example,
where each selects a small subset of coordinates. This can greatly reduce the amount of data needed to learn each component. The difficulty is determining the groups and representing interactions across them. A strongly interacting objective may not be adequately described by low-order additive terms, even when each individual variable appears weak in isolation.
Local methods provide another strategy. A trust-region method limits search to a region around a promising point and adapts its extent according to progress. TuRBO, introduced in 2019, uses local Gaussian-process models within adaptive trust regions and can maintain several regions simultaneously. Local modeling avoids requiring one stationary model to explain an entire complex domain. It does not, however, make local success equivalent to a global guarantee, and mechanisms for restarting or allocating resources among regions remain important.
Learned representations can exploit previous experiments, related tasks, or structured inputs. For example, a neural embedding may map a large configuration description into a smaller space on which a Gaussian process operates. This connects Bayesian optimization with dimensionality reduction and transfer learning. The representation must preserve distinctions important for optimization, not merely reconstruct inputs or predict average outcomes well. Jointly learning the representation and uncertainty from very little target data is especially difficult.
There is no universal dimensional threshold beyond which Bayesian optimization ceases to work. A hundred-variable problem with strong locality or sparsity can be easier than a ten-variable problem containing narrow discontinuities and weak predictive structure. Evaluation cost matters as well: expensive experiments can justify sophisticated structural models, whereas inexpensive objectives may favor broader direct-search strategies. Effective dimension, interaction structure, noise, and available prior information are more informative than coordinate count alone.
Convergence, regret, and the limits of theoretical guarantees
Theoretical analysis often measures performance through regret. For maximization, cumulative regret after evaluations is
This criterion penalizes every poor evaluation and is natural when each deployed decision has an immediate consequence. Simple regret instead evaluates the quality of a final recommendation:
The distinction matters for experimental optimization. A highly informative but low-performing experiment can improve the final answer while increasing cumulative regret.
Bayesian optimization is closely related to the multi-armed bandit problem. In continuum-armed settings, correlations between candidate decisions allow an observation at one point to inform others. Gaussian-process upper confidence bound analysis, notably work presented by Niranjan Srinivas and collaborators in 2010, connected regret to the information obtainable from observations under the selected kernel. This helped explain how kernel structure influences learnability rather than treating all search spaces as equally difficult.
A typical complexity measure is maximum information gain,
where denotes mutual information and represents an allowed collection of evaluation locations. Under suitable assumptions, regret bounds have schematic forms involving
with additional constants or logarithmic factors depending on the result. Sublinear cumulative regret implies that average regret vanishes. This expression is not a blanket guarantee for every Gaussian-process optimizer; it summarizes a family of analyses with specific assumptions.
Those assumptions may place the objective under the stated Gaussian-process prior or require a bounded norm in an associated reproducing kernel Hilbert space. Noise conditions, domain regularity, kernel parameters, and the accuracy of acquisition maximization also matter. Smoothness assumptions constrain how narrow or irregular an undiscovered optimum can be. Without such structure, finitely many observations cannot rule out an arbitrarily large peak in an unvisited region.
Practical implementations often estimate kernel parameters, use approximate posteriors, and only approximately optimize acquisitions. A theorem proved for a fixed known kernel does not automatically cover these adaptations. Similarly, asymptotic convergence says little by itself about performance after twenty or fifty expensive experiments. Finite-budget behavior depends strongly on constants, initialization, and whether the model’s assumptions resemble the real objective.
Misspecification remains a central limitation. Overly smooth kernels can suppress abrupt changes, and overconfident uncertainty estimates can prevent exploration of neglected regions. Heavy-tailed noise, nonstationarity, and hidden changes in experimental conditions can invalidate nominal uncertainty bounds. Robust modeling and diagnostic checks can expose some failures, but theoretical guarantees remain statements about defined mathematical conditions rather than unconditional certificates of global optimality.
Applications, benchmarking, and interpretation of results
A major application is hyperparameter optimization in machine learning. The input may specify regularization strengths, learning rates, architecture choices, or data-processing settings, while the objective is a validation score after training. Jasper Snoek, Hugo Larochelle, and Ryan Adams’s 2012 work helped popularize Gaussian-process Bayesian optimization for this purpose. The setting naturally combines expensive evaluations, noisy outcomes, and parameters whose useful ranges span several orders of magnitude.
The objective being optimized must be interpreted carefully. Performance on a repeatedly consulted validation set is not identical to performance on new data. Searching many configurations can overfit the validation criterion, and training randomness can obscure small differences. Cross-validation, independent final evaluation, and explicit accounting for repeated seeds address different aspects of this problem. Bayesian optimization improves the allocation of search effort; it does not remove statistical uncertainty in the resulting performance claims.
In engineering, candidates may be designs evaluated by finite-element simulations, computational fluid dynamics, or other numerical models. Relevant constraints can include stress, weight, efficiency, and manufacturing feasibility. Surrogate-based optimization is attractive when one evaluation takes hours, but the result remains conditional on simulator fidelity. A design that optimizes a numerical approximation can exploit artifacts of that approximation, making high-fidelity confirmation distinct from the optimization process itself.
Laboratory applications include reaction-condition selection, formulation development, and materials discovery. A candidate can encode composition, temperature, processing time, or instrument settings. Here, observation noise may include batch effects, sample variability, and drift. Automated platforms can close the loop between model fitting and physical experimentation, but operational restrictions—available reagents, setup costs, and allowable experiment sequences—must be represented if proposed experiments are to be executable.
Benchmarking requires more than comparing the best displayed curve. Relevant baselines include random search, suitable space-filling designs, and established optimization methods matched to the domain. Comparisons should specify initial data, evaluation budgets, parallel resources, acquisition overhead, and failure handling. Counting evaluations is appropriate when costs are similar; elapsed time or monetary cost is more informative when they differ. Repeated independent runs reveal sensitivity to initialization and stochastic outcomes.
Finally, reported success depends on how the incumbent is measured. In a noisy task, the best observed score is optimistically selected, whereas independently reevaluated performance estimates the quality of the recommendation more directly. Synthetic benchmark functions can isolate mathematical behavior but may not reproduce real noise, conditional parameters, or experiment failures. Historical experiment logs also require care: a new optimizer may select points absent from the log, so replaying only recorded observations cannot generally establish how the optimizer would have performed prospectively.