{
  "id": "hyperparameter-optimization",
  "title": "Hyperparameter Optimization",
  "lang": "en",
  "source_lang": "en",
  "summary": "Hyperparameter optimization selects learning-system configurations through systematic evaluation, balancing predictive performance, computational cost, and statistical reliability.",
  "category": "computer-science",
  "subcategory": "artificial-intelligence",
  "type": "concept",
  "aliases": [
    "Hyperparameter tuning",
    "Hyperparameter search",
    "HPO"
  ],
  "generated_at": "2026-11-04",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/hyperparameter-optimization/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/hyperparameter-optimization/",
      "json": "https://aiwiki.page/zh/hyperparameter-optimization.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "Hyperparameter optimization is the process of selecting the configuration settings that govern how a machine-learning system is constructed, trained, or evaluated. These settings, called hyperparameters, include regularization strengths, learning rates, tree depths, network architectures, and preprocessing choices. Unlike model parameters estimated directly from training examples, hyperparameters are usually chosen by an outer procedure that compares the results of multiple training runs.\n\nThe subject lies at the intersection of mathematical optimization, statistical model selection, and computing systems. A configuration that works well for one dataset, training budget, or hardware platform may perform poorly under another. Hyperparameter optimization therefore concerns not only finding high predictive accuracy but also defining a meaningful evaluation protocol, allocating computation efficiently, and distinguishing genuine improvements from random variation. Its methods range from simple grids and random sampling to probabilistic search, adaptive resource allocation, and differentiation through training."
    },
    {
      "heading": "Mathematical formulation and scope",
      "text": "A learning algorithm maps a configuration and a dataset to a fitted model. If \\(\\lambda\\) denotes a hyperparameter configuration and \\(\\mathcal{A}\\) denotes the training procedure, a basic formulation is\n\n\\[\\hat{\\lambda}\\in\\operatorname*{arg\\,min}_{\\lambda\\in\\Lambda} \\widehat{R}_{\\mathrm{val}}\\!\\left(\\mathcal{A}(D_{\\mathrm{train}};\\lambda)\\right),\\]\n\nwhere \\(\\Lambda\\) is the search space, \\(D_{\\mathrm{train}}\\) is the training data, and \\(\\widehat{R}_{\\mathrm{val}}\\) is an estimated validation risk. For classification, this risk might be the fraction of incorrect predictions; for regression, it might be mean squared error. The training objective and the selection criterion need not coincide: a classifier may minimize a differentiable loss function during training but be selected using average precision, calibration error, or another application-specific measure.\n\nThe distinction between parameters and hyperparameters is functional rather than absolute. In a neural network, weights are normally fitted from data, while the learning rate and the strength of regularization are selected externally. Yet an outer optimization procedure can learn regularization coefficients, and some training procedures adapt learning rates automatically. The relevant distinction is between quantities updated by the inner learning process and decisions controlled by the outer selection process. Pipeline decisions, such as choosing a feature representation or selecting between a support vector machine and a random forest, can also be encoded as hyperparameters.\n\nThis structure often gives rise to a bilevel optimization problem:\n\n\\[\\min_{\\lambda\\in\\Lambda}L_{\\mathrm{val}}(\\theta^*(\\lambda),\\lambda), \\qquad \\theta^*(\\lambda)\\in\\operatorname*{arg\\,min}_{\\theta} L_{\\mathrm{train}}(\\theta,\\lambda).\\]\n\nThe inner problem fits model parameters \\(\\theta\\); the outer problem selects \\(\\lambda\\). In practice, training may stop before reaching a minimum, or its outcome may depend on initialization, minibatch order, and numerical behavior. It is consequently often more accurate to regard the objective as the performance of a specified training procedure under a specified budget, rather than as the performance of an exactly optimized model.\n\nA trial is one evaluation of a proposed configuration, possibly including several folds or random seeds. If its observed score is \\(Y(\\lambda)=f(\\lambda)+\\varepsilon\\), the term \\(\\varepsilon\\) can include training randomness and finite-sample evaluation error. The search procedure observes \\(Y\\), not the unknown expected performance \\(f\\). It may also observe runtime, memory use, and intermediate metrics. Objectives involving latency or memory constraints require distinguishing a genuinely infeasible configuration from one that merely receives a poor predictive score.\n\nMost practical searches seek a strong feasible configuration within a finite budget, not a certified global optimum. The budget may be expressed in trials, elapsed time, accelerator-hours, or monetary cost. These definitions are not interchangeable: one trial can train a small linear model in seconds, while another can train a large neural network for days."
    },
    {
      "heading": "Search-space design and parameterization",
      "text": "A search space defines which learning systems can be considered. It can contain continuous quantities, integers, categorical choices, and conditional variables. A regularization coefficient is typically continuous; the number of trees is integer-valued; an optimizer choice is categorical. A kernel bandwidth matters only when a corresponding kernel method is selected. Such dependencies make many realistic spaces hierarchical rather than simple rectangular boxes.\n\nParameterization affects both coverage and the assumptions made by a search algorithm. For a positive coefficient spanning several orders of magnitude, uniform sampling in logarithmic coordinates often expresses the intended uncertainty better than uniform sampling in the original units. If \\(\\lambda\\) lies between \\(10^{-6}\\) and \\(10^{-1}\\), sampling\n\n\\[u\\sim\\operatorname{Uniform}(-6,-1), \\qquad \\lambda=10^u\\]\n\nallocates equal probability to each decade. Uniform sampling of \\(\\lambda\\) itself concentrates most trials near the upper end when viewed on a logarithmic scale. Neither choice is universally correct: the sampling distribution is part of the search design, not a neutral implementation detail.\n\nConditional structure is particularly important for algorithm selection. A search over support vector machines and random forests might expose a regularization parameter for the former and a feature-subsampling fraction for the latter. Treating all variables as active for every trial introduces meaningless coordinates. Tree-structured representations, explicit activation conditions, or separate subspaces can prevent this problem. Constraints can additionally exclude invalid combinations, such as incompatible solver options or model dimensions that exceed available memory.\n\nHyperparameters interact. A larger learning rate may require a different warm-up schedule; a deeper neural network may require stronger regularization; increasing batch size may alter both optimization behavior and hardware utilization. Searching each coordinate independently can miss these relationships. Conversely, adding many weakly relevant dimensions can make search unnecessarily difficult. Parameter importance is therefore conditional on the ranges, datasets, and algorithms examined. A variable that appears unimportant in a narrow interval may be decisive outside it, and an observed interaction need not transfer to a different model family.\n\nThe search space can encompass the whole modeling pipeline. Feature scaling, missing-value handling, feature selection, and data augmentation can influence the best model configuration. For gradient boosting, learning rate and number of boosting rounds are closely related; for deep networks, optimizer settings interact with initialization and normalization. A search that includes preprocessing must evaluate those steps within the same data-splitting boundaries as model fitting. Otherwise, apparent gains can reflect information reaching the model through the preprocessing procedure rather than a better configuration."
    },
    {
      "heading": "Validation, selection bias, and generalization",
      "text": "Hyperparameter optimization relies on an estimate of performance on observations not used to fit model parameters. A validation set supplies this signal, while a test set is reserved for evaluating the selected procedure. Repeatedly consulting the test set to choose configurations converts it into part of the tuning process. The resulting score no longer provides an independent assessment of the final choice. This remains true when test feedback is used informally, such as deciding which architecture to investigate next.\n\nCross-validation reduces dependence on a single partition by fitting and evaluating models across several splits. In \\(K\\)-fold cross-validation, each fold serves once as validation data while the remaining folds provide training data. Averaging the fold scores produces an estimate used to rank configurations. The computational cost is approximately multiplied by \\(K\\), although actual runtime depends on training-set size and implementation. Folds are not independent experiments because their training sets overlap, so the standard deviation of fold scores cannot automatically be interpreted as the standard error of an independent sample mean.\n\nThe splitting scheme must match the prediction task. Randomly splitting temporally ordered observations can allow future information to influence predictions about the past. Grouped data require keeping related observations together when the intended task concerns previously unseen groups. For example, records from one individual or images derived from the same original source can create data leakage if distributed across training and validation partitions. Preprocessing statistics, feature selection, and learned transformations must also be fitted using only the training portion of each split.\n\nSelection itself introduces optimism. If many configurations have similar true performance but noisy observed scores, the smallest observed error tends to belong to a configuration that benefited from favorable noise. Thus, increasing the search budget can improve the best reported validation score without improving generalization. Adaptive searches can intensify this effect because later configurations respond to earlier validation outcomes. Repeating trials with multiple seeds measures some training variability, but it does not remove uncertainty caused by repeatedly selecting against the same finite validation sample.\n\nNested cross-validation separates tuning from performance estimation. Within each outer training split, an inner procedure selects hyperparameters; the corresponding outer test fold then evaluates that complete selection procedure. The resulting estimate concerns the procedure—including its search budget and selection rule—rather than one fixed configuration. It can be expensive because the search is repeated for every outer split. After assessment, a final model may be tuned and fitted using the available development data, but its configuration need not match those selected in the individual outer folds.\n\nThese issues connect hyperparameter optimization with overfitting and distribution shift. A configuration can overfit the validation process even when each individual model is appropriately regularized. Moreover, an unbiased estimate for one data distribution does not guarantee performance after deployment conditions change. The objective, split design, and reporting protocol therefore determine what a successful search result actually establishes."
    },
    {
      "heading": "Grid search, random search, and direct methods",
      "text": "Grid search evaluates a finite Cartesian product of candidate values. If \\(d\\) hyperparameters each receive \\(k\\) values, the full grid contains \\(k^d\\) configurations. Five parameters with ten values each require 100,000 trials. This exponential growth makes dense grids impractical in high-dimensional spaces. Grids remain useful when there are only a few meaningful choices, when an exact finite menu must be compared, or when structured results are needed for visualization and sensitivity analysis.\n\nA grid's efficiency depends strongly on how many coordinates substantially affect performance. Suppose only two out of ten hyperparameters are influential. A grid repeats the same values of those two parameters across many combinations of the other eight. Random search instead samples complete configurations from a specified distribution, usually producing many more distinct values along each influential continuous coordinate. James Bergstra and Yoshua Bengio's 2012 study, “Random Search for Hyper-Parameter Optimization,” made this argument influential in machine learning and reported empirical comparisons supporting random search as a strong baseline.\n\nRandom search also supports a simple coverage calculation. If a desirable region occupies probability mass \\(p\\) under the sampling distribution, then \\(n\\) independent trials hit it at least once with probability\n\n\\[1-(1-p)^n.\\]\n\nFor \\(p=0.01\\), 100 trials give a probability of about \\(0.634\\); approximately 299 trials are needed to exceed \\(0.95\\). This calculation does not require knowing the geometry of the region, but it does depend on its probability under the chosen distribution. A poor parameterization can make valuable configurations extremely unlikely.\n\nSpace-filling methods modify sampling to obtain more even coverage. Quasi-Monte Carlo methods, including Sobol sequences, distribute points systematically over a continuous domain. Latin hypercube designs stratify each coordinate so that its marginal range is more evenly represented. Such methods can improve initial coverage, particularly in moderate dimensions, but conditional variables and irregular constraints complicate their application. More uniform coverage is also not equivalent to concentrating evaluations near promising configurations.\n\nDirect optimization methods can exploit observed objective values without fitting a probabilistic surrogate. Coordinate search varies one component at a time; pattern search explores neighborhoods; simplex-based methods maintain a geometric set of candidate points. These approaches can be effective in low-dimensional continuous spaces, but mixed categories, noisy scores, and costly evaluations can undermine their assumptions. Rounding continuous proposals into integer settings may also create repeated evaluations or distort neighborhood relationships.\n\nGrid and random search remain important experimental controls. They are straightforward to implement, naturally parallel, and comparatively easy to audit. A sophisticated optimizer's advantage is meaningful only relative to such baselines under comparable search spaces and resource budgets. Equal numbers of trials alone may be misleading if one method preferentially selects much more expensive models."
    },
    {
      "heading": "Bayesian optimization and surrogate models",
      "text": "Bayesian optimization addresses expensive objectives by using previous observations to guide future evaluations. It typically combines a surrogate model of performance with an acquisition function that selects promising configurations. The surrogate estimates how the objective varies across the search space and represents uncertainty or predictive variation. The acquisition function balances exploiting configurations expected to perform well against exploring configurations whose performance is uncertain. Updating the surrogate after each observation makes the search adaptive.\n\nA common surrogate is a Gaussian process, which defines a distribution over functions. Given observations, it yields a predictive mean \\(\\mu(\\lambda)\\) and standard deviation \\(\\sigma(\\lambda)\\) under its modeling assumptions. Covariance functions encode relationships between configurations, including smoothness and characteristic length scales. These assumptions are consequential: a model that works well for smooth continuous parameters may poorly represent categorical switches or abrupt failures. Conventional exact Gaussian-process inference also requires operations whose cost grows cubically with the number of observations, motivating approximations and alternative surrogates for large histories.\n\nFor a minimization problem, expected improvement measures the expected amount by which a candidate beats a reference value \\(f_{\\mathrm{best}}\\):\n\n\\[\\operatorname{EI}(\\lambda) = \\mathbb{E}\\!\\left[\\max\\{f_{\\mathrm{best}}-F(\\lambda),0\\}\\right].\\]\n\nUnder a Gaussian predictive distribution with positive standard deviation, it becomes\n\n\\[\\operatorname{EI}(\\lambda) = (f_{\\mathrm{best}}-\\mu(\\lambda))\\Phi(z) +\\sigma(\\lambda)\\phi(z), \\qquad z=\\frac{f_{\\mathrm{best}}-\\mu(\\lambda)}{\\sigma(\\lambda)},\\]\n\nwhere \\(\\Phi\\) and \\(\\phi\\) are the standard normal cumulative distribution and density functions. This expression illustrates how either a favorable mean or substantial uncertainty can make a candidate attractive. Noisy observations require care in defining the reference and distinguishing latent performance from measurement noise.\n\nOther surrogates include random forests and density estimators. The tree-structured Parzen estimator, introduced in work by Bergstra and colleagues in 2011, models distributions of configurations associated with better and worse outcomes, rather than directly modeling a Gaussian conditional distribution of performance. The sequential model-based algorithm configuration framework, SMAC, also introduced in 2011, used random-forest-based models to address algorithm-configuration spaces containing categorical and conditional parameters. These methods helped extend model-guided search beyond small, entirely continuous domains.\n\nPractical Bayesian optimization must solve an inner optimization problem to maximize its acquisition function. Although acquisition evaluations are usually much cheaper than full model training, this step is not free and can become difficult in large mixed spaces. Batch selection further complicates the process because outcomes for pending trials are unknown. Parallel methods may account for pending evaluations through hypothetical outcomes or choose a diverse set jointly, rather than repeatedly proposing nearly identical configurations.\n\nCost-aware and constrained variants incorporate additional predictions. A candidate might have attractive expected improvement but require ten times the training time of an alternative. Search can model evaluation cost, estimate constraint satisfaction, or optimize an acquisition function reflecting improvement per resource unit. Such choices define a different search policy from maximizing improvement per trial. Their effectiveness depends on how reliably both performance and cost can be inferred from the available observations."
    },
    {
      "heading": "Multi-fidelity optimization and early resource allocation",
      "text": "A full training run is often unnecessary to identify an unpromising configuration. Multi-fidelity optimization uses cheaper approximations, called fidelities, to obtain information before committing the full budget. A fidelity might be the number of training epochs, the fraction of training data, image resolution, simulation accuracy, or the number of evaluation folds. Its usefulness depends on whether low-cost observations predict outcomes at the target fidelity. Low-fidelity rankings need not match full-budget rankings.\n\nSuccessive halving begins by evaluating many configurations with small budgets, retaining only a fraction and assigning more resources to the survivors. With reduction factor \\(\\eta>1\\), each stage retains roughly \\(1/\\eta\\) of the candidates and multiplies their individual resource budgets by \\(\\eta\\). The total nominal resource expenditure per stage is then approximately constant. For example, 81 configurations trained for one unit can be reduced to 27 trained for three units, then nine for nine units, three for 27 units, and one for 81 units. The precise accounting depends on whether later allocations continue earlier training or restart it.\n\nHyperband, introduced by Lisha Li and colleagues, appeared as a preprint in 2016 and in the Journal of Machine Learning Research in 2018. It runs several successive-halving brackets that use different trade-offs between the initial number of configurations and the starting budget. Brackets beginning with many low-budget trials explore broadly but make strong early elimination decisions; brackets beginning with fewer, better-funded trials make less aggressive assumptions about early performance. Hyperband distributes resources across these alternatives instead of selecting a single initial allocation.\n\nAsynchronous successive halving, commonly called ASHA, allows workers to promote or stop trials without waiting for an entire stage to finish. This reduces synchronization delays when training times vary. It is particularly useful on distributed infrastructure, where a synchronous stage can leave many workers idle while waiting for one slow trial. Asynchronous decisions nevertheless depend on which results have arrived, so execution order can affect which configurations receive additional resources.\n\nPruning usually relies on a learning curve, but curves can cross. A configuration with a small learning rate or a long warm-up period may initially look weak and later outperform faster starters. Similarly, a model that performs well on a small data subset may not benefit as much from additional data as another model. Low-fidelity evaluation is therefore an information source with potential bias, not merely a noisier version of the full evaluation. Early elimination can systematically remove configurations whose advantages emerge late.\n\nMulti-fidelity methods can be combined with surrogate modeling. BOHB, introduced in 2018, combined Hyperband-style allocation with model-based configuration proposals. Resource allocation should also be distinguished from early stopping used as part of an individual model's training rule. A search scheduler stops trials to allocate resources among competitors; a training procedure may stop because validation performance has ceased improving. When training duration itself affects the final selected model, its interpretation must remain consistent across search and final fitting."
    },
    {
      "heading": "Gradient-based, evolutionary, and population-based approaches",
      "text": "When training and validation objectives are differentiable with respect to continuous hyperparameters, the outer objective can be optimized using a hypergradient. Consider a training update\n\n\\[\\theta_{t+1}=\\theta_t-\\alpha_t\\nabla_\\theta L_{\\mathrm{train}}(\\theta_t,\\lambda).\\]\n\nDifferentiating the validation loss through a sequence of such updates accounts for how \\(\\lambda\\) changes the final parameters. This can optimize quantities such as learning rates, regularization coefficients, or example weights. Unlike black-box search, it uses the internal structure of training rather than only the final score.\n\nReverse-mode differentiation through training resembles backpropagation through a long computational graph. Saving all intermediate states can be memory-intensive, while recomputation trades additional computation for reduced storage. Truncated differentiation considers only part of the trajectory and therefore changes the gradient estimate. Forward-mode approaches can avoid storing the entire history but may become expensive when the number of optimized hyperparameters is large. The appropriate method depends on the relative numbers of model parameters, hyperparameters, and training steps.\n\nImplicit differentiation provides another route. If the trained parameters satisfy the stationarity condition\n\n\\[\\nabla_\\theta L_{\\mathrm{train}}(\\theta^*,\\lambda)=0,\\]\n\nand the necessary smoothness and nonsingularity conditions hold, then\n\n\\[\\frac{d\\theta^*}{d\\lambda} = -H^{-1} \\frac{\\partial^2 L_{\\mathrm{train}}} {\\partial\\theta\\,\\partial\\lambda},\\]\n\nwhere \\(H\\) is the Hessian with respect to model parameters. Implementations generally seek Hessian-vector products and approximate linear solves rather than constructing and inverting a full Hessian. The resulting derivative describes sensitivity around a stationary solution; incomplete convergence and nonconvexity complicate its interpretation.\n\nEvolutionary algorithms instead maintain a population of configurations and generate new candidates through mutation, recombination, and selection. They can accommodate discrete architectures, mixed variables, and several competing objectives without requiring differentiability. Their population structure supports parallel evaluation and diverse exploration. They can also consume many trials, and their behavior depends on encoding, mutation operators, population size, and selection pressure. An evolutionary representation that frequently produces invalid configurations wastes computation unless constraints are incorporated into the operators or repair process.\n\nPopulation-based training, introduced by Max Jaderberg and colleagues in 2017, combines model training with periodic population-level intervention. Underperforming members can copy the weights and hyperparameters of stronger members, after which their hyperparameters are perturbed. This produces time-varying schedules rather than selecting only a fixed configuration before training. Because weights are inherited, population members are not independent trials trained from scratch.\n\nThese approaches solve related but distinct problems. Differentiable methods require suitable continuous structure; evolutionary methods can explore discontinuous choices; population-based training changes the training trajectory through interaction among workers. Comparing their output therefore requires specifying whether a “configuration” means fixed settings, a learned schedule, an architecture, or a complete procedure that transfers model state between runs."
    },
    {
      "heading": "Distributed execution, software, and operational objectives",
      "text": "Hyperparameter optimization is also a scheduling and experiment-management problem. A trial may require one processor, several graphics processing units, or a distributed training job spanning multiple machines. The search controller proposes configurations, while an execution layer assigns resources, collects intermediate results, and records outcomes. Parallel computing shortens elapsed search time but does not automatically reduce total computation. Highly parallel execution can also reduce adaptivity because many proposals are issued before earlier results become available.\n\nA useful experiment record contains more than the hyperparameter vector and final metric. It includes the data version, preprocessing procedure, software environment, random seed, hardware allocation, stopping rule, and checkpoint information. Version control identifies changes in training code, while data lineage records how datasets were derived. Reproducibility can still be limited by nondeterministic numerical kernels or changes in execution order. Repeating a configuration therefore requires reconstructing the relevant training procedure, not merely copying a list of numbers.\n\nFailures need explicit treatment. An out-of-memory error, numerical divergence, worker interruption, and deliberate pruning are different outcomes. Assigning all of them the same worst score can distort a surrogate model or conceal infrastructure problems. Conversely, repeatedly retrying configurations that cannot fit the available hardware wastes resources. Search systems commonly distinguish failed, completed, pruned, and pending trials and use checkpointing to recover useful progress. Resuming a trial may also require restoring optimizer state, data-order state, and random-number-generator state.\n\nSoftware systems expose different combinations of these functions. Optuna, introduced in 2019, provides programmable search spaces, samplers, and pruning mechanisms. Ray Tune supports distributed experiment execution and integrates multiple search algorithms and schedulers. Scikit-learn's grid and randomized search interfaces integrate configuration evaluation with cross-validation. Such software names do not identify a single optimization method: the same execution platform can run random search, Bayesian optimization, or successive halving, and each combination can have different statistical and computational behavior.\n\nOperational objectives often extend beyond predictive accuracy. A model intended for inference on a constrained device may face limits on memory, latency, or power consumption. A ranking system may have to satisfy a maximum response time while preserving predictive quality. These settings motivate multi-objective optimization, in which no single configuration necessarily dominates all others. A Pareto-efficient configuration is one for which improving one objective requires worsening another among the alternatives considered.\n\nHardware-dependent metrics require controlled measurement. Latency depends on batch size, precision, software kernels, and whether data transfer and preprocessing are included. A search that optimizes training speed can favor different models from one that optimizes deployment speed. For large-scale fine-tuning, even the configuration-search overhead can be substantial, making constraints on total resource use central to the problem definition rather than secondary implementation details."
    },
    {
      "heading": "Historical development, benchmarking, and unresolved limitations",
      "text": "Hyperparameter optimization grew out of older practices in statistical model selection, experimental design, and algorithm configuration. Selecting smoothing parameters, polynomial degrees, or regularization coefficients predates contemporary deep learning. Sequential optimization of expensive functions also has a longer history; the efficient global optimization work of Donald Jones, Matthias Schonlau, and William Welch in 1998 helped establish a widely used framework combining surrogate models and expected improvement. Machine learning brought these ideas into settings with noisy training runs, heterogeneous configuration spaces, and increasingly expensive evaluations.\n\nSeveral developments around 2011–2012 helped shape the modern field. Tree-structured Parzen estimators and SMAC addressed structured configuration problems, while Bergstra and Bengio established random search as an important baseline. Jasper Snoek, Hugo Larochelle, and Ryan Adams's 2012 paper, “Practical Bayesian Optimization of Machine Learning Algorithms,” helped popularize Gaussian-process-based tuning for machine-learning systems. Subsequent work emphasized scalable parallel execution, multi-fidelity allocation, and integration with automated machine learning, where choosing preprocessing and model families becomes part of the search.\n\nBenchmark design strongly affects conclusions about optimizer quality. A useful comparison controls the search space, dataset splits, evaluation metric, initial information, and budget. Performance can be plotted against completed trials, cumulative compute, or wall-clock time, but these answer different questions. A method that reaches a good result in few evaluations may still be slower if proposal generation is expensive or its chosen trials take longer. Comparing only the single best run can conceal substantial variation across search seeds and tasks.\n\nBenchmarks use several kinds of objective functions. Direct training is realistic but expensive. Tabular benchmarks store previously measured results for a finite collection of configurations, making repeated optimizer comparisons inexpensive and reproducible. Surrogate benchmarks approximate a training objective with a fitted predictive model. Each representation has limitations: finite tables restrict available configurations, while surrogate models can smooth away discontinuities or reproduce their own modeling biases. Learning-curve benchmarks support resource-allocation studies but must represent the relationship between partial and full training faithfully.\n\nPrior experience can accelerate a new search. Configurations successful on related tasks may provide initial candidates or inform a surrogate, connecting optimization with transfer learning and meta-learning. The central difficulty is deciding which experience transfers. Differences in data size, feature distributions, architecture, or training budget can make previously successful settings misleading. Reused configurations also complicate comparisons if one optimizer receives substantial prior information and another starts without it.\n\nHigh-dimensional spaces, weak low-fidelity signals, expensive evaluation, and adaptive validation reuse remain persistent limitations. Performance differences between optimizers can be smaller than differences caused by search-space construction or evaluation design. Even a carefully optimized configuration is tied to a particular task and budget. Claims of superiority therefore require explicit identification of the configuration space, information available to the optimizer, resource accounting, and independent assessment protocol."
    }
  ],
  "links": [
    "machine-learning",
    "hyperparameter",
    "mathematical-optimization",
    "training-data",
    "loss-function",
    "learning-rate",
    "regularization",
    "kernel-method",
    "support-vector-machine",
    "random-forest",
    "artificial-neural-network",
    "feature-scaling",
    "feature-selection",
    "data-augmentation",
    "gradient-boosting",
    "validation-set",
    "test-set",
    "cross-validation",
    "data-leakage",
    "generalization",
    "nested-cross-validation",
    "overfitting",
    "distribution-shift",
    "bayesian-optimization",
    "gaussian-process",
    "learning-curve",
    "early-stopping",
    "backpropagation",
    "graphics-processing-unit",
    "parallel-computing",
    "version-control",
    "data-lineage",
    "neural-network-inference",
    "multi-objective-optimization",
    "fine-tuning",
    "transfer-learning"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}