An objective function is a function that assigns a numerical value to each candidate solution in a mathematical optimization problem. Optimization seeks a candidate with the smallest or largest objective value, subject to any restrictions on the available choices. The function expresses what counts as better within the mathematical model: lower expenditure, greater output, smaller prediction error, or another specified criterion. It is distinct from the decision variables being chosen and the constraints defining which choices are permissible. (stanford.edu)
Mathematical formulation
A scalar minimization problem can be written as
[ \min_{x\in\mathcal F} f(x), ]
where (x) represents the decision variables, (\mathcal F) is the feasible set, and (f) is the objective function. Variables may be continuous, discrete, or a mixture. Constraints commonly take the form (g_i(x)\leq 0) and (h_j(x)=0). An unconstrained problem allows every point in the function’s specified domain. (stanford.edu)
The optimal value is
[ p^\star=\inf_{x\in\mathcal F}f(x). ]
An optimal solution (x^\star) satisfies (f(x^\star)=p^\star). The value and the solution are different objects: several solutions may share one optimal value, and an infimum need not be attained. A maximization problem can be converted into minimization by replacing (f) with (-f). (stanford.edu)
For illustration, minimizing (f(x)=(x-3)^2) over all real (x) gives (x^\star=3) and value zero. Restricting the feasible set to (0\leq x\leq2) changes the solution to (x^\star=2), with value one. This illustrates why an objective alone does not determine the answer: its domain and constraints are equally important.
Structure and optimality
The mathematical structure of the objective helps determine which solution methods apply. Linear programming uses a linear objective with linear equality and inequality constraints. Quadratic objectives include squared-error expressions, while nonlinear objectives may involve products, exponentials, or other nonlinear relationships. Discrete restrictions lead to problems such as integer programming, even when the objective itself is linear. (web.stanford.edu)
In convex optimization, the objective is a convex function and the feasible set is convex. Every local minimum is then globally optimal. Strict convexity implies that there can be at most one minimizer on a convex feasible set, although it does not by itself guarantee existence. Nonconvex problems may contain multiple local minima and saddle points. (web.stanford.edu)
For a differentiable objective, the gradient describes first-order variation. At an interior local minimum of an unconstrained problem, the gradient must vanish. This condition is not generally sufficient: a stationary point may instead be a maximum or saddle point. Gradient descent uses negative-gradient steps to reduce an objective, while Newton’s method also uses second-order information. With constraints, boundary solutions need not have zero gradient; the Karush–Kuhn–Tucker conditions describe optimality under appropriate assumptions. (web.stanford.edu)
Objectives in statistical learning
In machine learning, the decision variables are often model parameters. A loss function measures discrepancy between a prediction and its target; the training objective commonly aggregates losses across training data. Terminology varies, and “loss,” “cost,” and “objective” sometimes denote the same expression. More precisely, an objective can contain both a data-fitting loss and additional terms. (deeplearningbook.org)
A common form is
[ J(\theta)=\frac{1}{n}\sum_{i=1}^{n} \ell\bigl(h_\theta(x_i),y_i\bigr) +\lambda R(\theta). ]
Here (h_\theta) is the predictive model, (\ell) is the example-level loss, (R) is a regularization penalty, and (\lambda\geq0) controls its strength. Without the penalty, minimizing the average training loss is empirical risk minimization. Penalties can encode preferences for particular parameter configurations rather than prediction fit alone. (deeplearningbook.org)
For linear regression, ordinary least squares minimizes the sum of squared residuals. Dividing by the number of observations produces a mean squared error objective with the same minimizers when no other term changes. Maximum likelihood estimation maximizes a likelihood function, or equivalently minimizes its negative logarithm where the likelihood is positive. (deeplearningbook.org)
The population objective is often an expected value of loss under the data-generating distribution. Its empirical counterpart is calculated from a finite sample. Reducing training objective values therefore does not automatically establish improved generalization; an expressive model may exhibit overfitting. (deeplearningbook.org)
Equivalent objectives and numerical behavior
Some transformations preserve optimal solutions. Adding a constant or multiplying by a positive constant leaves minimizers unchanged. More generally, applying a strictly increasing transformation preserves the ordering of feasible objective values. Such transformations can nevertheless change convexity, derivatives, and numerical behavior, so equivalent mathematical objectives need not produce identical computational trajectories. (web.stanford.edu)
Scaling also matters for composite objectives. Multiplying every term by the same positive constant preserves minimizers, but rescaling only the data-fitting term changes its balance against regularization. For example, replacing a summed loss with an averaged loss while keeping the same penalty coefficient generally changes the optimization problem. (deeplearningbook.org)
Multiple criteria
Multi-objective optimization considers several objectives simultaneously, such as minimizing both manufacturing cost and product weight. A feasible solution is Pareto optimal if no other feasible solution improves one criterion without worsening another. This extends the idea of Pareto efficiency to general optimization models and typically yields a collection of trade-off solutions rather than one universally best choice. (direct.mit.edu)
A common scalarization minimizes a weighted sum,
[ F(x)=\sum_{k=1}^{m}w_k f_k(x). ]
The weights and the scales of the component objectives affect the selected trade-off. Weighted sums are useful but can fail to recover some Pareto-optimal solutions in nonconvex problems. Alternative formulations optimize one criterion while placing bounds on the others, thereby representing different preferences through constraints. (ocw.mit.edu)