Empirical risk minimization (ERM) is a principle in machine learning for choosing a model that minimizes average loss on a sample of observations. It replaces an unknown population-level objective with a quantity computable from data. Most commonly formulated for supervised learning, ERM provides a framework for understanding both familiar estimation methods and the conditions under which fitting observed examples leads to reliable predictions on unseen examples. It specifies a learning objective, rather than a particular computational algorithm. (cs.princeton.edu)
Mathematical formulation
Let the training data be (S=((x_1,y_1),\ldots,(x_n,y_n))), where (x_i) is an input and (y_i) its associated target. Choose a hypothesis class (\mathcal H) of candidate predictors and a loss function (\ell(h(x),y)). The standard statistical formulation assumes that examples are drawn independently from the same joint probability distribution (P). The population risk is the expected value
[ R(h)=\mathbb E_{(X,Y)\sim P}[\ell(h(X),Y)]. ]
Because (P) is generally unknown, this expectation cannot ordinarily be evaluated directly. The empirical risk is instead
[ \widehat R_S(h)=\frac1n\sum_{i=1}^{n}\ell(h(x_i),y_i), ]
and an empirical risk minimizer satisfies
[ \widehat h_S\in\operatorname*{arg,min}_{h\in\mathcal H} \widehat R_S(h). ]
There may be several minimizers, and a minimum need not be attained. An approximate ERM rule returns a predictor whose empirical risk is within a specified tolerance of the infimum. The choice of (\mathcal H), loss, and tie-breaking rule therefore helps define the resulting learning procedure. (cs.huji.ac.il)
Examples and connections
In linear regression, minimizing the mean squared error over affine predictors gives ordinary least squares. For a predictor (h_{w,b}(x)=w^\top x+b), the objective is
[ \frac1n\sum_{i=1}^{n}(w^\top x_i+b-y_i)^2. ]
Multiplying this objective by a positive constant does not change its minimizers. (cs229.stanford.edu)
For binary classification, zero–one loss equals one when a predicted label is incorrect and zero otherwise. Its empirical risk is the fraction of misclassified training examples. Practical methods often minimize a surrogate loss instead: logistic regression uses binary cross-entropy, while a soft-margin support vector machine combines hinge loss with a complexity penalty. Minimizing a surrogate is not identical to directly minimizing the misclassification rate. (cs.princeton.edu)
ERM also connects to maximum likelihood estimation. For independent observations and a conditional probabilistic model (p_\theta(y\mid x)), choosing loss (-\log p_\theta(y\mid x)) makes empirical-risk minimization equivalent to maximizing the conditional likelihood. This equivalence depends on the chosen probabilistic model and loss; not every ERM objective is a likelihood objective. (cs229.stanford.edu)
Generalization and statistical guarantees
Small training loss alone does not ensure small population risk. A sufficiently flexible model may fit accidental sample details or noise, producing overfitting. The central statistical question is therefore generalization: whether the sample-selected predictor performs well on new observations from the relevant population. (cs.huji.ac.il)
For a fixed predictor with integrable loss, the law of large numbers supports convergence of empirical risk to population risk. However, ERM selects its predictor after inspecting the sample. Pointwise convergence for each fixed predictor does not by itself control this data-dependent choice. A standard sufficient condition is uniform convergence over the entire hypothesis class. (stat.cmu.edu)
If
[ \sup_{h\in\mathcal H}|R(h)-\widehat R_S(h)|\leq\varepsilon, ]
then an exact empirical minimizer satisfies
[ R(\widehat h_S)-\inf_{h\in\mathcal H}R(h)\leq2\varepsilon. ]
An empirical optimization tolerance (\eta) adds (\eta) to this bound. Thus, controlling the largest empirical–population discrepancy controls excess risk relative to the best predictor in the chosen class. It does not establish optimality among all possible predictors. (stat.berkeley.edu)
For finite classes and bounded losses, concentration inequalities yield high-probability bounds involving sample size and the logarithm of class size. Infinite classes require more refined complexity measures, including VC dimension in binary classification and Rademacher complexity for loss classes. Algorithmic stability provides another route to generalization analysis by examining sensitivity to changes in training examples. (cs.princeton.edu)
Regularization and model selection
Regularization modifies the objective to discourage selected forms of model complexity:
[ \widehat h_\lambda\in \operatorname*{arg,min}_{h\in\mathcal H} \left[\widehat R_S(h)+\lambda\Omega(h)\right]. ]
Here (\Omega) is a penalty and (\lambda\geq0) controls its strength. This is regularized ERM, distinct from minimizing empirical loss alone. Constraints on the hypothesis class offer a related way to limit flexibility. Such restrictions may reduce estimation error while excluding predictors with lower achievable population risk. (stat.cmu.edu)
A validation set or cross-validation can support comparisons between candidate classes and penalty strengths. A separate test set evaluates the selected procedure on observations not used in fitting or selection. These evaluation mechanisms complement ERM rather than replace its training objective. (cs.huji.ac.il)
Computation and scope
Solving an ERM problem is a task in mathematical optimization. Some objectives admit closed-form solutions; others use iterative methods such as stochastic gradient descent. Statistical guarantees for an exact minimizer do not automatically establish that a practical optimizer finds one. Approximate optimization introduces an additional error component. (cs229.stanford.edu)
ERM targets average loss under the distribution represented by its training sample. Under distribution shift, that distribution differs from the deployment distribution, so low training-distribution risk need not imply low deployment risk. Weighted empirical objectives can address certain specified shifts when appropriate distributional assumptions hold, but ordinary ERM supplies no general guarantee against arbitrary changes in the data-generating process. (classic.d2l.ai)