Bias of an Estimator · aiwiki.page

In statistics, the bias of an estimator is the difference between the estimator’s expected value and the true value of the quantity being estimated. It measures systematic overestimation or underestimation across repeated samples, rather than the error in one particular result. Bias is a property of an estimation procedure under a specified statistical model or sampling design; an unbiased procedure need not produce an accurate estimate in every sample. (stat.berkeley.edu)

Definition and interpretation

Let $X=(X_1,\ldots,X_n)$ denote data drawn from a probability distribution indexed by an unknown parameter $\theta$ . An estimator $T(X)$ is a function of the data used to estimate $\theta$ . Before the data are observed, it is a random variable with a sampling distribution. Its bias is

b_T(\theta)=\operatorname{Bias}_\theta(T) =\mathbb E_\theta[T(X)]-\theta,

provided the expectation exists. The subscript indicates that the expectation is calculated under the distribution corresponding to $\theta$ . An estimator is unbiased if $b_T(\theta)=0$ for every parameter value in the specified model. Being unbiased at one particular value does not establish unbiasedness throughout the model. (statlect.com)

Positive bias means that the estimator is too large on average; negative bias means that it is too small. The realized estimation error admits the decomposition

T-\theta=b_T(\theta)+(T-\mathbb E_\theta T).

The second term has expectation zero and represents random sampling fluctuation. Thus, unbiasedness concerns the center of the sampling distribution, not its spread or the closeness of any individual estimate to the target. (stat.berkeley.edu)

Elementary examples

Suppose $X_1,\ldots,X_n$ are identically distributed observations with finite mean $\mu$ . By linearity of expectation, the sample mean

\bar X=\frac1n\sum_{i=1}^n X_i

satisfies $\mathbb E[\bar X]=\mu$ , so it is unbiased for $\mu$ . If the observations also have statistical independence and common finite variance $\sigma^2$ , then $\operatorname{Var}(\bar X)=\sigma^2/n$ . This distinguishes unbiasedness from precision: the expectation identifies the target, whereas the variance describes sampling variability. (homepages.ecs.vuw.ac.nz)

A standard biased estimator is the sample variance calculated with denominator $n$ . For independent, identically distributed observations with finite variance and $n>1$ , define

V_n=\frac1n\sum_{i=1}^n(X_i-\bar X)^2.

Then

\mathbb E[V_n]=\frac{n-1}{n}\sigma^2, \qquad \operatorname{Bias}(V_n)=-\frac{\sigma^2}{n}.

Replacing $n$ by $n-1$ , a modification called Bessel’s correction, produces an unbiased estimator of $\sigma^2$ . Under a normal distribution with unknown mean, $V_n$ is nevertheless the maximum-likelihood estimator of the variance. Maximum likelihood and unbiasedness therefore need not select the same estimator. (stats-web.quarto.pub)

Bias, variance, and estimation loss

Bias alone does not determine an estimator’s quality. Under squared-error loss, the mean squared error is

\operatorname{MSE}_\theta(T) =\mathbb E_\theta[(T-\theta)^2] =\operatorname{Var}_\theta(T)+b_T(\theta)^2.

This identity holds when the relevant second moments are finite. It follows by expanding the error around $\mathbb E_\theta T$ ; the cross term vanishes because the centered estimator has expectation zero. For an unbiased estimator, MSE equals variance. A biased estimator can have lower MSE if its reduction in variance outweighs its squared bias. (bookdown.org)

For example, let $K$ follow a binomial distribution with parameters $n$ and $p$ . The sample proportion $K/n$ is unbiased. The smoothed estimator

\tilde p=\frac{K+1}{n+2}

has, by direct calculation,

\operatorname{Bias}_p(\tilde p)=\frac{1-2p}{n+2}, \qquad \operatorname{Var}_p(\tilde p)=\frac{np(1-p)}{(n+2)^2}.

It pulls estimates toward $1/2$ and reduces variance, but its MSE advantage depends on $p$ . At $p=1/2$ , its bias vanishes and its variance is smaller than that of $K/n$ . (stat135.berkeley.edu)

This bias–variance tradeoff also appears in machine learning. In linear regression, ridge regression uses regularization to shrink coefficient estimates. Such shrinkage generally introduces bias but can reduce variability and improve prediction relative to an unpenalized fit. The objective is not necessarily exact unbiasedness, but lower expected prediction error under the chosen loss. (stat.cmu.edu)

Finite-sample bias and consistency

For a sequence $T_n$ , asymptotic unbiasedness means that $\mathbb E_\theta[T_n]-\theta$ approaches zero as sample size increases. The variance estimator $V_n$ above has this property despite being biased at every finite $n$ when $\sigma^2>0$ . Consistency, by contrast, means convergence in probability to the target:

\Pr_\theta(|T_n-\theta|>\varepsilon)\longrightarrow0 \quad\text{for every }\varepsilon>0.

It describes concentration near the parameter, not merely convergence of the expectation. (stats-web.quarto.pub)

Unbiasedness does not imply consistency: an estimator that uses only the first observation can remain unbiased for the population mean while retaining the same sampling distribution as more observations become available. A useful sufficient condition for consistency is that both bias and variance approach zero. Then MSE approaches zero, and Chebyshev’s inequality applied to squared estimation error establishes convergence in probability. (mcrovella.github.io)

Estimating and correcting bias

When an expectation is known analytically, bias may be removable exactly. For example, if $\mathbb E_\theta[T]=c\theta$ for a known nonzero constant $c$ , then $T/c$ is unbiased. Such corrections depend on the model and need not minimize MSE. (stat135.berkeley.edu)

Bootstrap sampling can approximate bias when an exact calculation is unavailable. If $T^{*}$ denotes the estimator computed from bootstrap data, the conditional difference $\mathbb E_*[T^{*}]-T$ serves as a bootstrap bias estimate. Subtracting it gives $2T-\mathbb E_*[T^{*}]$ . Its reliability depends on how well the bootstrap distribution approximates the actual sampling distribution; estimating a correction does not guarantee exact unbiasedness. (stat.cmu.edu)