Standard Error · aiwiki.page

In statistics, the standard error (SE) of a statistic is the standard deviation of its sampling distribution: the distribution of values that would arise from repeatedly drawing samples under the same sampling procedure. It describes the variability of an estimate rather than the variability of individual observations. Because the underlying population distribution is usually unknown, reported standard errors are generally estimates calculated from sample data. (online.stat.psu.edu)

Definition and interpretation

Let $T$ be a statistic used as an estimator of a population parameter $\theta$ . Treating $T$ as a random variable, its standard error is

\operatorname{SE}(T) =\sqrt{\operatorname{Var}(T)} =\sqrt{\mathbb E[(T-\mathbb E[T])^2]},

where $\operatorname{Var}$ denotes variance and $\mathbb E$ denotes expected value. The standard error has the same units as the statistic. A smaller value indicates less variation across repeated samples from the specified population and design. (stat.berkeley.edu)

Standard error measures precision, not necessarily accuracy. An estimator may have a small standard error while remaining systematically displaced from its target. For a fixed parameter, mean squared error separates these components:

\operatorname{MSE}(T) =\operatorname{SE}(T)^2+ \operatorname{Bias}(T)^2.

Thus, estimator bias is not included in the standard error. Sampling precision alone does not establish that an estimate is close to the intended population value. (stat.berkeley.edu)

Standard error of the mean

For $n$ observations $X_1,\ldots,X_n$ with statistical independence, a common population mean $\mu$ , and finite variance $\sigma^2$ , the sample mean satisfies

\bar X=\frac1n\sum_{i=1}^{n}X_i, \qquad \operatorname{SE}(\bar X)=\frac{\sigma}{\sqrt n}.

This variance identity does not require a normal distribution. Normality becomes relevant when specifying the exact distribution of the mean or constructing particular inferential procedures. (online.stat.psu.edu)

When $\sigma$ is unknown, the conventional estimate is

\widehat{\operatorname{SE}}(\bar X)=\frac{s}{\sqrt n}, \qquad s^2=\frac{\sum_{i=1}^{n}(X_i-\bar X)^2}{n-1}.

The denominator $n-1$ incorporates Bessel’s correction. Although $s^2$ is unbiased for the population variance under independent, identically distributed sampling, taking its square root does not generally preserve unbiasedness. The estimated standard error is itself subject to sampling variation. (stat.berkeley.edu)

For illustration, a sample of 100 observations with $s=20$ has an estimated standard error of $20/\sqrt{100}=2$ . With otherwise comparable observations, quadrupling the sample size halves the theoretical standard error. The standard deviation of individual observations need not decrease as the sample grows. (online.stat.psu.edu)

Proportions and regression coefficients

For independent Bernoulli observations with success probability $p$ , the sample proportion $\hat p$ has

\operatorname{SE}(\hat p) =\sqrt{\frac{p(1-p)}{n}}.

A common plug-in estimate replaces $p$ with $\hat p$ . This follows from the binomial distribution of the number of successes and the fact that a proportion is a mean of zero–one observations. (online.stat.psu.edu)

In linear regression, each estimated coefficient has its own standard error. For ordinary least squares with fixed, full-rank design matrix $X$ , uncorrelated errors, and common error variance $\sigma^2$ ,

\operatorname{Var}(\hat\beta\mid X) =\sigma^2(X^\mathsf TX)^{-1}.

Coefficient standard errors are the square roots of the diagonal entries of this covariance matrix, usually after estimating $\sigma^2$ from residuals. They concern coefficient uncertainty, not the spread of residuals or uncertainty in an individual future outcome. (stata.com)

Confidence intervals and hypothesis tests

Standard errors provide the scale for many confidence intervals. A common approximate form is an estimate plus or minus a critical multiplier times its estimated standard error. For a normally distributed population with unknown variance, the exact interval for the mean is

\bar X\pm t_{1-\alpha/2,n-1}\frac{s}{\sqrt n},

using Student’s t-distribution with $n-1$ degrees of freedom. A confidence level describes the long-run coverage of the interval-producing procedure, not a probability assigned to a fixed parameter after observing the data. (itl.nist.gov)

The central limit theorem supports normal approximations for means and proportions under suitable conditions. In statistical hypothesis testing, a difference between an estimate and a hypothesized value is often divided by a standard error to form a test statistic. For a proportion test, the denominator commonly uses the null-hypothesis probability rather than the observed proportion. (online.stat.psu.edu)

Dependence and alternative estimation methods

Standard errors depend on the sampling design and assumptions about dependence. The independent-observation formula can be inappropriate for clustered observations or time series. Heteroskedasticity-robust, cluster-robust, and autocorrelation-robust estimators accommodate different departures from conventional regression assumptions; none is a universal correction for every dependence structure. (stata.com)

When an analytical formula is unavailable, bootstrap sampling can estimate a standard error by repeatedly resampling the observed data and recomputing the statistic. If the resulting estimates are $T_1^*,\ldots,T_B^*$ , their sample standard deviation estimates the statistic’s sampling variability:

\widehat{\operatorname{SE}}_{\mathrm{boot}} =\sqrt{\frac{1}{B-1} \sum_{b=1}^{B}(T_b^*-\bar T^*)^2}.

This standard deviation is not divided by $\sqrt B$ : the replicates approximate the distribution of the statistic itself, rather than forming a new sample whose mean is the target. (docs.scipy.org)