Panel data are observations on multiple units—such as individuals, households, firms, schools, or countries—recorded at more than one point in time. They combine a cross-sectional dimension, which distinguishes units, with a temporal dimension, which follows those units through successive observations. A dataset containing annual employment records for the same firms, for example, is a panel; separate annual surveys of unrelated firms are repeated cross sections rather than an individual-firm panel.
The distinctive value of panel data is that they reveal both differences between units and changes within units. Repeated observations can help separate persistent characteristics from changing circumstances, distinguish adjustment processes from static associations, and investigate the timing of events. These advantages have made panels central to econometrics, social research, administrative statistics, and many other fields. However, repeated observations are not independent replications: persistent unobserved differences, dependence over time, missing observations, and feedback between variables require explicit treatment.
Panel analysis therefore involves more than applying an ordinary regression to a larger dataset. Its methods depend on the source of identifying variation, assumptions about unobserved characteristics, the relative numbers of units and periods, and the purpose of the analysis. Description, prediction, and causal inference can require different models even when they use the same records.
Data structure, study designs, and historical development
A conventional panel records an outcome and a vector of explanatory variables , where identifies units and identifies observation periods. A balanced panel observes every unit in every period, yielding unit-period records. An unbalanced panel contains different numbers of observations across units. Unbalancedness is a structural description, not by itself evidence of defective data: firms may enter a market, countries may begin reporting at different dates, or a survey may deliberately introduce new respondents.
A panel differs from a single time series because it contains multiple units whose trajectories can be compared. It differs from repeated cross sections because unit identity persists across observations. Repeated cross sections can nevertheless form a panel of stable aggregates, such as region-year averages. A pseudo-panel groups successive cross-sectional respondents into cohorts defined by relatively stable characteristics, such as birth decade. Cohort averages are then followed over time, but their changing sample composition introduces measurement and sampling issues absent from a genuine individual panel.
Panels are often described as short when is large relative to , and long when many periods are available for each unit. A household survey with 10,000 households observed over five waves exemplifies a short panel; annual observations for 20 countries over 60 years have a substantial time dimension. These distinctions matter because the mathematical justification of an estimator may rely on the number of units increasing while the number of periods remains fixed, or instead on a long time horizon. An estimator appropriate under one limiting framework need not perform well under another.
Institutional developments helped make panel analysis a major research field. The University of Michigan’s Panel Study of Income Dynamics began in 1968 and follows individuals and family units over time. The German Socio-Economic Panel began in 1984, while the British Household Panel Survey began in 1991. Such studies made it possible to examine income mobility, employment transitions, and household change rather than only comparing population snapshots. Administrative tax, employment, business, and educational records subsequently expanded the scale and frequency of available panels.
The statistical foundations developed through several related traditions. Error-components models formalized dependence arising from persistent unit characteristics, while econometric work emphasized the possibility that those characteristics correlate with observed regressors. Important contributions include Yair Mundlak’s 1978 treatment of correlated individual effects, Stephen Nickell’s 1981 analysis of dynamic-panel bias, and Manuel Arellano and Stephen Bond’s 1991 dynamic-panel estimator. These developments established that the observation pattern, substantive question, and assumptions about unobserved variables jointly determine how a panel can be analyzed.
Linear models and the sources of identification
A standard linear panel model is
where is an intercept, is a coefficient vector, is a time-invariant unit effect, is a period effect common to all units, and is an idiosyncratic disturbance. The unit effect may represent persistent managerial quality, geographic characteristics, or enduring household preferences. Period effects can represent common changes in prices, regulations, or aggregate economic conditions. The additive structure is an assumption: it does not encompass every way in which units can respond differently to common events.
Panel variation has two basic components. Between-unit variation concerns differences in unit averages, such as whether firms with higher average capital expenditure also have higher average output. Within-unit variation concerns departures from those averages, such as whether a firm’s output rises in years when its capital expenditure is unusually high. These relationships need not coincide. High-productivity firms might consistently invest more, while short-run investment increases within a firm could have delayed rather than immediate effects on output.
Applying pooled ordinary least squares without accounting for can create endogeneity if persistent omitted characteristics correlate with regressors. For example, a regression relating training expenditure to productivity may confound training with durable differences in management. The resulting coefficient mixes within-firm changes and between-firm differences. A panel makes these sources of variation distinguishable, but does not automatically determine which one identifies the relationship of interest.
A frequently used assumption in static models is strict exogeneity:
Here conditional expectation expresses the requirement that the disturbance in any period have zero conditional mean given the complete observed regressor history and the unit effect. This is stronger than requiring only contemporaneous noncorrelation. It rules out, for example, a current unexpected productivity shock influencing a firm’s future investment, unless that feedback is represented elsewhere in the model.
Identification also requires sufficient independent variation in the regressors after the relevant effects are removed. A characteristic that never changes within a unit cannot identify a coefficient through within-unit comparisons. Likewise, a national interest rate observed by every domestic firm in a given year cannot be separated from unrestricted year effects in a single-country firm panel. These are problems of statistical identification and matrix rank, not shortcomings that a larger number of observations necessarily resolves.
Finally, a common slope vector assumes that the same marginal relationship applies across units and periods. When effects differ, a pooled coefficient can summarize a particular weighted combination rather than a universal parameter. Its interpretation consequently depends on both the model and the distribution of variation in the data.
Fixed effects and first differences
The fixed-effects model treats unit-specific intercepts as unrestricted nuisance parameters and allows them to correlate with the observed regressors. Its central operation is to remove persistent additive heterogeneity before estimating slopes. In the one-way model without period effects, define , and similarly define and over the observed periods for unit . Subtracting unit means gives
The unit effect disappears because it is constant across that unit’s observations. Ordinary least squares on these transformed variables produces the within estimator.
An equivalent approach includes a separate indicator variable for each unit, subject to an appropriate intercept normalization. This least-squares dummy-variable formulation makes the model easy to interpret, although explicitly constructing millions of indicators can be computationally inefficient. Modern implementations often absorb fixed effects through demeaning or iterative projection. For a balanced panel with both unit and period effects, the transformation
removes both sets of additive effects. Simple double demeaning is not generally sufficient for an arbitrarily unbalanced panel; the projection must account for the actual observation pattern.
First differencing offers another way to eliminate a time-invariant unit effect:
where . With two periods, first differences and the within estimator yield the same slope estimates for the corresponding model. With more periods, they generally differ in efficiency because they transform the disturbance differently. If level disturbances are serially uncorrelated, the differenced disturbance has negative first-order serial correlation. Conversely, differencing can be attractive when disturbances in levels are highly persistent.
Fixed effects remove only the particular heterogeneity represented by the model. Unit effects absorb characteristics that remain constant, while period effects absorb shocks common to all units with the same additive impact. Neither automatically controls for changing management quality, unit-specific exposure to common shocks, or simultaneous determination of outcomes and regressors. Adding unit-specific linear trends permits different baseline trajectories but also changes the identifying variation: coefficients then depend on deviations from those fitted trends.
The within transformation has substantive costs. Time-invariant regressors disappear, and slowly changing variables may retain little usable variation. Measurement error can become relatively more important after persistent signal has been removed. A fixed-effects coefficient on job training, for example, compares an individual’s outcomes across periods with different training exposure; it does not directly measure why consistently trained individuals differ from consistently untrained individuals.
Under standard short-panel conditions, estimating many unit intercepts does not by itself make the slope estimator inconsistent in a static linear model. That result does not extend automatically to dynamic or nonlinear specifications. The phrase “fixed effects” therefore describes a treatment of heterogeneity, not a general guarantee of unbiased estimation or causal interpretation.
Random effects and correlated-effects models
A random-effects model represents unit heterogeneity through a probability distribution rather than estimating an unrestricted intercept for every unit. In the conventional linear specification, has mean zero and variance , while has variance . If the idiosyncratic disturbances are uncorrelated over time, the composite disturbance satisfies
Observations from the same unit are correlated because they share the same persistent component. This error-components structure provides a direct statistical account of within-unit dependence.
The conventional random-effects estimator requires the unit effect to be uncorrelated with the regressors, together with suitable assumptions about the idiosyncratic disturbance. Under the standard model, generalized least squares exploits the resulting covariance matrix to combine within-unit and between-unit information efficiently. For a balanced panel, its transformation can be written
with
When the unit component is negligible, approaches zero and the transformation resembles pooled regression. As the unit component becomes dominant, approaches one and the transformation approaches within demeaning.
The contrast between fixed and random effects is often misunderstood as depending solely on whether the sampled units are literally random. In econometric practice, the crucial distinction concerns assumptions about the relationship between unobserved unit heterogeneity and the regressors. A random sample of households does not establish that persistent household preferences are uncorrelated with income or education. Conversely, a finite collection of named institutions can still be modeled hierarchically when that model suits the inferential purpose.
Correlated random-effects models relax the conventional orthogonality restriction. Mundlak’s formulation represents the unit effect using unit-level regressor averages:
where the residual component is assumed appropriately unrelated to the regressor history. Including separates within-unit changes from systematic between-unit differences. In the standard balanced linear setting, with the relevant means included and a common specification, the coefficients on time-varying regressors reproduce the fixed-effects within slopes. Related formulations use more detailed functions of the regressor history.
The Hausman test, introduced in Jerry Hausman’s 1978 specification-testing work, compares estimators that should agree under a maintained null model but differ when its orthogonality restrictions fail. Its conventional fixed-versus-random-effects form contrasts the two slope estimates using an estimated covariance of their difference. The classical calculation depends on particular efficiency and covariance assumptions; robust alternatives require corresponding adjustments.
Random-effects models also support prediction of unit-specific outcomes through partial pooling. Units with limited information receive estimates drawn more strongly toward the model’s population prediction. This can improve prediction, but it does not independently validate the exogeneity assumptions needed to interpret a slope as a structural or causal effect.
Dependence, uncertainty, and statistical inference
Repeated observations change how uncertainty must be calculated. A panel containing 1,000 firms over ten years has 10,000 records, but these records usually do not provide as much independent information as 10,000 unrelated firms observed once. Persistent productivity, common management practices, and serially correlated shocks make observations from the same firm dependent. Conventional regression formulas that treat all disturbances as independent can therefore substantially understate standard errors, even when the coefficient estimator remains consistent.
Cluster-robust standard errors allow disturbances to have arbitrary covariance within specified clusters, subject to assumptions that support averaging across clusters. If firms form independent clusters, a unit-clustered covariance estimator has the sandwich form
with suitable finite-sample corrections and with interpreted as the design matrix actually used in estimation, including any relevant transformation. Unlike a simple heteroskedasticity-robust estimator, the middle term retains cross-products of residuals from different periods within the same unit.
The appropriate clustering level depends on the dependence and assignment structure, not simply on the dataset’s row identifiers. If a policy is assigned at the regional level, firms in the same region may share both treatment and unobserved shocks. Firm-level clustering alone can then miss important dependence across firms. Two-way clustering can accommodate dependence within units over time and within periods across units, under suitable conditions. It is not a universal correction for every spatial or network pattern.
The number of independent clusters is often more important for inference than the number of records. With few clusters, conventional large-sample approximations may be inaccurate, especially when cluster sizes or treatment exposure are highly uneven. Degrees-of-freedom adjustments, specialized small-sample corrections, and wild cluster bootstrap procedures address different aspects of this problem. No correction creates independent policy variation when treatment changes in only a handful of groups.
Cross-sectional dependence creates a separate challenge. Country outcomes may respond to global financial conditions; neighboring regions may share weather; firms may be linked through suppliers. Period indicators remove common additive shocks but not necessarily shocks with different unit-specific loadings. Depending on the sampling framework and time dimension, inference may require explicit common-factor models, spatial dependence structures, or covariance estimators that exploit a long temporal record.
Uncertainty also concerns what population or process the analysis represents. A panel of every municipality in a country is a census of those municipalities, but a regression can still involve uncertainty about disturbances, policy assignment, future periods, or a broader underlying process. Confidence intervals must be interpreted relative to that inferential framework. A narrow interval obtained under an implausible independence assumption is not evidence that the substantive relationship is precisely known.
Dynamic panels and endogenous regressors
Many panel outcomes depend on their own history. Employment adjusts gradually, firms accumulate capital, and household income may remain above or below its long-run level for several periods. A simple dynamic specification is
The coefficient represents persistence conditional on the included variables and unit effect. Under a stable common-coefficient model with , a permanent change in a regressor has a long-run multiplier . This interpretation additionally assumes that the remaining model structure and regressor process support the contemplated permanent change.
The lagged dependent variable creates a problem for the within estimator. After demeaning, is generally correlated with the transformed disturbance because the lagged outcomes contain past disturbances that also contribute to the disturbance mean. Nickell’s 1981 analysis established the resulting short-panel bias in standard settings. Increasing the number of units while holding the number of periods fixed does not eliminate it; its leading magnitude in familiar cases is of order .
First differencing removes , but it does not remove the endogeneity:
The regressor contains , while , so they are correlated. Under suitable restrictions on the serial dependence of the original disturbances, deeper outcome lags can serve as instrumental variables. A typical moment restriction is
Such restrictions are assumptions about the data-generating process, not consequences of differencing alone.
Arellano and Bond’s 1991 difference estimator combines these restrictions through the generalized method of moments. Regressors can be classified as strictly exogenous, predetermined, or endogenous, with different admissible instruments for each category. A predetermined variable may respond to earlier shocks but not the current shock; an endogenous variable may correlate with the current disturbance and therefore require deeper lags or external instruments.
When outcomes are highly persistent, lagged levels can be weak instruments for changes. System GMM, associated with work by Arellano and Olympia Bover in 1995 and Richard Blundell and Stephen Bond in 1998, combines differenced equations with level equations instrumented by lagged differences. The additional moment restrictions require assumptions about initial conditions and the relationship between changes and unit effects.
Diagnostics examine serial correlation and overidentifying restrictions, but passing a test does not establish instrument validity. First-order correlation in differenced errors is expected when level disturbances are serially uncorrelated; second-order correlation is more directly relevant to the usual lag-instrument restrictions. Instrument counts can also grow rapidly with the number of periods, overfitting endogenous variables and weakening specification tests. Dynamic-panel estimates therefore depend critically on the particular instruments, lag ranges, and moment restrictions used.
Policy evaluation and causal interpretation
Panel data support causal analysis by documenting outcomes before and after an intervention for both exposed and comparison units. Their principal advantage is temporal and comparative: researchers can observe whether groups already differed, examine the timing of changes, and remove certain persistent differences. These features do not make observational panels equivalent to randomized experiments. A policy may be introduced precisely because outcomes are deteriorating, and individuals may change behavior in anticipation of treatment.
A basic difference-in-differences design compares the outcome change in a treated group with the change in an untreated group:
Here and label treated and comparison groups rather than time periods. In a two-period setting, the estimator can be obtained from a regression with group effects, a period effect, and a treatment-by-post-period indicator. Panel observations allow the same units to be followed, although difference-in-differences can also be implemented with repeated cross sections under suitable conditions.
The central identifying restriction is parallel trends: absent treatment, the treated group’s average outcome would have changed like the comparison group’s outcome. Fixed effects remove baseline level differences but do not establish parallel counterfactual changes. Other relevant assumptions concern anticipation, spillovers, and composition. If a regional subsidy moves employment from untreated neighboring regions into treated regions, the comparison group may itself be affected, changing the interpretation of the estimate.
Event-study specifications replace a single treatment indicator with indicators for time relative to treatment. They describe the timing of estimated changes and can reveal differences before implementation. However, statistically insignificant pre-treatment coefficients do not prove parallel trends; tests may have low power, and some violations arise only near implementation. Conversely, an apparent pre-treatment change can reflect anticipation or the construction of the estimator rather than a simple failure of the substantive design.
Staggered adoption introduces additional complications when units begin treatment in different periods. A conventional two-way fixed-effects regression may use already-treated units as comparisons for later-treated units. If treatment effects differ across cohorts or evolve with exposure duration, the resulting coefficient need not equal a straightforward average treatment effect and can contain negative weights. Cohort-and-period estimators instead define particular treatment effects and compare treated cohorts with never-treated or not-yet-treated units under explicit assumptions.
Causal interpretation also depends on the estimand. The effect of initially adopting a policy, the effect after five years of exposure, and the effect on units that remain observed are different quantities. Changes in treatment intensity, reversals, and migration can make a single binary indicator inadequate. Panels help describe these processes, but causal identification still comes from an assignment mechanism or a defensible counterfactual argument, not from repeated measurement alone.
Nonlinear models, missing observations, and data management
Not all panel outcomes fit a linear conditional-mean model. Employment status is binary, patent counts are nonnegative integers, and time until an event requires a duration model. Panel extensions of logistic regression, count regression, and other generalized linear models represent these outcomes more directly. However, methods that eliminate additive effects in linear regression do not generally eliminate unit heterogeneity in nonlinear models, and direct estimation of one intercept per unit can produce an incidental-parameters problem when the number of periods remains small.
Conditional fixed-effects logit provides an important special case. Under its maintained logistic specification and conditional independence assumptions, conditioning on a unit’s total number of successes removes the unit intercept from the conditional likelihood. Units whose observed outcomes are always zero or always one provide no information for that conditional slope likelihood. Fixed-effects Poisson estimation offers another useful case: slope consistency can rely on a correctly specified conditional mean without requiring equality of the conditional mean and variance, although inference must address dispersion and dependence.
Random-effects nonlinear models integrate over a specified distribution of unit heterogeneity, often using maximum likelihood or Bayesian methods. Their results can depend on distributional assumptions and on whether the heterogeneity is allowed to correlate with covariates. In nonlinear models, conditional coefficients and population-averaged effects are generally different quantities. A coefficient describing changes conditional on a latent unit effect cannot automatically be read as the marginal change in the population’s average outcome probability.
Missing observations create problems distinct from model choice. Panel attrition occurs when units cease participating or otherwise disappear from observation. If low-income households are more likely to leave a survey, estimates based only on continuing respondents may misrepresent income trajectories. Yet disappearance need not always be a missing-data event: firm closure may be an outcome of substantive interest, and death or retirement may define the end of an applicable observation period. The meaning of an absent record must therefore be established before selecting a statistical treatment.
Weighting by estimated observation probabilities and imputation can address missingness under specified assumptions. Neither reconstructs information without restrictions. Identification becomes especially difficult when observation depends on unobserved outcomes even after conditioning on available history. Refreshment samples can provide information about later population composition, but do not automatically recover every missing individual trajectory.
Reliable construction is equally important. Unit identifiers must remain stable through firm mergers, household splits, institutional reorganization, and changes in reporting systems. Dates must distinguish reference periods from publication dates, and variables must retain comparable definitions. Long-format storage usually places one unit-period observation on each row; checks for duplicate identifiers, irregular gaps, and inconsistent units prevent many analytical errors.
Finally, predictive evaluation must respect the intended use. Randomly splitting records can place the same units in training and test sets or expose the model to later observations when predicting earlier ones. Time-based evaluation assesses forecasting, whereas holding out entire units assesses generalization to previously unseen units. These designs test different capabilities and can yield substantially different assessments of the same panel model.