{
  "id": "quasi-experiment",
  "title": "Quasi-experiment",
  "lang": "en",
  "source_lang": "en",
  "summary": "A quasi-experiment estimates causal effects without randomized assignment by using a design that supports a credible comparison with an untreated counterfactual.",
  "category": "mathematics",
  "subcategory": "probability-statistics",
  "type": "concept",
  "aliases": [
    "Quasi-experimental study",
    "quasi-experimental design"
  ],
  "generated_at": "2026-11-04",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/quasi-experiment/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/quasi-experiment/",
      "json": "https://aiwiki.page/zh/quasi-experiment.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "A quasi-experiment is an empirical study that seeks to estimate the causal effect of an intervention, exposure, or policy without assigning units to treatment conditions through investigator-controlled randomization. Instead, it uses features of treatment allocation, institutional rules, timing, or comparison groups to approximate the counterfactual comparison that a randomized experiment would provide. Quasi-experiments are central to causal inference when random assignment is impractical, unavailable, or incompatible with how an intervention is implemented.\n\nThe defining challenge is not simply the absence of randomization but the possibility that treated and untreated units would have experienced different outcomes even without treatment. Credible quasi-experimental designs address that problem through explicit assumptions and evidence about the assignment process. Major approaches include interrupted time series, difference-in-differences, regression discontinuity, and instrumental-variable designs. Some classifications also include carefully designed matched comparison studies, although the boundary between quasi-experimental and other observational research varies across disciplines."
    },
    {
      "heading": "Origins, scope, and relationship to experimentation",
      "text": "Quasi-experimental reasoning predates the term itself. Investigators have long used administrative boundaries, institutional changes, and externally imposed events to construct comparisons that resemble experiments. John Snow’s investigation of London cholera in 1854, for example, compared mortality among households supplied by different water companies. Because their customers were intermingled geographically but received water from differently contaminated sources, the comparison offered stronger evidence than a simple association between neighborhood conditions and disease. Its credibility depended on the circumstances of water supply rather than on investigator assignment.\n\nThe terminology became influential through Donald T. Campbell and Julian C. Stanley’s Experimental and Quasi-Experimental Designs for Research, first published in 1963. Their framework distinguished randomized experiments from designs lacking random assignment and organized threats to valid causal interpretation. Thomas D. Cook and Campbell expanded this program in Quasi-Experimentation: Design & Analysis Issues for Field Settings in 1979. William R. Shadish, Cook, and Campbell’s 2002 treatment further integrated experimental and quasi-experimental approaches to generalized causal inference.\n\nA quasi-experiment differs from a randomized controlled trial primarily in how treatment assignment is justified. In a randomized trial, assignment probabilities are known by design, and randomization makes treatment statistically independent of pretreatment characteristics in the assignment distribution. It does not guarantee identical groups in every realized sample. In a quasi-experiment, comparability instead follows from substantive knowledge, design restrictions, and assumptions about what outcomes would have been without treatment.\n\nA natural experiment is closely related but not identical. The expression usually emphasizes an external event or institutional process that generates exposure variation outside the researcher’s control. “Quasi-experiment” more directly describes the inferential design used to analyze such variation. A policy introduced at different dates may generate a natural experiment and be analyzed through several quasi-experimental methods. Conversely, an investigator may arrange a nonrandomized program evaluation that is quasi-experimental without arising from a naturally occurring event.\n\nThe category should not be equated with all nonrandomized research. A cross-sectional regression of income on educational attainment is observational, but it is not ordinarily considered a compelling quasi-experiment merely because it includes control variables. A quasi-experimental claim generally requires an identifiable comparison strategy and a reason that the relevant treatment variation can be separated from other determinants of the outcome. Disciplinary usage nevertheless differs: program evaluation often uses the term broadly, while applied econometrics commonly reserves it for designs with a particularly explicit source of plausibly exogenous variation."
    },
    {
      "heading": "Counterfactuals, assignment mechanisms, and causal estimands",
      "text": "The potential-outcomes framework makes the central problem precise. For unit \\(i\\), let \\(Y_i(1)\\) denote the outcome under treatment and \\(Y_i(0)\\) the outcome without treatment. If \\(D_i\\) indicates treatment receipt, the observed outcome is\n\n\\[Y_i=D_iY_i(1)+(1-D_i)Y_i(0).\\]\n\nThe individual treatment effect is \\(Y_i(1)-Y_i(0)\\), but only one of the two potential outcomes is observed. A design therefore constructs information about the missing counterfactual from other units, other periods, or an assignment rule. The resulting target may be an average effect across the population, an effect among treated units, or an effect for a narrower subgroup.\n\nFor example, the average treatment effect on treated units is\n\n\\[\\operatorname{ATT} = E[Y_i(1)-Y_i(0)\\mid D_i=1],\\]\n\nwhere \\(E[\\cdot]\\) denotes an expected value. A raw treated–untreated difference decomposes as\n\n\\[\\begin{aligned} E[Y_i\\mid D_i=1]-E[Y_i\\mid D_i=0] ={}&\\operatorname{ATT}\\\\ &+E[Y_i(0)\\mid D_i=1] -E[Y_i(0)\\mid D_i=0]. \\end{aligned}\\]\n\nThe second line is a baseline selection difference. It captures why observed outcome differences need not equal causal effects. Programs may deliberately enroll people with worse expected outcomes, while voluntary participation may attract people with greater motivation or resources.\n\nIdentification concerns whether the causal quantity can be recovered from the available data under specified assumptions. Estimation concerns how it is calculated from a finite sample. These are separate questions: a precisely estimated regression coefficient may fail to identify the intended effect. More observations can reduce sampling uncertainty without removing bias from an invalid comparison.\n\nDifferent quasi-experimental designs replace randomization with different identifying restrictions. Regression discontinuity uses continuity of untreated and treated potential outcomes around a threshold. Difference-in-differences uses restrictions on untreated changes over time. Instrumental variables use an excluded source of treatment variation. Matching approaches generally require that measured pretreatment variables account for the relevant treatment selection.\n\nThese assumptions concern an assignment mechanism, not just an equation. Knowledge of who implemented a rule, whether exceptions were allowed, and whether participants could anticipate treatment can be more important than a model’s goodness of fit. Directed acyclic graphs can clarify proposed causal relationships and distinguish confounders from mediators and colliders, but they do not establish that the depicted relationships are correct.\n\nThe treatment itself must also be sufficiently well defined. A nominally identical training program may differ in duration or content across sites. Effects can depend on these versions, on treatment timing, and on exposure among neighboring units. Consequently, an estimand must specify not only “treated versus untreated” but also the population, intervention, comparison condition, outcome, and follow-up period."
    },
    {
      "heading": "Nonequivalent comparison groups and baseline measurements",
      "text": "The nonequivalent comparison-group design compares an intervention group with a group selected without random assignment. Schools adopting a new curriculum, hospitals introducing a scheduling system, or municipalities implementing a service reform may be compared with institutions that continue existing practice. The design becomes more informative when both groups have pretreatment measurements, but baseline data alone do not guarantee a credible counterfactual.\n\nA post-treatment-only comparison is vulnerable to confounding whenever the determinants of treatment also affect the outcome. If better-resourced schools adopt a curriculum first, later achievement differences may reflect resources rather than the curriculum. Selecting schools with similar prior achievement, enrollment, and socioeconomic composition can improve comparability, but unmeasured differences in leadership or family involvement may remain.\n\nA common analysis adjusts the follow-up outcome for baseline outcome and other pretreatment covariates:\n\n\\[Y_{i,\\mathrm{post}} = \\alpha+\\tau D_i+\\beta Y_{i,\\mathrm{pre}} +\\gamma^\\top X_i+\\varepsilon_i.\\]\n\nThis linear regression describes an adjusted comparison. Interpreting \\(\\tau\\) causally requires assumptions about assignment conditional on the included variables and about the suitability of the model. An analysis of change scores instead compares \\(Y_{i,\\mathrm{post}}-Y_{i,\\mathrm{pre}}\\). The two approaches need not agree, particularly when baseline outcomes are measured with error or treatment selection depends on baseline status.\n\nRegression to the mean is especially important when interventions target extreme observations. Suppose a program enrolls schools after an unusually low annual test score. Even without an effective intervention, some schools will improve because temporary adverse influences dissipate. A comparison group selected using the same rule helps distinguish this statistical tendency from treatment effects. Multiple baseline measurements can show whether the low score represented a sustained condition or a temporary fluctuation.\n\nSeveral design additions strengthen nonequivalent comparisons. Repeated pretreatment outcomes reveal prior trajectories. A comparison outcome that the intervention should not affect can help detect broader changes or measurement artifacts. Groups facing similar eligibility rules or operational circumstances may be more credible than groups chosen solely because their observed averages match. These additions address specific alternative explanations rather than mechanically making a study experimental.\n\nSome superficially similar designs provide very weak evidence. A single treated group measured before and after an intervention cannot ordinarily distinguish treatment effects from aging, seasonal patterns, economic changes, or alterations in measurement. Campbell and Stanley classified such simple designs as “pre-experimental.” Modern terminology is less uniform, but the underlying limitation remains: observing improvement after an intervention establishes temporal ordering, not that the intervention caused the improvement."
    },
    {
      "heading": "Interrupted time series and changes in outcome trajectories",
      "text": "An interrupted time-series design uses repeated observations before and after a clearly defined intervention. Instead of asking only whether the post-intervention average differs from the pre-intervention average, it examines whether the intervention coincides with a departure from an established trajectory. Outcomes may include monthly traffic injuries, weekly service use, daily pollution measurements, or annual institutional performance.\n\nA basic segmented model is\n\n\\[Y_t = \\beta_0+\\beta_1t +\\beta_2I(t\\geq T_0) +\\beta_3(t-T_0)I(t\\geq T_0) +\\varepsilon_t,\\]\n\nwhere \\(T_0\\) is the intervention date and \\(I(\\cdot)\\) is an indicator function. The coefficient \\(\\beta_2\\) represents an immediate level change, while \\(\\beta_3\\) represents a change in slope. This specification distinguishes, for example, a sudden reduction in an outcome from a gradual improvement accumulating after implementation.\n\nThe implied causal comparison is between the observed post-intervention trajectory and the trajectory expected in the intervention’s absence. That expectation depends on the stability of the pretreatment process. A linear extrapolation can be misleading if the outcome was already approaching a ceiling, following a nonlinear cycle, or responding to another evolving influence. The expected response shape should therefore reflect the intervention mechanism: some changes are immediate, others delayed, temporary, or gradually phased in.\n\nBecause adjacent observations often resemble one another, time-series dependence matters. Ignoring serial correlation may produce misleading standard errors. Seasonal cycles, changes in population size, varying exposure denominators, and altered reporting practices can also mimic intervention effects. Count outcomes may require models that reflect their distribution, while rates require attention to how their denominators are constructed.\n\nThe central substantive threat is a concurrent event. If a service reform and a funding increase begin together, an interrupted series may identify their combined consequences rather than the reform alone. A controlled interrupted time series adds a comparison series exposed to common background influences but not to the intervention. Multiple comparison series or outcomes can make some rival explanations less plausible, although each must have a defensible relationship to the treated series.\n\nThere is no universally sufficient number of observations. Information depends on outcome variability, dependence, seasonality, intervention timing, and the expected response. Many observations over a short period may reveal little about an annual cycle. Conversely, a long observation window may span other institutional changes that undermine comparability. Precision and credibility therefore depend on the temporal structure of the data, not simply on counting observations before and after the interruption."
    },
    {
      "heading": "Difference-in-differences and staggered policy adoption",
      "text": "Difference-in-differences compares the change in an outcome for treated units with the change for untreated units. With one pretreatment and one post-treatment period, its basic form is\n\n\\[\\widehat{\\tau}_{\\mathrm{DiD}} = (\\overline Y_{T,\\mathrm{post}}-\\overline Y_{T,\\mathrm{pre}}) - (\\overline Y_{C,\\mathrm{post}}-\\overline Y_{C,\\mathrm{pre}}).\\]\n\nSubtracting changes removes fixed group differences and common time changes. Unlike a comparison of post-treatment levels, this design permits treated and comparison groups to begin at different outcome levels.\n\nIts central identifying assumption is parallel untreated trends: in the absence of treatment, the groups would have experienced the same expected outcome change. This is not an assumption that their observed levels are equal. It is also not established simply because a pretreatment trend test is statistically insignificant. Such tests may have low power, and similar historical trends need not persist through a new economic or institutional shock.\n\nAnticipation and changes in group composition create additional problems. If firms adjust employment before a regulation formally begins, the nominal pretreatment period may already contain treatment effects. If a policy changes migration or survey participation, changes in the measured population may be mistaken for changes in outcomes among a stable population. Spillovers can contaminate the comparison group when neighboring jurisdictions or interconnected markets respond to the same intervention.\n\nA well-known application is David Card and Alan Krueger’s 1994 study of fast-food restaurants following New Jersey’s April 1992 minimum-wage increase from $4.25 to $5.05 per hour. Restaurants in eastern Pennsylvania provided a comparison group. The study became an influential illustration of policy-based causal analysis and generated subsequent debate over data and specification. Its methodological significance lies in comparing changes across jurisdictions rather than treating a single post-policy difference as causal evidence.\n\nWith multiple periods, researchers often estimate event-study models that display outcome differences by time relative to adoption. These can reveal dynamic responses and pretreatment discrepancies. However, when treatment begins at different dates and effects vary across groups or over time, conventional two-way fixed-effects regressions may combine comparisons in problematic ways, including using already-treated units as controls for later-treated units.\n\nModern approaches therefore often estimate effects separately by adoption cohort and calendar period, using never-treated or not-yet-treated units where appropriate, before aggregating to a stated target. The aggregation weights matter: an average across units, across adoption cohorts, or across years since treatment answers different questions. The identifying assumptions must also specify the scale on which trends are parallel, since parallel changes in outcome levels do not imply parallel changes in logarithms or percentages."
    },
    {
      "heading": "Regression discontinuity and threshold-based assignment",
      "text": "A regression discontinuity design exploits a rule under which treatment changes at a threshold in an observed assignment variable. Examples include scholarship eligibility based on an examination score, program access based on household income, and administrative obligations triggered by population size. Donald Thistlethwaite and Donald Campbell introduced the design in 1960 in research on recognition and college scholarship-related outcomes.\n\nIn a sharp design, treatment is determined by the threshold:\n\n\\[D_i=I(X_i\\geq c),\\]\n\nwhere \\(X_i\\) is the running variable and \\(c\\) the cutoff. If the conditional means of both potential outcomes are continuous at \\(c\\), the outcome discontinuity identifies a treatment effect at that boundary:\n\n\\[\\tau_{\\mathrm{RD}} = \\lim_{x\\to c^+}E[Y_i\\mid X_i=x] - \\lim_{x\\to c^-}E[Y_i\\mid X_i=x].\\]\n\nThe reasoning is local: units just above and below the threshold are expected to be similar apart from their treatment status.\n\nThe continuity requirement does not mean outcomes must be unrelated to the running variable. Examination scores may strongly predict later attainment. The design allows that relationship to vary smoothly while attributing an abrupt jump at the cutoff to treatment, provided no other relevant factor changes discontinuously there. A coincident eligibility rule for a second program may instead make the estimand a combined effect.\n\nIn a fuzzy design, crossing the cutoff changes the probability of treatment rather than determining treatment perfectly. Some eligible people decline participation, or some ineligible people receive exceptions. The ratio of the outcome discontinuity to the treatment-probability discontinuity can identify a local effect for people whose participation is changed by the threshold, under additional instrumental-variable assumptions.\n\nDiagnostic evidence includes the distribution of the running variable, continuity of pretreatment characteristics, and institutional records about assignment. Precise manipulation is a concern when participants or administrators can sort individuals across the threshold using information related to potential outcomes. A density test can detect some forms of bunching, but neither a smooth density nor balanced observed covariates proves that sorting is absent. Rounding and inherently discrete scores require separate consideration.\n\nEstimation usually emphasizes observations near the cutoff, often through local linear regressions with a bandwidth determining the neighborhood used. Narrow bandwidths improve locality but reduce precision; wider bandwidths demand more approximation of the outcome–score relationship. Bias correction and sensitivity to bandwidth choice are consequently important. The resulting estimate is ordinarily local to the threshold population. Extrapolating it to people far from the cutoff requires assumptions not supplied by the discontinuity itself."
    },
    {
      "heading": "Instrumental variables and externally induced treatment variation",
      "text": "Instrumental-variable designs use a variable that changes treatment receipt while satisfying restrictions that permit its variation to identify a causal effect. The instrument may be a lottery offer, an administrative assignment rule, or another source of treatment variation. A lottery can generate randomized encouragement even when actual participation remains nonrandom. Other instruments arise from nonrandom institutional processes and require a more substantive justification.\n\nLet \\(Z_i\\) be a binary instrument and \\(D_i\\) treatment receipt. The Wald ratio is\n\n\\[\\widehat{\\tau}_{\\mathrm{IV}} = \\frac{E[Y_i\\mid Z_i=1]-E[Y_i\\mid Z_i=0]} {E[D_i\\mid Z_i=1]-E[D_i\\mid Z_i=0]}.\\]\n\nIts denominator is the first-stage effect of the instrument on treatment. The numerator is the reduced-form effect on the outcome. Their ratio converts an outcome difference induced by the instrument into an effect per induced change in treatment participation.\n\nA causal interpretation requires several distinct conditions. Relevance means that the instrument changes treatment probability. Independence means that instrument assignment is unrelated to the relevant potential outcomes and treatment-response types, possibly conditional on pretreatment variables. The exclusion restriction requires the instrument to affect the outcome only through the treatment as defined. These conditions cannot be replaced by showing that the instrument predicts treatment strongly.\n\nFor a binary treatment and instrument, monotonicity rules out units who would take treatment when not encouraged but refuse it when encouraged. Together with the other assumptions, it yields a local average treatment effect for “compliers”: units whose treatment status is changed by the instrument. This is not necessarily the effect for all treated people or for the general population. “Local” here refers to responsiveness to a particular instrument, not necessarily geographic proximity.\n\nJoshua Angrist’s 1990 study using Vietnam-era draft lottery variation is a prominent application of this reasoning to military service and later earnings. Draft eligibility altered the probability of service, creating a first stage. Interpreting eligibility-based outcome differences as the effect of service requires attention to other pathways through which eligibility might affect education, work, or behavior. The example illustrates why institutional knowledge is part of identification rather than merely background description.\n\nWeak instruments create an additional statistical difficulty. When the first stage is small or imprecisely estimated, conventional approximations can perform poorly, producing unstable estimates and misleading intervals. Instrument strength is therefore relevant to inference, but strength does not establish validity. A strong instrument with a direct effect on the outcome can yield a precise estimate of the wrong quantity. Different valid instruments can also identify different complier effects when treatment effects are heterogeneous."
    },
    {
      "heading": "Matching, weighting, and synthetic comparison units",
      "text": "Matching and weighting seek to construct treated and comparison groups that resemble one another on pretreatment information. Their classification is contested: some methodological traditions call them quasi-experimental, while others distinguish covariate-adjusted observational studies from designs based on external assignment mechanisms. The substantive issue is whether the observed variables are sufficient to remove treatment-selection differences relevant to the outcome.\n\nThe propensity score, developed as a central balancing device by Paul Rosenbaum and Donald Rubin in 1983, is the conditional probability of treatment given observed covariates:\n\n\\[e(X_i)=P(D_i=1\\mid X_i).\\]\n\nUnder conditional exchangeability, \\((Y_i(0),Y_i(1))\\perp D_i\\mid X_i\\), and adequate overlap, adjustment for this probability can support identification of average treatment effects. Conditional exchangeability is a conditional independence assumption about potential outcomes. It does not follow from a well-fitting model of treatment assignment.\n\nMatching may pair units using individual covariates, a distance measure, or an estimated propensity score. Weighting instead gives observations different contributions to a target population. For example, inverse-probability weighting for a population average effect weights treated units by \\(1/e(X_i)\\) and untreated units by \\(1/[1-e(X_i)]\\). Different weighting rules target different populations, so balance and estimand must be considered together.\n\nOverlap is crucial. If some covariate profiles occur only among treated units, the data contain no directly comparable untreated observations for them. Extreme weights are a warning that estimates may depend heavily on a few units. Restricting the analysis to a region of common support can improve comparability but changes the population represented. Balance checks assess the constructed comparison on observed characteristics; they cannot reveal unmeasured confounding.\n\nSynthetic control addresses a related problem for interventions affecting one or a few aggregate units. A weighted combination of untreated donor units is chosen to reproduce the treated unit’s pretreatment outcomes and relevant characteristics. Alberto Abadie and Javier Gardeazabal’s 2003 study of the Basque Country and Abadie, Alexis Diamond, and Jens Hainmueller’s 2010 study of California’s tobacco-control program were influential applications.\n\nSynthetic control does not simply make an aggregate treatment randomly assigned. Its credibility depends on the donor pool, pretreatment fit, the absence of contamination, and assumptions connecting pre- and post-treatment outcome processes. Poor pretreatment fit undermines the proposed counterfactual. Placebo interventions in donor units or at alternative dates can contextualize an observed gap, but their inferential meaning depends on comparability and the assumed assignment process. Neither matching nor synthetic control creates causal information solely by making graphs look similar."
    },
    {
      "heading": "Validity threats, uncertainty, and diagnostic evidence",
      "text": "Quasi-experimental validity has several dimensions. Internal validity concerns whether the estimated contrast represents a causal effect for the studied units and conditions. External validity concerns its relevance to other populations, settings, treatment versions, or periods. Construct validity concerns whether the measured treatment and outcome represent the intended concepts. A locally credible effect at an eligibility cutoff can have strong internal validity while offering limited information about a nationwide expansion.\n\nClassical threats include history, maturation, testing effects, instrument changes, attrition, and selection bias. History refers to other events coinciding with treatment; maturation to changes that would occur with time regardless of treatment. Instrumentation includes changes in definitions, reporting systems, or measurement procedures. If an intervention also changes how outcomes are recorded, an apparent effect may reflect ascertainment rather than the underlying phenomenon.\n\nPost-treatment adjustment presents another danger. Controlling for a variable affected by treatment can remove part of the effect being studied or create a spurious association. For example, conditioning on continued employment after a labor-market intervention may select different kinds of people in the treated and comparison groups. Similarly, restricting analysis to respondents observed at follow-up can induce bias if treatment affects response and response is related to potential outcomes.\n\nInterference arises when one unit’s treatment affects another unit’s outcome. A local policy may shift activity into neighboring areas, and educational interventions may affect untreated classmates. In such cases, the usual representation of each outcome as depending only on that unit’s treatment is incomplete. Researchers may distinguish direct effects, spillover effects, and effects of overall treatment saturation, but those quantities require designs and exposure definitions appropriate to the network or geographic setting.\n\nUncertainty must reflect the level at which treatment varies. Thousands of individual observations in a small number of treated regions do not provide thousands of independent policy assignments. Clustered errors, serial dependence, and few treated groups complicate inference. Conventional confidence intervals describe sampling uncertainty under an assumed model and design; they do not automatically incorporate uncertainty about the identifying assumptions.\n\nDiagnostic strategies include placebo dates, unaffected outcomes, alternative comparison groups, and sensitivity analysis. Each probes a particular vulnerability. A placebo outcome is useful only if the treatment should not affect it but relevant confounding could. Failure to reject a placebo effect is not proof of validity, especially with low statistical power. Conversely, a failed diagnostic may reveal that an otherwise sophisticated estimator is attached to an implausible research design."
    },
    {
      "heading": "Applications, reporting, and interpretation of evidence",
      "text": "Quasi-experiments are widely used in education, public health, labor-market research, environmental policy, and organizational evaluation. Their practical value is greatest when interventions operate at a scale or through a mechanism that makes investigator randomization unavailable. A national rule may have no untreated counterpart within the country, while a benefit threshold may offer a strong local comparison even without a conventional control group.\n\nThe substantive question determines which design is informative. An intervention introduced at a single known date may support interrupted time-series analysis. Different implementation dates across jurisdictions may support difference-in-differences. A strict eligibility threshold may support regression discontinuity. These are not interchangeable statistical techniques: each exploits a different feature of assignment and identifies effects under different assumptions.\n\nDetailed reporting therefore describes how treatment actually occurred. Relevant information includes eligibility rules, administrative discretion, announcement and implementation dates, compliance, exceptions, and concurrent programs. Outcome definitions and data coverage must be documented across groups and time. A discontinuity caused by a new recording system cannot be interpreted in the same way as a discontinuity in consistently measured behavior.\n\nResults require an explicit estimand and scale. An estimate may describe assignment to a program, receipt of treatment, or eligibility near a threshold. A change of two percentage points is different from a two-percent relative change. Effects measured after three months need not describe effects after three years. Aggregate and individual-level outcomes may also answer different questions, particularly when treatment changes population composition.\n\nTransparent analysis distinguishes design choices from subsequent exploration. Decisions about comparison units, intervention windows, bandwidths, outcome transformations, and subgroup analyses can materially affect results. Reporting alternative defensible specifications helps reveal dependence on these choices, while multiple testing matters when many outcomes or subgroups are examined. Prespecified analyses and reproducible code can reduce ambiguity without repairing an invalid assignment argument.\n\nEvidence across designs is often more informative than repetition of a single specification. A threshold-based study, a policy-timing study, and an encouragement design may address different populations and rely on different assumptions. Agreement can reduce concern about a particular source of bias; disagreement can reflect genuine effect heterogeneity rather than methodological failure. Comparisons must therefore preserve information about treatment content, institutional setting, target population, and follow-up duration rather than treating every estimate as a measurement of one universal effect."
    }
  ],
  "links": [
    "randomization",
    "causal-inference",
    "randomized-controlled-trial",
    "natural-experiment",
    "econometrics",
    "expected-value",
    "statistical-identification",
    "directed-acyclic-graph",
    "confounding",
    "linear-regression",
    "regression-to-the-mean",
    "indicator-function",
    "time-series",
    "standard-error",
    "difference-in-differences",
    "regression-discontinuity-design",
    "instrumental-variables",
    "propensity-score",
    "conditional-independence",
    "external-validity",
    "selection-bias",
    "confidence-interval",
    "sensitivity-analysis",
    "statistical-power",
    "education",
    "public-health",
    "labor-market",
    "multiple-testing"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}