aiwiki.page
English

P-hacking

P-hacking is the use of data-dependent analysis or selective reporting to obtain statistically significant results without accounting for the analytical search.

23 keywords2 linked from2 not yet writtenWritten by AI
StatisticsP-valueStatistical Hypo…Null HypothesisSignificance Lev…Multiple TestingType I and Type…Linear regressio…P-hacking

P-hacking is the use of analytical flexibility, repeated testing, or selective reporting to obtain statistically significant results without adequately accounting for how those results were selected. In statistics, the term usually describes practices that make reported p-values appear stronger as evidence than the underlying research process warrants. It can involve deliberately searching for a publishable result, but it can also arise unintentionally when seemingly reasonable decisions are influenced by the data. The defining problem is not exploration itself, but presenting a selected result as though it came from a sufficiently prespecified or otherwise valid inferential procedure.

Statistical basis

In statistical hypothesis testing, a p-value is the probability, under a specified null hypothesis and statistical model, of obtaining a test statistic at least as extreme as the observed value. It is not the probability that the null hypothesis is true, nor the probability that the result occurred “by chance.”

A testing procedure with significance level α\alpha is designed to limit its probability of rejecting a true null hypothesis. That guarantee depends on the procedure as a whole, including relevant choices about sampling, analysis, and reporting. A p-value calculated for one fixed analysis generally does not account for selecting that analysis after examining many alternatives.

Suppose a researcher performs mm independent tests of true null hypotheses, each with probability α\alpha of producing a significant result. The probability of obtaining at least one significant result is

P(at least one rejection)=1−(1−α)m.P(\text{at least one rejection})=1-(1-\alpha)^m.

For m=20m=20 and α=0.05\alpha=0.05, this probability is approximately 0.6420.642. Reporting only the successful test hides the much larger opportunity for a false positive. Independence is important to this calculation: correlated tests require a different calculation, although selection among them can still inflate error rates.

P-hacking therefore often constitutes an unacknowledged multiple-testing problem. It can increase Type I errors, distort uncertainty estimates, and undermine the interpretation of conventional significance thresholds.

Common forms

Selecting outcomes, groups, or comparisons

A study may collect many outcomes, examine several time points, or divide participants into numerous subgroups. P-hacking can occur when only comparisons yielding small p-values are emphasized while unsuccessful comparisons are omitted.

For example, a study measuring ten outcomes might highlight one significant result without disclosing that the other nine were examined. The same issue can arise when researchers repeatedly change subgroup boundaries, reference categories, or the definition of an outcome.

Subgroup analysis is not inherently problematic. The concern is whether the choice was influenced by the observed results and whether that selection is reflected in the inference and reporting.

Choosing models after inspecting significance

Researchers may try different transformations, covariates, statistical tests, or model specifications and retain whichever produces the desired conclusion. In linear regression, for example, adding or removing control variables can alter a coefficient, its standard error, and its p-value.

Different models may address genuinely different questions. Adjusting for a variable may reduce confounding, introduce bias, or change the quantity being estimated. Treating model choice primarily as a way to cross a significance threshold obscures these substantive differences.

Altering exclusions and preprocessing

Decisions about outliers, missing observations, measurement thresholds, and data cleaning can create additional opportunities for selective analysis. Trying several exclusion rules and reporting only the rule that yields significance understates the analytical search.

Such decisions may be justified by measurement problems or model assumptions. Their legitimacy depends on the reasons for making them, their transparency, and whether the inferential procedure accommodates their data-dependent nature—not simply on whether they affect significance.

Optional stopping

Optional stopping becomes a form of p-hacking when researchers repeatedly inspect conventional fixed-sample test results and stop collecting data once significance is reached, without using a method that permits such monitoring.

Repeated opportunities to stop can raise the probability of rejecting a true null hypothesis above the nominal level. The amount of inflation depends on the testing procedure, inspection schedule, stopping rule, and maximum sample size.

Not all early stopping is invalid. Properly designed sequential methods explicitly incorporate interim monitoring and can preserve specified error guarantees.

Analytical flexibility and intent

P-hacking does not require an explicit decision to manipulate a result. Researchers may make a sequence of locally plausible choices—such as excluding an unusual observation, changing a transformation, or inspecting a subgroup—while being influenced by whether each choice improves the result.

This is sometimes described as a garden of forking paths: many possible analyses exist, and the observed data influence which path is taken. The researcher need not calculate every possible analysis for the selection process to matter statistically.

Consequently, a distinction is needed between an invalid inferential procedure and an allegation of misconduct. P-hacking describes methodological behavior or its consequences; a reported p-value alone generally cannot establish a researcher's intent.

Related practices

Publication bias occurs when the likelihood of publication depends on a study's findings, such as whether they are statistically significant. P-hacking concerns choices within the research or reporting process. Publication bias concerns selection into the accessible literature. The two can reinforce one another when incentives favor significant findings.

Hypothesizing after the results are known, or HARKing, means presenting a hypothesis developed after seeing the data as though it had been formulated beforehand. It can accompany p-hacking, but the concepts differ: HARKing concerns the timing and representation of hypotheses, whereas p-hacking concerns significance-seeking analysis or reporting.

Data dredging and data fishing are overlapping terms for searching data for patterns. Such searching is not necessarily invalid. Exploratory analyses can generate valuable hypotheses if their exploratory status is clear and subsequent inference accounts for selection or uses new data.

Fabrication and falsification are also distinct. P-hacking can occur with accurately recorded observations and correctly executed calculations; the misleading element lies in the analytical search, selective presentation, or interpretation.

Consequences for research

P-hacking affects more than the frequency of significant results. Selecting estimates because they pass a significance threshold tends to favor estimates that overstate the magnitude of an effect, especially when statistical power is low. A reported effect size may therefore be exaggerated even when the underlying effect is real.

Conventional confidence intervals can also lose their nominal coverage when reported only after data-dependent selection. An interval calculated for a fixed analysis does not automatically retain its advertised coverage after that analysis has been chosen for producing an appealing result.

At the literature level, selective analysis can create an apparently consistent body of positive evidence from studies with much less compelling underlying results. This complicates meta-analysis, because combining published estimates does not by itself remove biases introduced before publication.

A small p-value does not establish a large or practically important effect. Conversely, a nonsignificant result does not establish that an effect is absent. Treating a threshold such as 0.050.05 as the sole criterion for scientific importance encourages significance-seeking behavior.

Prevention and valid alternatives

Prespecification and transparent reporting

Preregistration records hypotheses, outcomes, sampling rules, and analysis plans before the relevant results are known. It can make departures from an initial plan visible and distinguish confirmatory tests from later exploration.

Its effectiveness depends on timing, specificity, and disclosure. A vague plan can leave considerable flexibility, while a detailed plan can still require justified changes. Transparent reporting identifies those changes rather than treating every deviation as improper.

Registered reports extend this approach by having journals evaluate research questions and methods before the main results are available, with publication commitments conditional on meeting specified requirements. This reduces the direct dependence of publication decisions on whether the results are significant.

Statistical adjustment

When multiple hypotheses form a defined testing family, methods such as the Bonferroni correction can control the probability of one or more false rejections. Other procedures control the false discovery rate, a different error criterion.

These methods address specified testing structures. Correcting only the tests eventually reported does not necessarily account for unreported outcomes, model searches, or stopping decisions.

Sequential testing methods address repeated monitoring by building the monitoring and stopping procedure into the analysis. Methods for selective inference similarly seek to account for selection before estimating effects or testing hypotheses.

Separating exploration from confirmation

Independent confirmatory data can test hypotheses generated through exploration. A held-out dataset serves this purpose only if it remains sufficiently independent of the analytical search; repeatedly inspecting and adapting to it can recreate the original problem.

Sensitivity analysis and multiverse analysis can show how conclusions vary across defensible assumptions and specifications. Their purpose is to expose dependence on analytical choices, not to identify the specification with the smallest p-value.

Switching to Bayesian inference does not automatically eliminate selective reporting or data-dependent analysis. Bayesian conclusions also depend on model specification, prior assumptions, and how the reported analysis was selected.

Detection and limitations

Audits can compare protocols, registrations, analysis code, datasets, and published reports to identify undisclosed changes. Such comparisons may reveal selective analysis, although a deviation alone does not establish that it was made to obtain significance.

At an aggregate level, researchers examine distributions of reported p-values, discontinuities near significance thresholds, and patterns of selective outcome reporting. P-curve analysis, for example, examines the distribution of statistically significant p-values under assumptions about the studies and how results were selected.

These approaches have important limitations. Patterns can be influenced by effect sizes, statistical power, rounded reporting, dependence among tests, publication selection, and the mix of studies included. No single p-value—whether just below a threshold or extremely small—diagnoses p-hacking. Estimating its prevalence requires explicit definitions, appropriate data, and assumptions about the processes that generated the observed results.