{
  "id": "exploratory-data-analysis",
  "title": "Exploratory Data Analysis",
  "lang": "en",
  "source_lang": "en",
  "summary": "Exploratory data analysis uses visualization, numerical summaries, and iterative investigation to understand datasets, detect problems, and generate hypotheses before or alongside formal modeling.",
  "category": "mathematics",
  "subcategory": "probability-statistics",
  "type": "concept",
  "aliases": [
    "EDA",
    "exploratory analysis"
  ],
  "generated_at": "2026-11-05",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/exploratory-data-analysis/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/exploratory-data-analysis/",
      "json": "https://aiwiki.page/zh/exploratory-data-analysis.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "Exploratory data analysis (EDA) is an approach to statistics that investigates the structure, quality, and patterns of a dataset through graphical displays, numerical summaries, and iterative questioning. Its purposes include identifying unusual observations, understanding relationships, checking assumptions, and generating hypotheses or candidate models. Rather than beginning with a single fully specified model, EDA allows features of the observed data to influence subsequent questions and analytical choices. It is central to data science and complements, rather than replaces, formal statistical inference."
    },
    {
      "heading": "Origins and development",
      "text": "Exploratory practices long predate the term, but the modern approach is closely associated with the statistician John W. Tukey. His 1962 paper “The Future of Data Analysis” argued for data analysis as a field with its own practical questions, extending beyond mathematical statistics. His 1977 book Exploratory Data Analysis established a widely influential framework emphasizing graphical investigation, resistant summaries, transformations, and the discovery of unexpected structure.\n\nTukey distinguished exploratory work from confirmatory analysis without treating them as incompatible. Exploration develops questions and provisional explanations; confirmation evaluates specified claims using methods whose assumptions and error properties are understood.\n\nA famous illustration of the importance of graphical inspection is Anscombe’s quartet, introduced by Francis Anscombe in 1973. Its four datasets have nearly identical standard numerical summaries and fitted linear relationships but markedly different graphical structures. They demonstrate that a few summary statistics cannot adequately characterize a dataset.\n\nInteractive graphics and computational tools subsequently expanded EDA from small tables and hand-drawn displays to large, multidimensional datasets. Modern systems support filtering, linked plots, rapid aggregation, and reproducible analytical documents."
    },
    {
      "heading": "Objectives and principles",
      "text": "EDA commonly addresses several related questions:\n\n- What does the dataset represent? This includes the observational unit, population of interest, measurement procedures, units, and collection process.\n- Are the records internally consistent? Examples include duplicate identifiers, impossible values, inconsistent categories, and mismatched dates.\n- How are variables distributed? Important features include location, spread, skewness, multiple modes, and extreme values.\n- How do variables relate? Relationships may be nonlinear, differ between subgroups, or depend on time and context.\n- Which assumptions are plausible? Exploration can reveal features inconsistent with assumptions such as constant variability or independent observations.\n- Which questions merit further investigation? An unexpected pattern may motivate a new model, additional data collection, or a revised study design.\n\nEDA is iterative rather than a fixed checklist. A plot may suggest a subgroup comparison; that comparison may expose a coding error; correcting the error may change the apparent relationship.\n\nExploration is also not assumption-free. Variable selection, measurement scales, graphical encodings, and subgroup definitions all influence what becomes visible. Effective EDA makes these choices explicit and interprets patterns in the context of how the data were produced."
    },
    {
      "heading": "Understanding and checking the data",
      "text": "Structure and provenance\n\nInitial exploration establishes the number of records and variables, their types, and the meaning of a row. A row might represent a person, transaction, instrument reading, or aggregated region. Confusing these levels can produce misleading comparisons.\n\nIdentifiers and relationships between tables are particularly important. Combining tables can unintentionally duplicate observations or exclude unmatched records. Repeated measurements on the same entity also require attention because they generally do not satisfy statistical independence.\n\nCollection procedures constrain interpretation. A large convenience dataset can still exhibit selection bias, and extensive exploration cannot recover information about groups that were systematically excluded.\n\nMissingness and validity\n\nMissing values are examined by variable, record, group, and time. Their patterns may reveal instrument failures, changes in collection procedures, or differences in participation. A missing value must also be distinguished from a genuine zero, a value below a detection limit, or a quantity that does not apply.\n\nObserved patterns alone generally cannot establish the mechanism responsible for missingness. Decisions about missing-data imputation therefore require substantive knowledge and assumptions beyond an exploratory display.\n\nValidity checks include ranges, units, logical constraints, and consistency across related fields. A temperature recorded in two different unit systems, for example, may produce an apparent mixture of populations that is actually a processing error."
    },
    {
      "heading": "Core techniques",
      "text": "Univariate analysis\n\nUnivariate exploration examines one variable at a time. For numerical variables, common summaries include the sample mean, median, standard deviation, and selected quantiles. The mean and standard deviation can be strongly affected by extreme observations, whereas the median and interquartile range are more resistant.\n\nA histogram groups observations into intervals and displays their frequencies or densities. Its appearance depends on bin width and placement: broad bins can conceal structure, while narrow bins can exaggerate random variation. Smooth density estimates have an analogous dependence on their smoothing settings.\n\nA box plot summarizes the median, quartiles, and a convention-dependent representation of the tails. It is useful for comparing groups but may conceal multiple modes and other distributional features. Empirical cumulative distribution plots provide another view without requiring histogram bins.\n\nFor categorical variables, counts and proportions reveal dominant categories, rare levels, and inconsistent labels. Explicit denominators are essential when groups differ in size.\n\nBivariate and conditional analysis\n\nA scatter plot displays paired numerical observations and can reveal trends, curvature, clusters, changing variability, and unusual points. A correlation coefficient provides a compact numerical summary, but Pearson correlation describes linear association and may miss strong nonlinear relationships.\n\nRelationships between numerical and categorical variables can be examined using grouped distributions, individual-point displays, and tables of summaries. Two categorical variables are often explored with cross-tabulations, conditional proportions, or mosaic displays.\n\nConditioning on relevant variables is important because an aggregate relationship can differ from relationships within subgroups. Such changes may reflect confounding, differences in group composition, or other features of the data-generating process. An exploratory association does not by itself establish a causal effect; causal inference requires additional assumptions and an appropriate study design.\n\nMultivariate exploration\n\nWhen many variables are present, scatter-plot matrices, correlation matrices, heatmaps, and linked displays help identify broader structure. These methods become harder to interpret as dimensionality increases.\n\nPrincipal component analysis constructs directions that summarize variation, while cluster analysis searches for groups of similar observations. Both can support exploration, but their results depend on choices such as scaling, distance measures, and algorithm settings. A cluster is not automatically evidence of a naturally distinct population, and a low-dimensional projection can hide important differences.\n\nOrdered and temporal data\n\nFor time series, plotting observations in their original order is often more informative than examining their overall distribution alone. Time plots, lag plots, and seasonal comparisons can reveal trends, cycles, abrupt changes, and serial dependence.\n\nAggregation changes what can be seen. Daily measurements may expose short-lived events that monthly averages conceal, while longer-term summaries can clarify trends obscured by short-term variation. Irregular sampling and changes in measurement procedures can also create apparent temporal patterns."
    },
    {
      "heading": "Transformations, resistant summaries, and unusual observations",
      "text": "Transformations can make multiplicative relationships easier to inspect or reduce the visual dominance of a long tail. A logarithmic transformation, for example, changes ratios on the original scale into differences on the transformed scale. It also changes interpretation and cannot be applied directly to zero or negative values.\n\nScaling is especially consequential in distance-based analyses. If variables are measured in very different units, the variable with the largest numerical scale may dominate the results unless scaling is addressed.\n\nRobust statistics provides methods that are less sensitive to extreme observations or modest departures from assumptions. A common exploratory rule flags values outside\n\n\\[[Q_1-1.5\\,\\mathrm{IQR},\\;Q_3+1.5\\,\\mathrm{IQR}],\\]\n\nwhere \\(Q_1\\) and \\(Q_3\\) are the first and third quartiles and \\(\\mathrm{IQR}=Q_3-Q_1\\).\n\nThis rule identifies observations for investigation, not observations that must be deleted. An unusual value may be an error, a legitimate rare event, or evidence that several populations have been combined. Likewise, a point may be ordinary in each variable separately but unusual in its combination of values.\n\nComparing results with and without influential observations is a form of sensitivity analysis. Such comparisons reveal dependence on analytical choices but do not, by themselves, justify excluding data."
    },
    {
      "heading": "Relationship to modeling and inference",
      "text": "EDA and modeling often form a cycle. Exploratory findings suggest models, and model diagnostics generate new exploratory questions. After fitting linear regression, for example, residual plots can reveal curvature, changing spread, or dependence that the fitted model does not capture.\n\nThe distinction between exploration and confirmation concerns how claims are evaluated, not simply which techniques are used. A regression can be exploratory, and a graph can be part of a prespecified confirmatory analysis.\n\nA central limitation is that hypotheses suggested by a dataset have already benefited from inspection of that dataset. Applying ordinary hypothesis testing as though those hypotheses had been fixed in advance can produce misleading evidence. Searching many relationships also raises multiple testing concerns, even when the search is informal or primarily visual.\n\nIndependent replication, reserved data, or suitable methods that account for analytical selection can strengthen subsequent evaluation. Exploratory findings remain useful, but they must be distinguished from claims tested independently of the search that generated them.\n\nIn predictive machine learning, exploration can also compromise evaluation if held-out outcomes influence feature selection, preprocessing, or model choices. Cross-validation estimates predictive performance appropriately only when the evaluation procedure accounts for the relevant data-dependent steps."
    },
    {
      "heading": "Limitations and reproducibility",
      "text": "Exploratory displays can mislead through truncated axes, inappropriate scales, overplotting, or selective presentation. Smooth curves and apparent clusters may reflect graphical or algorithmic settings rather than stable structure. Very large datasets can make negligible differences visually prominent, whereas small datasets can produce striking patterns through random variation.\n\nEDA cannot by itself establish representativeness, prove distributional assumptions, or repair a weak experimental design. The absence of a visible pattern is not proof that no relationship exists.\n\nReproducible exploration preserves the original data, records transformations and exclusions, and documents the choices behind tables and figures. A useful analytical record separates verified data errors, descriptive findings, provisional explanations, and claims requiring independent evaluation. Code-based reports and versioned workflows help retain this distinction even when the investigation involves many iterations."
    }
  ],
  "links": [
    "statistics",
    "data-science",
    "statistical-independence",
    "selection-bias",
    "missing-data-imputation",
    "sample-mean",
    "median",
    "standard-deviation",
    "quantile",
    "histogram",
    "density-estimation",
    "box-plot",
    "scatter-plot",
    "correlation",
    "confounding",
    "causal-inference",
    "principal-component-analysis",
    "cluster-analysis",
    "time-series",
    "sensitivity-analysis",
    "linear-regression",
    "statistical-hypothesis-testing",
    "multiple-testing",
    "machine-learning",
    "cross-validation",
    "experimental-design"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}