{
  "id": "data-lineage",
  "title": "Data Lineage",
  "lang": "en",
  "source_lang": "en",
  "summary": "Data lineage records the origins, transformations, and movement of data, enabling users to trace outputs to their sources and assess downstream dependencies.",
  "category": "computer-science",
  "subcategory": "software",
  "type": "concept",
  "aliases": [
    "Data ancestry"
  ],
  "generated_at": "2026-11-03",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/data-lineage/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/data-lineage/",
      "json": "https://aiwiki.page/zh/data-lineage.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "Data lineage is the recorded history of where data comes from, how it is transformed, and where it moves or is used. It connects source data to intermediate and final outputs through operations such as copying, filtering, joining, aggregation, and model training. As a form of metadata, lineage describes relationships among data assets and processing activities rather than merely storing the data itself. It is used in data management to explain results, investigate errors, assess the effects of changes, and support accountability."
    },
    {
      "heading": "Scope and related concepts",
      "text": "A lineage record may describe a single query, a processing pipeline, or an organization’s interconnected data systems. Its scope can extend from operational applications and files through a data warehouse to analytical reports, exported datasets, and machine-learning models.\n\nTwo common directions of inquiry are:\n\n- Upstream tracing: identifying the sources and transformations that contributed to a selected output.\n- Downstream impact analysis: identifying the outputs, systems, or users potentially affected by a change to a source.\n\nData lineage overlaps with data provenance. Usage varies: provenance often emphasizes the evidence surrounding an object’s creation, including responsible people, software, methods, and circumstances, while lineage often emphasizes dependencies and movement through processing systems. There is no universally enforced boundary between the terms.\n\nLineage is also distinct from a data catalog or a database schema. A catalog helps users discover and understand assets; a schema describes their structure. Lineage explains how assets are related through derivation or movement. A catalog may display lineage, and schema information can help derive it, but neither alone supplies a complete processing history."
    },
    {
      "heading": "Representation and granularity",
      "text": "Lineage is commonly represented as a directed graph. Nodes can represent datasets, columns, processing jobs, individual executions, reports, or models. Edges describe relationships such as “reads,” “writes,” “derives from,” or “copies to.”\n\nA graph can distinguish data objects from the activities that produce them. For example:\n\nOrders ───────┐\n              ├── Revenue calculation ── Monthly revenue ── Dashboard\nExchange rates┘\n\nThis representation makes the transformation explicit rather than showing only a direct connection between input and output datasets. A knowledge graph can add relationships involving owners, business terms, policies, and software components.\n\nLevels of detail\n\nLineage can be captured at several granularities:\n\nGranularity | What it records | Typical purpose\nSystem level | Transfers between applications or platforms | Architectural mapping\nDataset or table level | Dependencies among files, tables, or collections | Pipeline monitoring and impact analysis\nColumn or field level | Relationships between input and output fields | Transformation explanation and sensitive-data tracking\nRecord or cell level | Contributions of particular records or values to an output | Detailed auditing and result explanation\n\nFiner granularity usually increases storage requirements and computational cost. A dependency between two tables does not establish that every record in one contributed to every result in the other.\n\nField-level lineage also requires distinctions among kinds of influence. In a query that sums sales only for records marked “approved,” the sales amount contributes to the output value, while the approval field determines which records are included. A system recording only direct value transformations may omit this filtering dependency.\n\nDesign-time and execution-specific lineage\n\nDesign-time lineage describes intended or possible dependencies inferred from pipeline definitions, queries, and configuration. Execution-specific lineage records what happened during a particular run, including the actual inputs, outputs, parameters, and execution status.\n\nThese views can differ. A pipeline may contain conditional branches, dynamically selected tables, or tasks that fail before producing output. A defined dependency therefore does not prove that data flowed along it during a particular execution.\n\nVersioning is essential to historical interpretation. A relationship between two current table names does not necessarily explain a report produced months earlier. More precise records identify dataset versions or snapshots, job runs, code revisions, and relevant timestamps."
    },
    {
      "heading": "How lineage is captured",
      "text": "Static analysis\n\nStatic methods inspect source code, configuration, or query definitions without executing the workload. For SQL, a parser can identify referenced tables, resolve column expressions, and derive relationships through joins, subqueries, and aggregations.\n\nStatic analysis can reveal dependencies before deployment and support change assessment. Its limitations include dynamically generated queries, unresolved schemas, external functions, and code whose behavior depends on runtime values. A stored procedure may construct a table name from a parameter that is unavailable during analysis.\n\nRuntime instrumentation\n\nRuntime methods collect information from executing systems. Query engines, schedulers, and processing frameworks can emit events identifying jobs, input datasets, output datasets, and execution times. Integration with platforms such as Apache Spark can capture dependencies from execution plans or processing events.\n\nRuntime capture provides evidence about actual activity, but the available detail depends on instrumentation. An event stating that a job read three tables and wrote one table does not, by itself, explain the relationships between their individual columns. Failed or partial executions also require careful interpretation.\n\nLogs, connectors, and manual records\n\nLineage can also be reconstructed from query histories, audit logs, orchestration metadata, and connectors to storage or analytics services. Manual documentation is useful for processes that automated tools cannot observe, including spreadsheet work and transfers between organizations.\n\nPractical systems often combine these methods. Automated records provide coverage at scale, while curated descriptions explain business meaning or fill gaps. The origin of each lineage assertion matters: an observed execution, a parsed dependency, and a manually entered relationship offer different kinds of evidence."
    },
    {
      "heading": "Technical foundations",
      "text": "Database research provides a more formal treatment of lineage. For a query over a relational database, provenance models may describe which input records contributed to an output and how those contributions were combined.\n\nFor example, suppose two input records independently satisfy a query and produce the same output tuple. Merely recording the set of source tables loses the distinction between these alternative derivations. More expressive models can preserve alternatives and combinations, including annotations that use addition for alternative derivations and multiplication for joint contributions. Such models support reasoning about query results, although their applicability depends on the query language and operations involved.\n\nOperational lineage systems often use coarser approximations because detailed query provenance can be expensive. Their central engineering problems include:\n\n- Identity resolution: recognizing the same asset across tools, aliases, and naming conventions.\n- Temporal consistency: connecting a run to the correct input and output versions.\n- Transformation semantics: distinguishing a copy from an aggregation, filter, or join.\n- Cross-system continuity: maintaining connections across organizational and technical boundaries.\n- Event reconciliation: handling missing, duplicated, delayed, or contradictory observations.\n\nIn stream processing, outputs may depend on event-time windows, late-arriving records, state, and repeated updates. Dataset-level edges alone may be insufficient to explain which events contributed to a particular result."
    },
    {
      "heading": "Applications",
      "text": "Error investigation and data quality\n\nLineage helps locate possible sources of incorrect, missing, or unexpectedly changed data. If several reports depend on the same transformed table, a shared upstream operation becomes a candidate for investigation.\n\nCombined with data-quality measurements, lineage can help determine which downstream assets may have been affected by a failed validation or stale input. It does not establish that every connected output is incorrect: some may use unaffected fields or exclude the problematic records.\n\nChange management\n\nBefore changing a field’s meaning, format, or availability, downstream tracing can identify dependent queries, pipelines, and reports. This supports coordination and testing during schema changes or platform migrations.\n\nDependency information may also support scheduling and selective recomputation. However, whether a task must run again depends on execution semantics and change-detection rules, not just graph connectivity.\n\nGovernance and audit\n\nWithin data governance, lineage provides evidence about the handling of important datasets. It can connect business definitions to technical implementations and help document how a reported figure was calculated.\n\nFor data privacy, lineage may help locate copies and derived datasets associated with sensitive information. It does not independently establish lawful processing, demonstrate effective deletion, or prove that a derived result cannot reveal sensitive information.\n\nMachine learning and scientific reproducibility\n\nIn machine learning, lineage can connect training data, preprocessing, feature engineering, training runs, and model artifacts. It can help investigate whether evaluation data entered a training pipeline, a form of data leakage, or identify models associated with a problematic dataset.\n\nLineage complements version control and reproducibility records. Reproducing an output may additionally require preserved data, software dependencies, configuration, random seeds, and execution-environment details. A dependency graph alone is not a reproducible experiment."
    },
    {
      "heading": "Development and standards",
      "text": "Data lineage developed from overlapping work in database provenance, scientific workflow systems, and enterprise data integration. Database research investigated explanations for query results, while workflow systems recorded computational steps and enterprise tools mapped transformations between operational and analytical systems.\n\nThe World Wide Web Consortium’s PROV family of specifications, published in 2013, established a general framework for representing provenance through entities, activities, agents, and their relationships. It is broader than enterprise data lineage and can describe the origins of many kinds of digital objects.\n\nOpenLineage provides an open specification for collecting lineage metadata around jobs, runs, and datasets, with extensible facets for additional information. Standardized representations can improve interoperability, but they do not eliminate differences in asset naming, capture coverage, or the meaning assigned to dependencies."
    },
    {
      "heading": "Limitations and security",
      "text": "Lineage is only as complete and accurate as its capture mechanisms. Uninstrumented applications, manual exports, external services, and undocumented transformations can create gaps. A visually continuous graph may therefore suggest more certainty than the evidence supports.\n\nA lineage edge establishes a recorded dependency, not necessarily a complete explanation of an output. It does not prove data correctness, statistical causation, or the absence of undiscovered inputs. Approximate methods may overstate dependencies, while incomplete capture may omit them.\n\nLineage metadata can itself be sensitive. It may expose confidential dataset names, business logic, system architecture, or relationships involving restricted information. Access control therefore applies to lineage records and their visualizations as well as to the underlying data. Capture time, evidence source, known coverage gaps, and version information are important for interpreting what a lineage record actually establishes."
    }
  ],
  "links": [
    "metadata",
    "data-management",
    "data-warehouse",
    "database-schema",
    "knowledge-graph",
    "source-code",
    "sql",
    "apache-spark",
    "relational-database",
    "stream-processing",
    "data-quality",
    "data-governance",
    "data-privacy",
    "machine-learning",
    "training-data",
    "feature-engineering",
    "data-leakage",
    "version-control",
    "interoperability",
    "causation",
    "access-control"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}