{
  "id": "data-catalog",
  "title": "Data Catalog",
  "lang": "en",
  "source_lang": "en",
  "summary": "A data catalog is a searchable inventory of data assets and their metadata, supporting discovery, interpretation, governance, and reuse.",
  "category": "computer-science",
  "subcategory": "software",
  "type": "concept",
  "aliases": [
    "Data catalogue"
  ],
  "generated_at": "2026-11-03",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/data-catalog/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/data-catalog/",
      "json": "https://aiwiki.page/zh/data-catalog.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "A data catalog is an organized, searchable inventory of data assets and the metadata that describes them. It helps users discover what data exists, understand its meaning and origin, identify responsible people, and determine how it may be accessed or used. Cataloged assets may include tables, files, datasets, data services, reports, and other resources. A catalog typically stores descriptions and references to these assets rather than serving as the primary store for the data itself."
    },
    {
      "heading": "Purpose and scope",
      "text": "Organizations often distribute data across operational systems, analytical platforms, departments, and external providers. Knowing that a dataset exists does not necessarily reveal whether it is relevant, current, trustworthy, or available for a particular purpose. A catalog brings this contextual information into a common discovery environment.\n\nA catalog can answer questions such as:\n\n- Which dataset contains a particular measure or business concept?\n- What does a field mean, and how is it calculated?\n- Who maintains the dataset?\n- How recently was it updated?\n- Which upstream sources and downstream reports depend on it?\n- What restrictions apply, and how can access be requested?\n\nCatalogs support data management, but their scope varies. An enterprise catalog may span many systems and business domains; a research catalog may emphasize methods, provenance, citation, and reuse conditions; a public-sector catalog may publish descriptions of openly available datasets. A platform-specific catalog may describe only assets within one service.\n\nDiscovery and access are distinct. A catalog can disclose that an asset exists without granting permission to retrieve it, although even that disclosure may require restrictions."
    },
    {
      "heading": "Information represented",
      "text": "Catalogs combine metadata obtained automatically from systems with information supplied or reviewed by people.\n\nMetadata category | Typical contents\nTechnical | Asset location, file format, columns, data types, schema, partitions, and relationships\nBusiness and semantic | Definitions, descriptions, approved terminology, calculation rules, and intended uses\nAdministrative | Owners, stewards, contacts, publication status, licenses, and retention requirements\nOperational | Update times, processing status, usage statistics, and service expectations\nQuality and provenance | Quality measurements, known limitations, source systems, and transformation history\nSecurity and policy | Sensitivity labels, access restrictions, and permitted-use classifications\n\nTechnical descriptions frequently include a database schema, while data lineage records relationships among sources, transformations, and outputs. Data quality information may describe completeness, validity, or freshness, but these properties require explicit definitions: a recently updated dataset is not necessarily accurate or suitable for every use.\n\nA business glossary provides definitions of terms such as “active customer” or “net revenue.” Linking those terms to physical fields helps distinguish business meaning from implementation. The same term may have different definitions in different domains, and a catalog must be able to represent such differences rather than silently treating them as equivalent."
    },
    {
      "heading": "Architecture and operation",
      "text": "Metadata acquisition\n\nCatalogs acquire metadata through connectors, scheduled scans, events, manual entry, and application programming interfaces. Connectors may inspect databases, data warehouses, file repositories, processing platforms, and reporting systems.\n\nTypical acquisition activities include:\n\n- Reading system metadata to identify tables, columns, and data types.\n- Inspecting files to infer structure and formats.\n- Collecting processing logs or parsing SQL to infer dependencies.\n- Importing descriptions, ownership records, and policy labels.\n- Profiling selected data to produce statistical summaries.\n\nProfiling is different from metadata extraction because it examines actual values. It may require additional permissions, generate processing costs, and expose sensitive information. Catalogs therefore differ in whether they retain sample values, store only aggregates, or avoid value inspection entirely.\n\nMetadata model and storage\n\nCollected information is mapped into a metadata model describing asset types, attributes, identifiers, and relationships. Stable identifiers help distinguish similarly named assets and preserve references when names or locations change.\n\nA catalog may use relational storage, document storage, graph storage, or combinations of these. Some represent metadata as a knowledge graph, enabling traversal from a report to its underlying tables, transformations, owners, and associated business concepts. A graph-based implementation is an architectural choice, not a defining requirement.\n\nIdentity resolution is a significant challenge. A replicated table, an exported file, and a report may represent related information without being the same asset. Incorrectly merging them can hide important differences; treating every copy as unrelated can make search and lineage difficult to interpret.\n\nSearch and interaction\n\nCatalog interfaces commonly provide keyword search, filters, browsing, asset pages, and programmatic access. Information retrieval techniques rank results using names, descriptions, terms, and other signals. Ranking may also consider usage, freshness, or review status, although popularity is not evidence of correctness.\n\nSome systems use machine learning to suggest classifications, detect similar datasets, or infer relationships. Suggested metadata remains subject to error and may require human review.\n\nMaintenance\n\nMetadata changes as source systems evolve. Catalog maintenance includes detecting new assets, updating descriptions and relationships, retiring obsolete entries, and preserving useful history. Scheduled harvesting introduces a delay between a source change and its appearance in the catalog; event-driven updates can reduce that delay but depend on reliable integration.\n\nA catalog can remain technically available while becoming less useful because its ownership records, definitions, or source references are stale."
    },
    {
      "heading": "Governance and security",
      "text": "A catalog commonly supports data governance by connecting assets to accountable owners, stewardship processes, classifications, and policies. Review workflows can distinguish an automatically discovered asset from one whose documentation has been checked for a stated purpose.\n\nSuch labels need defined criteria. “Certified” may indicate that an asset has an assigned owner, documented calculations, and specified quality checks; it does not establish universal fitness for use.\n\nCatalogs can integrate with access control systems to display permissions or route access requests. Actual enforcement may remain in the underlying storage or processing platform. A policy recorded in a catalog has no enforcement effect unless relevant systems or procedures apply it.\n\nMetadata itself may be sensitive. Asset names, field names, lineage, query histories, and samples can reveal confidential activities or personal information. Protecting data privacy therefore involves controlling catalog visibility as well as access to the underlying records. Permission-aware search also needs to account for snippets, relationship views, and aggregate results that might disclose restricted information."
    },
    {
      "heading": "Relationship to neighboring systems",
      "text": "Several related systems overlap with a data catalog without being identical to it:\n\n- Database system catalog: Internal metadata used by a database management system to manage objects such as tables, indexes, and permissions. An enterprise data catalog may collect this information across many systems.\n- Data dictionary: Detailed descriptions of fields, types, constraints, and meanings, often scoped to an application or database.\n- Business glossary: Definitions and relationships among business terms, which may exist independently of physical data assets.\n- Metadata repository: A broader category of metadata storage; a data catalog emphasizes discovery and contextual understanding.\n- Data marketplace: An environment that may add request, provisioning, subscription, or commercial exchange workflows to catalog-style discovery.\n\nA catalog is not inherently a query engine, a backup system, or a data integration platform. Products may combine these functions, but listing an asset does not imply that the catalog stores its contents or can execute queries against it."
    },
    {
      "heading": "Development and interoperability",
      "text": "Data catalogs developed from several traditions, including database dictionaries, enterprise metadata repositories, scientific data directories, and public data portals. As analytical environments expanded across platforms and organizational boundaries, catalogs increasingly combined automated discovery with business descriptions, ownership, and governance workflows.\n\nInteroperability matters when metadata must be exchanged between catalogs or aggregated across institutions. The World Wide Web Consortium developed the Data Catalog Vocabulary (DCAT), an RDF vocabulary for describing catalogs, datasets, data services, and distributions. A distribution represents an accessible form of a dataset, such as a downloadable file.\n\nShared vocabularies improve exchange but do not eliminate differences in meaning. Catalogs may still disagree about identifiers, classification schemes, licensing terms, or the boundaries of a dataset. Application profiles can specify additional conventions for particular communities."
    },
    {
      "heading": "Applications and limitations",
      "text": "Catalogs support analytical discovery, reuse of existing datasets, documentation, and impact analysis. In data engineering, lineage and ownership information help identify dependencies before changing a pipeline or field. Research and public-data catalogs help users assess provenance, coverage, and reuse conditions.\n\nTheir usefulness depends on coverage and metadata quality. Important limitations include:\n\n- Incomplete discovery: Unsupported systems, local files, and manually exchanged data may remain invisible.\n- Partial lineage: Dynamic code, external processing, and manual transformations can leave gaps in dependency records.\n- Semantic ambiguity: Similar names do not establish equivalent meanings, and matching schemas do not guarantee comparable data.\n- Unreliable automation: Inferred descriptions, classifications, and relationships can be incorrect.\n- Maintenance burden: Human knowledge about definitions, ownership, and appropriate uses requires continuing attention.\n- Uneven visibility: Security restrictions may prevent users from seeing enough context to assess an asset.\n\nCatalog effectiveness is therefore not measured simply by the number of indexed assets. Relevant measures include metadata freshness, ownership coverage, search success, unresolved documentation gaps, and whether users can identify data that is appropriate for their specific task."
    }
  ],
  "links": [
    "metadata",
    "data-management",
    "database-schema",
    "data-lineage",
    "data-quality",
    "application-programming-interface",
    "database",
    "data-warehouse",
    "sql",
    "knowledge-graph",
    "information-retrieval",
    "machine-learning",
    "data-governance",
    "access-control",
    "data-privacy",
    "database-management-system",
    "interoperability",
    "world-wide-web-consortium",
    "resource-description-framework",
    "data-engineering"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}