{
  "id": "query-optimizer",
  "title": "Query Optimizer",
  "lang": "en",
  "source_lang": "en",
  "summary": "A query optimizer selects an execution strategy for a database query by comparing equivalent plans, estimating their costs, and applying transformation rules.",
  "category": "computer-science",
  "subcategory": "software",
  "type": "concept",
  "aliases": [
    "Database query optimizer",
    "Query optimiser"
  ],
  "generated_at": "2026-11-04",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/query-optimizer/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/query-optimizer/",
      "json": "https://aiwiki.page/zh/query-optimizer.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "A query optimizer is a component of a database management system that selects how a query should be executed. Given a query, usually expressed in SQL, it considers alternative ways to access data, combine records, and compute results, then chooses a query execution plan. Its purpose is to reduce execution cost while preserving the query’s meaning. The selected plan is generally the best one found within the optimizer’s search space and planning budget, not a guaranteed globally optimal plan."
    },
    {
      "heading": "Role in query processing",
      "text": "SQL is largely a declarative programming language: a query specifies the result required rather than a detailed sequence of operations. The same query can often be executed in many ways, with performance differences of several orders of magnitude.\n\nFor example, a query joining customers to orders might:\n\n- Read both tables in full and combine their rows using a hash join.\n- Find a small set of customers first, then retrieve their orders through an index.\n- Read both inputs in a suitable order and use a merge join.\n\nWhich strategy is preferable depends on table sizes, filtering conditions, available indexes, data distributions, and execution resources.\n\nThe optimizer typically operates after parsing and semantic analysis, which resolve table names, column references, types, and permissions. It produces a plan for the execution engine. This resembles the relationship between optimization and code generation in a compiler, although database optimization depends especially heavily on estimates of data size and distribution.\n\nThe boundaries between query rewriting, optimization, and execution differ across database systems."
    },
    {
      "heading": "Logical and physical plans",
      "text": "Optimizers commonly distinguish between logical operations and physical implementations.\n\nA logical plan describes what operations are required, often using an extended form of relational algebra. Typical operations include filtering rows, selecting columns, joining relations, grouping, and aggregation.\n\nA physical plan specifies how those operations will run. It may choose:\n\n- A full table scan or an access path through a database index.\n- A nested-loop, hash, or merge implementation of a join.\n- Hash-based or sort-based aggregation.\n- An explicit sort or an input access path that already supplies the required ordering.\n- Serial execution or execution using multiple workers.\n- Materialization of intermediate results or pipelined delivery between operators.\n\nPhysical properties influence which alternatives are useful. An input already ordered by a join key, for example, may avoid sorting before a merge join. Consequently, the cheapest way to produce an intermediate result in isolation is not necessarily the cheapest way to produce it for a particular parent operation."
    },
    {
      "heading": "Semantics-preserving transformations",
      "text": "Query optimization relies on transformations that preserve the query’s observable results.\n\nCommon transformations include:\n\nPredicate pushdown. Apply a filter closer to the data source so that fewer rows reach later operators.\n\nProjection pruning. Remove columns that are not needed by subsequent operations, reducing intermediate row width and data movement.\n\nJoin reordering. Change the order of eligible joins to reduce intermediate result sizes or enable better access paths.\n\nSubquery transformation. Convert suitable subqueries into joins or other forms that can be optimized together with the surrounding query.\n\nConstant folding and simplification. Evaluate constant expressions and simplify predicates where semantics permit.\n\nPartition pruning. Exclude table partitions that cannot contain matching rows.\n\nThese transformations have conditions. SQL commonly uses multiset, or “bag,” semantics rather than pure set semantics, so duplicate rows must be preserved unless the query requests their removal. NULL, three-valued logic, outer joins, aggregation, and nondeterministic expressions impose additional restrictions.\n\nFor example, moving a predicate from an outer join’s condition into a subsequent filter can remove unmatched rows and change the result. A declared uniqueness constraint or foreign key may justify a transformation that would otherwise be unsafe. Optimizers must use constraints according to the database system’s rules for their validity and enforcement."
    },
    {
      "heading": "Rule-based and cost-based optimization",
      "text": "Rule-based optimization\n\nA rule-based optimizer applies predefined transformations or priorities. It might prefer an index access path when a suitable index exists, or push eligible filters below joins.\n\nRules are inexpensive to apply and remain fundamental even in sophisticated optimizers. However, fixed priorities cannot reliably account for differences in data size, selectivity, storage layout, and available memory. An index access path that is efficient for a few matching rows may be slower than a full scan when most rows qualify.\n\nCost-based optimization\n\nCost-based optimization compares candidate plans using estimates of resource consumption or execution time. An abstract formulation is:\n\n\\[P^*=\\operatorname*{arg\\,min}_{P\\in\\mathcal{P}(Q)}\\widehat{C}(P),\\]\n\nwhere \\(Q\\) is the query, \\(\\mathcal{P}(Q)\\) is the set of candidate plans considered, and \\(\\widehat{C}(P)\\) is the estimated cost of plan \\(P\\).\n\nCost models may account for:\n\n- Storage reads and writes.\n- Processor work, including expression evaluation and comparisons.\n- Memory requirements and temporary-storage spills.\n- Network communication.\n- Parallel execution overhead.\n- Startup latency and the cost of producing all result rows.\n\nCosts are often expressed in system-specific units rather than seconds. They are primarily intended to rank alternatives.\n\nThe optimization objective also matters. A query requesting only a few rows may benefit from a plan that produces its first results quickly, even if another plan would finish the complete result faster. Resource limits and workload priorities may further constrain plan selection.\n\nModern systems generally combine cost estimates with transformation rules and heuristics."
    },
    {
      "heading": "Cardinality estimation and statistics",
      "text": "Cardinality estimation predicts how many rows each operation will produce. These estimates strongly influence join order, join implementation, memory allocation, and access-path selection.\n\nFor a filter, a basic estimate is:\n\n\\[\\widehat{N}_{\\mathrm{out}}=N_{\\mathrm{in}}\\widehat{s},\\]\n\nwhere \\(N_{\\mathrm{in}}\\) is the estimated input row count and \\(\\widehat{s}\\) is the estimated selectivity, or fraction of rows satisfying the predicate.\n\nOptimizers may use table row counts, distinct-value counts, null fractions, histograms, frequent-value statistics, samples, and multicolumn statistics. Such information forms part of the database’s metadata.\n\nEstimation becomes difficult when data is skewed or columns are correlated. For example, predicates on city and postal code are usually not independent. Multiplying their individual selectivities can substantially underestimate the number of qualifying rows. Estimates can also become inaccurate when statistics are stale or predicates involve expressions whose behavior is difficult to model.\n\nErrors may compound across several joins. A severe underestimate can lead to repeated index lookups over a much larger input than expected, or to a hash operation that exceeds its memory allocation and spills to temporary storage.\n\nMore accurate estimates generally improve decisions, but they do not eliminate errors caused by incomplete plan search or an inaccurate cost model."
    },
    {
      "heading": "Searching the plan space",
      "text": "The number of possible plans grows rapidly with the number of joined relations. Optimizers must balance the potential benefit of further search against planning time and memory consumption.\n\nDynamic programming\n\nDynamic programming can build plans for smaller subsets of relations and combine them into larger plans. An optimizer retains useful alternatives for each subset, potentially distinguishing alternatives by properties such as ordering.\n\nKeeping only one plan per subset can be insufficient: a slightly more expensive intermediate plan may supply an ordering or distribution that saves substantial work later.\n\nMemo-based optimization\n\nMemo-based optimizers organize equivalent logical expressions and alternative physical implementations into shared groups. Transformation rules add alternatives, while optimization tasks explore them under required properties and cost bounds.\n\nThe Volcano and Cascades optimization frameworks are influential examples of this approach. Shared representations avoid repeatedly exploring equivalent subproblems and support extensible collections of rules.\n\nHeuristic and randomized search\n\nFor queries with many joins, exhaustive exploration may be impractical. Optimizers may restrict plan shapes, use greedy methods, limit search time, or apply a randomized algorithm.\n\nRestricting the search can reduce planning overhead but exclude good plans. Optimizer design therefore involves a trade-off between optimization effort and execution quality."
    },
    {
      "heading": "Parallel and distributed optimization",
      "text": "With parallel computing, an optimizer may select worker counts, divide scans, and choose how intermediate results are exchanged. More parallelism does not always improve performance: coordination, communication, and resource contention can outweigh the benefits.\n\nIn distributed computing, data location becomes another major planning concern. A distributed join may:\n\n- Broadcast a small input to the workers holding a larger input.\n- Repartition both inputs by their join keys.\n- Exploit inputs that are already compatibly partitioned.\n\nThe optimizer must consider communication volume, data skew, locality, and available resources. A plan with little total work can still have high elapsed time if one worker receives a disproportionate share of the data.\n\nFederated queries add differences in remote systems’ capabilities and costs. Pushing an operation to a remote source can reduce transferred data, but only when that source supports the required semantics and execution behavior."
    },
    {
      "heading": "Plan caching and adaptive execution",
      "text": "Database systems often cache plans to avoid repeating optimization. Reuse reduces planning overhead, but the same plan may not suit every parameter value. A predicate can be highly selective for one value and match much of a table for another.\n\nSystems address this problem in different ways, including generic reusable plans, parameter-sensitive alternatives, recompilation, and feedback from previous executions. Changes in schema, indexes, or statistics may invalidate cached plans or trigger reconsideration.\n\nAdaptive execution postpones or revisits some decisions using information observed while a query runs. A system may change a join strategy, adjust data exchanges, or reoptimize the remaining work after materializing an intermediate result. Adaptation has costs of its own and cannot necessarily recover work already performed.\n\nResearch also applies machine learning to cardinality estimation, cost prediction, and plan selection. Such methods must cope with changing data, unfamiliar queries, inference overhead, and the need for reliable fallback behavior."
    },
    {
      "heading": "Historical development",
      "text": "Cost-based relational query optimization became a central database technology during the 1970s. A landmark was the 1979 paper Access Path Selection in a Relational Database Management System, describing optimization in IBM System R. It established influential techniques for selecting access paths and join orders using statistics and estimated costs.\n\nSubsequent work broadened the range of transformations, physical operators, and search strategies. Extensible optimization frameworks helped separate the search machinery from database-specific rules. Parallel databases, data warehouses, and distributed analytical engines expanded optimization to include partitioning, communication, and increasingly diverse execution environments."
    },
    {
      "heading": "Inspection and limitations",
      "text": "Database systems commonly provide EXPLAIN-style facilities for inspecting plans. These may show operator choices, estimated row counts, estimated costs, access paths, and ordering or distribution properties. Execution-profiling variants can also report actual row counts and timing, although they generally execute the statement and may therefore have side effects.\n\nComparing estimated and actual row counts helps distinguish estimation errors from other causes of poor performance. Estimated costs from different database products are not generally comparable.\n\nEven a sophisticated optimizer has incomplete information. Runtime conditions such as cache contents, storage latency, available memory, concurrent workload, and network congestion are difficult to predict precisely. User-defined functions and remote data sources may be especially opaque.\n\nSome systems expose hints or plan controls that restrict optimizer choices. These can stabilize a particular workload, but they can also preserve an unsuitable plan after data or hardware changes. Query optimization therefore remains a constrained decision process under uncertainty, rather than a guarantee of the fastest possible execution."
    }
  ],
  "links": [
    "database-management-system",
    "sql",
    "declarative-programming",
    "compiler",
    "database-index",
    "foreign-key",
    "heuristic",
    "metadata",
    "dynamic-programming",
    "randomized-algorithm",
    "parallel-computing",
    "distributed-computing",
    "machine-learning",
    "ibm",
    "data-warehouse"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}