{
  "id": "mutation-testing",
  "title": "Mutation Testing",
  "lang": "en",
  "source_lang": "en",
  "summary": "Mutation testing evaluates software tests by introducing controlled changes into a program and measuring whether the tests detect them.",
  "category": "computer-science",
  "subcategory": "software",
  "type": "concept",
  "aliases": [
    "Mutation analysis",
    "Program mutation testing"
  ],
  "generated_at": "2026-11-05",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/mutation-testing/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/mutation-testing/",
      "json": "https://aiwiki.page/zh/mutation-testing.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "Mutation testing is a method of evaluating software tests by deliberately modifying a program and checking whether its tests detect the changes. Each modified version, called a mutant, usually contains one small alteration, such as replacing an arithmetic operator or reversing a condition. A test suite that distinguishes the mutant from the original program is said to kill that mutant. Mutation testing therefore examines the fault-detection capability of tests rather than merely whether they execute particular parts of a program."
    },
    {
      "heading": "Basic principle",
      "text": "A test suite can execute every statement in a program and still fail to detect incorrect behavior. For example, a test might call a function without checking its result, or check only inputs for which an incorrect implementation happens to behave correctly.\n\nMutation testing probes this weakness by introducing controlled changes through mutation operators. An operator is a rule for generating variants of a program, such as:\n\n- Replacing + with -.\n- Replacing < with <=.\n- Negating a Boolean condition.\n- Removing a method call.\n- Changing a return value to zero, an empty collection, or another predefined value.\n- Replacing a constant with a nearby value.\n\nMost conventional mutation testing uses first-order mutants, each produced by one application of a mutation operator. Mutants are normally tested independently, not accumulated into a progressively modified program.\n\nIf a test passes on the original program but fails on a mutant, it provides evidence that the suite detects that particular behavioral change. A surviving mutant identifies a change the tests did not distinguish, although it does not necessarily indicate a missing test: the change may preserve all relevant behavior.\n\nExample\n\nConsider a function written in Python:\n\ndef qualifies(age):\n    return age >= 18\n\nA mutation operator might produce:\n\ndef qualifies(age):\n    return age > 18\n\nTests checking only ages 17 and 19 pass on both versions. A test asserting that qualifies(18) is true kills the mutant.\n\nThis example illustrates the difference between execution and detection. The original tests execute the comparison, but they do not establish the intended behavior at its boundary."
    },
    {
      "heading": "Workflow and outcomes",
      "text": "A typical mutation-testing workflow consists of the following steps:\n\n- Establish a baseline. Run the unmodified program’s tests and confirm that the relevant tests pass.\n- Choose a scope. Select the code, mutation operators, and tests to include.\n- Generate mutants. Apply the selected operators to source code, an intermediate representation, or executable instructions.\n- Execute tests. Run each mutant against the relevant tests, usually stopping once a test kills it.\n- Classify results. Separate detected changes from survivors and execution problems.\n- Inspect useful survivors. Determine whether they expose missing inputs, inadequate checks, equivalent behavior, or an inappropriate mutation.\n\nCommon result categories include:\n\nOutcome | Meaning\nKilled | At least one relevant test detects the mutant.\nSurvived | Executed tests do not detect the mutant.\nNo coverage | No selected test executes the mutated location.\nInvalid | The generated mutant cannot be compiled or otherwise executed validly.\nTimed out | Execution exceeds a configured limit, potentially because the mutation creates nontermination.\nExecution error | Infrastructure or runtime problems prevent a reliable result.\n\nThese categories are tool-dependent. Some tools count timeouts as detected mutants; others report them separately. A timeout caused by an introduced infinite loop differs from one caused by an overloaded test machine, so reporting conventions matter."
    },
    {
      "heading": "Mutation score",
      "text": "The mutation score summarizes the proportion of relevant mutants detected by a test suite. A conceptual definition is:\n\n\\[\\operatorname{MS}=\\frac{K}{V-E}\\times 100\\%,\\]\n\nwhere:\n\n- \\(K\\) is the number of killed mutants;\n- \\(V\\) is the number of valid generated mutants;\n- \\(E\\) is the number of valid mutants equivalent to the original program.\n\nFor example, if 100 valid mutants include 10 equivalent mutants and tests kill 72, the score is \\(72/90=80\\%\\).\n\nIn practice, the exact number of equivalent mutants is usually unknown. Tools therefore often report an operational score based on all valid mutants, or on mutants remaining after explicit exclusions. Scores may also differ in their treatment of uncovered code, timeouts, and ignored mutations.\n\nConsequently, mutation scores are not directly comparable unless the mutation operators, program scope, test selection, exclusions, and classification rules are sufficiently similar. A score of 100% means that every counted mutant was detected; it does not establish that the program is correct or that all possible faults would be detected."
    },
    {
      "heading": "Strong, weak, and higher-order mutation",
      "text": "Strong mutation\n\nIn strong mutation testing, a test kills a mutant when the mutation leads to a difference in externally observed behavior that the test detects.\n\nSuccessful detection typically requires four conditions:\n\n- Reachability: execution reaches the mutated location.\n- Infection: the mutation creates a different program state.\n- Propagation: that difference reaches an observable result.\n- Revelation: a test check recognizes the result as incorrect.\n\nExecuting a mutated expression is therefore insufficient. An intermediate difference might be overwritten, canceled by later computation, or ignored by the test’s assertions.\n\nWeak mutation\n\nWeak mutation testing checks whether a mutation changes an internal state shortly after the mutated operation, rather than requiring the difference to propagate to a program output.\n\nThis can reduce execution cost or support test generation, but it answers a narrower question: whether the tests create an internal behavioral difference, not necessarily whether their final checks expose it. Intermediate variants, sometimes called firm mutation, observe behavior at a point between the immediate mutation site and final output.\n\nHigher-order mutation\n\nA higher-order mutant combines two or more mutations. Such combinations can model interacting changes and sometimes produce mutants that are harder to detect than their constituent first-order mutants.\n\nHigher-order mutation also greatly expands the search space. Research includes methods for selecting combinations that provide useful information without exhaustively enumerating them."
    },
    {
      "heading": "Equivalent mutants",
      "text": "An equivalent mutant behaves identically to the original program for every input and execution permitted by the adopted program model. No behavioral test can distinguish it within that model.\n\nFor example, replacing x > 0 with x >= 1 preserves the condition’s meaning when x is restricted to integers. The same replacement need not preserve meaning for floating-point values.\n\nEquivalence depends on the language semantics, allowed inputs, and observable behavior. A change may be irrelevant under an application’s input contract but observable when that contract is violated. Mutants may also be indistinguishable from one another without being equivalent to the original program, creating redundancy in the mutation set.\n\nGeneral program equivalence is undecidable for sufficiently expressive programming languages, a limitation related to the halting problem. Automated tools can eliminate some equivalent mutants through syntactic rules, compiler analysis, or restricted formal verification, but cannot solve the problem universally.\n\nA surviving mutant therefore requires interpretation. It may indicate a genuine test weakness, an equivalent transformation, an infeasible execution path, or behavior outside the intended requirements."
    },
    {
      "heading": "Relationship to other testing techniques",
      "text": "Code coverage\n\nCode coverage measures which program structures a test suite executes, such as statements or branches. Mutation testing asks whether tests detect selected changes to those structures.\n\nHigh coverage does not guarantee a high mutation score: tests can execute code without checking meaningful results. Mutation testing nevertheless depends on execution coverage, since an ordinary execution-based test cannot detect a mutation in code it never reaches. Coverage information is also useful for selecting which tests to run against each mutant.\n\nTest oracles\n\nA test oracle determines whether observed behavior is acceptable. Mutation testing can reveal weak oracles: a test may supply an appropriate input and exercise the relevant operation but fail to assert the property affected by a mutant.\n\nKilling a mutant does not itself establish that the original behavior is correct. It shows that a test distinguishes the original from a particular alternative, under the assumption that the test’s expected behavior is justified.\n\nFault injection and fuzzing\n\nMutation testing is related to fault injection, but its conventional purpose is to assess test adequacy by changing a program. Fault injection more broadly includes introducing events such as network failures or resource exhaustion to study system behavior.\n\nIn fuzz testing, mutation often means changing inputs to generate new test cases. Conventional mutation testing changes the program under test instead. Both uses employ controlled variation, but their targets and immediate objectives differ."
    },
    {
      "heading": "Computational cost and practical techniques",
      "text": "A straightforward implementation executes an entire test suite separately for every mutant. For \\(m\\) mutants and a test-suite runtime of \\(t\\), test execution alone can approach \\(m t\\), excluding generation, compilation, and setup costs.\n\nCommon cost-reduction techniques include:\n\n- Coverage-based test selection: running only tests that can reach the mutation.\n- Early termination: stopping a mutant’s test run after its first reliable failure.\n- Selective mutation: using a restricted set of mutation operators.\n- Sampling: evaluating a subset of generated mutants.\n- Incremental analysis: concentrating on changed code and affected tests.\n- Parallel execution: testing independent mutants concurrently.\n- Mutant schemata: embedding multiple alternative mutations into an instrumented program and selecting among them during execution.\n- Static filtering: removing some invalid, duplicate, or provably equivalent mutants before testing.\n\nThese techniques involve trade-offs. Sampling introduces uncertainty, selective operators narrow the fault model, and concurrent execution may increase resource contention. Shared state, nondeterministic behavior, or tests whose results depend on execution order can also make outcomes unreliable.\n\nMutation analysis can be integrated into continuous integration, but its cost often favors a limited scope, such as recently changed modules, rather than an unrestricted analysis on every change."
    },
    {
      "heading": "History and rationale",
      "text": "Mutation testing emerged as a research approach in the 1970s. Early foundational work includes Richard Hamlet’s 1977 work on compiler-assisted testing and Richard DeMillo, Richard Lipton, and Fred Sayward’s 1978 paper on selecting test data.\n\nTwo ideas are commonly associated with its rationale:\n\n- The competent programmer hypothesis proposes that programmers generally produce programs close to their intended implementations, making small syntactic changes useful models of some faults.\n- The coupling effect proposes that tests capable of detecting simple faults will also tend to detect many more complex faults composed of, or associated with, those simpler faults.\n\nThese ideas motivate the technique rather than guarantee its effectiveness for every system. The relationship between detecting artificial mutants and detecting real defects depends on the mutation operators, program, tests, and kinds of defects involved."
    },
    {
      "heading": "Applications and limitations",
      "text": "Within software engineering, mutation testing is used to assess regression tests, locate gaps in assertions and boundary cases, compare test-generation methods, and guide automated test generation. It can provide a more discriminating experimental measure than code coverage alone.\n\nIts central limitation is that generated mutations represent a chosen fault model. Common operators may expose mistakes in arithmetic or branching while doing little to model missing functionality, architectural errors, misunderstood requirements, or failures caused by interactions with external systems.\n\nMutation testing also cannot supply an independent specification. Adding an assertion solely to kill a mutant may encode incidental implementation behavior rather than a required property. A useful interpretation of a survivor therefore relates the changed behavior to an actual requirement, not merely to the objective of increasing a score."
    }
  ],
  "links": [
    "python-programming-language",
    "source-code",
    "halting-problem",
    "compiler",
    "formal-verification",
    "parallel-computing",
    "software-engineering"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}