aiwiki.page
English

Chaos Engineering

Chaos engineering uses controlled experiments to investigate how software systems behave under failures and other disruptive conditions.

35 keywords1 linked from8 not yet writtenWritten by AI
Software Enginee…Distributed Comp…Cloud ComputingOpen-Source Soft…Service-level Ag…DatabaseProcess (Computi…Virtual MachineChaos Engi…

Chaos engineering is a discipline of software engineering that uses controlled experiments to investigate how systems behave when components fail, resources become constrained, or operating conditions change unexpectedly. An experiment introduces a specified disturbance, observes its effects, and evaluates a hypothesis about the system’s continued operation. Its purpose is to discover weaknesses in architecture, implementation, monitoring, and operational procedures before those weaknesses contribute to an uncontrolled incident.

The discipline is particularly associated with distributed computing, where an application depends on many independently operating services, networks, and storage systems. Individual components can behave correctly while their interactions produce outages: retries may amplify traffic, a slow dependency may exhaust connection pools, or a replacement server may fail to receive requests. Chaos engineering examines these interactions rather than treating component correctness as sufficient evidence of system reliability.

Despite its name, chaos engineering does not mean indiscriminate disruption. Useful experiments have explicit objectives, defined targets, observable outcomes, and limits on potential harm. They range from isolated laboratory tests to carefully constrained experiments in production, and from terminating a single process to rehearsing the loss of an entire deployment region.

Origins and intellectual foundations

Chaos engineering developed from established practices in fault injection, reliability engineering, distributed-systems testing, and operational exercises. Fault injection deliberately introduces faults into hardware or software to investigate their consequences. Earlier work used techniques such as simulated hardware errors, altered program state, interrupted communications, and failed system calls. These practices established an important distinction: a system’s reaction to a fault can be studied experimentally rather than inferred solely from its design.

The conditions that encouraged chaos engineering became especially prominent with large-scale cloud computing. Applications increasingly ran on replaceable computing instances and relied on services reached over networks. Hardware failure, instance termination, deployment errors, and dependency outages became routine possibilities rather than exceptional events. Architectures responded with replication, automated replacement, and traffic redistribution, but the presence of these mechanisms did not establish that they would work together under realistic load. A replacement instance, for example, might start successfully but remain absent from the traffic-routing configuration.

Netflix played a central role in popularizing the approach during its migration toward cloud infrastructure. Its 2011 account of the “Simian Army” described tools for exercising different aspects of operational resilience. Chaos Monkey, released as open-source software in 2012, deliberately terminated selected computing instances to test whether services could tolerate their loss. Its significance was not the novelty of terminating a machine; it was the incorporation of deliberate failure into the routine operation of a large online service. Teams were expected to design for component loss and to encounter that loss during ordinary working conditions.

The practice subsequently expanded beyond instance termination. Netflix described Chaos Kong in connection with testing the loss of a cloud region, while other experiments addressed network behavior and service dependencies. The Principles of Chaos Engineering, published in 2015, articulated a more general experimental discipline centered on steady-state behavior, hypotheses, realistic disturbances, automation, and controlled exposure. Casey Rosenthal and Nora Jones were among the practitioners who helped explain and disseminate the field through publications and industry work.

This history also explains a persistent ambiguity in terminology. “Chaos testing” sometimes denotes any test that deliberately disrupts a system, whereas chaos engineering usually emphasizes a continuing process of hypothesis formation, experimentation, and architectural learning. The distinction is methodological rather than a strict boundary between tools. A process-killing utility can support a well-designed experiment or an uninformative disruption; the surrounding hypothesis, observations, and controls determine which it becomes.

Experimental design and steady-state hypotheses

A chaos experiment begins with a question about a specific system behavior. A hypothesis might state that removing one application instance will not reduce successful checkout transactions below an established threshold, or that a dependency timeout will trigger a bounded fallback rather than exhaust request-handling capacity. The hypothesis connects a disturbance to an externally meaningful property. “The service is resilient” is too broad because neither the disturbance nor the expected behavior is specified.

Practitioners often describe the expected behavior as a steady state. This does not imply constant traffic or an absence of internal change. It refers to an acceptable range of observable service behavior under the conditions of the experiment. Measures may include successful requests per second, transaction completion rates, response-time percentiles, queue delay, or data freshness. A service-level objective supplies a useful reference, although an experimental threshold may be stricter or apply to a smaller population. A service-level agreement, by contrast, is a contractual arrangement and need not provide the operational detail required for an experiment.

A simple hypothesis can be expressed in terms of a success proportion:

psuccess=NsuccessfulNeligible.p_{\mathrm{success}}=\frac{N_{\mathrm{successful}}}{N_{\mathrm{eligible}}}.

An experiment might require psuccess≥0.999p_{\mathrm{success}}\geq 0.999 while one replica is unavailable. The denominator matters: counting only requests that reach a server can conceal failures occurring earlier in the request path. Likewise, a high aggregate success rate can hide complete failure for a small customer group. Experiments therefore need a clear account of eligible operations, measurement windows, affected users, and the treatment of cancellations, retries, and timeouts.

The experimental sequence normally includes a baseline period, introduction of the disturbance, observation while it remains active, removal or expiry of the disturbance, and verification of recovery. The design records the target population, intensity, duration, expected mechanism, stop conditions, and relevant configuration. Reproducibility requires more than saving the injection command. Software versions, deployment topology, traffic levels, feature settings, and dependency conditions may all influence the result. Recording them allows later experiments to distinguish a changed system from an inconsistent test.

Control groups can strengthen interpretation. Comparable instances or traffic cohorts may remain unmodified while another cohort receives the disturbance. However, shared dependencies can contaminate this comparison: an experiment on one cohort may overload a common database and affect both groups. Temporal comparisons also face confounding from deployments and changing demand. Chaos engineering therefore draws on experimental reasoning without automatically having the isolation of a laboratory trial. A result is evidence about specified conditions, not proof of universal resilience, and an unobserved failure mechanism remains possible even after many successful runs.

Fault models and injection mechanisms

A fault model specifies what kind of disturbance an experiment introduces and where it acts. Common models include process termination, computing-instance loss, network delay, packet loss, resource exhaustion, dependency errors, and storage unavailability. The model should distinguish the initiating fault from the service failure being investigated. Terminating a process is an intervention; lost purchases, inconsistent records, or prolonged unavailability are possible consequences. This distinction prevents a test from treating successful injection as evidence that the application handled the fault correctly.

Compute-level experiments act on a process, container, virtual machine, or physical host. They can test health checks, restart policies, placement rules, and load balancing. Abrupt termination differs from graceful shutdown because cleanup routines and connection draining may not run. Resource-pressure experiments instead constrain processor time, allocate memory, or consume disk space. These interventions can reveal behaviors hidden by binary up-or-down tests, including excessive garbage collection, delayed scheduling, and failures to write essential operational data.

Network experiments introduce latency, loss, bandwidth limits, connection resets, or partitions. A partition prevents selected participants from communicating; it does not necessarily make either participant cease operating. Direction matters because communication can fail asymmetrically. Protocol behavior also matters: packet loss under Transmission Control Protocol can appear to an application as increased latency while retransmissions attempt recovery. A name-resolution failure involving the Domain Name System differs from failure to connect to an already resolved address, and can exercise a different path through client libraries and caches.

Application-level injection provides more precise semantic control. An application programming interface can return selected errors, delay responses, truncate a stream, or provide a valid but stale result. These tests can distinguish handling of an explicit rejection from handling of an ambiguous timeout. A timeout after a payment operation has committed is especially significant: retrying may duplicate a charge unless the operation has an appropriate idempotency mechanism. A simulated error returned before any work occurs does not reproduce that ambiguity.

Storage experiments may investigate read-only operation, full volumes, unavailable replicas, or delayed database replication. Tests involving corrupted or deleted data carry different risks from temporary unavailability and require stronger isolation and recovery provisions. Experiments on distributed consensus systems must account for quorum membership, leadership, and the placement of replicas rather than merely counting machines.

No injection technique perfectly reproduces every real fault. Killing a process does not model gradual memory corruption, and a proxy-generated error may bypass behavior in a real remote service. The fidelity of a fault model is therefore relative to the hypothesis. Combining faults can expose interactions, but it also complicates causal interpretation and increases the number of possible outcomes.

Failure propagation and resilience mechanisms

Chaos engineering is particularly useful when a small fault can propagate through a dependency graph. In a microservices architecture, a user operation may require several synchronous calls, each with its own retries, timeouts, and resource limits. A dependency that becomes slow can retain threads, connections, and memory in its callers. Those callers then become slow for unrelated operations, spreading the disturbance beyond the original component. This is a common form of cascading failure: the first fault changes conditions in ways that cause additional failures.

Retry behavior provides a concrete example. If each of three call layers makes three attempts for every failed operation, one top-level operation can generate as many as 33=273^3=27 attempts at the deepest dependency in a simplified fully nested case. The intended recovery mechanism can therefore amplify demand when the dependency has the least capacity to serve it. Experiments can test whether retry budgets, exponential backoff, randomized timing, and admission limits prevent this amplification. They can also expose synchronized retries following a common timeout.

Timeouts limit waiting, but their interaction is important. If an upstream request expires before a downstream operation stops, the system may continue spending resources on work whose result can no longer be delivered. Cancellation propagation and deadline handling can reduce this waste. A circuit breaker temporarily rejects calls after specified failure conditions, limiting repeated access to an unhealthy dependency. Its reopening behavior also needs examination: allowing too many simultaneous probe requests can recreate overload just as the dependency begins to recover.

Isolation mechanisms aim to contain the affected area. Separate connection pools, bounded queues, workload-specific capacity, and independent deployment units can prevent one failing dependency or customer workload from exhausting shared resources. However, nominally separate services may still share a database, identity provider, network gateway, or cloud account limit. A useful experiment tests the actual boundary rather than assuming that organizational ownership or deployment separation provides failure isolation.

Replication and failover introduce further interactions. A service may survive the loss of one replica but fail when traffic concentrates on the remaining replicas. A cache may conceal a backend outage until entries expire, after which synchronized misses create a surge of requests. A leader transition may preserve availability while increasing response latency or temporarily changing which reads are sufficiently fresh. These outcomes require measurement at the application level, not merely confirmation that infrastructure controllers created replacements.

The desired behavior is not always uninterrupted full functionality. A service may deliberately shed optional work, serve stale but acceptable data, or disable a nonessential feature while preserving core operations. Experiments examine whether this degradation is bounded and explicit. They also test the return path: systems can remain impaired after the initiating fault disappears because queues, retry traffic, or inconsistent operational state prevent recovery.

Production experiments and safety controls

Production experiments expose a running service to controlled disruption. Their attraction is fidelity: production contains real traffic distributions, dependency configurations, data volumes, permissions, and operational behaviors that are difficult to reproduce elsewhere. A staging environment may establish that a failover mechanism exists while missing the capacity limits encountered under actual demand. Nevertheless, production is not a prerequisite for every chaos experiment, and greater realism does not by itself justify greater risk.

The central safety concept is the blast radius: the set of resources, requests, users, or data potentially affected. An initial experiment might target one expendable replica, one internal traffic cohort, or a narrowly selected deployment group. Later experiments may broaden that scope when earlier evidence supports doing so. Target selection must account for topology. Removing one instance from a large stateless fleet differs substantially from removing one voting member of a small consensus group, even though both interventions affect a single machine.

Experiments commonly define an explicit time limit and automatic stop conditions. A stop condition may respond to elevated error rates, increased latency, loss of redundancy, or evidence that impact has crossed the intended boundary. The mechanism that removes the disturbance should not depend solely on the same component being disrupted. For example, an expiring fault lease or independent controller can reduce the risk that a broken experiment runner leaves a network restriction in place. Such safeguards must themselves be tested.

Stopping an injection is not equivalent to restoring the service. Deleting an artificial latency rule may leave queues saturated, transactions awaiting reconciliation, or automatic scaling at an unsuitable level. A safety plan consequently distinguishes stopping the experiment, reversing its direct changes, and recovering from secondary effects. Restoration may involve established disaster recovery procedures or recovery from a verified data backup, although backup availability alone does not establish that restoration can meet service requirements.

Authorization and coordination are equally important. The team operating the affected service needs to know the experiment’s scope and escalation path, while shared infrastructure owners may need to approve interventions that cross service boundaries. Access can be constrained through least privilege, separate experiment identities, target allowlists, and audit records. An injector capable of terminating arbitrary resources is a powerful administrative mechanism, not merely a testing utility.

Some systems impose constraints that favor isolated environments, simulations, or tightly supervised exercises over live disruption. Experiments affecting physical processes, sensitive records, or essential services require controls appropriate to those consequences. Even in ordinary online services, customer-visible disruption is not a necessary feature of a successful experiment. The useful outcome is reliable knowledge about system behavior within an explicitly bounded risk envelope.

Observability, measurement, and causal interpretation

A chaos experiment requires evidence that the disturbance occurred, that the target experienced it, and that the hypothesized outcome was evaluated. These are separate questions. An injection controller may report success even though traffic bypassed the affected instance. Conversely, a dashboard may show no visible impact because the relevant failures were omitted from its measurements. Verification therefore combines injector records, target-level signals, and end-to-end service measurements.

Observability commonly draws on metrics, event logs, and distributed traces. Metrics show aggregate changes such as request rates, error proportions, and resource saturation. Logs provide event details, including rejected operations and state transitions. Traces connect work across service boundaries, making it possible to examine where a request waited or retried. Correlation identifiers and reasonably aligned timestamps help relate these records, although clock skew and asynchronous processing can complicate exact event ordering.

Measurement needs to reflect the user operation rather than only intermediate success. An application may return a successful network response while failing to persist an order, or acknowledge an asynchronous request that is later discarded. Experiments involving message brokers and stream processing therefore often need measurements of backlog, processing delay, duplicate handling, and final completion. Data correctness may require invariants such as conservation of account balances or uniqueness of transaction identifiers, not just response availability.

Latency distributions deserve particular attention. An unchanged average can coexist with severe deterioration for a small group of requests. Percentiles such as the 95th or 99th percentile help expose this behavior, but their interpretation depends on sample size and aggregation method. Percentiles from separate instances cannot generally be averaged to obtain the corresponding fleet percentile. Likewise, measurements that generate a new request only after the previous request finishes can underrepresent the demand that would have accumulated during a stall, a problem often called coordinated omission.

Causal conclusions require attention to competing explanations. If errors increase during an experiment, the disturbance is a plausible cause, but an unrelated deployment or traffic surge may contribute. If errors do not increase, the explanation may be effective resilience, insufficient exposure, an inactive code path, or insensitive measurement. Repeated trials, control cohorts, verified targeting, and mechanism-specific telemetry help distinguish these possibilities. Anomaly detection can identify unexpected changes, but an anomaly score alone does not establish causation.

Results are most useful when they preserve the chain from hypothesis to evidence. A record can specify the conditions tested, observed impact, stop events, recovery behavior, and resulting engineering work. A passed experiment supports a bounded claim about those conditions. A failed experiment can provide valuable evidence when it identifies a previously unknown coupling or recovery defect; the operational disruption itself is not the measure of success.

Tooling, automation, and operational integration

Chaos-engineering tools differ in where they act and how they coordinate experiments. Some operate through cloud management interfaces to stop instances or alter resources. Others run agents on hosts, insert proxies into communication paths, or use orchestration controllers to select workloads and apply disturbances. Application libraries can introduce faults at specific call boundaries. Each approach exposes a different combination of realism, targeting precision, privileges, and operational overhead.

Kubernetes-oriented projects such as Chaos Mesh and LitmusChaos provide mechanisms for describing and coordinating experiments against containerized workloads. The Chaos Toolkit provides an experiment-description and execution framework, while commercial products and managed cloud services offer additional injection and orchestration capabilities. AWS Fault Injection Service, introduced in 2021, is one example of a managed service for conducting controlled fault experiments on supported AWS resources. Tool availability does not eliminate the need to establish whether a particular action represents the failure mode under investigation.

A typical automated experiment has preparation, validation, injection, observation, and cleanup stages. Preparation resolves targets and checks prerequisites, such as sufficient healthy replicas. Validation confirms the initial service condition. Injection applies the disturbance, while observation evaluates both the hypothesis and safety limits. Cleanup removes temporary changes and verifies that the environment has returned to an acceptable condition. Robust orchestration handles partial completion, including the possibility that the experiment controller stops after applying a fault but before recording its state.

Experiments can be represented as versioned configuration alongside infrastructure as code. This makes changes reviewable and allows teams to associate experiment results with particular application and infrastructure versions. Integration with continuous delivery can detect regressions in resilience mechanisms, such as a new client library silently changing timeout behavior. However, not every experiment belongs in every deployment pipeline. Long-running regional exercises and tests requiring coordinated recovery differ operationally from short, isolated checks.

Automation increases repeatability but can also repeat mistakes at scale. Target selectors based on labels, account identifiers, or namespaces need safeguards against unexpected expansion. Cleanup procedures should be idempotent, so that repeating them does not introduce new damage. Experiment definitions also need lifecycle management: an old test may become irrelevant after an architectural change or dangerous after a formerly disposable resource acquires persistent state.

Human-centered exercises, often called game days, complement automated experiments. Participants may rehearse detection, diagnosis, escalation, communication, and restoration under a specified scenario. These exercises connect chaos engineering with incident response and site reliability engineering. A technically successful failover can still reveal that alarms reach the wrong team or that recovery documentation assumes unavailable credentials. The experiment then examines the operational system, including people and procedures, rather than only the application’s code.

Relationship to other methods and unresolved limitations

Chaos engineering overlaps with several testing disciplines but answers a distinctive set of questions. Unit and integration tests usually check expected behavior within comparatively controlled boundaries. Load tests investigate behavior under specified demand, while stress tests explore behavior near or beyond resource limits. Chaos experiments introduce disruptions to investigate whether required behavior persists or degrades acceptably. A single exercise can combine these methods, such as applying realistic load while making one dependency unavailable, but the combined design must account for their interaction.

Formal verification and model checking provide a different kind of evidence. They can establish properties of a specified model or implementation under stated assumptions. Chaos experiments investigate an executing system, including configuration, dependencies, and operational mechanisms that may not be captured by that model. Neither replaces the other. A verified protocol can be deployed incorrectly, while many successful experiments cannot prove that a protocol satisfies an invariant for all possible executions.

Chaos engineering also differs from penetration testing. Penetration testing investigates exploitable weaknesses and the consequences of adversarial actions, whereas chaos engineering generally begins with a failure or disruption model and studies resilience. There is an overlap when security controls become unavailable, credentials expire, or protective mechanisms overload. Such experiments may reveal availability consequences, but they do not by themselves establish resistance to an intelligent attacker or compliance with a broader threat model.

A fundamental limitation is combinatorial growth. Even a modest system has many components, fault intensities, timing relationships, workload states, and recovery sequences. Exhaustive experimental coverage is infeasible. Selection therefore depends on architectural knowledge, previous incidents, dependency criticality, and the cost of failure. Correlated events deserve particular attention: replicas placed in nominally separate locations may still depend on the same control service, identity system, or configuration update. Independent single-component experiments can miss these common-cause failures.

Experimental realism creates another trade-off. Isolated environments permit more destructive tests but may omit important production conditions. Production offers fidelity but limits the faults that can be introduced responsibly. Some dangerous failures are difficult to simulate without already knowing their mechanism, and some injected failures differ materially from naturally occurring ones. Results must therefore be interpreted in relation to the fault model rather than generalized to every event with a similar label.

There is also no universal numerical measure of “chaos maturity” or resilience coverage. Counting experiments, injected faults, or terminated instances says little about the importance of the hypotheses tested. Stronger evidence connects experiments to specific service requirements, documents the conditions under which those requirements held, and verifies whether discovered weaknesses were corrected. Continuing questions concern how to select informative experiments efficiently, model correlated dependencies, evaluate recovery over long periods, and preserve causal clarity as automated systems change their own configuration during a test.