{
  "id": "latency",
  "title": "Latency",
  "lang": "en",
  "source_lang": "en",
  "summary": "Latency is the elapsed time between a defined initiating event and its corresponding response, reflecting propagation, processing, waiting, and communication delays.",
  "category": "computer-science",
  "subcategory": "networks-security",
  "type": "concept",
  "aliases": [
    "Response latency",
    "time delay"
  ],
  "generated_at": "2026-11-06",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/latency/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/latency/",
      "json": "https://aiwiki.page/zh/latency.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "Latency is the elapsed time between a specified initiating event and a corresponding observable result. In computer science, it may describe the time required to execute an instruction, retrieve data from memory, deliver a network packet, complete a service request, or display a response to user input. Its meaning depends on precisely defined endpoints: a network response arriving at a device is not the same event as the application processing that response or the screen displaying its contents.\n\nLatency matters because computing systems coordinate activities across components that operate at different speeds and distances. A processor may perform operations in nanoseconds while an intercontinental exchange takes tens or hundreds of milliseconds. Waiting can dominate the execution of otherwise efficient software, and occasional long delays can determine whether a service meets its requirements. Understanding latency therefore involves physical limits, hardware organization, communication protocols, scheduling, statistical measurement, and application design."
    },
    {
      "heading": "Definitions, boundaries, and related measures",
      "text": "A latency measurement requires a starting event, an ending event, and a time base. For a storage read, the interval might begin when an application submits a request and end when the requested bytes become available to that application. Alternatively, a device manufacturer might measure only the interval from command receipt to completion inside the device. Both are meaningful, but they describe different boundaries. Comparisons become misleading when measurements bearing the same label include different portions of a system.\n\nResponse time commonly includes both waiting and active service. A request may wait for a worker, execute on a processor, wait for storage, and then return through a network. Service time usually refers to time spent receiving service at a particular resource, although conventions differ across fields. End-to-end latency includes all activities between the chosen endpoints. It is not necessarily the sum of every activity performed: operations that run concurrently overlap, and only activities on the completion-determining dependency path contribute directly to the final interval.\n\nLatency differs from throughput, the rate at which work completes, and from bandwidth, the capacity or transfer rate of a communication channel. A system can have high throughput and high latency simultaneously. A freight train illustrates the distinction: it can transport a large volume per hour while taking a long time to deliver any one shipment. In computing, a deeply pipelined processor or a large batch-processing service can likewise complete many operations per second without completing an individual operation quickly.\n\nUnits vary with scale. Processor and memory operations are often discussed in nanoseconds; local device operations in microseconds or milliseconds; wide-area communication in milliseconds; and long-running jobs in seconds or longer. These are conventions rather than fixed boundaries. A processor cycle lasts \\(1/f\\) seconds at frequency \\(f\\), so a 3 GHz clock has a cycle period of approximately \\(0.333\\) nanoseconds. An instruction taking several cycles has a latency distinct from the interval at which the processor can begin independent instructions of the same kind.\n\nLatency also differs from a deadline and from a timeout. A deadline specifies when an operation must finish to satisfy a requirement; a timeout specifies when a component stops waiting or takes another action. Neither directly measures how long the underlying operation actually takes. A timed-out request may continue executing remotely unless cancellation reaches and stops it. Similarly, “real-time” describes a relationship between correctness and timing constraints, not merely a claim that a system is fast."
    },
    {
      "heading": "Physical limits and network delay",
      "text": "In a packet network, end-to-end delay is conventionally separated into propagation, transmission, processing, and queueing components. Propagation is the time a signal takes to traverse a medium. Transmission, also called serialization delay, is the time required to place a packet’s bits onto a link. Processing includes activities such as inspecting headers and determining a forwarding destination. Queueing is the time a packet waits before a resource can serve it. A simplified path model is\n\n\\[T_{\\mathrm{path}}=\\sum_i\\left(\\frac{d_i}{v_i}+\\frac{L_i}{R_i}+p_i+q_i\\right),\\]\n\nwhere \\(d_i\\) is link distance, \\(v_i\\) propagation speed, \\(L_i\\) transmitted length, \\(R_i\\) link rate, and \\(p_i\\) and \\(q_i\\) processing and queueing delays.\n\nPropagation imposes a physical floor. The speed of light in vacuum is exactly \\(299{,}792{,}458\\) metres per second; signals in typical optical fibre travel at approximately \\(2\\times10^8\\) metres per second. A 1,000 km fibre route therefore requires about 5 milliseconds in one direction before equipment and other delays are included. Actual cable paths are longer than straight-line geographic distances. Faster processors or higher link capacities do not remove this distance-dependent component.\n\nTransmission delay depends on packet length and link rate rather than route length. Serializing 1,500 bytes at 1 gigabit per second takes 12 microseconds, excluding additional framing and physical-layer overhead. At 10 megabits per second, the same bytes take 1.2 milliseconds. In a store-and-forward packet-switching network, a device may receive a packet before forwarding it, introducing serialization delay at successive links. Cut-through equipment can begin forwarding earlier, but still cannot eliminate propagation or the time required to transmit all the bits.\n\nRound-trip time, or RTT, measures an outward journey and a return journey together with whatever intervening processing the measurement includes. The familiar ping utility estimates a round-trip exchange, not application completion time. One-way delay cannot generally be inferred by dividing RTT by two because routes, traffic conditions, and scheduling can differ by direction. A router may also handle diagnostic traffic differently from ordinary application traffic.\n\nNetwork latency varies with congestion, route selection, wireless contention, retransmissions, and link-layer scheduling. A satellite connection illustrates the distinction between geography and implementation: a geostationary satellite orbits about 35,786 km above Earth, so a path up to that satellite and back down introduces substantial propagation delay even before terrestrial routing is considered. Low-Earth-orbit systems reduce orbital propagation distance, but their total delay also depends on ground-station placement, routing, and network load."
    },
    {
      "heading": "Queueing, utilization, and latency distributions",
      "text": "Queueing explains why latency often rises sharply as a system approaches capacity. When requests arrive at a busy resource, they wait until earlier work completes. Average utilization alone does not fully determine waiting time: variability in arrival intervals and service durations also matters. A resource with irregular bursts or occasional long operations can produce long waits even when its average load appears moderate. Scheduling policies determine which requests absorb those waits.\n\nQueueing theory provides mathematical models of this behaviour. In the idealized M/M/1 model, arrivals follow a Poisson process, service times are independent and exponentially distributed, and one server handles requests. If the arrival rate is \\(\\lambda\\) and the service rate is \\(\\mu\\), with \\(\\lambda<\\mu\\), mean time in the system is\n\n\\[W=\\frac{1}{\\mu-\\lambda}.\\]\n\nWith \\(\\mu=1{,}000\\) requests per second, increasing arrivals from 500 to 900 requests per second raises mean response time from 2 to 10 milliseconds. These values describe the model, not a universal prediction for real servers, but the rapid rise near saturation is broadly important.\n\nLittle’s law, established by John D. C. Little in 1961, relates average population, arrival rate, and time in a stable system:\n\n\\[N=\\lambda W.\\]\n\nHere \\(N\\) is the average number of items within the chosen boundary, \\(\\lambda\\) the long-run average arrival rate, and \\(W\\) the average time spent inside. The law does not require exponential service times. Its usefulness depends on consistent boundaries and compatible averaging assumptions: counting only executing requests while measuring latency that includes queued requests would mix different systems.\n\nLatency is therefore usually characterized as a distribution rather than a single number. The median describes the middle observation; the 95th, 99th, or 99.9th percentile describes increasingly rare delays. Tail latency refers to this slow end of the distribution. A service can have an acceptable mean while a significant minority of requests experience very long waits. Percentiles from different machines or time windows cannot generally be averaged to obtain the percentile of their combined observations.\n\nVariation in delay is often called jitter, especially in communications, although different standards define its exact statistic differently. In interactive media, variation can be as disruptive as a consistently longer delay because packets no longer arrive at the expected rhythm. Queues can absorb bursts, but they exchange packet loss or rejected work for additional waiting. Bufferbloat describes excessive delay caused by persistently full network buffers. It demonstrates why adding buffering can improve loss measurements while worsening responsiveness."
    },
    {
      "heading": "Processors, memory, and storage",
      "text": "At the hardware level, latency measures the time between making a request or issuing an operation and obtaining a usable result. A central processing unit distinguishes instruction latency from reciprocal throughput. An execution unit might accept an independent operation every cycle even though each operation takes several cycles to produce its result. A chain of dependent instructions must wait for successive results, whereas independent instructions may overlap through pipelining and out-of-order execution.\n\nThe memory hierarchy exploits differences in capacity, cost, bandwidth, and access latency. Registers and nearby caches offer rapid access to small working sets; main memory provides much larger capacity at greater delay; persistent storage is generally slower still. A hit in a small CPU cache may take only a few processor cycles, while a main-memory access on many contemporary systems takes tens to more than a hundred nanoseconds. Exact values depend on processor design, clocking, memory configuration, access pattern, and contention.\n\nAn application's effective memory latency can exceed a device’s basic access time. A virtual address may require translation, possibly involving page-table accesses. A cache miss may require ownership changes under a cache-coherence protocol. In non-uniform memory access systems, memory attached to another processor socket takes a different path from local memory. A page fault may invoke the operating system, and, if data must be retrieved from storage, the delay can increase by several orders of magnitude.\n\nStorage combines physical access and software overhead. A rotating disk must position its head and wait for the desired sector to rotate into place. At 7,200 revolutions per minute, one revolution takes about 8.33 milliseconds, yielding an average rotational wait of about 4.17 milliseconds under a uniform-position assumption, before seek and transfer costs. Flash memory avoids mechanical motion but still has different read, program, and erase behaviours. Controller scheduling, internal cleanup, error correction, and device queues affect observed latency, especially at high percentiles.\n\nHardware and software often hide latency rather than remove it. Prefetching requests data before it is needed; out-of-order execution performs unrelated work while an operation waits; multiple outstanding storage or memory requests improve overlap. Such techniques work poorly for strict dependency chains, such as following pointers whose next address is unknown until the current load completes. The distinction is central to performance analysis: increasing concurrency may raise throughput while leaving the latency of a single dependent access unchanged or even increasing it through contention."
    },
    {
      "heading": "Operating systems and application execution",
      "text": "Between hardware service and application-visible completion lie scheduling, synchronization, and runtime activities. A runnable thread may wait for processor time before it can execute. A blocked thread may need to be awakened and scheduled after an input/output operation completes. Interrupt handling, kernel work, and context switching add further intervals. Consequently, a device completing an operation does not imply that the waiting application immediately observes completion.\n\nCPU scheduling determines how competing tasks share execution resources. Time-sharing policies balance responsiveness and fairness, while real-time policies may prioritize tasks according to fixed priorities or deadlines. Priority inversion occurs when a high-priority task waits for a resource held by a lower-priority task that is itself delayed by other work. Priority inheritance can reduce this problem by temporarily raising the holder’s priority. These mechanisms address scheduling-related delay rather than making the protected operation intrinsically faster.\n\nSynchronization introduces additional waiting. A mutex can serialize access to a shared resource, and contention can turn a short protected operation into a substantial latency bottleneck. A spinlock consumes processor time while waiting, whereas a blocking lock can involve suspension and rescheduling. Neither approach is universally faster: the result depends on lock duration, contention, processor availability, and execution context. False sharing, in which independent variables occupy the same cache line, can also introduce coherence traffic and delay without explicit lock contention.\n\nLanguage runtimes contribute characteristic latency patterns. Garbage collection may suspend application threads for some phases, although collector designs differ greatly in pause behaviour. Just-in-time compilation can make early requests slower than later ones. Loading classes, mapping files, establishing connections, and populating caches create warm-up effects. In serverless computing, a cold invocation may additionally require preparing an execution environment, whereas a warm invocation can reuse an existing one. These are distinct operating conditions and are normally reported separately when relevant.\n\nAsynchronous programming allows a thread to do other work rather than block during an operation, but it does not automatically shorten that operation’s elapsed time. An event loop can serve many outstanding requests efficiently while still becoming unresponsive if a callback performs lengthy synchronous computation. Likewise, moving work to a worker pool introduces a queue and a handoff. Application latency reflects both the work performed and the architecture through which completion is delivered; apparent “idle time” may represent necessary waiting on an external dependency rather than unused optimization potential."
    },
    {
      "heading": "Distributed services, databases, and critical paths",
      "text": "In distributed computing, an operation often consists of several communications and local computations whose dependencies define a critical path. Three sequential remote calls each taking 20 milliseconds contribute approximately 60 milliseconds before local overhead. If those calls are independent and issued concurrently, their contribution is closer to the slowest completion time. Parallelism changes the dependency structure, but shared-resource contention can offset the expected improvement.\n\nFan-out makes tail latency especially important. Suppose an aggregator waits for all \\(n\\) independent, identically distributed subrequests, and each subrequest finishes within time \\(t\\) with probability \\(F(t)\\). The probability that all finish within that time is\n\n\\[P(T_{\\max}\\leq t)=F(t)^n.\\]\n\nIf each has a 99% chance of completing within a threshold, 100 such requests have only about a 36.6% chance of all meeting it. Real services often violate the independence assumption because requests share networks, hosts, or storage, but the example explains why a moderately rare delay can become common at the user-facing level.\n\nA database transaction can include query execution, lock acquisition, log persistence, and communication with replicas. Acknowledging a write after it reaches volatile memory has different latency and durability implications from acknowledging it only after durable storage. Write-ahead logging places log records on the recovery path, while group commit shares the cost of a persistence operation among several transactions. Grouping can improve throughput but may make an individual transaction wait for a group to form.\n\nDatabase replication introduces choices about when a result is considered committed or visible. Synchronous replication can require communication with other nodes before acknowledgment. Asynchronous replication may acknowledge earlier but allows replicas to lag. Distributed consensus protocols add communication and coordination to establish an agreed order of operations; their latency depends on network placement, protocol state, failures, and implementation. Eventual consistency changes some coordination requirements but does not mean that all operations are immediate or that application-level dependencies disappear.\n\nLoad balancing can reduce queueing by distributing requests, but only if the selection policy accounts adequately for available capacity and workload differences. Retries and hedged requests can reduce the latency experienced by some callers, yet they consume additional resources. During overload, uncontrolled retries can increase contention and lengthen queues. Time budgets therefore commonly propagate across service boundaries: a downstream operation receives only part of the caller’s remaining deadline rather than an unrelated, full-length timeout at every stage."
    },
    {
      "heading": "Web delivery, interactive media, and inference",
      "text": "For the World Wide Web, user-visible latency includes more than transferring a document. A navigation may require a Domain Name System lookup, connection establishment, security negotiation, request transmission, server processing, and rendering in a web browser. Cached name resolution or reused connections can omit some of these steps. Subresources and scripts introduce additional dependencies, so the first returned byte, the first visible content, and the point when an interface responds reliably are different milestones.\n\nProtocol design affects the number of round trips on this path. Establishing a conventional Transmission Control Protocol connection normally uses a three-way handshake. Transport Layer Security version 1.3, standardized in 2018, reduced the handshake exchanges needed for secure communication compared with common earlier configurations. QUIC, standardized in 2021, integrates transport establishment with TLS and supports multiple streams without TCP’s cross-stream head-of-line blocking from packet loss. It does not eliminate congestion, propagation, or application dependencies. Early data can reduce waiting in eligible resumed connections, but replay considerations restrict its use.\n\nA content delivery network places copies of content or application functionality closer to users. This can shorten propagation paths, reduce origin traffic, and avoid repeatedly computing the same response. Its benefit depends on cacheability, cache-hit rate, routing, and where dynamic data resides. Serving a static image from a nearby node is different from completing a personalized transaction that still requires communication with a distant database.\n\nInteractive audio and video must trade delay against continuity. A receiver’s jitter buffer delays playback to accommodate irregular arrival times; a larger buffer can reduce interruptions while increasing conversational delay. Audio processing also occurs in blocks: 256 samples at 48 kHz represent about 5.33 milliseconds of audio per block. Total input-to-output latency includes additional capture, processing, buffering, transport, and playback stages. In games and virtual reality, input sampling, simulation, rendering, display refresh, and scanout all contribute. At 60 Hz, one frame period is approximately 16.67 milliseconds.\n\nNeural-network inference introduces related distinctions. For an autoregressive large language model, time to first token includes the work needed before the first output token appears, while inter-token latency characterizes subsequent generation. Total response time also depends on output length. Batching requests on a graphics processing unit can raise throughput but introduce queueing; processing long prompts competes with ongoing generation. Streaming exposes partial results sooner without necessarily reducing the time required to finish the complete answer."
    },
    {
      "heading": "Measurement and experimental design",
      "text": "Accurate latency measurement begins by specifying the event boundaries and clock. For elapsed time within one machine, a monotonic clock avoids discontinuities caused by adjustments to civil time. Wall clocks can be corrected or stepped, making naïve timestamp subtraction unreliable. High-resolution counters also require attention to platform guarantees, virtualization, and measurement overhead. A clock displaying nanoseconds does not necessarily provide nanosecond accuracy, stability, or effective resolution.\n\nOne-way measurements across machines additionally depend on clock synchronization. Network Time Protocol estimates clock relationships through timestamp exchanges, while Precision Time Protocol can support much tighter synchronization in suitable controlled networks, especially with hardware timestamping. Neither makes uncertainty disappear. Clock offset, drift, asymmetric paths, and timestamp placement influence results. Round-trip measurements avoid requiring synchronized endpoint clocks, but they cannot by themselves isolate the outward path from the return path.\n\nA representative experiment controls workload as well as instrumentation. Relevant variables include request size, connection reuse, concurrency, arrival pattern, data distribution, cache state, and background activity. A closed-loop benchmark sends new work after previous work completes; if the service slows, the generator may unintentionally reduce its request rate. An open-loop benchmark schedules arrivals independently, exposing queue growth under a specified offered load. Neither model is inherently correct for every application, but they answer different questions.\n\nCoordinated omission occurs when a measurement process fails to represent requests or waiting intervals that would have occurred during a slowdown. For example, a client that stops generating work while awaiting a stalled response can report an artificially favourable distribution for a workload whose arrivals should continue independently. Corrections require explicit assumptions about the intended arrival process. Similar care is needed with timeouts and errors: reporting only successful, completed requests can conceal the slowest or failed attempts and understate the experience of callers.\n\nDistributed tracing records causally related spans across services, making critical paths and remote waits more visible. Sampling can limit overhead but may omit rare events unless the sampling strategy deliberately preserves them. Instrumentation itself can alter execution through logging, allocation, synchronization, or network traffic. Reliable reports therefore identify sample counts, measurement windows, error rates, and relevant uncertainty. At the 99.9th percentile, 1,000 observations leave only about one observation beyond the nominal threshold; estimating extreme tails reliably generally requires substantially more data and attention to workload changes."
    },
    {
      "heading": "Techniques for reducing and controlling latency",
      "text": "Latency optimization begins with identifying which activities determine completion. Amdahl’s law captures the limit imposed by an unaffected fraction of execution: if fraction \\(p\\) is accelerated by factor \\(s\\), ideal speedup is\n\n\\[S=\\frac{1}{(1-p)+p/s}.\\]\n\nAlthough originally associated with parallel processing, the same reasoning applies to latency reduction. Making computation twice as fast has limited effect when most response time is spent waiting for a remote service. Conversely, reducing network delay may barely affect a workload dominated by local computation.\n\nCaching avoids repeated work or communication when a stored result can be reused. A database index can reduce the amount of data examined; maintaining a reusable connection can avoid establishment handshakes; moving computation closer to its data can remove transfers. Each changes a specific part of the critical path. Cache invalidation and freshness requirements constrain reuse, while indexes impose storage and update costs. Compression can reduce transfer time on a constrained link but adds encoding and decoding work, so its effect depends on message size, processing capacity, and network conditions.\n\nConcurrency and speculation seek to overlap or anticipate work. Independent requests can execute in parallel; prefetching can begin a likely future access; a hedged request can send another attempt if the first is unusually slow. These techniques expend additional resources or depend on useful predictions. Cancellation is important because an abandoned request may otherwise continue using processor time, storage bandwidth, or remote capacity. Parallelism that exceeds the capacity of a shared dependency can worsen both mean and tail latency.\n\nQueue control addresses waiting directly. Admission control rejects or defers work before queues become unbounded. Backpressure propagates capacity limits toward producers. Short queues, appropriate scheduling, and sufficient spare capacity can reduce response-time variability, but the design must define what happens to work that cannot be served. Active queue management in networks can signal congestion or discard packets before buffers remain persistently full. Priority scheduling can protect latency-sensitive traffic while shifting delay to lower-priority traffic, rather than eliminating resource scarcity.\n\nBatching demonstrates the recurring trade-off between efficiency and individual delay. Combining operations amortizes fixed overhead and may greatly increase device utilization, but an item can wait for a batch to fill. A bounded batching timer limits this deliberate delay. Similarly, polling can avoid wake-up overhead at the cost of processor use and energy. Service-level agreements and internal objectives translate these trade-offs into measurable requirements, ideally specifying the operation, percentile, observation window, workload conditions, and treatment of errors rather than relying on an undefined promise of “low latency.”"
    },
    {
      "heading": "Historical development and timing-critical systems",
      "text": "Latency has been a concern throughout computing, but the dominant sources have changed. Early machines exposed delays associated with storage media, arithmetic circuitry, and peripheral devices. Magnetic drums and delay-line memories made access time closely dependent on the physical organization of stored data. As processors became faster, the disparity between computation and data movement became more prominent. Cache memories, pipelining, and speculative execution developed partly to bridge or conceal those differences rather than to make every component equally fast.\n\nInteractive time-sharing in the 1960s shifted attention toward the delay experienced by a person issuing a command. Batch systems emphasized efficient completion of large collections of work; interactive systems also needed acceptable response times for individual users. Networked computing extended the problem across machines. With the growth of the Internet and commercial web services, connection setup, geographic distance, server contention, and browser execution became interdependent parts of user-visible performance.\n\nLarge-scale service architectures made rare delays increasingly consequential. A request assembled from many components could encounter a slow dependency even when each component was usually fast. Jeffrey Dean and Luiz André Barroso’s 2013 article “The Tail at Scale” gave a widely cited account of this problem and techniques for mitigating it. The underlying issues connect modern service engineering to older queueing and reliability questions: shared resources, variability, and synchronization can make aggregate behaviour differ substantially from the behaviour of one isolated component.\n\nTiming-critical systems require more than favourable averages. A real-time operating system supports mechanisms intended to make scheduling and response more predictable. In hard real-time applications, missing a deadline can constitute system failure; in soft real-time applications, the consequence is usually degraded usefulness or quality. Worst-case execution-time analysis, interrupt-latency bounds, resource arbitration, and restrictions on dynamic behaviour can therefore matter more than peak throughput. A measured maximum is not automatically a proven worst-case bound, because testing may not encounter every relevant combination of states and interference.\n\nControl systems illustrate why predictability matters. A sensor measurement, computation, and actuator update form a feedback loop. Delay changes the loop’s phase relationships and can reduce stability margins; variation makes the effective timing harder to model. In human–computer interaction, perceptibility likewise depends on the task: text entry, conversational speech, remote manipulation, and immersive displays have different timing demands. No single numerical threshold defines acceptable latency for all of them. The relevant requirement may concern average responsiveness, a high percentile, a strict deadline, synchronization between streams, or the age of the information on which an action is based."
    }
  ],
  "links": [
    "computer-science",
    "bandwidth",
    "speed-of-light",
    "packet-switching",
    "router",
    "central-processing-unit",
    "memory-hierarchy",
    "cpu-cache",
    "cache-coherence",
    "non-uniform-memory-access",
    "operating-system",
    "flash-memory",
    "thread-computing",
    "cpu-scheduling",
    "mutex",
    "garbage-collection",
    "serverless-computing",
    "asynchronous-programming",
    "event-loop",
    "distributed-computing",
    "database-transaction",
    "write-ahead-logging",
    "database-replication",
    "distributed-consensus",
    "eventual-consistency",
    "load-balancing",
    "world-wide-web",
    "domain-name-system",
    "web-browser",
    "transport-layer-security",
    "virtual-reality",
    "neural-network-inference",
    "large-language-model",
    "graphics-processing-unit",
    "amdahls-law",
    "database-index",
    "service-level-agreement",
    "time-sharing",
    "internet",
    "real-time-operating-system",
    "human-computer-interaction"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}