Inference batching is the execution of multiple inputs or requests together during neural-network inference, rather than processing each separately. It is widely used in machine learning systems to increase throughput and reduce the computing cost per prediction. By combining work into larger operations, batching can make better use of processors, memory bandwidth, and shared model parameters. Its benefits depend on the workload: collecting requests may introduce waiting time, larger batches require more memory, and differences between requests can make scheduling more complex.
Operating principle
Many neural networks apply the same sequence of operations and the same learned parameters to different inputs. Batching adds a dimension representing these inputs, allowing the underlying numerical libraries to execute them together.
For a simplified linear layer, a single input vector is multiplied by a weight matrix. With a batch of inputs, these vectors become rows of an input matrix, and the operation takes the form
where contains the batched inputs, contains the shared weights, and the bias is broadcast across the rows. The number of arithmetic operations generally grows with the batch size, but the execution time may initially grow more slowly.
On a graphics processing unit (GPU), batching can:
- expose enough independent work to occupy more execution units;
- amortize accelerator-kernel launch and framework overhead;
- reuse model weights across multiple inputs;
- turn small numerical operations into larger, more efficient operations.
Similar benefits can occur on a central processing unit or a tensor processing unit, although the optimal batch size depends on the hardware and software implementation.
Batching is not the same as running independent model instances concurrently. Concurrent instances may duplicate weights or compete for resources; a batch normally combines inputs within shared model operations. A serving system can nevertheless use both approaches.
Main batching strategies
Static batching
Static batching forms a group of inputs before execution and keeps that group fixed for the relevant computation. It is common in offline workloads, such as classifying an image collection, producing document embeddings, or scoring a dataset.
Because inputs are already available, offline systems can often assemble large batches without introducing user-visible collection delays. The main constraints are memory capacity, input compatibility, and processing efficiency.
“Static” can also describe a fixed configured batch size. The term does not necessarily imply that every input has the same shape or that the model was compiled exclusively for that batch size.
Dynamic batching
Dynamic batching combines requests arriving independently at a model-serving endpoint, often through an application programming interface. A scheduler holds requests briefly in a queue and groups compatible ones for execution.
Typical controls include:
- a maximum batch size;
- preferred batch sizes;
- a maximum delay for collecting requests;
- shape and model-version compatibility;
- request priorities or deadlines.
At low arrival rates, waiting for a large batch can worsen response time without providing much computational benefit. At high arrival rates, useful batches may form almost immediately. A batch-collection delay limit does not necessarily bound total queueing time: requests may still wait while the device processes earlier work.
Continuous batching
Continuous batching, also called iteration-level batching in some systems, is especially important for large language models that generate variable-length outputs.
In a simple fixed-membership generation batch, shorter requests may finish while longer requests continue, leaving unused capacity unless the implementation can replace completed requests. Continuous batching allows the active set to change at scheduling boundaries: completed requests leave, and new requests enter.
The term describes a scheduling approach rather than one universal algorithm. Implementations differ in whether they mix prompt processing and token generation, divide prompts into chunks, or impose separate limits on active sequences and scheduled tokens.
Microbatching and shape-aware batching
A microbatch is a smaller subdivision of a larger logical batch. Microbatching can limit temporary memory use, support execution pipelines, or improve scheduling granularity. Its meaning depends on the system; it does not automatically imply continuous batching.
Shape-aware batching groups inputs with similar dimensions, such as images of similar resolution or text sequences of similar length. Length-based grouping is often called bucketing. It reduces padding and wasted computation, although delaying requests to find compatible partners can increase waiting time.
Throughput, latency, and memory
The central trade-off is between aggregate efficiency and individual response time.
If a batch of equally weighted requests takes seconds to execute, its execution throughput is
Batching improves throughput relative to single-request execution when . This condition does not imply that an individual request finishes faster. A request may wait for the batch to form and then experience the full batch execution time.
A simplified response-time decomposition is
where includes waiting for available capacity and batch formation, while includes such costs as preprocessing, transfers, and communication.
Increasing batch size usually improves throughput only up to a point. Eventually the processor, memory bandwidth, or another resource becomes saturated. Further increases can provide little benefit, increase latency, or cause an out-of-memory failure.
Memory requirements include model parameters, temporary activations, intermediate workspaces, inputs, and outputs. Parameters are generally shared across the batch, whereas many other allocations grow with batch size or input length. Consequently, a limit expressed only as a number of requests may be insufficient for workloads with highly variable input sizes.
Operational evaluation commonly considers median and tail latency, throughput under realistic arrival patterns, memory consumption, and cost per completed request. For interactive services, the relevant objective is often the highest sustainable throughput that still satisfies a service-level agreement, rather than peak throughput measured with an unlimited backlog.
Batching in autoregressive language models
Batching becomes more complex in transformer language models because each request has both a prompt-processing phase and a sequential generation phase.
Prefill and decode
During prefill, the model processes prompt tokens and constructs the state needed for generation. Many prompt positions can be processed in parallel, although attention and input-length constraints affect the cost.
During decode, each active sequence typically advances by one token per ordinary generation iteration. Successive tokens within a sequence depend on earlier tokens, but independent sequences can be processed together.
Decode at small batch sizes is often constrained by moving model weights from device memory rather than by arithmetic capacity alone. Batching allows weight access to support more sequences, improving utilization. As batches and sequence lengths grow, attention computation and cached-state traffic can become increasingly important; the bottleneck is not fixed across configurations.
Variable lengths and cached state
Prompts and generated responses vary in length. Padding every sequence to the longest one can waste computation and memory, although optimized implementations can avoid some of this work.
Transformer serving commonly maintains a key–value cache for each active sequence. This cache stores attention keys and values from previously processed tokens. For a given model and cache representation, its size grows approximately with the total number of cached tokens across active sequences, not merely the request count.
This makes memory management part of batching policy. A scheduler may have room for many short requests but only a few long ones. Block-based cache allocation and related techniques can reduce allocation waste, but they do not remove the underlying capacity constraint.
Token-budget scheduling
Language-model schedulers often constrain the total number of tokens processed in an iteration rather than using only a fixed request-count limit. An incoming prompt may contribute many tokens, while an active decoding sequence may contribute just one.
Long prompt computations can delay ongoing generation if scheduled as large uninterrupted units. Chunked prefill divides prompt processing into smaller portions, allowing finer-grained interleaving with decode work.
These decisions affect distinct user-visible measurements:
- Time to first token: the delay before generation begins.
- Inter-token latency: the delay between successive generated tokens.
- End-to-end latency: the time until the response is complete.
- Output-token throughput: the rate of generated tokens across requests.
Improving one measurement can worsen another. Larger prompt batches, for example, may improve prompt-processing efficiency while disrupting the responsiveness of active decoding requests.
Compatibility and correctness
Requests can be batched only when the execution system can represent and process them together. Relevant constraints include the model and version, data type, input dimensions, supported numerical operations, and execution backend. Different decoding settings or model adapters may require extra bookkeeping or specialized support rather than necessarily preventing batching.
For a model whose examples are mathematically independent, batching should preserve each example’s intended computation. Bit-for-bit equality is not guaranteed: different batch shapes may select different kernels or change floating-point reduction order.
Batch normalization requires particular care. In standard evaluation mode, it uses stored statistics rather than recomputing them from the current inference batch. A model deliberately using current-batch statistics can instead make one input’s output depend on the other inputs present. Incorrectly leaving a model in training mode may also activate training-specific behavior.
Stochastic generation introduces another distinction. Different scheduling or batch arrangements can change random-number consumption unless the serving system manages randomness independently for each request. Identical prompts therefore need not produce identical sampled outputs across batching configurations.
Deployment considerations and limitations
Batching is useful in computer vision, speech recognition, embedding generation, recommendation systems, and generative-model serving. Offline processing generally tolerates larger batches, while interactive workloads place tighter limits on waiting time.
In distributed deployments, batching interacts with load balancing. Spreading requests across many replicas can reduce the arrival rate at each replica and make efficient batches harder to form. Concentrating requests can improve batching efficiency but increase queueing or reduce spare capacity.
Batching is also distinct from model parallelism. It determines which inputs execute together, whereas model parallelism distributes model computation across devices. Both techniques can be combined.
Practical limitations include:
- Insufficient traffic: too few compatible requests arrive together.
- Heterogeneous workloads: long or expensive requests delay shorter ones.
- Memory pressure: activations or cached state limit the active batch.
- Fairness constraints: policies favoring short or easily packed requests can disadvantage other workloads unless starvation is controlled.
- Cancellation overhead: abandoned requests may retain resources until safely removed.
- Limited responsiveness: device operations may not be preemptible at the granularity needed by urgent requests.
The appropriate policy therefore depends on the model, hardware, arrival distribution, input lengths, and latency requirements. A batch size that performs well in a saturated benchmark may behave poorly under sparse or bursty production traffic.