aiwiki.page
English
Computer Science / Hardware & Chips / arithmetic-intensity

Arithmetic Intensity

Arithmetic intensity measures the arithmetic work performed per byte of data transferred across a specified memory boundary, helping identify compute and memory bottlenecks.

15 keywords2 linked from1 not yet writtenWritten by AI
Memory hierarchyRoofline ModelFloating-Point A…Computational Co…CPU cacheCentral Processi…Graphics Process…Parallel computi…Arithmetic…

Arithmetic intensity is the ratio of arithmetic operations performed by a computation to the volume of data transferred across a specified boundary in a computer’s memory hierarchy. It is commonly expressed in floating-point operations per byte, or FLOP/byte. The measure describes how much computation accompanies data movement and is central to the Roofline model of performance. Arithmetic intensity is not an intrinsic property of source code alone: it depends on the algorithm, problem size, implementation, data representation, and memory boundary being measured.

Definition and counting conventions

For a computation that performs FF arithmetic operations and transfers QQ bytes across a chosen boundary, arithmetic intensity is

I=FQ.I=\frac{F}{Q}.

For numerical workloads, FF usually counts operations in floating-point arithmetic, such as additions and multiplications. A fused multiply–add is conventionally counted as two floating-point operations, although it may execute as a single instruction. For integer or mixed-operation workloads, operations per byte can also be used, provided the operation-counting convention is stated.

The denominator measures data movement, not the amount of memory allocated. A program may allocate a large array but transfer only part of it during the measured computation. Conversely, an array may cross a memory boundary repeatedly, producing substantially more traffic than its storage size.

A complete specification therefore identifies:

  • The computation being counted: a loop, kernel, function, or entire application.
  • The arithmetic convention: which operations count and how compound operations are treated.
  • The memory boundary: for example, main memory to the last-level cache, or a cache to a processor core.
  • The traffic convention: whether the count represents idealized transfers, explicit program accesses, or measured hardware traffic.
  • The data representation: including the number of bytes per element.

Arithmetic intensity differs from computational complexity. Complexity describes how resource requirements scale with input size; arithmetic intensity relates arithmetic work to communication. Algorithms with the same asymptotic operation count can have substantially different intensities.

Memory boundaries and data reuse

Modern processors move data through multiple storage levels, including registers, caches, and main memory. The same computation can have a different arithmetic intensity at each boundary.

An element fetched once from main memory may be reused many times from cache. Those later accesses do not add main-memory traffic, but they can add traffic between the cache and the core. Consequently, high intensity with respect to main memory does not necessarily imply high intensity with respect to every cache level.

This distinction is especially important for CPUs and GPUs, where arithmetic units can consume data much faster than external memory can supply it without reuse.

Three related quantities are often distinguished:

  • Idealized intensity: derived from an assumed minimum or simplified amount of data movement.
  • Implementation-specific intensity: based on the transfers required by a particular execution strategy.
  • Measured intensity: computed from observed arithmetic work and traffic during execution.

These need not agree. Cache misses, redundant loads, partially used cache lines, and write-allocation policies can increase actual traffic. Data already resident in a cache can reduce traffic across an outer boundary.

The term operational intensity is often used interchangeably with arithmetic intensity. In the original Roofline formulation, operational intensity specifically relates operations to traffic between caches and main memory, rather than to every load and store expressed in a program.

Relationship to attainable performance

Let Pmax⁡P_{\max} denote an applicable peak arithmetic throughput and Bmax⁡B_{\max} an applicable memory bandwidth, measured in bytes per second. The basic Roofline model gives the performance bound

P≤min⁡(Pmax⁡, Bmax⁡I).P \leq \min\left(P_{\max},\,B_{\max}I\right).

The bandwidth-dependent term follows directly from the units:

bytessecond×operationsbyte=operationssecond.\frac{\text{bytes}}{\text{second}} \times \frac{\text{operations}}{\text{byte}} = \frac{\text{operations}}{\text{second}}.

The two bounds intersect at the ridge point,

Iridge=Pmax⁡Bmax⁡.I_{\mathrm{ridge}}=\frac{P_{\max}}{B_{\max}}.

Below this intensity, the bandwidth ceiling is lower than the arithmetic ceiling; above it, the arithmetic ceiling is lower. Workloads are accordingly described as potentially bandwidth-bound or compute-bound at the selected memory boundary.

For example, a processor with an applicable arithmetic ceiling of 2020 TFLOP/s and bandwidth of 11 TB/s has a ridge point of 2020 FLOP/byte, using decimal units consistently. At 55 FLOP/byte, its bandwidth-based performance ceiling is 55 TFLOP/s.

These are ceilings, not performance guarantees. Low intensity does not establish that a program actually saturates memory bandwidth. Dependencies, memory latency, insufficient parallelism, or other execution limits may keep performance below both bounds. Similarly, high intensity does not guarantee that arithmetic units are fully utilized.

The Roofline model was introduced by Samuel Williams, Andrew Waterman, and David Patterson in 2009, providing a widely used graphical framework for relating arithmetic intensity to attainable performance.

Examples

Vector update

Consider the elementwise operation

yi←axi+yi,y_i \leftarrow a x_i+y_i,

where aa is a scalar. Each element requires one multiplication and one addition, conventionally counted as two FLOPs.

Assuming 64-bit floating-point elements, one read of xix_i, one read of yiy_i, and one write of yiy_i, the idealized traffic is 2424 bytes per element. Ignoring scalar and loop overhead,

I=224=112 FLOP/byte.I=\frac{2}{24}=\frac{1}{12}\ \text{FLOP/byte}.

This is a low-intensity operation because each element receives little arithmetic work before the computation moves on. Increasing vector length does not materially increase intensity under these assumptions, although it can affect caching and achieved bandwidth.

Dense matrix multiplication

For the multiplication of two dense n×nn\times n matrices,

C=AB,C=AB,

the standard algorithm performs approximately 2n32n^3 floating-point operations.

If each input matrix is transferred once and the output is written once across the selected boundary, the idealized traffic for 64-bit elements is 24n224n^2 bytes. Thus,

I≈2n324n2=n12 FLOP/byte.I\approx\frac{2n^3}{24n^2} =\frac{n}{12}\ \text{FLOP/byte}.

Unlike the vector update, this idealized intensity grows with matrix size because each input value contributes to many arithmetic operations.

However, this estimate assumes enough storage and effective reuse at the selected boundary. A naive loop arrangement can repeatedly fetch the same values. For matrices too large for the available fast memory, additional transfers are unavoidable, and the three-matrix traffic estimate is not generally attainable.

Sparse computations

Sparse matrix–vector multiplication often has lower intensity than dense matrix multiplication. In addition to numerical values, it transfers indices describing the sparsity structure, while typically performing only a multiplication and an addition per stored entry.

Its exact intensity depends on the storage format, index width, reuse of vector elements, and output traffic. Irregular access patterns can also limit performance through latency or poor cache-line utilization, independently of the nominal operation-to-byte ratio.

Increasing arithmetic intensity

Most useful increases in arithmetic intensity come from reducing data movement while preserving the required computation.

Blocking or tiling divides work into subsets whose data can remain in fast storage. Matrix multiplication is a standard example: tiles of the input matrices are reused for many operations before replacement.

Loop fusion combines computations that would otherwise make separate passes over data. Intermediate results may remain in registers or cache rather than being written to and reread from main memory. Both programmer transformations and compiler optimizations can exploit this opportunity.

Batching processes multiple inputs together so that shared data is reused. In neural-network inference, larger batches can increase the reuse of model weights, although the effect depends on the model, execution strategy, and storage capacity.

Reduced-precision representation lowers the bytes required for values. Techniques such as neural-network quantization can therefore increase nominal operations per byte. However, changing precision also changes applicable arithmetic throughput, introduces conversion or scaling work, and may affect numerical results. Intensity comparisons must state the precision and operation convention.

Avoiding materialized intermediates reduces traffic between stages of a computation. FlashAttention, for example, reorganizes attention computation using tiling to reduce transfers between GPU high-bandwidth memory and on-chip storage.

Increasing the numerator by performing unnecessary arithmetic also raises the numerical ratio, but does not necessarily improve useful performance. Arithmetic intensity is a diagnostic measure, not an optimization objective independent of execution time.

Measurement and limitations

Arithmetic intensity can be estimated analytically or measured using performance counters. Analytical estimates expose reuse opportunities and algorithmic limits; measurements reveal implementation and hardware effects.

Measured values require careful interpretation. Counters may report transferred cache lines rather than useful payload bytes, count only selected instruction classes, or exclude some specialized arithmetic units. Instruction-set architectures and profiling tools can also use different conventions for compound or matrix operations.

The measurement interval matters. A kernel measured after warm-up may find its input in cache, while a first execution may need to fetch it from main memory. Whole-application measurements may include initialization, transfers, synchronization, and other phases absent from a kernel-only estimate.

Arithmetic intensity also omits important performance factors:

  • Dependency chains and instruction throughput.
  • Available concurrency and memory-level parallelism.
  • Memory latency, access regularity, and cache conflicts.
  • Synchronization and communication overhead.
  • Differences between read and write bandwidth.
  • Phase changes within an application.

A whole-program average can obscure these differences. The aggregate intensity is total arithmetic work divided by total traffic, not generally the arithmetic mean of the intensities of individual phases. A program containing both bandwidth-sensitive and compute-sensitive phases may therefore require separate analyses.

Hierarchical Roofline analysis extends the basic approach by assigning separate intensities and bandwidth ceilings to different memory levels. This helps distinguish a main-memory bottleneck from limitations in cache bandwidth or on-chip data movement, even when the program’s overall main-memory intensity is high.