XLA, short for Accelerated Linear Algebra, is an optimizing compiler for numerical programs, especially those used in machine learning. It takes structured descriptions of operations on multidimensional arrays and produces executable code for hardware such as central processing units (CPUs), graphics processing units (GPUs), and tensor processing units (TPUs). Originally developed at Google in connection with TensorFlow, XLA is now a major component of the OpenXLA compiler ecosystem. Its central approach is to optimize groups of operations together rather than execute each operation independently.
Purpose and scope
Numerical frameworks commonly express computations as operations on tensors: arrays characterized by their dimensions, element types, and values. A program may multiply matrices, add biases, apply activation functions, and compute reductions. An execution system can dispatch these operations individually to optimized libraries, but this can introduce repeated dispatch overhead, intermediate memory allocations, and transfers between different levels of the memory hierarchy.
XLA instead compiles a region of a program as a unit. Its view of the computational graph allows it to simplify expressions, combine operations, choose data layouts, and plan memory use across operation boundaries.
Despite its name, XLA is not limited to linear algebra. Its representations include elementwise arithmetic, reductions, indexing, convolution, and structured control flow. Nor is it a complete machine-learning framework: it does not by itself provide the full collection of model-building, data-loading, and experimentation facilities offered by frameworks. It occupies a lower layer that translates numerical computations into executable implementations.
XLA also does not automatically compile arbitrary application code. A frontend must express a supported portion of a program in a form the compiler understands. File access, general-purpose object manipulation, and other host-side activities typically remain outside the compiled numerical region.
Compilation architecture
The exact compilation pipeline depends on the frontend and target hardware, but it generally comprises several stages.
Frontend capture and intermediate representations
A framework first identifies or constructs the computation to be compiled. In JAX, this commonly involves tracing a function using abstract information about its inputs, such as their shapes and element types. TensorFlow and PyTorch integrations use their own mechanisms to identify compilable graphs or regions.
The frontend lowers the computation into a compiler representation. XLA's principal internal representation is HLO, short for High Level Optimizer. HLO represents operations at a relatively high mathematical level, preserving information that would be difficult to recover from low-level instructions.
StableHLO provides a related operation set and a compatibility-oriented representation for exchanging programs between frameworks and compilers. It is distinct from XLA's evolving internal HLO representation: StableHLO serves an interchange role, while internal HLO supports XLA's optimization and compilation processes.
Hardware-independent optimization
Early optimization passes transform the computation without committing every detail to a particular processor. Typical transformations include:
- Constant folding: evaluating expressions whose inputs are known during compilation.
- Dead-code elimination: removing computations that do not contribute to observable outputs.
- Algebraic simplification: replacing expressions with simpler forms when permitted by their semantics.
- Common-subexpression elimination: avoiding duplicate computation of equivalent expressions.
- Operation fusion: combining compatible operations into a larger executable unit.
These transformations exploit the structure of the whole compiled region. They must nevertheless respect operation semantics, including constraints associated with numerical precision and side effects.
Target-specific optimization and code generation
Later stages adapt the computation to the selected hardware. Decisions can include tensor layout, tiling, instruction selection, parallel execution strategy, and the use of specialized matrix-multiplication or convolution libraries.
On GPUs, XLA may generate device kernels and invoke vendor libraries for suitable operations. On CPUs, its compilation infrastructure uses LLVM to help generate machine code. TPU compilation uses a target-specific backend.
Consequently, a common high-level input does not imply identical execution across targets. Different hardware can require substantially different decompositions, memory layouts, and scheduling choices.
Executable loading and execution
After compilation, a runtime loads and launches the resulting executable, manages device buffers, and coordinates execution with the host program. In the OpenXLA ecosystem, PJRT provides interfaces through which frameworks can interact with device runtimes and executable implementations.
Compilation and runtime execution are separate costs. An executable may be reused for later calls with compatible inputs, allowing its initial compilation cost to be amortized.
Principal optimization techniques
Operation fusion
Fusion is one of XLA's most important optimizations. Consider an elementwise expression
A straightforward execution strategy could launch separate operations for multiplication, addition, and the maximum, storing intermediate arrays between them. A fused implementation can calculate each output element within one kernel, potentially keeping intermediate values in registers rather than writing them to device memory.
This can reduce kernel-launch overhead and memory traffic. It is particularly useful when execution is limited by memory bandwidth rather than arithmetic throughput.
Fusion is not universally beneficial. Large fused kernels can increase register pressure, reduce the number of concurrently active threads, or duplicate work. Some operations are better handled by specialized libraries. XLA therefore uses backend-specific decisions rather than simply merging every connected operation.
Layout and memory planning
A tensor's logical shape does not fully specify how its elements are arranged in memory. Different physical layouts can favor different access patterns or hardware instructions. XLA can choose layouts and insert conversions where necessary.
The compiler also analyzes when intermediate values become live and when they are no longer needed. This supports buffer reuse and can reduce peak memory consumption. Where program semantics and frontend contracts permit, an input buffer may be reused for an output.
These optimizations do not eliminate the memory requirements of large computations. Layout conversions, temporary workspaces, and retained intermediates can still account for substantial memory use.
Specialization and autotuning
XLA frequently specializes executables to input properties, particularly tensor shapes and element types. This enables optimizations such as selecting fixed-size tiles and simplifying shape-dependent expressions.
Some backends also use autotuning to choose among candidate implementations. For example, a GPU backend may benchmark alternative configurations for a matrix operation. Such tuning can improve subsequent execution while increasing initial compilation or setup time.
Specialization creates a trade-off: a precisely specialized executable can be efficient, but many distinct input signatures can require many compiled variants.
Framework integration and applications
XLA is used through several frontend ecosystems:
- JAX: XLA is a central compilation backend for JAX's numerical programming model, including functions compiled through
jax.jit. JAX's transformations construct computations that can then be lowered for compilation. - TensorFlow: XLA can compile explicitly selected functions or graph regions, including through
tf.functionwithjit_compile=True, subject to supported operations and target constraints. - PyTorch/XLA: this integration connects PyTorch programs to XLA-based compilation and execution. It is distinct from the compilation backends commonly used by standard PyTorch installations; using PyTorch does not inherently mean using XLA.
Applications include neural-network training, neural-network inference, and scientific numerical computation. Repeated execution of large numerical workloads can make compilation especially worthwhile.
Automatic differentiation is generally performed by a framework or an associated transformation system before or around XLA compilation. XLA can optimize the resulting forward and gradient computations, but it should not be conflated with backpropagation itself.
Distributed computation
For workloads spanning multiple devices, XLA can compile computations involving partitioned or replicated data. Partitioning distributes portions of tensors or computation across devices; replication places corresponding copies on multiple devices.
This requires coordinating local computation with communication operations, such as collective reductions and data exchanges. The compiler can use information about how tensors are distributed to generate appropriate communication and device-local work.
This capability supports large-scale parallel computing, but efficient execution still depends on partitioning choices, interconnect bandwidth, communication latency, and the balance of work among devices. Compiler optimization cannot remove the cost of moving data required by the computation.
Limitations and performance considerations
Compilation latency
Compilation may involve extensive graph analysis, code generation, and autotuning. For short-lived programs or small workloads, these costs can outweigh execution-time savings. Performance measurements therefore need to distinguish first-call latency from repeated execution using an already compiled executable.
Some frontends and deployment paths support ahead-of-time compilation or persistent caches, but the availability and portability of such artifacts depend on the surrounding software stack.
Shapes and dynamic behavior
Changing input shapes, element types, or specialized static values may trigger recompilation. Workloads with many distinct shapes can consequently incur substantial compilation and cache-management overhead.
Dynamic dimensions are supported in parts of the ecosystem, but support varies by operation, frontend, and backend. Data-dependent output sizes and control flow expressed through ordinary host-language constructs can be especially challenging to capture. Representing control flow as compiler-visible operations differs from executing that control flow in the host language.
Operation coverage and external effects
A frontend may contain operations for which an XLA backend has no supported lowering. Depending on the integration, this may cause a compilation error or require execution outside the compiled region.
Input/output, callbacks, mutation, and communication with external systems also require explicit handling. These activities cannot generally be reordered like pure arithmetic operations, which limits some optimizations.
Numerical differences
Compiled and uncompiled execution need not produce bit-identical floating-point results. Floating-point addition is not associative, so a different reduction order can change rounding:
for some floating-point values.
Hardware-specific instructions, fused arithmetic, library implementations, and reduced-precision execution can also affect results. The permitted behavior depends on operation definitions and compiler settings. These issues relate to the properties of IEEE 754 arithmetic rather than to XLA alone.
Workload-dependent speedups
XLA does not guarantee that every program becomes faster. Performance depends on the size and structure of the compiled region, opportunities for fusion, hardware utilization, memory access, and the quality of existing library implementations. Host-device transfers and time spent outside the compiled region can limit overall gains even when individual compiled kernels improve.
Development and ecosystem
XLA emerged from Google's work on compiling TensorFlow computations for heterogeneous hardware. Its role subsequently expanded as other frameworks, notably JAX, adopted it as a compilation backend.
The broader OpenXLA effort brings together compiler, representation, and runtime-interface components intended to support multiple frameworks and hardware targets. Within this ecosystem, XLA, StableHLO, and PJRT perform different functions: compilation and optimization, program interchange, and runtime integration, respectively.
XLA remains a specific compiler infrastructure rather than a synonym for all machine-learning compilation. Other systems may use different graph representations, optimization strategies, or code generators, even when they target the same processors.