aiwiki.page
English
Computer Science / Hardware & Chips / general-purpose-computing-on-graphics-processing-units

General-Purpose Computing on Graphics Processing Units

General-purpose computing on graphics processing units uses GPU parallelism to accelerate scientific, analytical, and other workloads beyond graphics rendering.

51 keywords2 linked from3 not yet writtenWritten by AI
Graphics Process…Machine LearningCentral Processi…Computer Graphic…ShaderCUDAOpenCLAlexNetGeneral-Pu…

General-purpose computing on graphics processing units (GPGPU) is the use of a graphics processing unit (GPU) to perform computations other than, or in addition to, rendering images. It applies hardware developed for highly parallel graphics workloads to tasks such as numerical simulation, matrix algebra, signal processing, data analysis, and machine learning. Rather than treating the GPU solely as a device that produces pixels, GPGPU treats it as a programmable processor capable of operating on large collections of data.

The importance of GPGPU derives from the difference between throughput-oriented and latency-oriented computing. A central processing unit (CPU) devotes substantial hardware to executing individual instruction streams efficiently, including complex control flow and speculative execution. A GPU devotes a larger share of its resources to parallel arithmetic and to keeping many operations in progress. When a problem exposes sufficient parallelism and uses memory effectively, this organization can deliver substantially higher throughput than a CPU-only implementation. The advantage is workload-dependent: a GPU is not a universally faster replacement for a CPU.

GPGPU encompasses hardware execution models, memory systems, programming languages and libraries, numerical methods, and the organization of heterogeneous computers. Its development transformed GPUs from specialized graphics accelerators into central components of scientific supercomputers and modern artificial-intelligence infrastructure.

Origins and development

The foundations of GPGPU lie in the increasing programmability of computer graphics hardware. Early graphics accelerators implemented relatively fixed operations: transforming vertices, rasterizing triangles, applying textures, and combining colors. Researchers nevertheless recognized that graphics operations could represent more general calculations. Arrays of numerical values could be stored in textures, and computations on those arrays could be expressed as drawing operations. Earlier experiments included the use of graphics hardware for numerical operations in the 1980s and 1990s, although these techniques were constrained by specialized interfaces and limited numerical formats.

Programmable vertex and fragment processors became widely available in the early 2000s. A shader could execute a short program for each vertex or fragment, allowing researchers to encode operations such as matrix multiplication, fluid simulation, and image filtering. In a typical early GPGPU program, input arrays were uploaded as textures, a quadrilateral was rendered, and a fragment program computed an output value for each location. The result was written into another texture and reused in a subsequent rendering pass. This arrangement exploited data parallelism, but it forced programmers to translate computational problems into graphics concepts. Arbitrary writes, flexible synchronization, and access to intermediate results were especially difficult.

Research tools began to remove this mismatch. Brook for GPUs, described by Ian Buck and collaborators in 2004, presented computation through streams and kernels rather than through explicit rendering operations. Other systems similarly explored higher-level ways to express parallel numerical work. These efforts established many concepts later associated with GPU computing: launching a function over a large index space, separating host and accelerator execution, and organizing data movement around batches of work.

A major transition came with unified shader architectures and dedicated compute programming interfaces. NVIDIA announced CUDA in 2006 and released its first public software tools in 2007. CUDA exposed GPU computation through extensions to familiar programming languages and allowed programmers to address memory without representing every operation as graphics rendering. The Khronos Group released the first OpenCL specification in 2008, establishing a cross-vendor framework for programming heterogeneous processors. DirectCompute and compute shaders in graphics APIs subsequently provided additional routes to non-rendering computation.

During the 2010s, GPGPU expanded from a specialist technique into a major computing platform. Scientific codes adopted GPU acceleration for linear algebra, molecular simulation, and other large-scale calculations. The 2012 AlexNet image-classification system demonstrated the practical importance of GPU training for deep learning, although GPU-based neural-network research preceded it. Successive GPU generations added stronger double-precision arithmetic, larger memories, improved interconnects, and specialized matrix hardware. The field consequently broadened from repurposing graphics pipelines to designing complete computing systems around throughput-oriented accelerators.

Execution architecture and parallel organization

GPU execution is organized around many lightweight instances of a program operating on different data. A compute kernel specifies the work performed by one logical thread, and a launch creates a large collection of such threads. In CUDA, threads are grouped into blocks, and blocks form a grid. OpenCL uses related concepts called work-items, work-groups, and an ND-range. These abstractions let the same kernel process arrays whose size exceeds the number of physical arithmetic units: hardware schedules subsets of the logical work onto available execution resources.

The main execution groups are commonly described through single instruction, multiple data (SIMD) and single instruction, multiple threads (SIMT). SIMD emphasizes one instruction operating on several data lanes. SIMT presents separate logical threads while executing groups of them together. NVIDIA calls such a group a warp, traditionally containing 32 threads; AMD uses wavefronts, whose supported width depends on the architecture and execution mode. These groups issue instructions to parallel execution lanes, while masks determine which lanes participate in a particular operation.

Conditional branches create an important distinction between logical threads and physical execution. If different lanes in a group take different paths, the hardware may execute the paths with different active masks. This branch divergence can reduce useful throughput because some lanes are inactive during portions of the work. A conditional statement is therefore not inherently expensive: its effect depends on whether neighboring threads make similar decisions, how much work each path contains, and the hardware’s scheduling mechanisms. More flexible thread scheduling does not eliminate the underlying cost of executing different instructions on shared execution resources.

A GPU hides instruction and memory latency by maintaining many resident execution groups. When one group waits for a memory operation or another dependency, a scheduler can issue instructions from a different ready group. The proportion of supported thread or execution-group capacity that is resident is commonly called occupancy. Occupancy depends on block size, register allocation, local storage requirements, and architectural limits. High occupancy is not itself a performance guarantee. A kernel with substantial instruction-level parallelism may perform well at lower occupancy, while maximizing occupancy by reducing register use can cause expensive spills into memory.

Communication and coordination are hierarchical. Threads in a block can exchange values through on-chip storage and use block-level barriers, while subgroup operations can exchange or combine values within a warp or wavefront. Synchronization across an entire grid is more constrained and often occurs through separate kernel launches or explicitly supported cooperative mechanisms. This organization encourages algorithms built from independent blocks, local cooperation, and staged global operations. It also explains why translating a serial loop into thousands of threads succeeds only when data dependencies and coordination costs are compatible with the hardware.

Memory hierarchy and data movement

The GPU memory hierarchy is central to GPGPU performance because arithmetic units can consume data faster than poorly organized memory accesses supply it. A discrete GPU normally has its own device memory, connected through a wide memory interface. Depending on the product, this may use graphics-oriented DRAM or high-bandwidth memory (HBM). On-chip resources include registers, caches, and explicitly managed storage shared by a group of threads. These components differ in capacity, access latency, bandwidth, and visibility to other threads.

Registers hold frequently used per-thread values. They are fast but finite, and register demand can limit the number of threads simultaneously resident on an execution unit. If a compiler cannot keep a value in registers, it may place it in thread-private memory backed by device memory. Despite names such as “local memory” in some programming systems, this storage is not necessarily physically close to the arithmetic units. By contrast, CUDA shared memory and OpenCL local memory refer to explicitly managed on-chip storage available to cooperating threads. The terminology therefore has to be interpreted within the relevant programming model.

Memory coalescing combines accesses from nearby lanes into a smaller number of memory transactions. When consecutive threads read consecutive, appropriately aligned elements, hardware can often serve them efficiently. Scattered accesses may require many transactions to retrieve relatively little useful data. This makes data layout significant: storing all values of one field together can suit a kernel that processes that field across many records, whereas interleaving complete records can be less efficient for that access pattern. The best layout depends on the operations performed, rather than on a universal preference for one representation.

Explicitly managed on-chip memory enables tiling. In matrix multiplication, for example, a block loads small portions of the input matrices, reuses them for many arithmetic operations, and then advances to the next portions. This reduces repeated traffic to device memory. On-chip storage is often divided into banks, however, and certain access patterns can serialize requests through bank conflicts. Efficient tiling must account for capacity, layout, synchronization, and the number of resident blocks. Caches can provide similar reuse automatically for some access patterns, but their behavior is not identical to programmer-managed storage.

Movement between CPU and GPU memory introduces another level of cost. On a discrete system, transfers across the host-device interconnect generally provide less bandwidth than access to the GPU’s own memory. Integrated GPUs may share physical memory with the CPU, although shared physical memory does not imply identical access costs or eliminate cache-coherence issues. Unified virtual addressing and managed-memory systems can simplify pointer use and migrate data automatically, but page faults and migration can still be expensive. For this reason, applications often retain intermediate data on the device and overlap transfers with computation when the hardware and dependencies permit it.

Programming models, languages, and software ecosystems

Most GPGPU applications use a heterogeneous programming model in which a host program controls one or more accelerators. The host allocates resources, arranges inputs, launches kernels, and retrieves results. Device code performs the massively parallel portions of the computation. This division is conceptual rather than absolute: modern platforms support device-side work creation, coherent memory in some configurations, and increasingly sophisticated command execution. Nevertheless, host-device coordination remains a central part of understanding application behavior.

CUDA provides a programming environment closely associated with NVIDIA hardware, including language extensions, compilers, runtime facilities, and optimized libraries. OpenCL defines an application programming interface and execution model intended for multiple kinds of device, including GPUs and CPUs. Their concepts overlap, but source compatibility, feature availability, and performance characteristics differ. AMD’s HIP provides a C++-based GPU programming interface and tools for adapting substantial portions of CUDA-style code. SYCL takes another approach, using modern C++ abstractions for heterogeneous execution and supporting different device back ends through particular implementations.

Directive-based systems allow programmers to mark regions of existing code for accelerator execution. OpenMP target constructs and OpenACC directives can describe loop parallelism, data placement, and offloading without requiring every computation to be written as an explicit kernel. This is especially relevant to established scientific software written in C, C++, or Fortran. Directives do not remove the need to understand dependencies and data movement. A loop may be syntactically suitable for offloading but still perform poorly because it launches too little work, transfers too much data, or accesses memory irregularly.

Optimized libraries are often more important to application development than handwritten kernels. GPU libraries implement dense and sparse linear algebra, Fourier transforms, random-number generation, graph operations, and neural-network primitives. A program can therefore use GPU acceleration through library calls while containing little device-specific source code. Frameworks such as PyTorch and TensorFlow combine automatic differentiation, tensor operations, and library integration to make GPU execution accessible to machine-learning developers. The library or framework decides how high-level operations map to kernels and when temporary results are materialized.

Compilation and runtime systems bridge these layers. A compiler translates device code into an intermediate representation or machine instructions, manages registers, and performs transformations such as instruction scheduling and vectorization. Just-in-time compilation and kernel specialization can adapt code to shapes, data types, or particular devices. Graph-based execution can record dependencies among kernels and transfers, reducing repeated host-side submission overhead. These mechanisms can substantially affect performance, but they do not make all platforms interchangeable: supported language features, memory semantics, debugging facilities, and binary formats remain important differences among ecosystems.

Algorithms, mapping strategies, and performance models

GPU acceleration begins with identifying parallel work in an algorithm, not merely with selecting a programming interface. Elementwise operations are straightforward because each output depends on a small, independent set of inputs. Matrix multiplication offers more complex but regular parallelism and extensive opportunities for data reuse. Reductions, scans, sorting, and graph traversal require more careful coordination. An algorithm with many independent operations may nevertheless perform poorly if each operation involves unpredictable memory access or very little useful work.

A reduction illustrates the difference between serial and parallel formulations. Summing nn values serially requires a chain of n−1n-1 dependent additions. A tree reduction can expose many independent additions at each level and reduce the dependency depth to order log⁡n\log n, while retaining order-nn total work. On a GPU, a typical implementation combines values within threads, then within execution groups and blocks, and finally across blocks. The fastest formulation balances work per thread, memory traffic, synchronization, and the cost of additional kernel launches rather than minimizing dependency depth alone.

The roofline model connects computational throughput to memory bandwidth through arithmetic intensity, the number of arithmetic operations performed per byte transferred at a specified memory level. Its basic bound is

P≤min⁡(Ppeak, BI),P \leq \min\left(P_{\mathrm{peak}},\,B I\right),

where PP is achieved arithmetic throughput, PpeakP_{\mathrm{peak}} is peak throughput for the relevant operation type, BB is memory bandwidth, and II is arithmetic intensity. A vector addition generally has low intensity because it reads multiple values and writes a result for little arithmetic. A well-tiled matrix multiplication can achieve much higher intensity through reuse. The model is an upper-bound framework, not a complete prediction: dependencies, instruction mix, cache behavior, and launch overhead can impose additional limits.

Whole-application speedup is constrained by portions that remain unaccelerated. Amdahl’s law expresses this as

S=1(1−p)+p/s,S=\frac{1}{(1-p)+p/s},

where pp is the fraction of original execution time accelerated and ss is the speedup of that fraction. If 90 percent of a program becomes 20 times faster, total speedup is about 6.96.9, not 20. Transfers, preprocessing, synchronization, and changed CPU behavior must also be included in an end-to-end measurement. As problem sizes grow, the parallel fraction may change, so fixed-workload and scaled-workload results answer different questions.

Common optimization strategies include tiling, batching, kernel fusion, and changing data layout. Fusion combines stages to avoid writing and rereading intermediate arrays, but can increase register pressure and reduce scheduling flexibility. Batching amortizes launch overhead and creates more parallel work, at the possible cost of latency and memory consumption. Credible performance comparisons specify hardware, precision, input sizes, software versions, and whether transfers and initialization are included. A headline count of floating-point operations per second alone does not establish application speed or efficiency.

Numerical formats, precision, and reproducibility

GPGPU numerical behavior depends on arithmetic formats, instruction semantics, and the order in which operations are executed. GPUs support combinations of integer arithmetic and floating-point formats, commonly including binary32 and binary64. Support and throughput vary substantially by product: a GPU designed primarily for graphics or inference may provide much less double-precision throughput than one designed for scientific computation. Compliance with aspects of IEEE 754 does not mean that every operation, compiler mode, or mathematical-library function has identical behavior across all devices.

Floating-point arithmetic is not associative. In finite precision, (a+b)+c(a+b)+c can differ from a+(b+c)a+(b+c) because intermediate results are rounded. Parallel reductions change the grouping of operations, so they can produce different answers from a serial implementation even when both are correct under their stated arithmetic rules. Fused multiply-add evaluates a multiplication and addition with a single final rounding, which often improves accuracy but can change results relative to separate instructions. Handling of subnormal values, approximate transcendental functions, and compiler transformations can also affect outputs.

Mixed-precision computing uses different formats for different stages of a calculation. A matrix product may multiply low-precision operands while accumulating into a higher-precision result. Specialized matrix units make such operations particularly efficient. Formats such as binary16 and bfloat16 offer different compromises: binary16 has more fraction bits but a narrower exponent range, while bfloat16 has the same exponent-field width as binary32 but fewer fraction bits. Lower precision reduces storage and data movement as well as potentially increasing arithmetic throughput.

Numerical methods can exploit low-precision speed without simply accepting low-precision answers. In iterative refinement for a linear system Ax=bAx=b, an approximate solution can be computed using a lower-precision factorization. The residual r=b−Axr=b-Ax is then evaluated more accurately, a correction equation is solved, and the solution is updated. Convergence depends on factors including the conditioning of AA, the arithmetic used, and the quality of the approximate solve. In neural-network training, loss scaling can help prevent small gradients from underflowing in formats with limited range, although the details depend on the chosen representation and optimizer.

Reproducibility has several levels. Repeating a calculation on one device with fixed software may be easier than reproducing it across architectures, compiler versions, or different numbers of GPUs. Atomic updates and scheduling-dependent reduction orders can introduce run-to-run differences. Deterministic algorithms, fixed reduction structures, and more accurate accumulation can reduce variation, but may require additional time or storage. Numerical validation therefore distinguishes bitwise equality from agreement within a justified error tolerance, and separates ordinary rounding differences from genuine defects such as races or invalid memory accesses.

Scientific, analytical, and artificial-intelligence applications

Dense linear algebra is particularly well suited to GPUs because many operations combine regular memory access with substantial reuse. Matrix multiplication, factorizations, and related kernels underpin simulations, optimization, signal processing, and machine learning. Fast Fourier transforms also benefit from parallel execution, although their staged communication patterns require careful memory organization. Sparse calculations are less uniform: a sparse matrix stores only selected entries, reducing unnecessary arithmetic and storage, but its irregular indexing can limit bandwidth utilization and create unequal work among threads.

In scientific computing, GPUs accelerate particle interactions, mesh-based differential equations, Monte Carlo methods, and many other workloads. Molecular dynamics provides a representative example. Forces among atoms can be calculated in parallel, with neighbor lists limiting the short-range pairs that need evaluation. Long-range electrostatic methods may combine particle calculations with mesh operations and Fourier transforms. The application must balance these stages, update spatial data structures, and coordinate communication across devices. Programs such as GROMACS, NAMD, and LAMMPS have developed GPU-capable execution paths, with supported features and the extent of acceleration varying by version and configuration.

Image processing and computer vision map naturally onto data-parallel hardware because images contain large grids of related samples. Convolutions, geometric transformations, feature extraction, and optical-flow calculations can process many pixels concurrently. Yet not every stage is equally regular: connected-component analysis and graph-based methods involve dependencies and irregular access. GPU implementation therefore ranges from simple per-pixel kernels to multi-stage algorithms that reorganize the problem around local neighborhoods, hierarchical processing, or sparse work queues.

Artificial-intelligence workloads have become a major driver of GPU architecture and software. Training an artificial neural network repeatedly evaluates forward computations, calculates gradients through backpropagation, and updates parameters. Large matrix products and convolutions provide substantial parallel work, while specialized matrix instructions accelerate supported formats. Transformer models add attention operations, normalization, and data rearrangement to this mix. FlashAttention illustrates an algorithmic response to memory constraints: it reorganizes attention computation to reduce transfers between high-bandwidth memory and on-chip storage rather than merely increasing arithmetic throughput.

Neural-network inference introduces a different balance of objectives, including latency, throughput, memory capacity, and request variability. Large batches can use hardware efficiently, while small interactive requests may leave portions of a device underutilized. Outside AI, GPUs accelerate database operations such as scans, filtering, aggregation, and some joins, as well as password-search and cryptographic hashing workloads. Suitability depends on algorithm structure: repeated independent hashes offer extensive parallelism, whereas memory-hard constructions deliberately make memory capacity and data access central costs. Across these fields, acceleration depends on the complete processing pipeline, not just its most parallel stage.

Multi-GPU systems and distributed execution

A single GPU is limited by its memory capacity, arithmetic throughput, and power envelope. Larger applications use multiple devices within one computer or across a network of computers. This extends GPGPU into distributed computing, where communication patterns and partitioning become as important as individual kernel efficiency. Supercomputers such as Summit, which entered service in 2018 using IBM POWER9 processors and NVIDIA V100 GPUs, demonstrate the combination of general-purpose CPUs, many accelerators, and high-performance interconnects.

Data parallelism assigns different input partitions to different devices while maintaining equivalent computational state. In neural-network training, each device can process part of a batch, compute local gradients, and participate in a collective operation that combines those gradients. Model parallelism instead divides a model’s parameters or operations across devices. Tensor parallelism partitions individual tensor computations, while pipeline parallelism assigns successive groups of layers to different devices. These strategies solve different capacity and throughput problems and can be combined in one training system.

Collective communication includes broadcast, gather, reduce, all-reduce, and all-to-all exchanges. An all-reduce, for example, combines values from all participants and distributes the result back to each participant. Ring-based implementations divide data into chunks and distribute communication over several steps; tree-based methods provide different latency and bandwidth trade-offs. The most effective strategy depends on message size, device count, and network topology. GPU-aware implementations of the Message Passing Interface can accept device buffers, while specialized collective libraries optimize common accelerator communication patterns.

Physical topology constrains these operations. Devices may communicate through the host interconnect, dedicated GPU links, switches, or network adapters. Peer-to-peer transfers and direct movement between network interfaces and device memory can reduce host staging, but support depends on hardware, drivers, and system configuration. A nominal aggregate bandwidth does not imply that every pair of devices can communicate at that rate simultaneously. Placement of communicating tasks, contention, and the relation between local and remote links all influence performance.

Scaling also creates memory and coordination overheads. A domain-decomposed simulation exchanges boundary or “halo” data between neighboring partitions, while graph algorithms may communicate irregular updates. As each device’s local problem becomes smaller, communication and synchronization can consume an increasing fraction of runtime. Overlapping independent computation with transfers helps only when genuine independence exists and the hardware can make concurrent progress. Fault recovery, checkpointing, and load balancing become more significant as device counts rise, particularly when work is uneven or a long-running calculation cannot simply restart from the beginning.

Practical limits, correctness, and portability

The limits of GPGPU follow from the same architectural choices that create its advantages. Small tasks may not contain enough parallel work to amortize launch and transfer costs. Long dependency chains cannot be accelerated merely by adding execution lanes. Pointer-heavy data structures, unpredictable branching, and irregular memory access can leave arithmetic resources underused. CPUs remain important for control-intensive processing, low-latency tasks, and coordination. A heterogeneous design often assigns different stages to the processor best suited to their characteristics rather than attempting to move an entire application onto the GPU.

Correctness is complicated by extensive concurrency. A race condition occurs when conflicting accesses are not properly ordered, and a result depends on execution timing. Barriers and atomic operations address particular coordination needs but have specific scopes and memory-ordering rules. A block-level barrier cannot generally synchronize unrelated blocks, and an atomic update to one variable does not automatically establish every required ordering for other data. Incorrect assumptions can remain hidden on one workload and fail when scheduling, launch dimensions, or compiler behavior changes.

GPU software also interacts with operating-system and deployment constraints. The device driver manages resources, command submission, memory mappings, and hardware access. Display-attached systems may impose execution-time limits on long-running kernels. Shared accelerators require scheduling and isolation among workloads, especially in cloud computing environments. Hardware partitioning can allocate subsets of device resources on supported products, but the available isolation properties and supported features vary. These operational considerations affect whether a computation can run reliably, not just how fast its kernels execute.

Reliability includes both software errors and hardware faults. Scientific accelerators may provide error-correcting memory and mechanisms to report or contain particular faults. Such mechanisms do not detect every possible error in computation. Applications may additionally use residual checks, conservation laws, redundant calculations, or periodic checkpoints according to their requirements. Energy and thermal limits also matter: sustained performance can differ from brief measurements, and utilization affects energy per completed task. GPU acceleration can improve energy efficiency for suitable workloads, but device peak power alone neither proves nor disproves such an improvement.

Portability has several meanings: code may compile on multiple platforms, produce equivalent results, or achieve competitive performance on each. These are distinct achievements. A kernel tuned for one execution-group width, register allocation, cache arrangement, or matrix instruction may behave differently elsewhere. Higher-level programming systems and standardized interfaces reduce some adaptation costs, while architecture-specific libraries often provide stronger optimization for particular operations. The continuing technical challenge is to preserve useful abstractions without concealing the data movement, synchronization, numerical assumptions, and resource constraints that determine GPU behavior.