CUDA

CUDA (Compute Unified Device Architecture) is a parallel computing platform and programming model developed by NVIDIA. It enables software developers to use NVIDIA graphics processing units (GPUs) for general-purpose computing (that is, General-purpose computing on graphics processing units), rather than being limited to graphics rendering. CUDA encompasses programming language extensions, compilers, runtime libraries, drivers, and numerical libraries. It is one of the most widely used ecosystems for general-purpose GPU computing, with a particularly dominant position in deep learning and scientific computing.

History

GPUs were originally designed for graphics rendering. Before CUDA, programmers who wanted to use GPUs for general-purpose computing typically had to rely on graphics interfaces such as OpenGL or DirectX shaders, disguising computational problems as graphics tasks—a significant barrier to entry. NVIDIA’s G80 architecture (the GeForce 8800 series), introduced around 2006, featured a unified shader architecture that paved the way for general-purpose computing, and CUDA emerged soon afterward. According to NVIDIA’s release archives, CUDA 1.0 was released in June 2007.

CUDA has since continued to expand alongside successive generations of hardware. CUDA 3.0 (2010) enhanced C++ support; CUDA 5.0 (2012) introduced features such as dynamic parallelism; and CUDA 6 (2014) introduced Unified Memory, allowing the CPU and GPU to share a single address space and reducing the burden of manually managing data transfers. In 2012, AlexNet was trained on two GPUs, an achievement widely regarded as one of the milestones in the large-scale adoption of GPUs for deep learning. Since then, CUDA has become increasingly integrated with deep learning frameworks.

Programming Model

A CUDA program typically consists of two parts: the host (the CPU and its memory) and the device (the GPU and its memory). The host prepares data, allocates device memory, and launches computations; the device executes functions called kernels, with large numbers of threads running in parallel.

Threads are organized hierarchically: multiple threads form a thread block, and thread blocks form a grid. Threads within the same block can access shared memory and synchronize with one another, while threads in different blocks generally cannot synchronize directly. At the hardware level, threads are scheduled in groups of 32, called warps. If threads within the same warp take different branches, performance suffers—a phenomenon known as thread divergence.

The following minimal vector addition example illustrates how to define and launch a kernel:

__global__ void vecAdd(const float* a, const float* b, float* c, int n) {
    int i = blockIdx.x * blockDim.x + threadIdx.x;
    if (i < n) c[i] = a[i] + b[i];
}
// Host-side launch: vecAdd<<<(n + 255) / 256, 256>>>(a, b, c, n);

CUDA natively supports C and C++, using syntax extensions—such as the triple-angle-bracket launch syntax above—to specify parallel execution. It can also be used through Fortran interfaces and Python bindings, such as CuPy and Numba's CUDA module.

Hardware Architecture

A CUDA device consists of multiple streaming multiprocessors (SMs), each containing numerous arithmetic units and capable of keeping multiple thread blocks resident simultaneously. Unlike CPUs, which consist of a small number of complex cores, GPUs pursue high throughput using large numbers of relatively simple cores.

The GPU memory hierarchy includes registers, shared memory (on-chip memory shared by threads within a thread block), L1 and L2 caches, and global device memory. Program performance often depends on how effectively this hierarchy is used—for example, by reducing accesses to global device memory, coalescing memory accesses, and avoiding bank conflicts in shared memory. Data transfers between the host and the device take place over interconnects such as PCIe or NVLink and are often a performance bottleneck.

NVIDIA uses compute capability version numbers to identify the feature sets of different GPUs. Architectures such as Volta, Turing, Ampere, Hopper, and Blackwell correspond to different version numbers. Newer architectures introduce specialized units such as Tensor Cores, and CUDA libraries are optimized to take advantage of these units.

Software Stack

The CUDA software stack is typically divided into three layers: the driver handles communication with the hardware; the runtime API provides commonly used functions such as cudaMalloc; and the toolkit includes the compiler, debugging and profiling tools, and a range of numerical libraries.

The nvcc compiler separates and compiles source files containing a mixture of host and device code. Device code is usually first compiled into PTX, an architecture-independent intermediate representation, which the driver or toolchain then translates into machine code (SASS) for a specific GPU. This layered design allows PTX written for older hardware to potentially run on newer hardware.

Driver and toolkit versions must meet certain compatibility requirements. NVIDIA's release notes specify the minimum driver version required for each toolkit version. In practice, deep learning frameworks such as PyTorch and TensorFlow are typically compiled against a specific version of the CUDA toolkit. If the toolkit version is incompatible with the driver or framework version, a program may be unable to use the GPU and silently fall back to running on the CPU.

Core Libraries and Applications

Much of CUDA’s value comes from its extensive library ecosystem, which includes:

cuBLAS: A dense linear algebra library that provides a foundation for Linear Algebra operations;
cuSPARSE and cuSOLVER: Libraries for sparse matrix operations and solvers;
cuFFT: A library for fast Fourier transforms; see Fast Fourier Transform;
cuDNN: A library of fundamental operations for deep learning, including convolution, normalization, and attention;
Thrust: A parallel algorithms library similar to the C++ Standard Template Library;
NCCL: A collective communication library for multiple GPUs and nodes;
TensorRT: An optimization toolkit for inference deployment;
CUTLASS: A template library for writing high-performance kernels for matrix multiplication and other operations.

CUDA is widely used for Deep Learning training and inference, Molecular Dynamics simulations, climate and fluid simulations, image and video processing, financial risk calculations, cryptography, and astrophysics. Many large language models and scientific computing applications rely on CUDA to run, while protein structure prediction systems such as AlphaFold also require substantial GPU computing resources.

Versions

CUDA Toolkit releases are identified by version numbers. According to NVIDIA’s release archive, as of September 2026, the latest stable release is the 13.4 series, with version 13.4.2 released in September 2026; earlier series, including 13.3 and 13.2, remain available in the archive. With major version upgrades, NVIDIA typically drops support for older GPU architectures. For example, some third-party compilations indicate that CUDA 13.x no longer supports the Maxwell, Pascal, and Volta architectures, while retaining support for Turing and later architectures. Because sources differ on such information, users should consult NVIDIA’s official release notes when choosing a version.

Competition and Ecosystem

CUDA is proprietary technology that runs only on NVIDIA GPUs, one of its most frequently cited characteristics. Several vendors and standards organizations offer alternatives. For example, the Khronos Group's OpenCL is an open, cross-vendor standard. AMD offers HIP and the ROCm platform; HIP's syntax closely resembles CUDA's, making code migration easier. SYCL and Intel's oneAPI also target heterogeneous computing. In addition, GPU-oriented programming languages such as OpenAI's Triton reduce reliance on CUDA to some extent.

Analysts generally agree that CUDA's dominance stems not only from the technology itself, but also from its long-established developer community, vast body of existing code, libraries optimized over many years, and deep integration with major frameworks. This ecosystem-level Network Effect makes it difficult for alternative platforms to quickly sway developers' choices, even when they are competitive in performance or cost. This phenomenon has prompted discussions about Switching Cost and market dominance, and the debate continues.

Limitations and Controversies

CUDA’s main limitations include the following:

Vendor lock-in: Code generally runs only on NVIDIA hardware, and moving it to other platforms often requires rewriting it or using translation tools.
Performance portability: Kernels optimized for one architecture generation may not perform efficiently on another. Parameters such as thread block size and memory layout may need adjustment.
Numerical precision: GPUs perform floating-point operations in a different order from CPUs, and operations such as parallel reductions may produce numerical results that differ slightly from their serial counterparts. This poses challenges for reproducibility in scientific computing. For related issues, see Floating-Point Arithmetic.
Programming complexity: Issues involving the memory hierarchy, synchronization, and divergence make writing efficient CUDA code a considerable challenge. Many users therefore access CUDA indirectly through higher-level frameworks rather than writing kernels themselves.
Data transfer overhead: If the amount of computation is small relative to the volume of data transferred between the host and device, the speedup provided by the GPU may be limited.

CUDA’s closed-source nature and market position have also made it a subject of policy debate. Some observers have raised concerns about the concentration of computing resources, standard-setting, and fair competition.

References

CUDA Toolkit Archivedeveloper.nvidia.com
cuda-toolkit · PyPIpypi.org
Release Notes Release 13.4 NVIDIA Corporation Sep 11, 2026docs.nvidia.com
CUDA Programming Guide — CUDA Programming Guidedocs.nvidia.com
Cuda Toolkit Archivedeveloper.nvidia.com
Introduction to CUDA Programming - GeeksforGeeksgeeksforgeeks.org
What is CUDA? Parallel programming for GPUsinfoworld.com
What Is CUDA?supermicro.com
Nvidia's parallel computing platformtheneuralbase.com
CUDA News Today: NVIDIA Toolkit, AMD ROCm, and AI Framework Releases (2026)spheron.network