{
  "id": "high-bandwidth-memory",
  "title": "High-Bandwidth Memory",
  "lang": "en",
  "source_lang": "en",
  "summary": "High-bandwidth memory is a stacked DRAM technology that uses wide, short connections to supply processors with high data throughput.",
  "category": "computer-science",
  "subcategory": "hardware",
  "type": "concept",
  "aliases": [
    "HBM",
    "high bandwidth memory",
    "high-bandwidth stacked memory"
  ],
  "generated_at": "2026-11-02",
  "model": "pro/gpt-6.1-sol",
  "url": "https://aiwiki.page/en/high-bandwidth-memory/",
  "translations": {
    "zh-Hans": {
      "url": "https://aiwiki.page/zh/high-bandwidth-memory/",
      "json": "https://aiwiki.page/zh/high-bandwidth-memory.json"
    }
  },
  "sections": [
    {
      "heading": "",
      "text": "High-bandwidth memory (HBM) is a family of dynamic random-access memory technologies that combines vertically stacked memory dies with an exceptionally wide interface to a nearby processor. Rather than obtaining bandwidth primarily by increasing the signaling speed of a relatively narrow connection, HBM transfers data across thousands of closely spaced connections. It is used with graphics processing units, computing accelerators, and some central processing units where the rate of moving data is a major constraint on performance.\n\nHBM is both a memory-interface technology and an advanced packaging system. Its practical implementation depends on vertical connections through memory dies, fine-pitch connections between chips, and a package capable of linking the memory to a processor. These features provide high bandwidth and favorable energy consumption per transferred bit, but also make HBM more expensive and less readily replaceable than conventional memory modules. Its development has consequently been closely connected with high-performance computing and, increasingly, the training and operation of large artificial intelligence models."
    },
    {
      "heading": "Memory organization and operating principles",
      "text": "Like other forms of DRAM, HBM stores information in cells whose electrical charge must be periodically refreshed. Accessing data involves selecting a row, sensing its contents, transferring requested data, and eventually preparing the bank for another operation. Stacking does not remove these underlying DRAM operations. HBM’s principal advantage comes from allowing many operations and transfers to proceed in parallel, rather than from making each individual memory cell dramatically faster.\n\nAn HBM stack contains several memory dies and supporting interface circuitry, commonly including a base die beneath the memory layers. Vertical electrical connections carry data, commands, addresses, power, and control signals through the stack. The processor communicates with the stack through a memory controller and physical interface designed for the relevant HBM generation. Several stacks may surround one processor, providing additional capacity and multiple independent paths for data movement.\n\nThe wide interface is divided into channels. A first-generation HBM stack has eight 128-bit channels, producing an aggregate data width of 1,024 bits. Later generations reorganize this interface to increase concurrency and improve access efficiency. HBM2 introduced a pseudo-channel mode in which a 128-bit channel can operate as two 64-bit portions with partly independent operation. HBM3 uses sixteen 64-bit channels and provides further subdivision into pseudo-channels. The precise sharing of command and timing resources matters: a pseudo-channel is not simply an entirely separate memory device.\n\nWithin each channel, banks provide another level of parallelism. A memory controller can schedule work across banks while an individual bank is occupied by activation, precharge, or refresh. Sequential and well-distributed requests can therefore sustain much higher throughput than a sequence of dependent requests to the same bank. Address mapping—how addresses select stacks, channels, banks, rows, and columns—has a direct influence on how effectively an application uses the available hardware.\n\nCapacity, interface width, and stack height are related but distinct properties. Adding more memory layers usually increases capacity; it does not necessarily widen the external interface or proportionally increase bandwidth. Likewise, a processor with more HBM stacks generally has greater aggregate bandwidth only if it also provides the controllers, package connections, and internal data paths needed to use them. The resulting memory system is an organized collection of parallel resources, not a single uniformly accessible reservoir whose peak transfer rate is available to every request."
    },
    {
      "heading": "Three-dimensional stacking and package integration",
      "text": "HBM depends on through-silicon vias (TSVs), conductive structures that pass through a silicon die. TSVs allow signals to travel vertically between memory layers instead of requiring each die to connect separately to the outside of a package. Manufacturing these connections involves creating and insulating openings, filling them with conductive material, thinning wafers, and aligning the resulting dies with sufficient precision for dense vertical interconnects.\n\nThe dies are joined using fine-pitch interconnections, traditionally microbumps, with materials and processes that provide both electrical contact and mechanical support. Increasing stack height makes assembly more demanding: many thin dies must remain aligned, individual connections must function reliably, and the stack must fit within package-height and thermal constraints. The base die supplies interface and support functions whose extent varies between implementations. The presence of a base die does not, by itself, mean that application computation takes place inside the memory.\n\nA common arrangement places the processor and several HBM stacks side by side on a silicon interposer. This structure contains dense wiring that connects the processor’s memory interfaces to the stacks. The interposer then connects to a package substrate, which provides larger-scale power and signal connections to the circuit board. The arrangement is often called 2.5-dimensional integration: the memory is internally stacked in three dimensions, while the main processor and memory stacks occupy neighboring positions on a shared interconnect structure.\n\nThe interposer is important because an ordinary circuit board cannot economically route thousands of extremely short, closely spaced connections at comparable density. Keeping these connections short also reduces electrical loading and simplifies some aspects of signal transmission. Other packaging approaches can use embedded silicon bridges or different high-density redistribution structures. HBM therefore requires suitable advanced interconnection, but it is not defined by one manufacturer’s particular packaging process.\n\nPackaging imposes physical limits on the complete device. More stacks require additional area around the processor, more routing, more power delivery, and a larger cooling surface. Large interposers may require specialized fabrication and lithographic stitching. Mechanical deformation, differences in thermal expansion, and stresses introduced during assembly must also be controlled. A processor design can consequently be limited by package size or manufacturing practicality before it exhausts the theoretical capabilities of its memory interface.\n\nUnlike a socketed memory module, HBM is normally part of the processor package and is not field-upgradable. A failed stack or interconnect can affect the usability of the entire expensive assembly. Testing and repair provisions are therefore integral to the technology rather than secondary manufacturing details."
    },
    {
      "heading": "Standards and historical development",
      "text": "HBM developed from work on stacked DRAM and wide memory interfaces involving memory manufacturers, processor designers, and standards organizations. JEDEC published the original JESD235 HBM standard in 2013. An early prominent commercial deployment was AMD’s Fiji graphics processor, used in Radeon R9 Fury products introduced in 2015. Fiji combined four first-generation HBM stacks with the GPU on an interposer, providing 4 GB of memory and an aggregate bandwidth of 512 GB/s. It demonstrated the technology’s compact physical layout while also illustrating the capacity limitations of early stacks.\n\nFirst-generation HBM used a 1,024-bit data interface per stack, with implementations around 1 gigabit per second per data pin. The resulting bandwidth was approximately 128 GB/s per stack. HBM2 increased supported capacity and signaling rates while retaining the 1,024-bit aggregate data width. At 2 gigabits per second per pin, one stack provides 256 GB/s of theoretical bandwidth. HBM2 became important in computing accelerators and high-end graphics products, including NVIDIA’s Pascal-generation P100 accelerator introduced in 2016.\n\nThe following table shows representative interface limits or rates rather than a guarantee for every product:\n\nGeneration or designation | Aggregate data width per stack | Representative per-pin rate | Corresponding theoretical bandwidth per stack\nHBM | 1,024 bits | 1.0 Gb/s | 128 GB/s\nHBM2 | 1,024 bits | 2.0 Gb/s | 256 GB/s\nHBM2E | 1,024 bits | 3.2 Gb/s | 409.6 GB/s\nHBM3 | 1,024 bits | 6.4 Gb/s | 819.2 GB/s\nHBM4 | 2,048 bits | 8.0 Gb/s | 2,048 GB/s\n\nHBM2E became a widely used designation for enhanced HBM2 products offering higher rates and capacities. JEDEC published HBM3 as JESD238 in 2022. HBM3 raised the standardized transfer rate to 6.4 gigabits per second per pin, increased channel parallelism, and expanded provisions for capacity and reliability. HBM3E subsequently became an industry designation for higher-speed implementations of the HBM3 family. The “E” labels are useful commercial distinctions, but they should not automatically be treated as separate standards with the same status as a new numbered generation.\n\nJEDEC published the HBM4 standard in 2025. A central architectural change was doubling the aggregate data-interface width to 2,048 bits per stack and increasing the channel count to 32. At 8 gigabits per second per pin, that width corresponds to 2.048 terabytes per second per stack in decimal units. As with previous generations, a standard’s supported capabilities must be distinguished from the operating specifications of a particular shipping device.\n\nSuccessive generations are not interchangeable components. Changes to signaling, channel organization, power delivery, controller logic, and physical connections require coordinated processor and package designs. HBM’s historical progression has therefore involved complete platforms rather than a simple sequence of replacement memory chips."
    },
    {
      "heading": "Bandwidth, latency, and performance measurement",
      "text": "The theoretical bandwidth of a memory interface follows from its aggregate data width and per-pin transfer rate:\n\n\\[B_{\\text{peak}}=\\frac{W R}{8},\\]\n\nwhere \\(W\\) is the width in bits and \\(R\\) is the transfer rate per pin in transfers per second. For an interface transferring one bit per pin per transfer, a 1,024-bit stack operating at 6.4 gigabits per second per pin yields 819.2 GB/s. Four such stacks would have a combined theoretical bandwidth of 3.2768 TB/s. These figures use decimal gigabytes and terabytes; they are not identical to binary gibibytes and tebibytes.\n\nPeak bandwidth measures the interface’s capacity under ideal conditions, not the rate attained by arbitrary software. Refresh, row activation, precharge, read–write turnarounds, scheduling conflicts, and incomplete utilization of transferred data all reduce useful throughput. The application may also be unable to generate enough outstanding requests. A processor can have an extremely wide external interface while being limited by its internal interconnect, caches, instruction issue rate, or ability to track pending memory operations.\n\nLatency is a different quantity: the elapsed time between issuing a request and receiving its result. HBM’s short physical connections do not eliminate DRAM core timing. A chain of dependent accesses, such as following pointers through a linked structure, may show limited improvement because each request must wait for the preceding one. By contrast, a large stream of independent requests can exploit many channels and banks simultaneously. High bandwidth is therefore especially valuable when an application exposes substantial memory-level parallelism.\n\nThe relationship between computation and memory traffic can be described using the roofline model. If \\(I\\) is arithmetic intensity, measured in operations per byte transferred from memory, then an approximate performance ceiling is\n\n\\[P_{\\text{attainable}}\\leq \\min\\left(P_{\\text{compute}},\\,B_{\\text{sustained}}I\\right).\\]\n\nAn operation performing two floating-point operations for every eight bytes read has an intensity of only 0.25 operations per byte before accounting for output traffic. Even an extremely powerful processor will execute such work at a rate constrained by memory movement. Increasing data reuse can raise arithmetic intensity and sometimes improve performance more than increasing raw bandwidth.\n\nMeaningful measurement therefore requires more than quoting a vendor’s peak number. Tests must specify access pattern, working-set size, read–write mixture, transfer size, and the number of concurrent requests. They must also distinguish HBM traffic from traffic satisfied by on-chip caches. A benchmark fitting entirely in cache measures a different part of the memory hierarchy, even if its apparent transfer rate exceeds the external memory specification. Application-level measurements additionally need to separate data movement within a device from transfers between the device and its host system."
    },
    {
      "heading": "Power consumption, cooling, and reliability",
      "text": "HBM’s energy efficiency derives substantially from short interconnects and a wide interface that need not run each data pin as fast as narrower high-speed memory systems. A simplified expression for dynamic switching power is\n\n\\[P_{\\text{dynamic}}\\approx \\alpha C V^{2}f,\\]\n\nwhere \\(\\alpha\\) represents switching activity, \\(C\\) effective capacitance, \\(V\\) voltage, and \\(f\\) frequency. Shorter wiring can reduce capacitance, while lower signaling voltage reduces switching energy. Actual memory power also includes refresh, internal array operations, clocking, background activity, and losses in the supporting circuitry.\n\nEnergy per transferred bit and total power are not equivalent. An HBM subsystem can be more efficient per bit than another memory technology while consuming substantial total power because it moves far more data. Higher transfer rates, more stacks, and greater capacity all affect the package’s power requirements. Some energy is consumed even when useful bandwidth is low, so workload behavior and power-management states influence efficiency.\n\nThermal design is complicated by the physical stack. Heat must travel through multiple material interfaces, and temperatures need not be uniform across layers. The adjacent processor may also be a powerful heat source. A common heat spreader or cold plate must accommodate differences in component height, contact pressure, and thermal resistance. Raising memory temperature can increase leakage and refresh requirements and may force operation at a reduced rate. Cooling is thus a condition for sustained performance, not merely a way of preventing immediate failure.\n\nReliability mechanisms operate at several levels. DRAM devices may use internal correction and repair, while a complete memory subsystem can provide error-correcting codes, error reporting, redundant resources, and other protection. Internal correction is not identical to end-to-end protection of application data: an error occurring beyond the protected internal region may require separate detection. Consequently, an accelerator’s reliability specification cannot be inferred solely from the generation of HBM that it uses.\n\nTesting must identify defective memory cells, faulty vertical connections, and assembly problems. Manufacturers seek to use known-good dies before stacking and to test again at intermediate and final stages. Spare resources can repair some defects, but not every failure is recoverable. More layers and connections create more opportunities for a defect to affect the assembly, although mature processes, redundancy, and improved testing can mitigate this effect.\n\nPower delivery presents another integration problem. Large and rapidly changing currents can produce voltage fluctuations, requiring suitable distribution networks and decoupling. Reliable high-speed operation therefore depends jointly on memory design, package construction, controller behavior, cooling, and electrical regulation."
    },
    {
      "heading": "Applications in scientific computing and artificial intelligence",
      "text": "HBM is particularly useful for workloads whose processors would otherwise spend much of their time waiting for data. In scientific computing, examples include sparse matrix–vector multiplication, stencil calculations, graph processing, and some stages of numerical simulation. These operations often perform relatively little arithmetic for each byte fetched. Large numbers of execution units can make the bandwidth mismatch more severe: adding arithmetic hardware does little if the memory system cannot keep it supplied.\n\nNot all scientific applications benefit equally. Dense matrix multiplication can reuse blocks of data many times in on-chip storage, achieving high arithmetic intensity. Its performance may be limited by computation rather than HBM bandwidth once blocking is effective. Sparse operations have less predictable access and may transfer indices as well as numerical values. Their performance depends on data structure, locality, and access concurrency, so a higher advertised bandwidth does not translate directly into the same percentage improvement in execution time.\n\nIn deep learning, HBM holds model parameters, activations, gradients, optimizer state, and temporary working data. Training can require much more memory than the stored weights alone suggest. A model trained with the Adam optimizer, for example, commonly needs first- and second-moment state in addition to weights and gradients. Mixed-precision techniques complicate the accounting further because some implementations maintain higher-precision copies of selected values. The exact footprint depends on framework, numerical format, optimizer, and distributed-training strategy.\n\nFor large language models, inference has distinct phases. Processing an input prompt can expose considerable parallel computation, while generating subsequent tokens at a small batch size often involves repeatedly reading model weights for comparatively little new arithmetic. A rough bandwidth-based lower bound is\n\n\\[T\\geq \\frac{D}{B_{\\text{sustained}}},\\]\n\nwhere \\(D\\) is the amount of data that must actually be read from HBM during the step. This is not a complete latency model: caching, batching, communication, conditional computation, and overlap can alter the amount of traffic and its critical path.\n\nThe key–value cache used by transformer models creates additional capacity and bandwidth demands. It retains attention-related data from earlier tokens so that they do not need to be recomputed at every step. Its size depends on sequence length, batch size, layer count, attention configuration, and numerical representation. Longer contexts can therefore increase memory requirements even when the model’s parameters are unchanged.\n\nHBM does not solve communication between accelerators. A model split across several devices may spend substantial time exchanging activations, parameters, or partial results over a separate interconnect. Each device’s local HBM bandwidth and the system’s device-to-device bandwidth are different resources. Effective parallel computing requires balancing both rather than treating memory throughput as the only bottleneck."
    },
    {
      "heading": "Programming models and data placement",
      "text": "HBM is generally not addressed through a special application instruction simply because it is stacked memory. Software sees it through the processor’s memory model. On many discrete accelerators, it serves as local device memory allocated and accessed through facilities such as CUDA or other programming interfaces. A kernel reads and writes ordinary device addresses, while the hardware controller distributes the requests across HBM resources.\n\nThe distinction between local memory and host memory remains important. A discrete accelerator may have several terabytes per second of local memory bandwidth while its connection to the host provides far less. Repeatedly transferring data over PCI Express can therefore dominate an application even when the accelerator’s HBM is fast. Keeping intermediate results on the device, combining operations, and overlapping transfers with computation can reduce this imbalance. These are properties of the overall data path, not improvements to HBM itself.\n\nSome CPUs integrate HBM as an additional memory tier. Intel’s Xeon CPU Max series, introduced in 2022, incorporated 64 GB of HBM2E per processor package. Such systems can expose HBM through different operating modes, including use as directly addressable memory or as a hardware-managed cache for a larger conventional memory capacity. The exact mode determines whether the application or system software must explicitly manage placement and whether the working set can exceed local HBM capacity without leaving the high-bandwidth tier entirely.\n\nWhen multiple memory types or processor regions have different access costs, non-uniform memory access becomes relevant. Allocation policy, thread placement, and which processor first initializes a page can influence where data resides. A program with a suitable memory-access pattern may benefit substantially from careful placement, while a program that continually accesses a remote tier may fail to achieve the bandwidth available near its executing cores. Automatic placement can help, but it cannot eliminate the consequences of finite capacity and asymmetric connections.\n\nAt the kernel level, efficient use requires contiguous or otherwise well-organized accesses, enough parallel requests, and appropriate reuse in registers and caches. On GPUs, neighboring threads can often combine their requests into fewer memory transactions. Poorly aligned or scattered access may fetch many bytes that are never used. Blocking, fusion, and data-layout changes can reduce external traffic without altering the hardware.\n\nAlgorithms can also avoid unnecessary HBM transfers. FlashAttention, for example, reorganizes attention computation around tiled operations and on-chip storage rather than materializing the full attention matrix in external memory. This illustrates an important distinction: HBM supplies a high-capacity data path, while software transformations determine how much data must cross that path. Faster memory and better algorithms address related but separate parts of the performance problem."
    },
    {
      "heading": "Cost, competing technologies, and development constraints",
      "text": "HBM competes with other memory technologies through a combination of bandwidth, capacity, energy use, physical size, and cost. Conventional DDR SDRAM attached through memory modules offers large, comparatively economical, and often replaceable capacity. GDDR SDRAM supplies high bandwidth to many graphics processors using faster signaling over narrower per-device interfaces and circuit-board connections. HBM achieves a very wide interface in a small area near the processor, but requires a more elaborate package. None of these approaches is universally superior across all system sizes and workloads.\n\nHBM is also distinct from on-chip cache. Cache made from static random-access memory can provide very low latency and high local bandwidth, but it occupies expensive processor-die area and generally offers much less capacity. HBM occupies a middle position: much denser than a large SRAM cache and much closer to the processor than ordinary memory modules, while retaining DRAM’s access and refresh characteristics. Some systems use all three levels together.\n\nIts cost includes more than the DRAM cells. TSV processing, wafer thinning, precision stacking, testing, interposers or alternative interconnect structures, package substrates, and advanced assembly all contribute. The memory stacks must be available in the required capacity and speed grade, and the processor must be designed to match them. A package defect can jeopardize components that were individually valuable, making manufacturing yield and fault isolation economically significant.\n\nSK hynix, Samsung Electronics, and Micron Technology are major HBM suppliers. Delivering complete HBM-equipped processors also depends on foundries, packaging providers, substrate manufacturers, equipment makers, and test facilities. Capacity at one stage cannot automatically compensate for a shortage at another. Expanding DRAM wafer production alone, for example, does not necessarily provide the additional stacking and advanced-packaging capacity needed to ship more finished accelerators.\n\nFurther development involves several interacting directions: faster signaling, wider interfaces, more memory layers, denser DRAM dies, improved cooling, and more capable base-die circuitry. Each has different consequences. More layers can increase capacity without proportionally increasing interface bandwidth; wider interfaces require more connections and controller resources; higher signaling rates make electrical and thermal design harder. The balance among these changes is as important as any single headline specification.\n\nMore advanced base dies also create opportunities for integrating additional interface, test, or control functions. Such changes should be distinguished from processing in memory, in which computation is deliberately moved into or near memory arrays. Ordinary HBM remains a memory technology even when it uses sophisticated support logic. Whether a particular system places computation in the processor, the base die, or the memory device is an architectural choice, with separate implications for programmability, manufacturing, and data movement."
    }
  ],
  "links": [
    "graphics-processing-unit",
    "central-processing-unit",
    "artificial-intelligence",
    "jedec",
    "roofline-model",
    "memory-hierarchy",
    "error-correcting-code",
    "deep-learning",
    "large-language-model",
    "transformer-architecture",
    "parallel-computing",
    "cuda",
    "flashattention",
    "gddr-sdram",
    "static-random-access-memory",
    "micron-technology"
  ],
  "sources": [],
  "license": "https://creativecommons.org/licenses/by/4.0/"
}