HBM memory (high-bandwidth memory) is DRAM built as a vertical stack of chips and placed right next to a GPU or AI accelerator, so the two can exchange data over a very wide, short connection. That bandwidth is why AI chips depend on it: an Nvidia Rubin GPU reads its 288 GB of HBM4 at 19.2 TB/s, and an AMD Instinct MI355X reads 288 GB of HBM3E at 8 TB/s. Each HBM generation (HBM3, HBM3E, HBM4) has raised bandwidth per stack, and each accelerator generation has adopted it.
On this page
How HBM is built
Micron describes HBM as “a specialized, high-performance 3D-stacked SDRAM architecture” that “stacks multiple DRAM dies vertically,” connected through “through-silicon vias (TSVs) and microbumps.” SK hynix gives the same idea in one line: HBM “vertically interconnects multiple DRAM chips and dramatically increases data processing speed in comparison to conventional DRAM products.”
Three parts make it work:
- The stack. Several DRAM dies sit on top of one another: 8-high and 12-high in the HBM3E parts Samsung and Micron list, and up to 16-high under the HBM4 standard. More dies mean more capacity in the same footprint.
- TSVs. Through-silicon vias are vertical wires that run through each die, so signals pass straight down the stack instead of out to the edges.
- The interposer. Micron notes that HBM is “integrated directly into a system-in-package (SiP) alongside the CPU or GPU via a silicon interposer.” Because the memory sits beside the processor in the same package, the link between them can be far wider than a memory bus routed across a circuit board.
That width is the defining feature. Micron gives an HBM cube a 1,024-bit data interface, and the HBM4 standard doubles it to 2,048 bits. Bandwidth per stack is roughly the width multiplied by the speed of each pin: 2,048 bits at 8 Gb/s is 2 TB/s, the figure in the HBM4 standard.
Why AI accelerators need it
Running a large language model means reading the model’s weights from memory for every token generated. When the model is large and the batch of requests is small, the chip spends more time waiting for memory than doing math, so more memory bandwidth means more tokens per second in that regime.
Nvidia’s own pages show the effect. The H200 SXM has the same FP8 rating as the H100 SXM, 3,958 teraFLOPS with sparsity, but Nvidia moved it to HBM3e with 141 GB at 4.8 TB/s, which it describes as “nearly double the capacity” of the H100 with “1.4X more memory bandwidth.” The H200 upgrade was memory, not compute.
Capacity matters as much as speed. A model’s weights and its working memory for long contexts have to fit in HBM across the GPUs serving it, so more gigabytes per GPU means fewer GPUs per model.
HBM generations
The figures below come from the JEDEC standard and the memory makers’ own product pages, as of October 2026. Vendor parts often run faster than the standard’s baseline.
| Generation | Pin speed | Bandwidth per stack | Capacity per stack | Source |
|---|---|---|---|---|
| HBM3 | Up to 6.4 Gbps | Up to 819 GB/s | 16 GB, 24 GB | Samsung |
| HBM3E | Up to 9.2 Gbps | Up to 1,180 GB/s (Samsung); more than 1.2 TB/s (Micron) | 24 GB (8-high), 36 GB (12-high) | Samsung, Micron |
| HBM4, JEDEC standard | 8 Gb/s | Up to 2 TB/s | Up to 64 GB (16-high) | JEDEC |
| HBM4, vendor parts | Over 10 Gbps (SK hynix); over 11 Gbps (Micron); up to 13 Gbps (Samsung) | More than 2.8 TB/s (Micron); up to 3.3 TB/s (Samsung) | 36 GB (12-high); 48 GB 16-high sampling (Micron) | SK hynix, Micron, Samsung |
| HBM4E | Up to 16 Gbps | Up to 64 GB (16-high) | Samsung |
HBM3 to HBM3E raised the pin speed, from 6.4 Gbps to 9.2 Gbps on Samsung’s parts, and the 12-high stack brought 36 GB per stack.
HBM3E to HBM4 is the larger step. JEDEC published the HBM4 standard (JESD270-4) on April 16, 2025. It doubles the interface to 2,048 bits and the independent channels per stack from HBM3’s 16 to 32, allows stacks from 4-high to 16-high with up to 64 GB, and stays “backwards compatible with existing HBM3 controllers.” SK hynix said on September 12, 2025 that its HBM4 has 2,048 I/O terminals, “double from the previous generation,” and runs above 10 Gbps against the standard’s 8 Gbps. Micron states its HBM4 is in high-volume production as a 36 GB 12-high stack with more than 2.8 TB/s per stack, more than double its HBM3E.
HBM4E is the next step on Samsung’s product page: up to 16 Gbps and up to 64 GB in a 16-high stack, built on what Samsung calls a 1c DRAM process with a 4 nm logic base die.
Which AI accelerators use which HBM
Memory type, capacity and bandwidth below are as each accelerator’s maker lists them, as of October 2026.
| Accelerator | HBM generation | Memory | Bandwidth |
|---|---|---|---|
| Nvidia H100 SXM | HBM3 | 80 GB | 3.35 TB/s |
| Nvidia H100 NVL | HBM3 | 94 GB | 3.9 TB/s |
| Nvidia H200 | HBM3e | 141 GB | 4.8 TB/s |
| Nvidia DGX B200 (8 GPUs) | HBM3e | 1,440 GB total | 64 TB/s total |
| Nvidia GB200 Superchip (2 GPUs) | HBM3E | 372 GB | 16 TB/s |
| Nvidia Rubin GPU | HBM4 | 288 GB | 19.2 TB/s |
| AMD Instinct MI300X | HBM3 | 192 GB | 5.3 TB/s |
| AMD Instinct MI325X | HBM3E | 256 GB | 6 TB/s |
| AMD Instinct MI355X | HBM3E | 288 GB | 8 TB/s |
Two more data points sit outside the table. Nvidia’s architecture write-up gives the original H100 PCIe card HBM2e at more than 2 TB/s, one generation behind the SXM version. AMD says its MI400 series, launched on July 23, 2026, uses HBM4.
Read down the Nvidia rows and memory bandwidth per GPU grows from 3.35 TB/s on the H100 SXM to 8 TB/s per B200 (64 TB/s across eight in DGX B200) and 19.2 TB/s on Rubin, a 5.7x increase from H100 to Rubin (our calculation from Nvidia’s figures). On the AMD side, capacity grows from 192 GB on MI300X to 288 GB on MI355X, and bandwidth from 5.3 to 8 TB/s.
Full accelerator specs, including compute and power, are in the AI chips catalog; the data centers being built around these chips are tracked on AI data centers.
