AI inference vs training comes down to building a model versus running it. Large-scale training adjusts a model’s weights across many batches of data on tightly linked chips. Inference uses the trained weights to process requests, where response speed and the resources used per answer matter. Hardware vendors tune systems for different parts of this work. Their announcements distinguish planned products from available systems.
On this page
What training does
Training is a loop. The model reads a batch of data, predicts the next token, compares its prediction with the right answer and adjusts its weights to reduce the error. Microsoft describes the cycle in its Fairwater post as repeating “trillions of times until the system gets better at whatever it’s being trained to do.”
A conventional full-training step involves a forward pass (the prediction), a backward pass (working out how the trainable weights contributed to the error) and a weight update. For a large model that work is spread over thousands or hundreds of thousands of accelerators, which must exchange results after each step. Three things follow:
- Throughput matters more than latency. What counts is how many training tokens the whole cluster processes per day, not how fast any single answer arrives.
- The network is part of the computer. Chips wait on each other at every step, so the bandwidth between chips and racks sets how well a cluster scales.
- Memory holds more than the model. Training keeps weights, gradients, optimizer state and intermediate activations in memory at once.
What inference does
Inference runs only the forward pass, and it happens every time someone sends a prompt. It has two phases with different bottlenecks:
- Prefill processes the whole prompt at once. It is compute-heavy and parallel, much like a training step without the backward pass.
- Decode generates the answer one token at a time. For transformer language models, decode repeatedly accesses model weights and the accumulated key-value (KV) cache. At small batch sizes, memory traffic and communication can dominate; the bottleneck changes with batch size, context length and model architecture.
This is why inference hardware is judged on two numbers that pull against each other: latency, how quickly a single user gets tokens back, and throughput, how many tokens the system serves across all users. Batching many requests together raises throughput but can slow each individual answer. Agentic workloads, where a model calls tools and other models in long chains, make per-token latency more visible because small delays add up across steps.
Precision formats: FP8 and FP4
Both workloads use reduced-precision number formats to fit more operations into the same silicon and memory. Formats such as BF16, FP8 and FP4 offer different precision and resource requirements. A hardware specification alone does not establish which format will preserve a particular model’s quality. The choice depends on the model, the training or serving software, and validation of the resulting outputs.
Fewer bits per value means less memory per weight and more operations per second, so headline figures are often quoted at a chip’s lowest supported precision. Two cautions apply when reading spec sheets:
- Compare chips only at the same precision. An FP4 figure is not comparable with an FP8 or BF16 figure.
- Vendors use their own variants. AWS lists “configurable FP8” (cFP8) on Inferentia2 and MXFP8 and MXFP4 on Trainium3; Microsoft notes that Nvidia Blackwell supports low-precision formats such as FP4.
Inference-first hardware
Google TPU 8i. Google announced its eighth-generation TPUs on April 22, 2026 as two chips. In Google’s Cloud Next post, TPU 8i is the chip “for inference and reinforcement learning,” built for low latency. Google says it triples on-chip SRAM to 384 MB, raises HBM to 288 GB so large KV caches can stay on the chip, doubles inter-chip bandwidth to 19.2 Tb/s and adds a Collectives Acceleration Engine that cuts on-chip latency by up to 5x. Google claims 80% better performance per dollar for inference than the previous generation. Its April announcement described Cloud availability as forthcoming, so those announced specifications should not be read as a guarantee of access in every region.
Nvidia Groq 3 LPX. Nvidia’s LPX page describes a rack of 256 Groq 3 LPU chips, each with 500 MB of SRAM and 150 TB/s of SRAM bandwidth, for 128 GB of SRAM per rack. In one configuration described in Nvidia’s technical blog LPX operates beside Vera Rubin NVL72 racks, with Rubin GPUs handling prefill and the attention part of decode, and LPX handling latency-sensitive decode work such as mixture-of-experts layers. Nvidia said on August 24, 2026 that LPX is in full production. It reported an Artificial Analysis benchmark of 3,431 output tokens per second on Gemma 4 31B at 100,000 tokens of context, a figure Nvidia published.
AWS Inferentia2. AWS’s Inferentia page describes Inferentia2 as a chip for “deep learning (DL) and generative AI inference.” AWS’s product page lists 32 GB of HBM per chip; its architecture documentation specifies 32 GiB. Per AWS’s Neuron documentation, each chip delivers 190 TFLOPS in FP16, BF16, cFP8 and TF32, with 820 GiB/s of memory bandwidth.
Microsoft Azure Maia 300. Microsoft unveiled Maia 300 on October 12, 2026 for 2027, when Microsoft says it will begin scaling the Maia family to multi-gigawatt deployments as the primary inference system for its products and services. Microsoft lists 288 GB of HBM4 at 18.0 TB/s per chip and 96 accelerators in a 167 kW rack; our Maia 300 report has the rack details.
Intel GPU D-series. Intel named its Crescent Island inference GPU the Intel GPU D-series at the OCP Global Summit in October 2026. It takes the opposite approach to HBM-based designs: Intel quotes up to 480 GB of LPDDR5X memory, 1.5 TB/s of memory bandwidth and 1.3 petaflops of FP8 compute in a 350-watt, air-cooled PCIe card, and its D270 product page pitches it at long-context and high-concurrency serving. Intel’s third-quarter 2026 results, released October 29, still call the D-series “upcoming”.
Training-first hardware
Google TPU 8t. Google calls TPU 8t its “training powerhouse.” It packs 9,600 chips into a superpod with 121 exaflops of compute and 2 PB of shared memory, and Google says it delivers nearly 3x the compute performance per pod of the previous generation. Google says more than one million TPU chips can be connected into a single training cluster.
AWS Trainium3. AWS made Trn3 UltraServers generally available on December 2, 2025. AWS says each Trainium3 chip provides 2.52 PFLOPS of FP8 compute and 144 GB of HBM3e at 4.9 TB/s, and an UltraServer scales to 144 chips and 362 FP8 PFLOPS. AWS’s Neuron documentation lists 2,517 TFLOPS for both MXFP8 and MXFP4 and 671 TFLOPS in BF16. The name says training, but AWS now positions Trainium for “AI training and inference at scale.”
That last point is typical. The split is about which workload a chip is tuned for, not what it can run. Google’s TPU page describes TPU 8i as “optimized for post-training and inference,” and Nvidia calls Vera Rubin NVL72 “the flexible, general-purpose workhorse for training and inference.”
Side by side
| Training | Inference | |
|---|---|---|
| What runs | Forward pass, backward pass, weight update | Forward pass only (prefill, then decode) |
| Main goal | Cluster throughput, time to a finished model | Latency per user and tokens served per system |
| Usual bottleneck | Compute and chip-to-chip bandwidth | Memory bandwidth during decode |
| What sits in memory | Weights, gradients, optimizer state, activations | Weights and the KV cache |
| Precision | Depends on model and training recipe | Depends on model and serving configuration |
| Example hardware | Google TPU 8t, AWS Trainium3 | Google TPU 8i, Nvidia Groq 3 LPX, AWS Inferentia2, Microsoft Maia 300, Intel GPU D-series |
| Chip | Vendor focus | Memory per chip | Vendor figure |
|---|---|---|---|
| Google TPU 8t | Training | 2 PB per 9,600-chip superpod | 121 exaflops per 9,600-chip superpod |
| Google TPU 8i | Post-training and inference | 288 GB HBM, 384 MB SRAM | 80% better inference performance per dollar vs prior generation |
| Nvidia Groq 3 LPU (in LPX) | Low-latency inference | 500 MB SRAM | 150 TB/s SRAM bandwidth per chip |
| AWS Inferentia2 | Inference | 32 GB HBM | 190 TFLOPS FP16/BF16/cFP8 |
| Microsoft Maia 300 | Inference | 288 GB HBM4 | 18.0 TB/s memory bandwidth |
| Intel GPU D-series | Inference | Up to 480 GB LPDDR5X | 1.3 PFLOPS FP8 |
| AWS Trainium3 | Training and inference | 144 GB HBM3e | 2.52 PFLOPS FP8 |
Full vendor specs for these and other accelerators are in our AI chips catalog, and the facilities that house them are in the AI data centers tracker.
