Google launched new vLLM and SGLang inference engines for TPU on October 20, 2026, built on TorchTPU, a layer that runs PyTorch natively on Google’s accelerators. Both serving engines already ran on TPU through JAX-based backends; the new versions let teams write, customise and debug their TPU serving stack in PyTorch instead. Google announced them on the Google Developers Blog.

On this page

What Google released

Google built the two engines with the companies founded by each project’s creators: the vLLM backend with Inferact and the SGLang backend with RadixArk.

Underneath both is TorchTPU. Google says it makes the TPU a native PyTorch device: tensors are standard PyTorch tensors on a tpu device, code runs eagerly or compiles through torch.compile with XLA, and collectives go through torch.distributed. Each engine keeps its own scheduler, KV cache manager and API server, with TPU implementations underneath. Most models use the standard upstream PyTorch model definitions, and TPU-specific speed-ups come from custom kernels written in Pallas.

Google lists these supported features for both engines:

  • automatic prefix caching, KV cache offloading to host memory and Managed Lustre, tool calling and structured output;
  • speculative decoding with MTP, EAGLE-3 and n-gram drafters, among others;
  • data-parallel attention with expert parallelism, and context parallelism for long prompts;
  • disaggregated serving, with prefill and decode on separate TPU hosts;
  • vision-language and embedding models;
  • FP8, FP4 (MXFP4, NVFP4), INT8 and INT4 checkpoints.

KV cache transfers for disaggregated serving and offload go through TPU Sync, an open-source C++ library that Google says reads the device buffers behind PyTorch tensors directly, without an extra copy on the host.

Who runs it today

Google says both engines are running in production with preview customers. In the announcement, Meta’s PyTorch-TPU lead Zongwei Zhou says SGLang-TorchTPU “is running in production at Meta today.”

For vLLM, Google says it worked with Inferact on a Pallas kernel for Moonshot AI’s 2.8-trillion-parameter Kimi K3 that runs the whole decode step as one kernel, serving more than 700 tokens per second to a single user on Ironwood TPU. Google publishes its performance results through the SemiAnalysis InferenceX project; those figures are Google’s own submissions.

What stays the same

The JAX engines are not going away. Google says vLLM-JAX and SGLang-JAX will continue to be maintained and supported. On models, Google says the two engines together already cover most popular open model families, including Qwen, DeepSeek, GLM, Kimi, GPT-OSS and Gemma, and that it is working on day-0 support for new open models.

For teams evaluating a move, the practical point is that TorchTPU does not ask them to rewrite PyTorch serving code in JAX. It does not guarantee that every vLLM or SGLang feature behaves the same on TPU as on GPU, so the supported-feature list above is the place to start. TPU generations and specifications are in our Google TPU explainer and the AI chips comparison.