How fast will Gemma 4 26B A4B feel on your hardware?

Gemma 4 26B A4B speed on GPUs and Macs

Gemma 4 26B A4B is a 25.81B-parameter mixture-of-experts model (3.8B active per token). Smaller quantizations need less memory and run faster, at some cost in quality. Below is the expected speed on each piece of hardware at the best quantization that fits.

Hardware	Quantization	Memory	Generation	First token
H200 141GB (cloud)	Q4_K_M	Fits	775 tok/s	0.3 s
H100 80GB (cloud)	Q4_K_M	Fits	541 tok/s	0.3 s
A100 80GB (cloud)	Q4_K_M	Fits	329 tok/s	0.3 s
NVIDIA RTX 5090 32GB	Q4_K_M	Fits	289 tok/s	0.3 s
NVIDIA RTX PRO 6000 96GB	Q4_K_M	Fits	289 tok/s	0.3 s
AMD RX 7900 XTX 24GB	Q4_K_M	Fits	240 tok/s	0.3 s
Mac Studio M5 Ultra 512GB	Q4_K_M	Fits	227 tok/s	0.3 s
Mac Studio M3 Ultra 512GB	Q4_K_M	Fits	171 tok/s	0.3 s
NVIDIA RTX 4090 24GB	Q4_K_M	Fits	163 tok/s	0.3 s
AMD Radeon AI PRO R9700 32GB	Q4_K_M	Fits	161 tok/s	0.3 s
NVIDIA RTX 3090 24GB	Q4_K_M	Fits	151 tok/s	0.3 s
Mac Studio M4 Max 128GB	Q4_K_M	Fits	114 tok/s	0.3 s
AMD RX 9070 XT 16GB	Q4_K_M	Partial offload	92 tok/s	1.2 s
MacBook Pro M5 Max 128GB	Q4_K_M	Fits	87 tok/s	0.3 s
NVIDIA RTX 5080 16GB	Q4_K_M	Partial offload	83 tok/s	1.2 s
NVIDIA RTX 5070 Ti 16GB	Q4_K_M	Partial offload	81 tok/s	1.2 s
MacBook Pro M5 Pro 48GB	Q4_K_M	Fits	77 tok/s	0.3 s
NVIDIA RTX 4070 Ti Super 16GB	Q4_K_M	Partial offload	70 tok/s	1.2 s
AMD RX 7900 XT 20GB	Q4_K_M	Fits	66 tok/s	0.3 s
Mac Mini M4 Pro 64GB	Q4_K_M	Fits	57 tok/s	0.4 s
MacBook M4 Pro 24GB	Q4_K_M	Fits	57 tok/s	0.4 s
NVIDIA RTX 5060 Ti 16GB	Q4_K_M	Partial offload	55 tok/s	1.2 s
Intel Arc Pro B70 32GB	Q4_K_M	Fits	51 tok/s	0.3 s
MacBook M3 Max 48GB	Q4_K_M	Fits	50 tok/s	0.3 s
AMD Ryzen AI Max+ 395 128GB (Strix Halo)	Q4_K_M	Fits	49 tok/s	0.3 s
NVIDIA DGX Spark 128GB	Q4_K_M	Fits	44 tok/s	0.3 s
NVIDIA RTX 4070 12GB	Q4_K_M	Partial offload	44 tok/s	1.2 s
NVIDIA RTX 3060 12GB	Q4_K_M	Partial offload	38 tok/s	1.2 s
CPU-only (Ryzen 9 9950X, 96GB DDR5)	Q4_K_M	Fits	24 tok/s	0.4 s
MacBook M2 Pro 16GB	Q4_K_M	Doesn't fit	Won't run	—

Other models

Estimates for a short prompt on a single device. Measured numbers come from community benchmarks (r/LocalLLaMA, localscore.ai and more); others are derived from similar setups or projected from specs. Open source on GitHub.