How fast will Qwen 3.5/3.6 35B A3B feel on your hardware?

Qwen 3.5/3.6 35B A3B speed on GPUs and Macs

Qwen 3.5/3.6 35B A3B is a 35.95B-parameter mixture-of-experts model (3B active per token). Smaller quantizations need less memory and run faster, at some cost in quality. Below is the expected speed on each piece of hardware at the best quantization that fits.

HardwareQuantizationMemoryGenerationFirst token
H200 141GB (cloud)Q4_K_MFits1133 tok/s0.3 s
H100 80GB (cloud)Q4_K_MFits791 tok/s0.3 s
NVIDIA RTX 5090 32GBQ4_K_MFits423 tok/s0.3 s
NVIDIA RTX PRO 6000 96GBQ4_K_MFits423 tok/s0.3 s
AMD Instinct MI50 32GBQ4_K_MFits356 tok/s0.4 s
Mac Studio M5 Ultra 256GBQ4_K_MFits347 tok/s0.3 s
Mac Studio M5 Ultra 512GBQ4_K_MFits312 tok/s0.3 s
AMD RX 7900 XTX 24GBQ4_K_MPartial offload264 tok/s1.2 s
Mac Studio M3 Ultra 512GBQ4_K_MFits237 tok/s0.3 s
NVIDIA RTX 4090 24GBQ4_K_MPartial offload200 tok/s1.2 s
NVIDIA RTX 3090 24GBQ4_K_MPartial offload188 tok/s1.2 s
Mac Studio M4 Max 128GBQ4_K_MFits158 tok/s0.3 s
AMD Radeon AI PRO R9700 32GBQ4_K_MFits148 tok/s0.3 s
NVIDIA Tesla V100 32GBQ4_K_MFits105 tok/s0.3 s
AMD RX 7900 XT 20GBQ4_K_MPartial offload99 tok/s1.2 s
AMD Ryzen AI Max+ PRO 495 192GB (Gorgon Halo)Q4_K_MFits95 tok/s0.3 s
MacBook Pro M5 Max 128GBQ4_K_MFits95 tok/s0.3 s
NVIDIA RTX Spark N1X 128GB (laptop)Q4_K_MFits91 tok/s0.3 s
Mac mini M5 Pro 64GBQ4_K_MFits89 tok/s0.3 s
MacBook Pro M5 Pro 48GBQ4_K_MFits89 tok/s0.3 s
Intel Arc Pro B70 32GBQ4_K_MFits66 tok/s0.3 s
NVIDIA DGX Spark 128GBQ4_K_MFits64 tok/s0.3 s
AMD RX 9070 XT 16GBQ4_K_MPartial offload61 tok/s1.2 s
NVIDIA RTX 5080 16GBQ4_K_MPartial offload61 tok/s1.2 s
NVIDIA RTX 5070 Ti 16GBQ4_K_MPartial offload61 tok/s1.2 s
AMD Ryzen AI Max+ 395 128GB (Strix Halo)Q4_K_MFits53 tok/s0.3 s
NVIDIA RTX 5060 Ti 16GBQ4_K_MPartial offload51 tok/s1.2 s
NVIDIA RTX 3060 12GBQ4_K_MPartial offload39 tok/s1.2 s
CPU-only (Ryzen 9 9950X, 96GB DDR5)Q4_K_MFits33 tok/s0.4 s
MacBook M4 Pro 24GBQ4_K_MDoesn't fitWon't run—

Other models

Estimates for a short prompt on a single device. Measured numbers come from community benchmarks (r/LocalLLaMA, localscore.ai and more); others are derived from similar setups or projected from specs. Open source on GitHub.