Qwen 3.5/3.6 35B A3B is a 35.95B-parameter mixture-of-experts model (3B active per token). Smaller quantizations need less memory and run faster, at some cost in quality. Below is the expected speed on each piece of hardware at the best quantization that fits.
| Hardware | Quantization | Memory | Generation | First token |
|---|---|---|---|---|
| H200 141GB (cloud) | Q4_K_M | Fits | 1133 tok/s | 0.3 s |
| H100 80GB (cloud) | Q4_K_M | Fits | 791 tok/s | 0.3 s |
| NVIDIA RTX 5090 32GB | Q4_K_M | Fits | 423 tok/s | 0.3 s |
| NVIDIA RTX PRO 6000 96GB | Q4_K_M | Fits | 423 tok/s | 0.3 s |
| AMD Instinct MI50 32GB | Q4_K_M | Fits | 356 tok/s | 0.4 s |
| Mac Studio M5 Ultra 256GB | Q4_K_M | Fits | 347 tok/s | 0.3 s |
| Mac Studio M5 Ultra 512GB | Q4_K_M | Fits | 312 tok/s | 0.3 s |
| AMD RX 7900 XTX 24GB | Q4_K_M | Partial offload | 264 tok/s | 1.2 s |
| Mac Studio M3 Ultra 512GB | Q4_K_M | Fits | 237 tok/s | 0.3 s |
| NVIDIA RTX 4090 24GB | Q4_K_M | Partial offload | 200 tok/s | 1.2 s |
| NVIDIA RTX 3090 24GB | Q4_K_M | Partial offload | 188 tok/s | 1.2 s |
| Mac Studio M4 Max 128GB | Q4_K_M | Fits | 158 tok/s | 0.3 s |
| AMD Radeon AI PRO R9700 32GB | Q4_K_M | Fits | 148 tok/s | 0.3 s |
| NVIDIA Tesla V100 32GB | Q4_K_M | Fits | 105 tok/s | 0.3 s |
| AMD RX 7900 XT 20GB | Q4_K_M | Partial offload | 99 tok/s | 1.2 s |
| AMD Ryzen AI Max+ PRO 495 192GB (Gorgon Halo) | Q4_K_M | Fits | 95 tok/s | 0.3 s |
| MacBook Pro M5 Max 128GB | Q4_K_M | Fits | 95 tok/s | 0.3 s |
| NVIDIA RTX Spark N1X 128GB (laptop) | Q4_K_M | Fits | 91 tok/s | 0.3 s |
| Mac mini M5 Pro 64GB | Q4_K_M | Fits | 89 tok/s | 0.3 s |
| MacBook Pro M5 Pro 48GB | Q4_K_M | Fits | 89 tok/s | 0.3 s |
| Intel Arc Pro B70 32GB | Q4_K_M | Fits | 66 tok/s | 0.3 s |
| NVIDIA DGX Spark 128GB | Q4_K_M | Fits | 64 tok/s | 0.3 s |
| AMD RX 9070 XT 16GB | Q4_K_M | Partial offload | 61 tok/s | 1.2 s |
| NVIDIA RTX 5080 16GB | Q4_K_M | Partial offload | 61 tok/s | 1.2 s |
| NVIDIA RTX 5070 Ti 16GB | Q4_K_M | Partial offload | 61 tok/s | 1.2 s |
| AMD Ryzen AI Max+ 395 128GB (Strix Halo) | Q4_K_M | Fits | 53 tok/s | 0.3 s |
| NVIDIA RTX 5060 Ti 16GB | Q4_K_M | Partial offload | 51 tok/s | 1.2 s |
| NVIDIA RTX 3060 12GB | Q4_K_M | Partial offload | 39 tok/s | 1.2 s |
| CPU-only (Ryzen 9 9950X, 96GB DDR5) | Q4_K_M | Fits | 33 tok/s | 0.4 s |
| MacBook M4 Pro 24GB | Q4_K_M | Doesn't fit | Won't run | — |
Estimates for a short prompt on a single device. Measured numbers come from community benchmarks (r/LocalLLaMA, localscore.ai and more); others are derived from similar setups or projected from specs. Open source on GitHub.