GPT-OSS 120B is a 116.83B-parameter mixture-of-experts model (5.1B active per token). Smaller quantizations need less memory and run faster, at some cost in quality. Below is the expected speed on each piece of hardware at the best quantization that fits.
| Hardware | Quantization | Memory | Generation | First token |
|---|---|---|---|---|
| H100 80GB (cloud) | MXFP4 | Fits | 488 tok/s | 0.3 s |
| A100 80GB (cloud) | MXFP4 | Fits | 297 tok/s | 0.3 s |
| NVIDIA RTX PRO 6000 96GB | MXFP4 | Fits | 261 tok/s | 0.3 s |
| Mac Studio M5 Ultra 512GB | MXFP4 | Fits | 205 tok/s | 0.3 s |
| H200 141GB (cloud) | MXFP4 | Fits | 196 tok/s | 0.3 s |
| Mac Studio M3 Ultra 512GB | MXFP4 | Fits | 154 tok/s | 0.3 s |
| MacBook Pro M5 Max 128GB | MXFP4 | Fits | 115 tok/s | 0.3 s |
| Mac Studio M4 Max 128GB | MXFP4 | Fits | 103 tok/s | 0.3 s |
| AMD Ryzen AI Max+ 395 128GB (Strix Halo) | MXFP4 | Fits | 52 tok/s | 0.4 s |
| NVIDIA DGX Spark 128GB | MXFP4 | Fits | 40 tok/s | 0.3 s |
| NVIDIA RTX 5090 32GB | MXFP4 | Partial offload | 39 tok/s | 1.2 s |
| AMD Radeon AI PRO R9700 32GB | MXFP4 | Partial offload | 37 tok/s | 1.2 s |
| Intel Arc Pro B70 32GB | MXFP4 | Partial offload | 31 tok/s | 1.3 s |
| CPU-only (Ryzen 9 9950X, 96GB DDR5) | MXFP4 | Fits | 22 tok/s | 0.4 s |
| NVIDIA RTX 3090 24GB | MXFP4 | Doesn't fit | Won't run | — |
| NVIDIA RTX 4090 24GB | MXFP4 | Doesn't fit | Won't run | — |
| NVIDIA RTX 5060 Ti 16GB | MXFP4 | Doesn't fit | Won't run | — |
| AMD RX 7900 XTX 24GB | MXFP4 | Doesn't fit | Won't run | — |
| NVIDIA RTX 3060 12GB | MXFP4 | Doesn't fit | Won't run | — |
| NVIDIA RTX 5070 Ti 16GB | MXFP4 | Doesn't fit | Won't run | — |
| AMD RX 9070 XT 16GB | MXFP4 | Doesn't fit | Won't run | — |
| Mac Mini M4 Pro 64GB | MXFP4 | Doesn't fit | Won't run | — |
| NVIDIA RTX 5080 16GB | MXFP4 | Doesn't fit | Won't run | — |
| MacBook Pro M5 Pro 48GB | MXFP4 | Doesn't fit | Won't run | — |
| AMD RX 7900 XT 20GB | MXFP4 | Doesn't fit | Won't run | — |
| NVIDIA RTX 4070 Ti Super 16GB | MXFP4 | Doesn't fit | Won't run | — |
| MacBook M4 Pro 24GB | MXFP4 | Doesn't fit | Won't run | — |
| NVIDIA RTX 4070 12GB | MXFP4 | Doesn't fit | Won't run | — |
| MacBook M3 Max 48GB | MXFP4 | Doesn't fit | Won't run | — |
| MacBook M2 Pro 16GB | MXFP4 | Doesn't fit | Won't run | — |
Estimates for a short prompt on a single device. Measured numbers come from community benchmarks (r/LocalLLaMA, localscore.ai and more); others are derived from similar setups or projected from specs. Open source on GitHub.