How fast will GPT-OSS 120B feel on the NVIDIA RTX PRO 6000 96GB?

GPT-OSS 120B on the NVIDIA RTX PRO 6000 96GB

GPT-OSS 120B is a 116.83B-parameter mixture-of-experts model (5.1B active per token). At MXFP4 it needs about 64.1 GB of memory. The NVIDIA RTX PRO 6000 96GB has 89 GB usable, so it fits entirely in memory. Expect about 261 tokens/sec (95–427), with the first token after 0.3 s for a short prompt. This estimate is interpolated from benchmarks of similar setups.

Speed by quantization

Quantization	File size	Memory	Generation	First token	Source
MXFP4	63.4 GB	Fits	261 tok/s	0.3 s	Derived

GPT-OSS 120B on other hardware

H100 80GB (cloud) · 488 tok/s at MXFP4
A100 80GB (cloud) · 297 tok/s at MXFP4
Mac Studio M5 Ultra 512GB · 205 tok/s at MXFP4
H200 141GB (cloud) · 196 tok/s at MXFP4
Mac Studio M3 Ultra 512GB · 154 tok/s at MXFP4
MacBook Pro M5 Max 128GB · 115 tok/s at MXFP4
Mac Studio M4 Max 128GB · 103 tok/s at MXFP4
AMD Ryzen AI Max+ 395 128GB (Strix Halo) · 52 tok/s at MXFP4
NVIDIA DGX Spark 128GB · 40 tok/s at MXFP4
NVIDIA RTX 5090 32GB · 39 tok/s at MXFP4
AMD Radeon AI PRO R9700 32GB · 37 tok/s at MXFP4
Intel Arc Pro B70 32GB · 31 tok/s at MXFP4
CPU-only (Ryzen 9 9950X, 96GB DDR5) · 22 tok/s at MXFP4
NVIDIA RTX 3090 24GB · won't run
NVIDIA RTX 4090 24GB · won't run
NVIDIA RTX 5060 Ti 16GB · won't run
AMD RX 7900 XTX 24GB · won't run
NVIDIA RTX 3060 12GB · won't run
NVIDIA RTX 5070 Ti 16GB · won't run
AMD RX 9070 XT 16GB · won't run
Mac Mini M4 Pro 64GB · won't run
NVIDIA RTX 5080 16GB · won't run
MacBook Pro M5 Pro 48GB · won't run
AMD RX 7900 XT 20GB · won't run
NVIDIA RTX 4070 Ti Super 16GB · won't run
MacBook M4 Pro 24GB · won't run
NVIDIA RTX 4070 12GB · won't run
MacBook M3 Max 48GB · won't run
MacBook M2 Pro 16GB · won't run

Other models on the NVIDIA RTX PRO 6000 96GB

Qwen 3.6/3.8 27B · 43 tok/s at Q4_K_M
GLM-5.3 Flash · 90 tok/s at UD-IQ1_S
GPT-OSS 20B · 347 tok/s at MXFP4
Gemma 4 26B A4B · 289 tok/s at Q4_K_M
Qwen 3.5/3.6 35B A3B · 391 tok/s at Q4_K_M
Ornith 1.5 35B A3B · 308 tok/s at Q4_K_M
DeepSeek V4 Flash · 195 tok/s at UD-IQ1_M
Gemma 4 31B · 39 tok/s at Q4_K_M
Llama 3.1 8B · 75 tok/s at Q4_K_M
Qwen 3.5 9B · 98 tok/s at Q4_K_M
DeepSeek R1 Distill 14B · 47 tok/s at Q4_K_M
Ornith 1.5 9B · 96 tok/s at Q4_K_M
Muse Glimmer 30B · 28 tok/s at Q4_K_M
Qwen3 30B A3B · 278 tok/s at Q4_K_M

All models on the NVIDIA RTX PRO 6000 96GB · GPT-OSS 120B on all hardware

Estimates for a short prompt on a single device. Measured numbers come from community benchmarks (r/LocalLLaMA, localscore.ai and more); others are derived from similar setups or projected from specs. Open source on GitHub.