How fast will local models feel on the NVIDIA RTX PRO 6000 96GB?

Local LLM speed on the NVIDIA RTX PRO 6000 96GB

The NVIDIA RTX PRO 6000 96GB has 96 GB of memory and 1792 GB/s of memory bandwidth, which is what mostly decides generation speed. Below is the expected speed for each model at the best quantization that fits.

ModelQuantizationMemoryGenerationFirst token
Qwen 3.6/3.8 27BQ4_K_MFits46 tok/s0.3 s
Qwen 3.8 Flash-NextGSQ-RCO-IQ3_SFits280 tok/s0.3 s
GLM-5.3 FlashUD-IQ1_SPartial offload86 tok/s1.2 s
Gemma 4 26B A4BQ4_K_MFits313 tok/s0.3 s
GPT-OSS 20BMXFP4Fits375 tok/s0.3 s
Qwen 3.5/3.6 35B A3BQ4_K_MFits423 tok/s0.3 s
Gemma 4 31BQ4_K_MFits43 tok/s0.3 s
Qwen 3.5 9BQ4_K_MFits138 tok/s0.3 s
DeepSeek V4 FlashUD-IQ1_MFits210 tok/s0.3 s
DeepSeek V4.1 FlashQ2_KDoesn't fitWon't run—
GPT-OSS 120BMXFP4Fits282 tok/s0.3 s
Llama 3.1 8BQ4_K_MFits76 tok/s0.3 s
Gemma 4 12BQ4_K_MFits110 tok/s0.3 s
GLM-5.3UD-IQ1_SDoesn't fitWon't run—
Nemotron 3.5 Lightning 30B A3BQ4_K_MFits325 tok/s0.3 s

Other hardware

Estimates for a short prompt on a single device. Measured numbers come from community benchmarks (r/LocalLLaMA, localscore.ai and more); others are derived from similar setups or projected from specs. Open source on GitHub.