How fast will Nemotron 3.5 Lightning 30B A3B feel on the NVIDIA RTX PRO 6000 96GB?

Nemotron 3.5 Lightning 30B A3B on the NVIDIA RTX PRO 6000 96GB

Nemotron 3.5 Lightning 30B A3B is a 31.58B-parameter mixture-of-experts model (3B active per token). At Q4_K_M it needs about 26.0 GB of memory. The NVIDIA RTX PRO 6000 96GB has 89 GB usable, so it fits entirely in memory. Expect about 325 tokens/sec (156–494), with the first token after 0.3 s for a short prompt. This estimate is interpolated from benchmarks of similar setups.

Speed by quantization

Quantization	File size	Memory	Generation	First token	Source
Q4_K_M	25.3 GB	Fits	325 tok/s	0.3 s	Derived
Q8_0	35.0 GB	Fits	235 tok/s	0.3 s	Derived
FP16	65.8 GB	Fits	125 tok/s	0.3 s	Derived

Nemotron 3.5 Lightning 30B A3B on other hardware

H200 141GB (cloud) · 871 tok/s at Q4_K_M
H100 80GB (cloud) · 608 tok/s at Q4_K_M
NVIDIA RTX 5090 32GB · 385 tok/s at Q4_K_M
AMD Instinct MI50 32GB · 274 tok/s at Q4_K_M
Mac Studio M5 Ultra 256GB · 267 tok/s at Q4_K_M
Mac Studio M5 Ultra 512GB · 240 tok/s at Q4_K_M
Mac Studio M3 Ultra 512GB · 182 tok/s at Q4_K_M
AMD Radeon AI PRO R9700 32GB · 173 tok/s at Q4_K_M
NVIDIA Tesla V100 32GB · 163 tok/s at Q4_K_M
MacBook Pro M5 Max 128GB · 137 tok/s at Q4_K_M
Mac Studio M4 Max 128GB · 121 tok/s at Q4_K_M
AMD RX 7900 XTX 24GB · 96 tok/s at Q4_K_M
NVIDIA RTX 4090 24GB · 85 tok/s at Q4_K_M
NVIDIA RTX 3090 24GB · 83 tok/s at Q4_K_M
AMD Ryzen AI Max+ PRO 495 192GB (Gorgon Halo) · 73 tok/s at Q4_K_M
Mac mini M5 Pro 64GB · 68 tok/s at Q4_K_M
MacBook Pro M5 Pro 48GB · 68 tok/s at Q4_K_M
AMD Ryzen AI Max+ 395 128GB (Strix Halo) · 64 tok/s at Q4_K_M
AMD RX 7900 XT 20GB · 57 tok/s at Q4_K_M
NVIDIA RTX Spark N1X 128GB (laptop) · 54 tok/s at Q4_K_M
Intel Arc Pro B70 32GB · 53 tok/s at Q4_K_M
NVIDIA DGX Spark 128GB · 50 tok/s at Q4_K_M
AMD RX 9070 XT 16GB · 41 tok/s at Q4_K_M
NVIDIA RTX 5080 16GB · 41 tok/s at Q4_K_M
NVIDIA RTX 5070 Ti 16GB · 40 tok/s at Q4_K_M
NVIDIA RTX 5060 Ti 16GB · 35 tok/s at Q4_K_M
NVIDIA RTX 3060 12GB · 29 tok/s at Q4_K_M
CPU-only (Ryzen 9 9950X, 96GB DDR5) · 25 tok/s at Q4_K_M
MacBook M4 Pro 24GB · won't run

Other models on the NVIDIA RTX PRO 6000 96GB

Qwen 3.6/3.8 27B · 46 tok/s at Q4_K_M
Qwen 3.8 Flash-Next · 280 tok/s at GSQ-RCO-IQ3_S
GLM-5.3 Flash · 86 tok/s at UD-IQ1_S
Gemma 4 26B A4B · 313 tok/s at Q4_K_M
GPT-OSS 20B · 375 tok/s at MXFP4
Qwen 3.5/3.6 35B A3B · 423 tok/s at Q4_K_M
Gemma 4 31B · 43 tok/s at Q4_K_M
Qwen 3.5 9B · 138 tok/s at Q4_K_M
DeepSeek V4 Flash · 210 tok/s at UD-IQ1_M
DeepSeek V4.1 Flash · won't run
GPT-OSS 120B · 282 tok/s at MXFP4
Llama 3.1 8B · 76 tok/s at Q4_K_M
Gemma 4 12B · 110 tok/s at Q4_K_M
GLM-5.3 · won't run

All models on the NVIDIA RTX PRO 6000 96GB · Nemotron 3.5 Lightning 30B A3B on all hardware

Estimates for a short prompt on a single device. Measured numbers come from community benchmarks (r/LocalLLaMA, localscore.ai and more); others are derived from similar setups or projected from specs. Open source on GitHub.