How fast will local models feel on the H200 141GB (cloud)?

Local LLM speed on the H200 141GB (cloud)

The H200 141GB (cloud) has 141 GB of memory and 4800 GB/s of memory bandwidth, which is what mostly decides generation speed. Below is the expected speed for each model at the best quantization that fits.

Model	Quantization	Memory	Generation	First token
Qwen 3.6/3.8 27B	Q4_K_M	Fits	115 tok/s	0.3 s
GLM-5.3 Flash	UD-IQ3_XXS	Fits	287 tok/s	0.3 s
GPT-OSS 20B	MXFP4	Fits	306 tok/s	0.3 s
Gemma 4 26B A4B	Q4_K_M	Fits	775 tok/s	0.3 s
Qwen 3.5/3.6 35B A3B	Q4_K_M	Fits	1048 tok/s	0.3 s
Ornith 1.5 35B A3B	Q4_K_M	Fits	824 tok/s	0.3 s
DeepSeek V4 Flash	UD-Q3_K_XL	Fits	353 tok/s	0.3 s
GPT-OSS 120B	MXFP4	Fits	196 tok/s	0.3 s
Gemma 4 31B	Q4_K_M	Fits	106 tok/s	0.3 s
Llama 3.1 8B	Q4_K_M	Fits	393 tok/s	0.3 s
Qwen 3.5 9B	Q4_K_M	Fits	263 tok/s	0.3 s
DeepSeek R1 Distill 14B	Q4_K_M	Fits	215 tok/s	0.3 s
Ornith 1.5 9B	Q4_K_M	Fits	258 tok/s	0.3 s
Muse Glimmer 30B	Q4_K_M	Fits	75 tok/s	0.3 s
Qwen3 30B A3B	Q4_K_M	Fits	744 tok/s	0.3 s

Other hardware

Estimates for a short prompt on a single device. Measured numbers come from community benchmarks (r/LocalLLaMA, localscore.ai and more); others are derived from similar setups or projected from specs. Open source on GitHub.