How fast will GLM-5.3 Flash feel on your hardware?

GLM-5.3 Flash speed on GPUs and Macs

GLM-5.3 Flash is a 321.3B-parameter mixture-of-experts model (18B active per token). Smaller quantizations need less memory and run faster, at some cost in quality. Below is the expected speed on each piece of hardware at the best quantization that fits.

HardwareQuantizationMemoryGenerationFirst token
H200 141GB (cloud)UD-IQ3_XXSFits310 tok/s0.3 s
NVIDIA RTX PRO 6000 96GBUD-IQ1_SPartial offload86 tok/s1.2 s
Mac Studio M5 Ultra 256GBQ4_KFits63 tok/s0.3 s
MacBook Pro M5 Max 128GBUD-IQ1_SFits60 tok/s0.4 s
Mac Studio M4 Max 128GBUD-IQ1_SFits54 tok/s0.5 s
Mac Studio M5 Ultra 512GBUD-Q4_K_XLFits52 tok/s0.3 s
AMD Ryzen AI Max+ 395 128GB (Strix Halo)UD-Q2_K_XLFits25 tok/s0.5 s
NVIDIA RTX Spark N1X 128GB (laptop)UD-Q2_K_XLFits21 tok/s0.3 s
Mac Studio M3 Ultra 512GBUD-Q4_K_XLFits20 tok/s0.4 s
AMD Ryzen AI Max+ PRO 495 192GB (Gorgon Halo)UD-IQ4_XSFits20 tok/s0.5 s
NVIDIA DGX Spark 128GBUD-Q2_K_XLFits18 tok/s0.3 s
CPU-only (Ryzen 9 9950X, 96GB DDR5)UD-IQ1_SPartial offload13 tok/s2.0 s
NVIDIA RTX 3090 24GBUD-IQ1_SDoesn't fitWon't run—
NVIDIA RTX 5090 32GBUD-IQ1_SDoesn't fitWon't run—
AMD Radeon AI PRO R9700 32GBUD-IQ1_SDoesn't fitWon't run—
NVIDIA RTX 4090 24GBUD-IQ1_SDoesn't fitWon't run—
NVIDIA RTX 5060 Ti 16GBUD-IQ1_SDoesn't fitWon't run—
NVIDIA Tesla V100 32GBUD-IQ1_SDoesn't fitWon't run—
Intel Arc Pro B70 32GBUD-IQ1_SDoesn't fitWon't run—
AMD RX 7900 XTX 24GBUD-IQ1_SDoesn't fitWon't run—
NVIDIA RTX 5070 Ti 16GBUD-IQ1_SDoesn't fitWon't run—
NVIDIA RTX 3060 12GBUD-IQ1_SDoesn't fitWon't run—
NVIDIA RTX 5080 16GBUD-IQ1_SDoesn't fitWon't run—
Mac mini M5 Pro 64GBUD-IQ1_SDoesn't fitWon't run—
AMD Instinct MI50 32GBUD-IQ1_SDoesn't fitWon't run—
MacBook Pro M5 Pro 48GBUD-IQ1_SDoesn't fitWon't run—
AMD RX 9070 XT 16GBUD-IQ1_SDoesn't fitWon't run—
AMD RX 7900 XT 20GBUD-IQ1_SDoesn't fitWon't run—
MacBook M4 Pro 24GBUD-IQ1_SDoesn't fitWon't run—
H100 80GB (cloud)UD-IQ1_SDoesn't fitWon't run—

Other models

Estimates for a short prompt on a single device. Measured numbers come from community benchmarks (r/LocalLLaMA, localscore.ai and more); others are derived from similar setups or projected from specs. Open source on GitHub.