GLM-5.3 Flash is a 321.3B-parameter mixture-of-experts model (18B active per token). At UD-IQ1_S it needs about 93.8 GB of memory. The NVIDIA RTX PRO 6000 96GB has 89 GB usable, so it only fits with part of the model offloaded to system RAM, which slows it down. Expect about 90 tokens/sec (34–146), with the first token after 1.2 s for a short prompt. This estimate is projected from the hardware specs; no benchmarks exist yet.
| Quantization | File size | Memory | Generation | First token | Source |
|---|---|---|---|---|---|
| UD-IQ1_S | 93.1 GB | Partial offload | 90 tok/s | 1.2 s | Projected |
| UD-IQ1_M | 97.6 GB | Partial offload | 66 tok/s | 1.2 s | Projected |
| UD-IQ2_XXS | 101.8 GB | Partial offload | 53 tok/s | 1.2 s | Projected |
| UD-Q2_K_XL | 108.7 GB | Partial offload | 40 tok/s | 1.2 s | Projected |
| UD-IQ3_XXS | 120.4 GB | Partial offload | 28 tok/s | 1.2 s | Projected |
| UD-Q3_K_XL | 147.5 GB | Partial offload | 17 tok/s | 1.2 s | Projected |
| UD-IQ4_XS | 156.8 GB | Partial offload | 15 tok/s | 1.2 s | Projected |
| UD-Q4_K_XL | 199.7 GB | Partial offload | 9.3 tok/s | 1.2 s | Projected |
| UD-Q5_K_XL | 240.3 GB | Doesn't fit | Won't run | — | — |
| UD-Q6_K_XL | 291.8 GB | Doesn't fit | Won't run | — | — |
| Q8_0 | 341.0 GB | Doesn't fit | Won't run | — | — |
| FP16 | 642.8 GB | Doesn't fit | Won't run | — | — |
All models on the NVIDIA RTX PRO 6000 96GB · GLM-5.3 Flash on all hardware
Estimates for a short prompt on a single device. Measured numbers come from community benchmarks (r/LocalLLaMA, localscore.ai and more); others are derived from similar setups or projected from specs. Open source on GitHub.