Run GLM locally
What each open GLM model needs at every quantization, which one fits a 24 GB GPU, a 128 GB Mac or a server, which runtimes support it, and the commands that work. File sizes read from Hugging Face on October 26, 2026.
Count all the memory the model can use: your GPU's VRAM plus system RAM, or a Mac's unified memory. Each pick leaves room for the context and the operating system, because the download is only the weights.
| Memory | Run this | Download | Notes |
|---|---|---|---|
| 8 to 12 GBAn 8 GB GPU, or a 16 GB laptop | GLM-4.6V-Flash Q4_K_M | 6.2 GB | Add the 1.8 GB vision projector for image input. |
| 16 GBA 16 GB GPU | GLM-4.6V-Flash Q8_0or GLM-4.7-Flash UD-Q2_K_XLthe stronger coder, at 2-bit and a short context | 10.0 GB11.9 GB | GLM-4.7-Flash fits only at 2 to 3 bits here; it is at its best from 4-bit up. |
| 24 GBAn RTX 3090 or 4090 | GLM-4.7-Flash UD-Q4_K_XL | 17.5 GB | Unsloth's recommended setup. Leaves about 6 GB for context. |
| 32 to 48 GBAn RTX 5090, a 48 GB Mac | GLM-4.7-Flash Q6_K | 24.7 GB | 6-bit, with room for a long context. |
| 64 GBTwo 24 GB GPUs, a 64 GB Mac | GLM-4.7-Flash Q8_0or GLM-4.5-Air UD-Q2_K_XLa larger, older model at 2-bit | 31.8 GB47.4 GB | 8-bit GLM-4.7-Flash with its full 200K context. |
| 128 GBA 128 GB Mac, NVIDIA DGX Spark, a 128 GB AMD Ryzen AI Max machine | GLM-5.3 Flash UD-IQ3_XXSor GLM-5.3 Flash UD-Q2_K_XL11.6 GB smaller, for more context | 120.4 GB108.7 GB | The 3-bit file Unsloth demos on 128 GB devices. On a Mac, raise the GPU memory limit first (see the pitfalls). |
| 24 GB GPU + 128 GB RAM or moreA desktop with one RTX 4090 and a lot of system memory | GLM-5.3 Flash UD-IQ3_XXSor GLM-5.3 Flash UD-Q4_K_XLwith 256 GB of RAM or more | 120.4 GB199.7 GB | Keep the experts in system RAM with --cpu-moe and the rest on the GPU. Speed follows your RAM bandwidth (see the speeds people report). |
| 192 GBA 192 GB Mac or workstation | GLM-5.3 Flash UD-IQ4_XS | 156.8 GB | 88% top-1 agreement with the full model, against 82% at 3-bit. |
| 256 GBA 256 GB Mac or workstation | GLM-5.3 Flash UD-Q4_K_XL | 199.7 GB | 92% top-1 agreement. On a Mac, mlx-community's 4-bit MLX build (204 GB) is the alternative. |
| 512 GBA 512 GB Mac or a large-memory server | GLM-5.3 Flash UD-Q6_K_XLor GLM-5.3 UD-IQ3_XXSthe 753B flagship at 3-bit | 291.8 GB281.7 GB | Or GLM-5.3 Flash at 8-bit (341 GB). |
The default GLM-5.3 Flash checkpoint is FP8, about 306 GiB of weights before any context. vLLM's recipe serves it on four GPUs with tensor parallelism (one GB200 tray, or four H200s in its validation), on NVIDIA Hopper or newer, AMD MI350X and MI355X, or Huawei Ascend 950PR. The BF16 checkpoint needs about twice the memory; NVIDIA's NVFP4 build (204 GB) needs Blackwell GPUs.
320B total / 18B active
The model this site began with, and the strongest GLM that fits one machine: about 93 GB at 1-bit, 120 GB at 3-bit for a 128 GB computer, 200 GB at 4-bit.
| Quantization | Bits | Download | Top-1 agreement |
|---|---|---|---|
| UD-IQ1_S | 1-bit | 93.1 GB | 70.9% |
| UD-IQ1_M | 1-bit | 97.6 GB | 73.1% |
| UD-IQ2_XXS | 2-bit | 101.8 GB | 76.3% |
| UD-Q2_K_XL | 2-bit | 108.7 GB | 78.3% |
| UD-IQ3_XXS | 3-bit | 120.4 GB | 81.6% |
| UD-Q3_K_XL | 3-bit | 147.5 GB | 86.3% |
| UD-IQ4_XS | 4-bit | 156.8 GB | 88.2% |
| UD-Q4_K_XL | 4-bit | 199.7 GB | 92.2% |
| UD-Q5_K_XL | 5-bit | 240.3 GB | 94.3% |
| UD-Q6_K_XL | 6-bit | 291.8 GB | 95.2% |
| Q8_0 | 8-bit | 341.0 GB | not published |
| BF16 | 16-bit | 641.6 GB | reference |
30B total / ~3B active
The GLM for a single gaming GPU: about 17.5 GB at 4-bit, so it runs on a 24 GB card, and Ollama ships it as a one-line download.
| Quantization | Bits | Download |
|---|---|---|
| UD-Q2_K_XL | 2-bit | 11.9 GB |
| UD-IQ3_XXS | 3-bit | 12.9 GB |
| UD-Q3_K_XL | 3-bit | 13.8 GB |
| IQ4_XS | 4-bit | 16.3 GB |
| UD-Q4_K_XL | 4-bit | 17.5 GB |
| Q4_K_M | 4-bit | 18.3 GB |
| Q5_K_M | 5-bit | 21.4 GB |
| Q6_K | 6-bit | 24.7 GB |
| Q8_0 | 8-bit | 31.8 GB |
| BF16 | 16-bit | 59.9 GB |
9B (dense)
The smallest current GLM, with vision: about 6 GB at 4-bit, so it fits an 8 GB GPU or a laptop.
| Quantization | Bits | Download |
|---|---|---|
| UD-Q2_K_XL | 2-bit | 4.2 GB |
| Q3_K_M | 3-bit | 5.0 GB |
| Q4_K_M | 4-bit | 6.2 GB |
| Q5_K_M | 5-bit | 7.0 GB |
| Q6_K | 6-bit | 8.3 GB |
| Q8_0 | 8-bit | 10.0 GB |
| BF16 | 16-bit | 18.8 GB |
Runtimes
GLM-5.3 Flash mixes linear and sparse attention, which every runtime had to add support for, so it runs in fewer places than GLM-4.7-Flash. As of October 26, 2026:
| Runtime | GLM-5.3 Flash | GLM-4.7-Flash | Good to know |
|---|---|---|---|
| llama.cpp | Yes, text and vision, from build b11279 (September 30, 2026) | Yes | GLM-5.3 Flash support was merged into mainline on September 30, 2026, so Unsloth's fork is no longer needed (its guide still points to it). Unsloth updated its GGUFs for mainline on October 5, 2026. source |
| Ollama | Cloud only: glm-5.3-flash:cloud runs on Ollama's servers | Yes: ollama run glm-4.7-flash (19 GB, needs Ollama 0.14.3 or later) | Ollama's library has no local GLM-5.3 Flash; its only tag is the paid cloud model. GLM-4.7-Flash comes as q4_K_M (19 GB, the default), q8_0 (32 GB) and bf16 (60 GB). source |
| LM Studio Reported | Reported working with Unsloth's GGUF on a current runtime | Yes: GGUF and MLX builds (lmstudio-community) | LM Studio runs GGUFs through its llama.cpp engine, so GLM-5.3 Flash needs an engine built after the mainline merge; users reported it working from early October. source |
| MLX (Mac) | Through mlx-vlm: a 4-bit build of 204 GB, for 256 GB Macs | Yes, in mlx-lm: 4-bit 16.9 GB to 8-bit 31.8 GB | mlx-lm does not support GLM-5.3 Flash's architecture; mlx-vlm does. The 4-bit build is mixed 4/5/6-bit, 190 GiB. source |
| vLLM | Yes, version 0.29.0 or later, on datacenter GPUs | Yes | Needs FlashInfer 0.6.17 or newer for the sparse attention. On Hopper GPUs leave the KV cache in BF16: FP8 KV cache is not supported there for this model. source |
| SGLang | Yes, on datacenter GPUs | Yes | Z.ai's model card lists SGLang first and links its GLM-5.3 Flash cookbook. source |
| KTransformers | Yes: official FP8 weights on one RTX 40 or 50 series GPU plus 350 GB of RAM | Not covered | Reads the FP8 checkpoint directly, runs the experts on the CPU (an AVX-512 FP8 kernel) and supports up to the full 1M context. source |
| Unsloth Desktop | Yes | Yes | A desktop app for Mac, Windows and Linux that downloads the GGUF, offloads to RAM by itself and serves an API. source |
Commands
Four setups, from a gaming GPU to a GPU server. Each one ends with an OpenAI-compatible endpoint on localhost, so the same code that calls a hosted GLM API calls your own machine.
The shortest path: one command downloads the 19 GB 4-bit build and opens a chat.
ollama run glm-4.7-flash # or straight into Claude Code, on the local model ollama launch claude --model glm-4.7-flash
Ollama also serves it at http://localhost:11434 for your own code.
Downloads Unsloth's 4-bit GGUF from Hugging Face and serves it on port 8080, with Z.ai's recommended sampling.
llama-server -hf unsloth/GLM-4.7-Flash-GGUF:UD-Q4_K_XL \ --ctx-size 32768 \ --temp 1.0 --top-p 0.95 --min-p 0.01 --repeat-penalty 1.0 \ --port 8080
Point any OpenAI SDK at http://localhost:8080/v1. For tool calling, Z.ai recommends --temp 0.7 --top-p 1.0.
Needs llama.cpp b11279 or later. The 3-bit download is 120 GB in four shards; point llama-server at the first.
hf download unsloth/GLM-5.3-Flash-GGUF \
--local-dir GLM-5.3-Flash-GGUF \
--include "*UD-IQ3_XXS*"
llama-server \
--model GLM-5.3-Flash-GGUF/UD-IQ3_XXS/GLM-5.3-Flash-UD-IQ3_XXS-00001-of-00004.gguf \
--ctx-size 65536 -fa on \
--temp 1.0 --top-p 0.95 \
--chat-template-kwargs '{"reasoning_effort":"high"}' \
--port 8080One 24 GB GPU and lots of system RAM: add --cpu-moe to keep the experts in RAM. For image input, also download mmproj-F16.gguf and pass --mmproj. reasoning_effort takes low, high or max (the default, deepest and slowest).
Serves the official FP8 checkpoint with tool calling and reasoning parsed, from vLLM's own recipe.
vllm serve zai-org/GLM-5.3-Flash \ --tensor-parallel-size 4 \ --tool-call-parser glm47 \ --reasoning-parser glm47 \ --enable-auto-tool-choice
On Blackwell, add --kv-cache-dtype fp8; on Hopper leave it out. MTP speculative decoding: --speculative-config '{"method":"mtp","num_speculative_tokens":5}'.
Settings
Both models are tuned for these values; the wrong ones are the usual cause of loops and rambling.
temperature 1.0, top_p 0.95reasoning_effort low, high or max; the default is max, the slowest.clear_thinking=true in the chat template for chat use; it defaults to false.temperature 1.0, top_p 0.95; for tool calling 0.7 and 1.0--min-p 0.01 (llama.cpp's default is 0.05)Pitfalls
The errors people hit most, with the cause and the fix. The first two come from the switch to mainline llama.cpp on September 30, 2026, and Unsloth's update of its files on October 5, 2026.
unknown model architecture: 'glm5next'Your GGUF's first shard is from before October 5, 2026: Unsloth's day-one files named the architecture glm5next, and mainline llama.cpp calls it glm5-next.
Download the first shard (...-00001-of-0000N.gguf) and mmproj-F16.gguf again; the other shards are unchanged. hf download with the same --include fetches only what changed.
unknown model architecture: 'glm5-next'Your llama.cpp is older than build b11279, the first with GLM-5.3 Flash. Packaged installers can lag the source by days.
Update llama.cpp to b11279 or later (llama-server --version prints the build), or build it from source.
In Ollama's library GLM-5.3 Flash is a cloud model: the :cloud tag runs on Ollama's servers and is billed per token.
To run it on your machine, use llama.cpp, LM Studio or Unsloth Desktop with a GGUF. Locally, Ollama runs GLM-4.7-Flash.
Failed to load CLIP model when you send an imageThe vision projector file predates the mainline port.
Download mmproj-F16.gguf again (updated October 5, 2026) and pass it with --mmproj.
The file size is only the weights; the context needs memory of its own, which grows with every token you allow.
Lower --ctx-size, turn on flash attention with -fa on (it keeps llama.cpp's working memory flat as the context grows), or step down one quantization.
macOS lets the GPU use only part of unified memory by default, well below the total.
Raise the limit with sudo sysctl iogpu.wired_limit_mb=<megabytes>, set above the model's size and below your total memory (it resets on restart), or pick the next smaller file.
GGUFs made before January 21, 2026 used the wrong expert scoring function, and a repeat penalty makes it worse.
Download the GGUF again, set the repeat penalty to 1.0, and use Z.ai's sampling (temperature 1.0, top_p 0.95).
Speed
Only 18B of its 320B parameters work on each token, so it runs far faster than its size suggests, but speed still follows memory bandwidth: a server with many memory channels runs the same file much faster than a desktop. These are measurements people posted, with their setup, not promises.
| Setup | File | Result | Reported |
|---|---|---|---|
| 1 RTX 4090 + EPYC server with DDR5 (experts on the CPU) | GLM-5.3 Flash UD-Q4_K_XL | About 24 tokens/s generating and 130 tokens/s reading the prompt, steady out to 92K tokens of context, with 12.8 GB of VRAM in use | September 4, 2026 |
| 1 RTX 4090 + 128 GB DDR5 desktop | GLM-5.3 Flash IQ3_S | About 9 tokens/s generating at first, falling to 6 by 128K of context, on an early build | August 28, 2026 |
| 1 NVIDIA B200 (Unsloth's test) | GLM-5.3 Flash UD-IQ1_S | About 63 tokens/s generating, and 86 with MTP speculative decoding at 4K of context | September 4, 2026 |
Every open GLM model
The smallest, 4-bit and 8-bit GGUFs next to Z.ai's own weights. GLM-5.2's GGUFs are the same sizes as GLM-5.3's.
| Model | Parameters | Smallest | 4-bit | 8-bit | Z.ai weights |
|---|---|---|---|---|---|
| GLM-5.3 Flash | 320B total / 18B active | 93.1 GB | 199.7 GB | 341.0 GB | 328.3 GB FP8 |
| GLM-4.7-Flash | 30B total / ~3B active | 11.9 GB | 17.5 GB | 31.8 GB | 62.4 GB BF16 |
| GLM-4.6V-Flash | 9B (dense) | 4.2 GB | 6.2 GB | 10.0 GB | 20.6 GB BF16 |
| GLM-4.5-Air | 106B total / 12B active | 47.4 GB | 67.7 GB | 117.5 GB | 220.9 GB BF16 |
| GLM-5.3 | 753B total | 216.7 GB | 467.3 GB | 801.4 GB | 755.6 GB FP8 |
Local or hosted
For privacy, offline work or heavy use, yes. To save money on light use, rarely: hosted GLM-5.3 Flash costs cents per million tokens, and Z.ai's own API serves GLM-4.7-Flash free, while a machine that runs GLM-5.3 Flash well costs thousands.
Run GLM-5.3 Flash in a browser or over an API first, and download it once you know it suits your work.
Z.ai's API serves GLM-4.7-Flash at no charge, with a Standard API Key.
FAQ
ollama run glm-5.3-flash:cloud runs on Ollama's servers and is billed per token; there is no local GLM-5.3 Flash in Ollama's library. Ollama does run GLM-4.7-Flash locally with ollama run glm-4.7-flash.Once it runs, the next step is shipping a real product on a model like GLM, without wiring up the infrastructure yourself.