2026-08-28 · 10 min · glm · moe · quantization · apple-silicon · mlx · inference
orcarouter/GLM-5.3-Flash-MLX takes the 320B-total, 18B-active MoE covered in GLM-5.3-Flash: 45 layers, 11 of them expensive and quantizes it for Apple Silicon with a method called OrcaSAQ. Five builds ship as subfolders of one repo, ranging from a near-lossless 6-bit down to a "lite" 2-bit variant explicitly aimed at the smallest machine the model can be squeezed onto. That last build is the one worth reading the README slowly for — because the same document that ships it also specifies, in its own words, exactly how narrow "fits" turns out to be, and exactly what it doesn't reliably do once it's there.
Nothing about the base architecture is re-explained here — the hybrid linear-plus-sparse attention, the IndexPool indexer trick, and Manifold-Constrained Hyper-Connections all belong to the base model and are covered in the article linked above. This one is about what happens to those 320 billion parameters on the way down to a laptop.
| Repo | orcarouter/GLM-5.3-Flash-MLX — MLX quantization of zai-org/GLM-5.3-Flash |
| Method | OrcaSAQ — calibration-free, architecture-aware Sensitivity-Aware Quantization |
| Builds | 2bit-lite ~102GB · 2-bit ~145GB · 3-bit ~184GB · 4-bit ~204GB (repo root, recommended default) · 6-bit ~296GB |
| Quantized from | official FP8 release, ≈328GB decimal (consistent with the base article's own ≈306 GiB figure, converted from binary to decimal) |
| Always BF16 | 34 linear-attention layers, sparse indexer, mHC, norms, embed_tokens, lm_head, entire vision tower — never FP8 upstream, never quantized here |
| Re-quantized | only 48 tensors were FP8 upstream (4 projections × 11 sparse-attention blocks + the MTP block), fused by MLX into 173 config entries covering 37,338 actual tensors |
| Runtime | mlx-vlm (vision-language, not mlx-lm) ≥ 0.6.17, requires a build with glm5_next support |
| License | MIT, inherited from the base model |
| Checked and holding | the six-line bit-allocation table sums to exactly 37,338 tensors; the FP8 file size cross-checks against the base article's own figure; PPL, KL divergence and cosine similarity all agree on where the quality cliff sits |
| Checked and worth qualifying | 2bit-lite, marketed as the build made for a MacBook Pro, fits exactly one of nine common Mac memory tiers, needs a manual memory-limit workaround even there, and was verified only on a data-center GPU that isn't a Mac at all |
- parameters
- 347.87B
- repo size
- 1.61 TB
- architecture
- Glm5NextForConditionalGeneration
- license
- mit
- downloads
- 16.6K
- likes
- 32
- files
- 428
Five builds, and a cliff all three quality measures agree on
The README backs its quality claims with three independent measurements against the dequantized FP8 reference, run through the identical forward pass: perplexity, KL divergence and top-1 token agreement, and weight-space cosine similarity. They use different units and none of them is derived from the others, which makes it worth noticing that they tell the same story.
Switch metrics and the shape doesn’t change, only the units do — perplexity, KL divergence and cosine similarity are three different ways of asking the same question, and all three answer it the same way. On perplexity, the single step from 3-bit to 2-bit is larger than the entire gentle slide from 6-bit down through 3-bit, combined. 2bit-lite then pushes every one of the three measures higher still. The README’s own conclusion is blunt about what that means: pick this ladder for fit, not for quality.
Read any of the three metrics and the shape is identical: 6-bit is close enough to call near-lossless, the slide through 4-bit and 3-bit is gentle, and then something changes at 2-bit. The perplexity increase alone goes from under ten percent at 3-bit to fifty-seven percent at 2-bit and a hundred forty-one percent at 2bit-lite. The README's own summary of this is worth quoting exactly, because it is more honest than most quantization pitches: "everything down to 3-bit degrades gently, 2-bit costs a lot, and 2bit-lite costs a lot more. Pick it for fit, not for quality."
That sentence is the whole thesis of this page. 2bit-lite is not a value tier — it is a size tier, sold on what it fits rather than what it preserves. Whether it fits what its own marketing says it fits is the next question.
What OrcaSAQ actually raises, and what it never touches
OrcaSAQ's pitch is calibration-free, architecture-aware mixed precision: instead of running a calibration dataset to discover which weights are sensitive, it uses structural priors — a tensor's role in the architecture decides its bit budget, decided once, mechanically, the same way for every model it's applied to. The README states the rule for which tensors are even eligible to move: a tensor is re-quantized if and only if the FP8 release shipped it with a _scale_inv companion. Everything that was already BF16 upstream — because DeepSeek Sparse Attention's linear-attention layers, the sparse indexer, the hyper-connection machinery, the norms, the embedding and output heads, and the entire vision tower were never cast to FP8 in the base release — stays BF16 here regardless of which tier you download.
Only tensors that shipped with a _scale_inv companion in the original FP8 checkpoint are eligible to be re-quantized at all — the MoE and dense-MLP linears, plus the four projections of every sparse-attention block: 11 sparse layers at depth 3, 7, 11 through 43, plus the MTP block, twelve blocks times four projections is 48 tensors. Everything below the dashed line in this diagram was never FP8 to begin with, so it rides through at BF16 no matter which tier is selected — the bar simply does not move. MLX fuses each layer’s routed experts into one switch_mlp module, so 173 config entries are enough to carry per-tensor overrides for all 37,338 of them.
Within the eligible set, the policy is a small number of rules applied uniformly: expert and sparse-attention projections sit at the tier's base bit width; down_proj tensors — the ones whose input dimension is widest and therefore most sensitive to rounding — get one bit more; the shared expert, which fires on every single token rather than eight-of-many, gets two bits more. The six line items in that policy — 24,768 expert gate/up projections, 12,384 expert down_proj, a handful of dense-layer equivalents, 129 shared-expert tensors, and 48 sparse-attention projections — sum to exactly 37,338, the README's own stated count of quantized tensors, which is a clean confirmation that nothing in the transcription above drifted from the source table.
The 48 is worth pausing on. GLM-5.3-Flash has 45 transformer layers plus one MTP layer; only 11 of those 45 use sparse attention (the other 34 are linear-attention layers with no FP8 tensors to begin with), at depths 3, 7, 11 and every fourth layer up to 43. Eleven sparse blocks plus the MTP block is 12, and each contributes 4 projections: 12 times 4 is 48. That's the entire re-quantizable footprint of the attention stack — a small, precisely bounded set sitting inside a policy that otherwise spends its bit budget on the MoE experts that make up the overwhelming majority of the model's weight.
Nine Mac memory tiers, one that works
Every build's page lists a "Min RAM" figure, and it is the number that actually decides who can run what — not the file size, which undercounts the runtime overhead a Mac needs for the OS and the rest of the system. Checked across all five builds, the min-RAM figure runs consistently eight to ten percent above the file size, which is a tight and fairly uniform margin.
Of the forty-five cells in this grid — five builds across nine memory tiers — exactly one is a fit: the 128 GB tier running 2bit-lite. Regular 2-bit needs 160 GB, which no Mac configuration reaches — the 128 GB ceiling is short by 32 GB, and the gap only widens from there: 72 GB for 3-bit, 96 GB for 4-bit, 192 GB for 6-bit. 128 GB is also Apple’s most expensive unified-memory tier, available only on the top M4 Max / M5 Max configuration — the base and Pro-tier chips this site’s reader is more likely to actually own top out far lower.
Lay that requirement against Apple's published unified-memory tiers — 16, 18, 24, 32, 36, 48, 64, 96 and 128 gigabytes, spanning the whole current Mac lineup — and the picture is stark. Of nine tiers and five builds, forty-four of the forty-five combinations do not fit. Regular 2-bit needs 160GB, and no Mac configuration reaches it — the 128GB ceiling falls 32GB short, and every larger build widens the gap from there. The only cell that lights up is 2bit-lite on the 128GB tier, and 128GB is not an ordinary configuration: it is the single largest unified-memory option Apple sells, available only on the top M4 Max or M5 Max chip. The base and Pro-tier chips that make up most MacBook Pro sales top out far lower.
The MacBook Pro claim, checked against the same document
The promotional framing for this release draws a straight line from the base model's own pitch to a specific audience, paraphrased here: "We said GLM-5.3-Flash would run on a MacBook Pro. Our original quants didn't actually make that practical for most MacBook Pro users. So we went back to work. Introducing GLM-5.3-Flash 2-bit Lite — built specifically for MacBook Pro." Read on its own, that sentence promises broadened access: a fix aimed at "most" MacBook Pro owners who were previously shut out.
The README's own text, in the section that actually documents 2bit-lite, says something much narrower:
That sentence names one machine, not "most" MacBook Pros: the single, most expensive configuration Apple sells, sitting above every base and Pro-tier chip in the lineup, and it still requires a manual step most users configuring a Mac out of the box will never take — raising macOS's default ceiling on GPU-accessible memory by hand, because the default is not high enough to load 112GB of model into a 128GB machine. That is the opposite of widening access to "most" MacBook Pro users; it is narrowing the previous impossible-for-everyone situation down to possible-for-owners-of-the-single-rarest-configuration, with a terminal command required even then.
The headroom math sharpens the point. The README states that the same build on a single H200 — a data-center GPU with 141GB and no operating system competing for memory — leaves about 39GB free for the KV cache, computed against the 102GB file size. On a 128GB Mac the arithmetic is worse before the wired-memory limit is even raised: 128GB minus the 112GB min-RAM figure leaves only about 16GB of headroom, against 39GB on the H200 sitting right next to it in the same README section. The build that is supposedly for a MacBook Pro has less breathing room on a MacBook Pro than it does on the GPU the README verified it on instead.
Verified on an H200, marketed for a Mac
That last detail is not incidental — the README's own "Field notes" section for 2bit-lite opens by stating where testing actually happened.
The README verifies 2bit-lite on one Nvidia H200 — not a Mac, despite the entire point of this build being Mac fit. What that verification finds is a clean split: chat and short-form text are fine, and long code generation reproduces three separate failure modes. The README’s own recommendation follows directly from its own data — reach for 2bit-lite only when nothing larger fits, and reach for 2-bit or higher for coding and long-horizon agentic work. That is precisely the domain the base model is built and marketed to be strong at.
Multi-turn chat and short-form Q&A are reported stable, at roughly 10 tokens per second. Long code generation is where the README's own honesty is most useful: it names three separate, reproducible failure modes rather than a vague "may struggle." Repetition loops. Missing glue code — the overall shape of an answer is right, but load-bearing lines like imports and error handling silently disappear. Rewrite churn — the model keeps restarting an answer without ever committing to a final version. None of this is inferred from the quality tables; it is the vendor's own stated field observation, on the exact build being marketed to Mac owners.
This lands squarely on GLM-5.3-Flash's own selling point. The base model's README describes it as "approaching Claude Opus 4.8 on coding and agentic benchmarks," and the base article's own scoring of that table found GLM-5.3-Flash ahead of Opus 4.8 on nine of fourteen shared rows — most of them exactly the coding and agentic categories this quantization's field notes flag as unreliable. The variant positioned for the hardware most readers actually own is the one variant the vendor's own testing says not to trust for the thing the base model is supposed to be best at.
The ledger
Well supported. Every number in the bit-allocation policy and the three quality tables checks out against the README and against itself — the tensor counts sum exactly, the FP8 file size matches the base article's own figure once units are converted, and PPL, KL divergence and cosine similarity independently agree on where the degradation curve bends. The mechanical rule for what gets quantized (FP8-shipped tensors only) is precise enough to derive the 48-tensor sparse-attention footprint from first principles and get the same number the README states.
Thin. There is no ablation isolating what the base+1 and base+2 bit bumps individually buy versus a flat base-bit policy — OrcaSAQ's architecture-aware priors are stated as a design choice, not validated against a swept alternative in this document.
Mis-framed. A promotional pitch that reads as broadening MacBook Pro access describes, in the same vendor's own README, a build that fits exactly one Mac configuration — the rarest and most expensive one Apple sells — via a manual workaround, with real-world verification performed only on a data-center GPU, and a field-tested reliability gap on precisely the coding workload the underlying model is marketed to excel at. None of the individual facts are hidden; they are simply never stated in the same sentence as the marketing claim they undercut.
Related on this site: GLM-5.3-Flash for the base architecture this quantizes — the hybrid attention, IndexPool, and mHC that carry through unquantized here; Nemotron in NVFP4 for a very different point on the quantization spectrum, a model trained natively in 4-bit rather than quantized after the fact; and How LLM inference works for why the KV cache headroom this piece keeps returning to is the thing that actually runs out first at long context.