Laya on Apple silicon: 1.39×, not 50×, and K is a literal again

Yesterday's piece put a System One scorer in the reader's tab and then had to publish a hole in the middle of it. There is no GPU in the machine I write these on — /dev/dri does not exist — so Chrome handed back a SwiftShader adapter, every WebGPU number was a software rasterizer's number, and I refused all of them. What I could measure was WASM on four contended vCPUs: 72.8 / 128.3 / 214.1 ms at two options, 437.1 / 472.8 / 675.9 ms at sixteen, and 11.4 to 32.6 seconds for one sixteen-option decision on a 4-bit ModernBERT-large.

Somebody made the measurement I could not. mizorewww shipped two independent ports of Laya to Apple silicon on 19 and 20 September, benchmarked them on an M3 Max, and committed every timing sample, every rejected run and every export manifest alongside the prose.

I still cannot run them. There is no Apple silicon here either, so every performance figure in this article is Reported — read out of the repositories' own committed JSON, never reproduced. What I can do is read the conversion path, check the arithmetic, and find out what the three claims that reached me are actually claims about. All three turn out to be missing a denominator, and the repositories are not the ones who dropped it.

Two repos, not one, and both answers are yes

The name says Core ML and the pitch says MLX, so: both, in that order, by the same author, hours apart.

• mizorewww/laya-mlx — a native MLX runtime. The ModernBERT encoder, decision head, scorer and action head all reimplemented in MLX; tokenization through the Hugging Face Rust tokenizer; no PyTorch or Transformers at inference. 1,197 stars.
• mizorewww/laya-coreml — a Core ML export and a second, hand-written graph aimed at the Neural Engine. Its NOTICE names laya-mlx as the thing it is adapted from, at commit fc1df62. 242 stars.

Both are Apache-2.0 ports of NandhaKishorM/laya at 6a58191 — the same upstream commit the relational-choice piece quoted build_sequence from. Neither ships weights; both download the original checkpoints and convert them.

The Core ML repo is the more interesting one, because it did the thing nobody else in this story has done: it wrote a transformer graph specifically to land on the Neural Engine, and then integrated the machine's own power sensor over the run to find out whether it was worth it.

expand

Claim one: 50× faster than Jev

The first thing I did was grep for it.

$ git -C laya-coreml ls-files | wc -l
155
$ git -C laya-mlx ls-files | wc -l
130
$ git -C laya-coreml grep -Ili jev | wc -l
0
$ git -C laya-mlx  grep -Ili jev | wc -l
0
$ git -C laya-coreml grep -Ili typesafe | wc -l
0
$ git -C laya-mlx  grep -Ili typesafe | wc -l
0

Zero, across 285 tracked files. Two READMEs, two Chinese READMEs, sixteen engineering documents, seventy-one benchmark result files and nine commit messages: neither repository mentions Jev once, or TypeSafe once, or compares itself to a hosted API anywhere. The only thing a case-insensitive sweep turns up is three video containers and two git packfiles, which is what happens when you grep binaries for a three-letter string.

So the 50× is not a claim the repositories make, retracted or otherwise. It is a claim about them. And it is arithmetically reachable — you just have to pick the right denominator.

The three filled dots are the only speed ratios either repository publishes about itself, and all three compare the same workload on the same machine. The headline one is 1.394×, the Neural Engine graph against compiled MLX FP16 across three balanced cycles of alternating 20-second blocks, 65,598 completed predictions in total. The README states it without decoration — | Speed gain | 1× | **1.39×** | **1.42×** |, a row in a table — and then writes its own least flattering sentence, which is about the demo everyone is sharing:

The short ANE result does not establish a long-context advantage. A 600-step paired Snake check matches 600/600 actions, with zero deaths and zero shield interventions; the current ANE adapter's three sequential calls do not establish a consistent full-game speedup over compiled MLX.

The fastest path is the one the repository declines to claim a Snake win for.

The bars are what you get when you change the denominator from the same task on the same machine to an HTTPS round trip to somebody else's datacentre. Divide the published third-party Jev p50s — 236 ms and 276 ms from two independent benchmark repos, 338.6 ms from a third — by the Core ML port's 4.976 ms warm local single-question p50, and you get 47.4× to 68.0×. Fifty sits comfortably inside that. It is a real division of two real numbers, and it measures the Atlantic.

The browser piece refused exactly this comparison in its own favour, and the reason has not changed: one side is compute, the other side is compute plus TLS plus queueing plus the speed of light, and stripping the network from one side makes the ratio evaporate. The honest version is the weak one. A 322M encoder, warm, on a Neural Engine, answers a 91-token four-option question in about 5 ms, and a hosted decision API answers over the wire in about 250. That tells you the hosted figure is dominated by something other than arithmetic. It does not rank two models.

Claim two: 60 decisions per second, playing Snake

Sixty is in the repository. It is the row that fails.

The paced sweep asks the demo to hold a rate and counts how often a tick blows its budget; 60 requested decisions per second misses the deadline on 100% of ticks and settles at 50.68/s. What the Core ML runtime actually sustains, with no deadline at all, is 49.10, 49.89 and 49.99 decisions per second over three 600-step episodes — pooled 49.66/s, which is 20.14 ms per decision, not 16.7.

The arithmetic of why 60 fails is right there. Sixty decisions per second is a 16.67 ms budget. The three model calls alone are 16.32 ms at p50, so the budget leaves 0.35 ms for the planner, the Rich composition, the truecolor ANSI serialization and the game update — and in the same episodes those account for the 2.75 ms gap between the three-question p50 and the whole-tick p50 of 19.07 ms (25.96 ms at p95). You cannot fit that in 16.67 ms, and the benchmark says so by failing every tick rather than by rounding.

"Roughly 60" is defensible, but only on the other runtime. The MLX port sustains 63.61 moves/s over 2,400 uncapped steps (per-seed range 46.32 to 76.37), 75.40 moves/s on its opt-in compiled path, and 64.77 moves/s in a 20.01-second real-terminal recording. The reason for the gap is structural and the repository names it: Core ML executes questions sequentially, while MLX batches them. MLX runs the move's three questions as one B=3 forward pass. The Neural Engine bundle is exported at B=1 and has to call three times.

Which brings us to the part of the claim that was never specified.

How many options is a decision?

A Snake move is not one question with four options. It is three questions, and you can read them off laya_coreml/snake/policy.py directly:

# laya_coreml/snake/policy.py — one decision, three typed questions
questions = {
    "move": {"type": "choice", "instructions": "Choose the best safe move toward food.",
             "criteria": {m.direction: ... for m in moves}},   # 4 options, always
    "risk": {"type": "noul", "instructions": "Is a safe route available?"},   # 2
    "food": {"type": "noul", "instructions": "Is food reachable through empty cells?"},  # 2
}
output = self.agent.predict(state, questions)

moves is the full DIRECTIONS tuple, so the choice question always carries four options even when three of them are walls, and render_options in common.py expands every noul into a fixed [false, true] pair. Eight option markers per decision, spread over three forward passes. Per decision the runtime reports 150 to 155 input tokens, median 152, and that range is identical in every one of the twelve benchmark episodes the repository commits traces for.

That matters because a per-decision latency without an option count is not a latency. On the shared-sequence shape, cost is roughly linear in the total length of the packed sequence, so four short options and twenty-five long ones are not the same request at all. The single-question figure everybody quotes — 4.976 ms p50, 5.307 ms p95 — is one four-option question, 91 real tokens padded to 96. Three of those end-to-end calls is 14.93 ms; the measured three-question Snake API is 16.32 ms. The Snake questions are shorter than the benchmark's — about 51 tokens each against 91 — but both pad to the same fixed 96-token window, so the graph does identical work either way and only the host-side cost moves.

The repository draws the line itself, in bold, so that nobody downstream has to:

The single-question 4.98 ms result must not be advertised as the full Snake frame time.

expand

Claim three: at most 1 GB

This one is nearly true, and the interesting part is which quantity it is true of. It is not the weights, it is not process RSS, and it is not a budget the runtime enforces. It is MLX's peak allocator high-water mark for one short question, and it comes from a table the MLX repo publishes:

Check the first two columns against each other. 421,293,827 × 2 bytes = 842,587,654 = 803.55 MiB, which is the published 803.6 exactly; 321,908,995 × 2 = 613.99 MiB, which is the published 614.0. The weights are plain FP16, nothing else. So the answer to at what quantisation does 421M land under 1 GB is: no quantisation at all. The port is explicit — this is a parameter-name/dtype conversion, not quantization or retraining — and the source checkpoints already store FP16, so even asking for FP32 buys arithmetic precision, not weight precision.

That leaves 140.0 MiB of headroom between the weights and the peak, which is what one 93-token forward pass costs in activations. Ten questions at full context cost 1,029.4 MiB of activations on top of the same weights, and the total is 1,833.0 MiB — 1.79 GiB, comfortably over any reading of "1 GB". The claim is true of exactly one cell in that table and false of the cell two columns to its right.

Two more things the phrase quietly flattens. Peak MLX allocation is not process RSS, and the Core ML repo says so in as many words: Process RSS is recorded, including framework caches; it is not interchangeable with MLX active-memory statistics. And the download is a separate number nobody quotes: the published Core ML bundles are 557 to 848 MB each, because the host-side embedding table has to ship even though it never goes near the Neural Engine. Of the 679.9 MB ANE bundle only 251.91 MB is the Core ML body; the W8 variant shrinks that body to 129.29 MB and the bundle still only falls to 557.3 MB. Roughly 428 MB of both is embeddings and tokenizer, which is the clearest possible demonstration of where the bytes actually are.

The finding: K is a literal again

Here is the thing I would keep if I could keep only one, and it is the second independent time it has turned up.

The browser piece closed on rlcd-modernbert-151m's ONNX export, whose graph metadata reads logits[batch_size, 25] — batch symbolic, sequence symbolic, 25 a literal. The option count that is a loop bound in PyTorch had become a tensor shape at the edge. I flagged it as a small, checkable way in which the architecture does not survive the trip intact, and attached a falsifier.

It happens again here, on a different runtime, in a different framework, by a different author, against a different model. And it is worse.

In laya_coreml/convert.py, only two of the five model inputs get flexible shapes:

# laya_coreml/convert.py — max_options defaults to 32
inputs = {
    "input_ids":     torch.zeros((batch_size, length), dtype=torch.int32),
    "attention_mask": torch.ones((batch_size, length), dtype=torch.int32),
    "marker_pos":    torch.zeros((batch_size, max_options), dtype=torch.int32),
    "marker_mask":    torch.ones((batch_size, max_options), dtype=torch.int32),
    "qtype":         torch.zeros((batch_size,), dtype=torch.int32),
}
specs = [
    ct.TensorType(
        name=name, dtype=np.int32,
        shape=sequence_shape if name in ("input_ids", "attention_mask") else tuple(value.shape),
    )
    for name, value in inputs.items()
]

sequence_shape is a ct.EnumeratedShapes over a bucket list. Everything else is tuple(value.shape) — a literal. The runtime then enforces it with the politest possible error, which even tells you the remedy:

# laya_coreml/inputs.py
if any(len(item["markers"]) > shape["max_options"] for item in items):
    raise ValueError("Question exceeds the exported max_options; convert with a larger value")

That is the same class of thrown error the browser worker raised past its 25th candidate, stated as a limit instead of advertised as a feature. And on the Neural Engine path it hardens twice over: the adapter refuses to load a bundle that is not exactly K=32, asserts the graph's full input signature including a dense marker_map (1, L, 1, 32), and reshapes the output to (1, 32) unconditionally.

# laya_coreml/ane.py
if shape["batch_size"] != 1 or shape["max_options"] != 32 or shape["flexible"]:
    raise ValueError("ANE runtime requires a fixed B1/K32 bundle")
...
expected_shapes = {"embeddings": (1, width, 1, length), "full_mask": (1, length, 1, length),
                   "local_mask": (1, length, 1, length), "type_vectors": (1, width, 1, 1),
                   "marker_map": (1, length, 1, 32)}

And every published bundle carries its frozen shape in its own manifest, so this is not an inference about one export. It is an inventory.

K is never flexible in any of the six. Five are 32 and one is 4, and the one that is 4 is the Snake bundle, whose option capacity is four because a snake has four directions to choose between. That is the constraint in its purest form: a general-purpose decision model, exported into a graph that can physically only answer questions about a game with at most four answers.

The difference from the browser case is the direction of travel. In ONNX, batch and sequence stayed symbolic and only N froze. On the Neural Engine, all three freeze — B=1, L=96, K=32 — because the layout rewrite that makes the graph an ANE target (channel-first B,C,1,L activations, 1×1 convolutions for every projection, per-head attention, static masks) is a rewrite into fixed buffers. Going closer to the metal costs more of the shape, not less.

What the Snake decision actually is

Before anyone reads 49.66 decisions per second as a model playing Snake: read what the model is given.

# laya_coreml/snake/policy.py — the planner runs first, and it already knows
moves = game.moves()                                    # legality, safety, advance, eats
safe = [m for m in moves if m.safe]
preferred = max(safe, key=lambda m: m.advance).direction   # the answer
...
questions["move"]["criteria"] = {
    m.direction: ("Blocked. Collision."       if not m.legal
                  else "Unsafe. Traps the snake." if not m.safe
                  else "Safe. Eat food now. Best." if m.eats
                  else "Safe. Best route to food." if m.direction == preferred
                  else "Safe. Slower route.")
    for m in moves
}

A deterministic Hamiltonian-cycle planner computes legality, safety, progress and whether a move eats, picks the best one, and then writes that verdict into the option text. The model's job is to read four short labels, one of which says Safe. Best route to food., and put its mass there. A shield then clamps execution to the planner's safe set anyway.

The repository is completely straight about this. From its own benchmark page:

The model still receives the same exact planner features, so these runs do not test reasoning from an unprocessed board.

With the shield disabled, raw top-1 over 200 moves on three seeds scored 7, 6 and 7 with zero deaths. Across every mode: 4,920 decisions, zero deaths, four safety interventions. All of that is a legitimate demonstration of a fast local classifier in a loop, which is what it says it is. None of it is a demonstration of a model playing Snake, and the difference is the planner.

What I could not check, and what the repo could not either

The honest inventory of holes, mine first.

I did not run any of this. No Apple silicon, so every millisecond, watt and joule above is read out of committed JSON. I checked what is checkable from source: the export shapes, the option counts, the token totals, the parameter arithmetic, the grep.

The repository's own holes are more interesting, because it publishes them. CPU_AND_NE did not make the ordinary Core ML export use the Neural Engine at all — the compute plan put all 1,318 assigned operations on the CPU and the result was 81.34 ms p50, seven times slower than the CPU+GPU plan. The rewrite is what moved 6,390 nonconstant operations onto MLNeuralEngineComputeDevice, and even then the repo grades its own evidence: an MLComputePlan is anticipated placement, so it went and took a 15.97-second Instruments trace and found 3,124 Neural Engine prediction intervals, then noted that the table is global and cannot attach a PID to any of them.

RangeDim with CPU_AND_GPU produced wrong and non-repeatable answers — 47 of 63 on the SDPA graph, 20 of 63 with explicit attention — and FP32 did not fix it. Enumerated shapes did, and then tripped an MPSGraph compiler SIGTRAP on a sliced constant boolean mask. All of that is committed with "passed": false rather than deleted.

And the energy measurement: an IOReport counter returned roughly 38,021 W for the CPU and 2,068 W for the Neural Engine in a single sample, which propagated into a 40,089 W system reading. The entire affected run was rejected whole, without clipping the bad sample, and its raw record is still in the tree marked unusable. That is the behaviour that makes me believe the 2.784× energy number more than I believe most benchmark tables.

The gap the whole project leaves open, stated in its own words: the short-input advantage does not generalise. A real 1,024-token request through the Neural Engine graph takes 91.703 ms p50 against the MLX port's 51.98 ms. The 4.98 ms figure describes a 96-token window and nothing else.

What I would actually ship

• If the workload is short, fixed and repeated, the Neural Engine is worth the rewrite. 1.394× the speed and 2.784× the whole-system energy, on the same weights, with 59/59 fixture agreement and 0.002925 maximum probability drift. The energy ratio is the real result and the speed ratio is the one that travelled.
• If the workload is long or variable, use MLX. 91.7 ms against 52.0 ms at 1,024 tokens, and MLX batches questions where the ANE bundle calls sequentially. Fixed graphs are a bet on the shape of your traffic.
• Do not quote a per-decision latency without the option count and the question count. On this architecture they are the two variables that move it, and every circulating number for this project has both stripped off.
• Check max_options in the manifest before you design around a bundle. It is a published field, it is never flexible, and in one shipped bundle it is 4.

The answer to what does Jev-shaped inference cost on real Apple silicon is: about 5 ms for a small short question on a Neural Engine, about 7 to 14 ms in MLX, 1.394× rather than the 10× the project set out to find against a well-tuned GPU path, and roughly 800 MiB to 1.8 GiB of unified memory depending entirely on how many questions you ask at once. Every one of those is a smaller number than the one that travelled, and all of them come with a denominator.

Nothing here was executed. There is no Apple silicon in the machine this was written on, so every latency, watt and joule above is Reported: read out of the committed results in mizorewww/laya-coreml at 12b7501 and mizorewww/laya-mlx at fc1df62, both cloned rather than summarised, both Apache-2.0 ports of NandhaKishorM/laya at 6a58191. The shape inventory comes from benchmarks/results/hub-release.json; the Snake rates from benchmarks/results/coreml-snake.json; the memory and parameter figures from laya-mlx's BENCHMARKS.md; the energy and Neural Engine placement figures from docs/ANE_BENCHMARKS.md. Star counts are from the GitHub API on 20 September 2026. The "1 GB max memory" and "roughly 60 decisions per second" phrasings reached me through a third-party summary, not from either repository. Third-party Jev latencies are the ones collected in Jev scores zero. Companion pieces: Jev in the browser for the export constraint this one confirms, and Jev scores zero for what Laya's marker-token architecture is and is not.