# Laya vs Jev: the slower model gets an easier question

> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/laya-vs-jev
> date: 2026-09-21
> tags: explainer, llm, benchmarks, on-device-inference, architecture, agents
[`virajbhartiya/laya-vs-jev`](https://github.com/virajbhartiya/laya-vs-jev) puts
two decision models on the same obstacle course. **Laya** runs on the machine
through MLX. **Jev** is TypeSafe's hosted API over the network. The course is a
Python port of Chrome's offline dinosaur game, pinned to a Chromium commit, with
the original constants and collision boxes. Both models get the same *kind* of
question every move — pick one of `jump`, `duck`, `run` — and answer with
probabilities. Neither was trained on the game.

<RepoCard repo="virajbhartiya/laya-vs-jev" />

My first reaction was the cheap one: a real-time game that races local inference
against an HTTPS round trip measures the round trip, not the models. It does.
The repository says so on line 119 of its own demo document, in a section headed
**Keeping it fair to a slow model**, and says it better than I was going to:

> For Jev, at about 370 ms, it is one turn every 22 frames, while the window in
> which a jump clears an obstacle is only 18 to 28 frames wide. With one chance
> per obstacle, **Jev died at the first or second one even though it picked the
> best move every time.**

Then it ships two mitigations, and then concedes: *"What stays unfair is
physics."* That is a repository that has already had the argument with itself.

So the interesting question is not whether the race is fair. It is what an honest
head-to-head measures once you have taken latency out of play as far as it will
go. I read the harness, reproduced the arithmetic, and then ran the repository's
own planner over its own benchmark snapshots on a Linux box with no models and no
API key at all. The answer is that the harness is the experiment, and the thing
it varies most is not the model.

<Callout type="note">
Everything below is **read from source or computed locally**. There is no Apple
silicon here and no TypeSafe key, so I have raced nothing. The planner and the
engine are pure Python with no MLX dependency, which is why the new measurements
in this piece are possible at all — and which is also the article's main
practical finding.
</Callout>

## The arithmetic, checked

`laya_mlx/trex/engine.py:13` sets `FPS = 60`, so `FRAME_MS = 1000 / 60 = 16.67`.
The round-trip numbers land as advertised:

| | frames | ms |
|---|---:|---:|
| One Jev round trip at 370 ms | 22.2 | 370 |
| The doc's stated jump window | 18–28 | 300–467 |
| Laya's answer at ~33 ms | 2.0 | 33 |

The round trip is roughly the *whole* window. That is the entire story of why
one-question-at-a-time kills a hosted model here, and why the fix has to be
concurrency rather than speed: you cannot make the window wider, and you cannot
make the network shorter, so you buy more chances inside it.

I wanted the window from the physics rather than from the prose. The planner and
the engine import nothing but `math`, `sys`, `random` and `dataclasses`, so they
run anywhere. For each obstacle I sweep the takeoff delay and keep the longest
contiguous run of delays that survives the 160-frame horizon:

```python
# laya_mlx/trex/planner.py exposes everything this needs.
search = Search(World(snap, 160), (1, 1), 160)
ok, state = [], snap.trex
for d in range(70):
    jumped = search.press(state, "jump", d)
    if search.run_frames(jumped, "run", d, 160 - d) is not None:
        ok.append(d)                       # a takeoff at delay d clears it
    state = search.run_frames(state, "run", d, 1)
```

Over **616 obstacles that actually need a jump or a duck**, across five course
seeds, the window is **13 to 26 frames, median 18** — 217 to 433 ms. The doc's
18-to-28 sits about four frames higher than that at both ends: its lower bound is
my median. High birds are excluded; the dino runs under them, so their "window"
is 31 to 50 frames and would flatter the distribution.

<CadenceWindow />

**88.5% of jump-or-duck obstacles have a window of 22 frames or fewer.** At one
request in flight, the majority of obstacles offer a hosted model at most one
chance, and nothing guarantees that chance lands inside it. At six in flight
the turn comes every 6 frames, which is narrower than *every* window in the
sample. That number is not in the repository, and it is the one that shows the
mitigation working.

## The two mitigations, precisely

**Several requests in flight.** `Pilot` starts one worker thread per in-flight
slot, and `Pilot.stagger` decides how far apart they ask:

```python
@property
def stagger(self):
    # At most four questions per round trip: enough turns, and a hosted API's
    # rate limit is respected. Extra request slots only keep that schedule regular.
    return min(12, max(1, round(max(1.0, self.typical_frames) / min(self.inflight, 4))))
```

`--jev-inflight` defaults to 6 and accepts up to 8, but `min(self.inflight, 4)`
caps the *schedule* at four questions per round trip. At 22 frames of typical
latency that is a turn every 6 frames. Slots five and six do not buy more turns;
they absorb jitter so the six-frame cadence holds. The README does not say this,
and it is the difference between "I raised the flag to 8 and nothing changed" and
knowing why.

**Lockstep.** `--lockstep N` freezes each game while its own model answers, then
plays `N` frames and asks again. `Arena.tick_lockstep` simply returns while a
pilot is waiting — no physics step, no shield, nothing — so the game is a pure
function of its action sequence. `cli.build()` also forces `inflight` to 1 for
both players in this mode. The flag's own help text is the clearest statement of
intent in the repository: *"Removes latency from play to compare decisions
alone."*

**Discards.** An answer is asked on the premise that the keys stay as they are
until it lands. `Pilot` keeps an `epoch` counter that increments whenever an
applied answer changes the keys, and `premise_holds()` rejects anything asked in
a stale epoch, anything older than the last accepted sequence number, and
anything whose predecessor was itself dropped. The last clause is a Python chained
comparison — `self.bump_seq == decision.pending_seq is not None` — which reads as
"the answer that ended the old epoch is the one this answer was told about, and it
existed", and is easy to misread as an equality against a boolean. Rejected answers
are counted in `stats.discarded`, surface in the report as `answers_discarded` and
appear on screen as `skipped`. In the repository's own screenshot that counter
reads **1074 for Laya and 689 for Jev** — the fast player throws away more answers
than the slow one, because it changes the keys more often.

## The finding: the prompt is a function of latency

Here is the part I did not expect. The question is not fixed. `Pilot.work()`
calls `self.timing()`, which returns a tuple derived from that player's own
*measured* latency:

```python
def timing(self):
    """(first, gap, period) for Planner.plan, in frames."""
    if self.lockstep:
        return ((0, 0), (self.lockstep, self.lockstep), None)
    landing = (self.latency_frames, self.latency_frames + self.jitter_frames)
    if self.inflight == 1:
        return (landing, landing, None)
    return (landing, landing, self.longest_wait)
```

That tuple goes straight into `Planner.plan(snap, first, gap, period)`, which
decides which actions are safe, which one is `best`, and — this is the part that
matters — how far away the obstacle is *said to be*:

```python
travelled = sum(obstacle_step(world.speed.get(k, snap.speed), ahead.offset)
                for k in range(1, lo + 1))
distance = int(ahead.x - travelled - (snap.trex_x + TREX_WIDTH))
```

`lo` is the player's own latency in frames. So the distance printed in the state
line is where the obstacle will be *when this player's answer lands*. Two players
looking at the same physical frame are told two different numbers.

<TwoQuestions />

One frame. Laya is told the cacti are 172 px away and that waiting is best. Jev
is told they are 70 px away and that jumping is best. If Laya answers `jump` it
is scored a miss; if Jev answers `jump` on the same frame it is scored a hit.
Neither model is wrong, and neither prompt is wrong — the prompt is doing exactly
what it should, which is to describe the world the answer will arrive in. But it
means the two players are not taking the same exam.

I put the repository's own benchmark sample through this. `scripts/benchmark_trex_planner.py`
builds snapshots from seed 17 — 122 of them, called three times each for the 366
planner calls its doc reports — and I planned each one twice, once at Laya's
timing and once at Jev's, using two of the three regimes that script already
hardcodes:

| Over 122 snapshots, seed 17 | |
|---|---:|
| Identical question for both players | **13.9%** |
| Different distance in the state line | 95.1% |
| Different set of safe actions | 72.1% |
| Different recommended action | 23.0% |

Where both distances are defined, Jev's is smaller every single time, by 102 to
170 px, median 136. There is no snapshot in the sample where the slower player is
told the obstacle is further away.

## Which exam is harder

The `labeled` prompt marks exactly one option `Best.`, marks the other safe ones
`Safe.`, and marks unsafe ones `Unsafe. … Collision.` So "picked the planner's
best move" is a reading-comprehension score, which the docs say plainly: *"it
measures how fast and how reliably each model reads a labelled situation, not
whether it can work out dinosaur physics."*

The difficulty of that reading task is set by how many options are struck out
before the model sees them.

<SafeSpread />

Laya is shown all three actions as safe on **71.3%** of snapshots and has to
pick the one line that additionally says `Best` — where the three lines are
"Jumps too early", "Crouches; no benefit" and "Waits; acts later". Jev sees two
of three marked `Collision` on **59.8%**, and sees all three safe on 19.7%. A
more forced question is an easier question.

This is the answer to the number in the repository's hero screenshot.

<Figure
  src="/articles/laya-vs-jev/fig1.png"
  alt="A screenshot of the arena window at the end of a match. Two lanes, Laya on the left in blue and Jev on the right in orange, both showing a dinosaur mid-course with an 'Arrival shield saved it' notice. Both scores read 1,175 and the header says MATCH 0 to 2, Jev wins the match. Laya's stats: answer time 49 ms, model 24 ms, best move 71 percent, live saves 774 with 142 model vetoes, cost free. Jev's: answer time 383 ms, model 361 ms, best move 89 percent, live saves 92 with 0 model vetoes, cost 0.0195 dollars. At the bottom, a score-over-time chart, an answer-time-per-move chart, and two head-to-head bars: answer time median 49 ms against 383 ms, and BEST MOVE PICKED 71 percent against 89 percent. The footer reads 'Course designed by Jev: 192 obstacles so far, seed 7'."
  caption="The only head-to-head result the repository ships in full. Read the bottom-right pair: BEST MOVE PICKED, 71% for Laya and 89% for Jev, drawn as a direct comparison. Those two percentages were scored against prompts built from each player's own latency — the 89% belongs to the exam where two of three options were struck out before the model read them. Note also the veto counts, 142 for Laya and 0 for Jev: Jev never once proposed an action the planner had marked unsafe, which is the same fact as its 89% stated a second way. (virajbhartiya/laya-vs-jev, docs/assets/trex-arena-window.png, commit 2854178, Apache-2.0, flattened onto white for dark mode.) Used under the Apache License 2.0, with the project NOTICE and the Chromium BSD-3-Clause notice for the game sprites committed beside the image at /articles/laya-vs-jev/APACHE-2.0-LICENSE.txt, /articles/laya-vs-jev/NOTICE.txt and /articles/laya-vs-jev/CHROMIUM-BSD-3-CLAUSE.txt."
/>

## A prompt that says Safe about a collision

One thing falls out of the planner's fallback path that the README does not
surface. When no action survives every possible landing time, `Planner.plan`
downgrades to best effort and sets `robust = False`. If even the relaxed search
finds nothing, `plan.safe` stays all-`False` while `plan.best` keeps whatever
`plan_once` chose. `build_question` then formats the options like this:

```python
criteria = {
    a: f"Safe. {notes[a]}. Best." if a == best
    else f"Safe. {notes[a]}." if plan.safe[a]
    else f"Unsafe. {notes[a]}. Collision."
    for a in ACTIONS
}
```

The `a == best` branch is checked first and does not consult `plan.safe`, so the
recommended option is labelled `Safe.` unconditionally. On the 30 of 122
snapshots where Jev's timing leaves nothing safe, the prompt reads:

```text
state:    Dino runner game. The dino is in the air. Large cactus ahead, 0 px away.
  jump: Unsafe. Lands on the large cactus. Collision.
  duck: Unsafe. Drops onto the large cactus. Collision.
  run:  Safe. Lands on the large cactus. Best.
```

`Safe.` and `Lands on the large cactus.` in the same line. All 30 read this way.
At Laya's timing it happens on 14. It is a formatting bug rather than a physics
bug — the planner knows perfectly well the situation is lost, and the report
counts it in `best_effort_decisions` — but it is a contradictory string handed to
a model whose entire job is to read the string, and it lands on the slow player
twice as often. The `robust` flag is right there in the `Plan`; the format call
just never asks.

Related: `build_question`'s docstring says `labeled` *"never names the answer"*.
It does not name it in the *state* — that is what `--prompt guided` adds — but it
appends `Best.` to the winning option's criteria in both modes. The distinction
is real; the docstring is a word short.

## What is not in the repository

Thirty files in `benchmarks/results/`, inherited from
[`mizorewww/laya-mlx`](https://github.com/mizorewww/laya-mlx) and credited at
`README.md:62`. **None of them mentions Jev.** Those are Snake and single-question
latency results for the MLX runtime, and the README says so: *"those latency
measurements are not T-Rex gameplay measurements."*

What does exist is three head-to-head tables in `docs/TREX_DEMO.md` prose and the
screenshot above. What does not exist is any of it in a form you can re-analyse.
The document states it outright: *"Raw validation recordings and videos are local
artifacts and are not bundled with the repository."* No `artifacts/trex/*.json`,
no `.jsonl` recording, no crash sidecar. The harness is published; the evidence
is a picture of a scoreboard and some tables.

The document is also disciplined about what it will let those tables mean. The
historical M3 run carries *"The single-request schedule and host stalls prevent
using these results to judge the current system."* The latest match carries *"This
is a show result, not evidence of better model skill"* and *"The host dropped 5.83
seconds of stalled time; do not treat it as a clean latency benchmark."* The smoke
test carries *"A single live run is a smoke test, not evidence of a stable
winner."*

So: **anyone quoting a Laya-vs-Jev scoreline is quoting their own run**, on their
own Mac, on their own network, against a hosted model whose latency they do not
control. The repository never claims otherwise. This is the caveat that matters
more than anything else in this piece, and it is the one most likely to get lost
when a screenshot travels.

One more structural note, since it is easy to miss: `--course jev` is the
**default**, so by default one of the two competitors designs the obstacles both
must clear. The design is shared, seeded and computed ahead of play, so it is not
a latency advantage, and the doc reports Jev's difficulty rising with speed (rank
correlation 0.47) against Laya's 0.19. It is still a competitor setting the exam.
`--course random` removes it, and any comparison I would publish would use it.

## Does this measure decision quality at all?

Partly, and only in one mode. Here is the exact state of it.

**In real time, no.** The per-player `agreement_with_planner` in the report is
scored against a rubric built from that player's own latency, and I have measured
how far apart the two rubrics are: the same question on 13.9% of snapshots, a
different recommended action on 23.0%. The live window draws the two rates as a
head-to-head bar (`app.py:602`, `"BEST MOVE PICKED"`), and that bar is comparing
two different exams.

**In lockstep, yes — and it has been run once.** `timing()` returns
`((0,0),(N,N),None)` for every player, so the tuple comes from the flag instead of
the model and the rubric is shared. The doc's historical table has it:

| Lockstep, 6 frames per decision | Laya | Jev |
|---|---:|---:|
| Deaths | 0 | 0 |
| Picked the planner's best move | 74% | 100% |
| Shield saves | 30 | 0 |

That is the closest thing to a decision-quality comparison anyone has, and it is
published without a decision count, without a report file, inside a section the
document itself labels superseded. It is also not quite clean: each pilot freezes
independently, so the two games diverge at the first differing action and the
state distributions stop matching.

**Per-decision agreement between the models is not recoverable from anything the
harness emits.** I checked. `Pilot.think` computes
`agreed = same_effect(proposed, plan.best, plan.airborne)` and the recording's
per-frame panel carries `p` (the probability vector), `prop`, `exec`, `veto` and
a running `agree` ratio — but not `best` per decision, not the question, and no
join key onto the other player's decision at the same game frame. You could
difference `agree × n` between frames to recover most per-decision hit/miss flags.
You still could not answer *"on the frames where both models were asked the same
thing, how often did they pick the same action"*, because in real time there are
almost no such frames, and in lockstep the games have drifted apart.

There is also no way to give both players the same amount of work. `--duration`,
`--round-seconds` and `--rounds` are all wall-clock, and in lockstep a player's
game advances only while it is not waiting: six played frames per decision against
a two-frame wait for Laya and a twenty-two-frame wait for Jev, so roughly `6/8`
of real time against roughly `6/28`. The report exposes exactly this as
`game_time_ratio`. Ninety seconds on the clock is about seventy seconds of game
for one player and about twenty for the other. There is no `--decisions N`.

### The run I would actually do

Do not race them. The race is not needed, and the pieces to avoid it are already
in the repository and already model-free:

```python
from laya_mlx.trex.planner import Planner, snapshot
from laya_mlx.trex.backends import build_question, create, decide

snaps = build_snapshots(seed=17)          # exactly as scripts/benchmark_trex_planner.py does
planner = Planner()
backends = {k: create(k) for k in ("laya", "jev")}

for snap in snaps:
    # ONE timing tuple for both players: latency is a constant, not a variable.
    plan = planner.plan(snap, (0, 0), (6, 6), None)
    state, questions = build_question(plan, "labeled")
    row = {"best": plan.best, "safe": plan.safe, "state": state}
    for name, backend in backends.items():
        row[name] = decide(backend, state, questions).probabilities  # identical bytes
    print(row)
```

Fix one timing tuple. Generate the question set once. Ask both models the same
strings. Then you get a per-decision agreement rate, a per-decision comparison
against `plan.best`, and a paired probability distance — on identical inputs, with
no game, no clock, no network in the measurement, and it is reproducible by
anyone with a key because the question set is text you can commit. It is about
twenty lines against the existing API, and it is the first direct quality
comparison between these two models that would mean anything.

If you must do it in the game: `--lockstep 6 --unassisted --course random
--course-style original --seed 7`, one player per process, and tune `--duration`
per player until the two reports show equal `decisions`. The shields off matter —
with `--unassisted` the action sequence is the model's alone, so the run is
deterministic and someone else can reproduce it exactly.

## What I would take away

- **The harness is the more interesting artifact.** A deterministic 60 fps
  Chromium port, an exact-physics safety game with adversarial answer timing, a
  planner the repository clocks at 0.7 ms p50 for the fast player, and a replay
  format that re-renders rather than screen-captures. All of it is Apache-2.0 and
  none of it needs MLX; I did four new measurements on a Linux container with no
  model of any kind.
- **A latency-aware prompt is a good idea that breaks comparison.** Telling a slow
  model where the obstacle *will be* is obviously right for playing the game and
  quietly fatal for scoring two players against each other. If you build anything
  that adapts the prompt to the caller, you have made your metric
  caller-dependent, and nothing in the metric's name will warn you.
- **Concurrency, not speed, is the lever on a hosted model in a real-time loop.**
  Four questions per round trip turns a cadence wider than 88.5% of the windows
  into one narrower than all of them, without the API getting any faster.
- **Publish the report file.** One `--report out.json` committed next to the
  screenshot would make the tables re-analysable and the caveats checkable. The
  discipline in that document deserves it.

Related reading on this family: [the browser
piece](/articles/jev-in-the-browser), which refused a local-versus-hosted latency
comparison for this exact reason; [the Apple silicon
ports](/articles/laya-mlx), whose benchmark suite this repository inherits; and
[what a decision model cannot do](/articles/what-decision-models-cannot-do),
which is the architectural case for why none of these models can reason about the
game in the first place.

<ChangeMyMind>

<Falsifier claim="The two players are handed the same question on only 13.9% of the repository's own 122 benchmark snapshots.">
Build the snapshots the way `scripts/benchmark_trex_planner.py` does (seed 17, `invincible`, a jump every 39 frames, a snapshot every 31 while obstacles are on screen), then compare `build_question(planner.plan(snap, (1,3), (1,3), 2), "labeled")` against `build_question(planner.plan(snap, (18,24), (18,24), 6), "labeled")` on each. I get 17 of 122 identical. It needs no MLX and no API key: `engine.py`, `planner.py` and `backends.py` import only `math`, `random`, `os`, `queue`, `threading`, `time`, `dataclasses` and `pathlib` at module scope, and MLX is imported lazily inside `LayaBackend.serve`. On a machine without MLX you have to load them as a package of their own rather than through `laya_mlx/__init__.py`, which pulls in `Agent` and therefore `mlx.core` — copying `laya_mlx/trex/` out with its relative imports intact is enough. If a different sample set or a different pair of timing tuples gets close to parity, the divergence is an artifact of the two regimes that script hardcodes rather than a property of the harness, and the strength of my claim goes with it.
</Falsifier>

<Falsifier claim="The BEST MOVE PICKED bar compares two rates scored against different rubrics, so Jev's 89% against Laya's 71% is not evidence of better decisions.">
Run `--lockstep 6` — where `timing()` returns the same tuple for every player — and compare the two `agreement_with_planner` figures from the report. If the gap survives at a comparable decision count and a comparable number of game frames per player, the rubric was not carrying it and the real-time bar was closer to fair than I am saying. The repository's own lockstep table already shows a gap (74% against 100%), which is why I have called this partly open rather than settled; what would move me is that table with an `n` and a committed report file.
</Falsifier>

<Falsifier claim="The takeoff window is 13-26 frames, and one serial Jev round trip is wider than it on 88.5% of jump-or-duck obstacles.">
Sweep the takeoff delay per obstacle with `Search`/`World` at a 160-frame horizon, on seeds 3, 7, 11, 17 and 23, excluding high birds. I measure a median of 18 frames over 616 obstacles. A longer horizon, `--course-style original` instead of the staged default, or counting the high bird would all move it — the high bird alone runs 31 to 50 frames and drops the share to 79.2%. If the window is genuinely 18 to 28 as the doc says, the share below 22 frames falls and my "one chance per obstacle" framing is too strong, though the conclusion about concurrency does not change.
</Falsifier>

<Falsifier claim="No machine-readable head-to-head result is committed, so every published Laya-vs-Jev scoreline is somebody's single run.">
`git grep -Ili jev` returns 13 of 164 tracked files, and `benchmarks/results/` — 30 files — is not among them. `docs/TREX_DEMO.md` says raw recordings and videos are not bundled. If a later commit adds an `artifacts/trex/*.json`, a `.jsonl` recording or a CI job that runs the arena, this is out of date and the scorelines become checkable, which is the outcome I want.
</Falsifier>

<Falsifier claim="The best-effort prompt labels a doomed action 'Safe. … Best.', on 30 of 122 snapshots at Jev's timing.">
`build_question`'s dict comprehension checks `a == best` before it checks `plan.safe[a]`, so the recommended option is prefixed `Safe.` whether or not the planner believes it. Every zero-safe snapshot I generated reproduces it. If a later commit reorders those branches, or if `plan.best` is guaranteed safe on some path I have missed, this is fixed or wrong — note that `plan_once` does contain a guard, `if not safe[best] and any(safe.values())`, which by construction cannot fire when nothing is safe.
</Falsifier>

<Falsifier claim="Raising --jev-inflight past 4 buys no extra turns.">
`Pilot.stagger` divides by `min(self.inflight, 4)`, so 6 and 8 produce the same stagger as 4 at any latency. Instrument `stagger` across `--jev-inflight 1..8` and print it; if the values keep falling past 4, I have misread the expression. The weaker claim — that the extra slots still help by absorbing jitter, per the code's own comment — is not something I have measured at all.
</Falsifier>

</ChangeMyMind>

---

Read at `virajbhartiya/laya-vs-jev` commit `2854178`, 164 tracked files. Repository
Apache-2.0; game rules, sprites and sounds from The Chromium Authors under
BSD-3-Clause. Every planner measurement in this piece was computed locally from
that commit's `laya_mlx/trex/engine.py`, `planner.py` and `backends.py`; no model
was loaded and no API was called.