Laya vs Jev: the slower model gets an easier question

virajbhartiya/laya-vs-jev puts two decision models on the same obstacle course. Laya runs on the machine through MLX. Jev is TypeSafe's hosted API over the network. The course is a Python port of Chrome's offline dinosaur game, pinned to a Chromium commit, with the original constants and collision boxes. Both models get the same kind of question every move — pick one of jump, duck, run — and answer with probabilities. Neither was trained on the game.

My first reaction was the cheap one: a real-time game that races local inference against an HTTPS round trip measures the round trip, not the models. It does. The repository says so on line 119 of its own demo document, in a section headed Keeping it fair to a slow model, and says it better than I was going to:

For Jev, at about 370 ms, it is one turn every 22 frames, while the window in which a jump clears an obstacle is only 18 to 28 frames wide. With one chance per obstacle, Jev died at the first or second one even though it picked the best move every time.

Then it ships two mitigations, and then concedes: "What stays unfair is physics." That is a repository that has already had the argument with itself.

So the interesting question is not whether the race is fair. It is what an honest head-to-head measures once you have taken latency out of play as far as it will go. I read the harness, reproduced the arithmetic, and then ran the repository's own planner over its own benchmark snapshots on a Linux box with no models and no API key at all. The answer is that the harness is the experiment, and the thing it varies most is not the model.

The arithmetic, checked

laya_mlx/trex/engine.py:13 sets FPS = 60, so FRAME_MS = 1000 / 60 = 16.67. The round-trip numbers land as advertised:

The round trip is roughly the whole window. That is the entire story of why one-question-at-a-time kills a hosted model here, and why the fix has to be concurrency rather than speed: you cannot make the window wider, and you cannot make the network shorter, so you buy more chances inside it.

I wanted the window from the physics rather than from the prose. The planner and the engine import nothing but math, sys, random and dataclasses, so they run anywhere. For each obstacle I sweep the takeoff delay and keep the longest contiguous run of delays that survives the 160-frame horizon:

# laya_mlx/trex/planner.py exposes everything this needs.
search = Search(World(snap, 160), (1, 1), 160)
ok, state = [], snap.trex
for d in range(70):
    jumped = search.press(state, "jump", d)
    if search.run_frames(jumped, "run", d, 160 - d) is not None:
        ok.append(d)                       # a takeoff at delay d clears it
    state = search.run_frames(state, "run", d, 1)

Over 616 obstacles that actually need a jump or a duck, across five course seeds, the window is 13 to 26 frames, median 18 — 217 to 433 ms. The doc's 18-to-28 sits about four frames higher than that at both ends: its lower bound is my median. High birds are excluded; the dino runs under them, so their "window" is 31 to 50 frames and would flatter the distribution.

88.5% of jump-or-duck obstacles have a window of 22 frames or fewer. At one request in flight, the majority of obstacles offer a hosted model at most one chance, and nothing guarantees that chance lands inside it. At six in flight the turn comes every 6 frames, which is narrower than every window in the sample. That number is not in the repository, and it is the one that shows the mitigation working.

The two mitigations, precisely

Several requests in flight. Pilot starts one worker thread per in-flight slot, and Pilot.stagger decides how far apart they ask:

@property
def stagger(self):
    # At most four questions per round trip: enough turns, and a hosted API's
    # rate limit is respected. Extra request slots only keep that schedule regular.
    return min(12, max(1, round(max(1.0, self.typical_frames) / min(self.inflight, 4))))

--jev-inflight defaults to 6 and accepts up to 8, but min(self.inflight, 4) caps the schedule at four questions per round trip. At 22 frames of typical latency that is a turn every 6 frames. Slots five and six do not buy more turns; they absorb jitter so the six-frame cadence holds. The README does not say this, and it is the difference between "I raised the flag to 8 and nothing changed" and knowing why.

Lockstep. --lockstep N freezes each game while its own model answers, then plays N frames and asks again. Arena.tick_lockstep simply returns while a pilot is waiting — no physics step, no shield, nothing — so the game is a pure function of its action sequence. cli.build() also forces inflight to 1 for both players in this mode. The flag's own help text is the clearest statement of intent in the repository: "Removes latency from play to compare decisions alone."

Discards. An answer is asked on the premise that the keys stay as they are until it lands. Pilot keeps an epoch counter that increments whenever an applied answer changes the keys, and premise_holds() rejects anything asked in a stale epoch, anything older than the last accepted sequence number, and anything whose predecessor was itself dropped. The last clause is a Python chained comparison — self.bump_seq == decision.pending_seq is not None — which reads as "the answer that ended the old epoch is the one this answer was told about, and it existed", and is easy to misread as an equality against a boolean. Rejected answers are counted in stats.discarded, surface in the report as answers_discarded and appear on screen as skipped. In the repository's own screenshot that counter reads 1074 for Laya and 689 for Jev — the fast player throws away more answers than the slow one, because it changes the keys more often.

The finding: the prompt is a function of latency

Here is the part I did not expect. The question is not fixed. Pilot.work() calls self.timing(), which returns a tuple derived from that player's own measured latency:

def timing(self):
    """(first, gap, period) for Planner.plan, in frames."""
    if self.lockstep:
        return ((0, 0), (self.lockstep, self.lockstep), None)
    landing = (self.latency_frames, self.latency_frames + self.jitter_frames)
    if self.inflight == 1:
        return (landing, landing, None)
    return (landing, landing, self.longest_wait)

That tuple goes straight into Planner.plan(snap, first, gap, period), which decides which actions are safe, which one is best, and — this is the part that matters — how far away the obstacle is said to be:

travelled = sum(obstacle_step(world.speed.get(k, snap.speed), ahead.offset)
                for k in range(1, lo + 1))
distance = int(ahead.x - travelled - (snap.trex_x + TREX_WIDTH))

lo is the player's own latency in frames. So the distance printed in the state line is where the obstacle will be when this player's answer lands. Two players looking at the same physical frame are told two different numbers.

One frame. Laya is told the cacti are 172 px away and that waiting is best. Jev is told they are 70 px away and that jumping is best. If Laya answers jump it is scored a miss; if Jev answers jump on the same frame it is scored a hit. Neither model is wrong, and neither prompt is wrong — the prompt is doing exactly what it should, which is to describe the world the answer will arrive in. But it means the two players are not taking the same exam.

I put the repository's own benchmark sample through this. scripts/benchmark_trex_planner.py builds snapshots from seed 17 — 122 of them, called three times each for the 366 planner calls its doc reports — and I planned each one twice, once at Laya's timing and once at Jev's, using two of the three regimes that script already hardcodes:

Where both distances are defined, Jev's is smaller every single time, by 102 to 170 px, median 136. There is no snapshot in the sample where the slower player is told the obstacle is further away.

Which exam is harder

The labeled prompt marks exactly one option Best., marks the other safe ones Safe., and marks unsafe ones Unsafe. … Collision. So "picked the planner's best move" is a reading-comprehension score, which the docs say plainly: "it measures how fast and how reliably each model reads a labelled situation, not whether it can work out dinosaur physics."

The difficulty of that reading task is set by how many options are struck out before the model sees them.

Laya is shown all three actions as safe on 71.3% of snapshots and has to pick the one line that additionally says Best — where the three lines are "Jumps too early", "Crouches; no benefit" and "Waits; acts later". Jev sees two of three marked Collision on 59.8%, and sees all three safe on 19.7%. A more forced question is an easier question.

This is the answer to the number in the repository's hero screenshot.

expand

A prompt that says Safe about a collision

One thing falls out of the planner's fallback path that the README does not surface. When no action survives every possible landing time, Planner.plan downgrades to best effort and sets robust = False. If even the relaxed search finds nothing, plan.safe stays all-False while plan.best keeps whatever plan_once chose. build_question then formats the options like this:

criteria = {
    a: f"Safe. {notes[a]}. Best." if a == best
    else f"Safe. {notes[a]}." if plan.safe[a]
    else f"Unsafe. {notes[a]}. Collision."
    for a in ACTIONS
}

The a == best branch is checked first and does not consult plan.safe, so the recommended option is labelled Safe. unconditionally. On the 30 of 122 snapshots where Jev's timing leaves nothing safe, the prompt reads:

state:    Dino runner game. The dino is in the air. Large cactus ahead, 0 px away.
  jump: Unsafe. Lands on the large cactus. Collision.
  duck: Unsafe. Drops onto the large cactus. Collision.
  run:  Safe. Lands on the large cactus. Best.

Safe. and Lands on the large cactus. in the same line. All 30 read this way. At Laya's timing it happens on 14. It is a formatting bug rather than a physics bug — the planner knows perfectly well the situation is lost, and the report counts it in best_effort_decisions — but it is a contradictory string handed to a model whose entire job is to read the string, and it lands on the slow player twice as often. The robust flag is right there in the Plan; the format call just never asks.

Related: build_question's docstring says labeled "never names the answer". It does not name it in the state — that is what --prompt guided adds — but it appends Best. to the winning option's criteria in both modes. The distinction is real; the docstring is a word short.

What is not in the repository

Thirty files in benchmarks/results/, inherited from mizorewww/laya-mlx and credited at README.md:62. None of them mentions Jev. Those are Snake and single-question latency results for the MLX runtime, and the README says so: "those latency measurements are not T-Rex gameplay measurements."

What does exist is three head-to-head tables in docs/TREX_DEMO.md prose and the screenshot above. What does not exist is any of it in a form you can re-analyse. The document states it outright: "Raw validation recordings and videos are local artifacts and are not bundled with the repository." No artifacts/trex/*.json, no .jsonl recording, no crash sidecar. The harness is published; the evidence is a picture of a scoreboard and some tables.

The document is also disciplined about what it will let those tables mean. The historical M3 run carries "The single-request schedule and host stalls prevent using these results to judge the current system." The latest match carries "This is a show result, not evidence of better model skill" and "The host dropped 5.83 seconds of stalled time; do not treat it as a clean latency benchmark." The smoke test carries "A single live run is a smoke test, not evidence of a stable winner."

So: anyone quoting a Laya-vs-Jev scoreline is quoting their own run, on their own Mac, on their own network, against a hosted model whose latency they do not control. The repository never claims otherwise. This is the caveat that matters more than anything else in this piece, and it is the one most likely to get lost when a screenshot travels.

One more structural note, since it is easy to miss: --course jev is the default, so by default one of the two competitors designs the obstacles both must clear. The design is shared, seeded and computed ahead of play, so it is not a latency advantage, and the doc reports Jev's difficulty rising with speed (rank correlation 0.47) against Laya's 0.19. It is still a competitor setting the exam. --course random removes it, and any comparison I would publish would use it.

Does this measure decision quality at all?

Partly, and only in one mode. Here is the exact state of it.

In real time, no. The per-player agreement_with_planner in the report is scored against a rubric built from that player's own latency, and I have measured how far apart the two rubrics are: the same question on 13.9% of snapshots, a different recommended action on 23.0%. The live window draws the two rates as a head-to-head bar (app.py:602, "BEST MOVE PICKED"), and that bar is comparing two different exams.

In lockstep, yes — and it has been run once. timing() returns ((0,0),(N,N),None) for every player, so the tuple comes from the flag instead of the model and the rubric is shared. The doc's historical table has it:

That is the closest thing to a decision-quality comparison anyone has, and it is published without a decision count, without a report file, inside a section the document itself labels superseded. It is also not quite clean: each pilot freezes independently, so the two games diverge at the first differing action and the state distributions stop matching.

Per-decision agreement between the models is not recoverable from anything the harness emits. I checked. Pilot.think computes agreed = same_effect(proposed, plan.best, plan.airborne) and the recording's per-frame panel carries p (the probability vector), prop, exec, veto and a running agree ratio — but not best per decision, not the question, and no join key onto the other player's decision at the same game frame. You could difference agree × n between frames to recover most per-decision hit/miss flags. You still could not answer "on the frames where both models were asked the same thing, how often did they pick the same action", because in real time there are almost no such frames, and in lockstep the games have drifted apart.

There is also no way to give both players the same amount of work. --duration, --round-seconds and --rounds are all wall-clock, and in lockstep a player's game advances only while it is not waiting: six played frames per decision against a two-frame wait for Laya and a twenty-two-frame wait for Jev, so roughly 6/8 of real time against roughly 6/28. The report exposes exactly this as game_time_ratio. Ninety seconds on the clock is about seventy seconds of game for one player and about twenty for the other. There is no --decisions N.

The run I would actually do

Do not race them. The race is not needed, and the pieces to avoid it are already in the repository and already model-free:

from laya_mlx.trex.planner import Planner, snapshot
from laya_mlx.trex.backends import build_question, create, decide
 
snaps = build_snapshots(seed=17)          # exactly as scripts/benchmark_trex_planner.py does
planner = Planner()
backends = {k: create(k) for k in ("laya", "jev")}
 
for snap in snaps:
    # ONE timing tuple for both players: latency is a constant, not a variable.
    plan = planner.plan(snap, (0, 0), (6, 6), None)
    state, questions = build_question(plan, "labeled")
    row = {"best": plan.best, "safe": plan.safe, "state": state}
    for name, backend in backends.items():
        row[name] = decide(backend, state, questions).probabilities  # identical bytes
    print(row)

Fix one timing tuple. Generate the question set once. Ask both models the same strings. Then you get a per-decision agreement rate, a per-decision comparison against plan.best, and a paired probability distance — on identical inputs, with no game, no clock, no network in the measurement, and it is reproducible by anyone with a key because the question set is text you can commit. It is about twenty lines against the existing API, and it is the first direct quality comparison between these two models that would mean anything.

If you must do it in the game: --lockstep 6 --unassisted --course random --course-style original --seed 7, one player per process, and tune --duration per player until the two reports show equal decisions. The shields off matter — with --unassisted the action sequence is the model's alone, so the run is deterministic and someone else can reproduce it exactly.

What I would take away

• The harness is the more interesting artifact. A deterministic 60 fps Chromium port, an exact-physics safety game with adversarial answer timing, a planner the repository clocks at 0.7 ms p50 for the fast player, and a replay format that re-renders rather than screen-captures. All of it is Apache-2.0 and none of it needs MLX; I did four new measurements on a Linux container with no model of any kind.
• A latency-aware prompt is a good idea that breaks comparison. Telling a slow model where the obstacle will be is obviously right for playing the game and quietly fatal for scoring two players against each other. If you build anything that adapts the prompt to the caller, you have made your metric caller-dependent, and nothing in the metric's name will warn you.
• Concurrency, not speed, is the lever on a hosted model in a real-time loop. Four questions per round trip turns a cadence wider than 88.5% of the windows into one narrower than all of them, without the API getting any faster.
• Publish the report file. One --report out.json committed next to the screenshot would make the tables re-analysable and the caveats checkable. The discipline in that document deserves it.

Related reading on this family: the browser piece, which refused a local-versus-hosted latency comparison for this exact reason; the Apple silicon ports, whose benchmark suite this repository inherits; and what a decision model cannot do, which is the architectural case for why none of these models can reason about the game in the first place.

Read at virajbhartiya/laya-vs-jev commit 2854178, 164 tracked files. Repository Apache-2.0; game rules, sprites and sounds from The Chromium Authors under BSD-3-Clause. Every planner measurement in this piece was computed locally from that commit's laya_mlx/trex/engine.py, planner.py and backends.py; no model was loaded and no API was called.