{
  "claim": "Reversing a question's option list does not change AgentJev's answer, and neither does any other permutation of it. The answer changed on 0 of 50 reversed questions and on 0 of 2,520 permutations. The candidate vectors that come out of the backbone are bit-identical between the two orders, so the residual movement — worst case 9.5e-07 logits — is a few units in the last place of a float32, arriving inside the head's attention sum. The same head moves a candidate's logit by a median of 0.088 when you change which OTHER options are on the list. Position carries nothing; composition carries a lot.",
  "method": "Weights from aimeigaoshou/agent-jev, code from malevrigns/agent-jev at b66a8ce, float32 on CPU — the dtype jev_service/engine.py uses when --device is not cuda. Per question: build the caller's payload twice, once as published and once with the option list reversed, and push each independently through jev_service.contract.prepare, encode_paths, the Qwen3 backbone, AgentJevModel._score, the per-primitive temperature from temperatures.json and the softmax. Compare the two candidate-vector tensors. Because they come back bit-identical, scoring a permuted set of the forward vectors through _score is exactly what the service computes for a permuted request, so every permutation — 24 at four candidates, 120 at five — was then scored at the head and realigned. Leave-one-out drops each candidate in turn and re-scores the survivors; the size-controlled variant compares two different leave-one-out subsets that both contain the candidate being watched. Boolean questions are excluded: contract.prepare hardwires their candidate order to ['true', 'false'], so a caller has nothing to reverse.",
  "source": "https://huggingface.co/aimeigaoshou/agent-jev",
  "captured": "2026-09-22",
  "note": "The end-to-end run is 50 questions because a 0.6B model in float32 on four contended CPU cores is a minute or two per reversed pair; the permutation sweep is large because it reuses one backbone pass per question. Cardinality is 4 and 5 only, which is what this benchmark contains — the contract accepts 255 and I have not tested there. Nor have I tested bf16, which is what the GPU service runs and whose significand is eight bits against float32’s twenty-four. The questions are a seeded shuffle of the Choice and Score half of the split, so the draw is random but it is not stratified: top-1 agreement with the teacher on this subsample is 0.720 against 0.7925 on the full 2,000, which at this N is within sampling noise and is not a claim that the slice is representative.",
  "columns": [
    {
      "key": "check",
      "label": "perturbation"
    },
    {
      "key": "n",
      "label": "n",
      "align": "right"
    },
    {
      "key": "result",
      "label": "measured",
      "align": "right"
    },
    {
      "key": "isolated",
      "label": "a per-option scorer gives"
    }
  ],
  "rows": [
    {
      "check": "reverse the list — answer changed",
      "n": 50,
      "result": "0",
      "isolated": "0, by construction"
    },
    {
      "check": "reverse the list — worst |Δ| on a backbone vector",
      "n": 50,
      "result": "0.0",
      "isolated": "0, exactly"
    },
    {
      "check": "reverse the list — worst |Δlogit|",
      "n": 50,
      "result": "4.8e-07",
      "isolated": "0, exactly"
    },
    {
      "check": "reverse the list — worst |Δprobability|",
      "n": 50,
      "result": "6.0e-08",
      "isolated": "0, exactly"
    },
    {
      "check": "every permutation — answer changed",
      "n": "2,520",
      "result": "0",
      "isolated": "0, by construction"
    },
    {
      "check": "every permutation — worst |Δlogit|",
      "n": "2,520",
      "result": "9.5e-07",
      "isolated": "0, exactly"
    },
    {
      "check": "drop an option — median |Δlogit| on a survivor",
      "n": 780,
      "result": "0.088",
      "isolated": "0, exactly"
    },
    {
      "check": "drop an option — worst |Δlogit| on a survivor",
      "n": 780,
      "result": "0.71",
      "isolated": "0, exactly"
    },
    {
      "check": "swap which neighbour is present, set size fixed — median |Δlogit|",
      "n": 900,
      "result": "0.081",
      "isolated": "0, exactly"
    }
  ]
}