[{"title":"Two ways to read a decision out of a model","description":"The whole of how a System One model turns a caller-supplied option list into probabilities, in runnable Python. Same input, same output shape, two completely different mechanisms — and only one of them is sensitive to the order you list the options in.","date":"2026-09-18","lang":"python","tags":["llm","inference","calibration"],"kind":"snippets","slug":"two-ways-to-read-a-decision","body":"Both take `(evidence, question, options[])` and return one probability per option,\nfor an option list the caller chose at request time. Neither generates any text.\nNeither has the number of options baked into its weights.\n\nThat is where the similarity stops.\n\n```python\nimport torch, torch.nn.functional as F\n\n# ─────────────────────────────────────────────────────────────────────────────\n# FAMILY 1 — vocabulary readout.  (openjev, Bespoke Nimble, any LoRA-on-an-LLM)\n#\n# The options are TEXT IN THE PROMPT. The model only ever picks a letter.\n# One forward pass, exactly like generation — you just never sample.\n# ─────────────────────────────────────────────────────────────────────────────\ndef read_by_vocabulary(model, tok, evidence, question, options):\n    letters = \"ABCDEFGHIJKLMNOP\"[: len(options)]          # hard cap: 16 options\n    prompt = tok.apply_chat_template([\n        {\"role\": \"system\", \"content\":\n            \"Apply the supplied criterion to the supplied evidence. Choose exactly \"\n            \"one listed option. Respond with only its uppercase letter.\"},\n        {\"role\": \"user\", \"content\": json.dumps({\n            \"evidence\": evidence, \"criterion\": question,\n            \"options\": [{\"letter\": L, \"description\": o}\n                        for L, o in zip(letters, options)]})},\n    ], tokenize=False, add_generation_prompt=True)\n\n    # each letter must be ONE token that round-trips, or you cannot slice its row\n    slots = [tok.encode(L, add_special_tokens=False)[0] for L in letters]\n\n    ids = torch.tensor([tok.encode(prompt, add_special_tokens=False)])\n    with torch.inference_mode():\n        logits = model(ids).logits[:, -1, :]              # (1, 151936)  <- full vocab\n    return F.softmax(logits[0, slots], dim=-1)            # (N,)  renormalised over N rows\n\n\n# ─────────────────────────────────────────────────────────────────────────────\n# FAMILY 2 — per-option scalar scorer.  (cua-s1-forms, open-jev-deberta, and —\n# per the published disclosures — Jev itself)\n#\n# No letters. No vocabulary. The head's output dimension is ONE, and it runs\n# once per option. N is a loop bound.\n# ─────────────────────────────────────────────────────────────────────────────\nclass DecisionHead(torch.nn.Module):            # open-jev-deberta's real shapes:\n    def __init__(self, d=1024):                 #   0.weight [1024, 3072]\n        super().__init__()                      #   2.weight [1, 1024]\n        self.net = torch.nn.Sequential(         #   = 3,147,777 params\n            torch.nn.Linear(3 * d, d), torch.nn.GELU(), torch.nn.Linear(d, 1))\n\n    def forward(self, q, o):                    # q, o: (B, 1024) pooled spans\n        return self.net(torch.cat([q, o, q * o], dim=-1)).squeeze(-1)   # (B,)\n\ndef read_by_scoring(encoder, head, tok, evidence, question, options):\n    ctx = pool(encoder, tok, f\"{evidence}\\n{question}\")           # (1024,)\n    scores = torch.stack([                                        # one scalar EACH\n        head(ctx[None], pool(encoder, tok, o)[None])[0] for o in options\n    ])                                                            # (N,)\n    return F.softmax(scores, dim=-1)\n```\n\n## What actually differs\n\nThe first slices `N` rows out of a 151,936-wide vector that was computed anyway.\nThe second computes `N` separate numbers, one per option, from a head whose output\nlayer is `Linear(1024, 1)`.\n\nConsequences, in order of how much they will bite you:\n\n- **Order.** Family 1 is sensitive to it, because `A` and `B` are different tokens\n  carrying different learned priors. Measured on openjev's own committed\n  predictions: reversing the option list changes the answer on\n  **10 of 36 cases — 27.8%** — and confidently, not marginally. Family 2 has no\n  letters and no positions, so the question does not arise.\n- **Cardinality.** Family 1 caps at the number of single-token letters you can\n  find: `LETTERS = \"ABCDEFGHIJKLMNOP\"`, so 16. Family 2 runs the loop more times;\n  TypeSafe quote 255 for Jev, with a second stage above some threshold.\n- **Cost.** Family 1 is one forward pass regardless of `N`. Family 2 is `N` head\n  evaluations over a shared encoding — cheap per option, but linear in `N`.\n\n## Checking the first one is real\n\n`softmax` over the sliced logits *is* the returned distribution, and openjev\ncommits enough to verify it:\n\n```python\n# openjev commits option_logits AND probabilities for every row\nworst = max(\n    abs(a - b)\n    for row in predictions\n    for a, b in zip(softmax(row[\"option_logits\"]), row[\"probabilities\"])\n)\n# 144 rows, worst |diff| = 9.7e-08 — i.e. the 6-dp rounding in the JSON\n```\n\nWritten up at length in [RLCD is not constrained\ndecoding](/articles/rlcd-calibrated-decisions) and [A System One model in 706,048\nparameters](/articles/cua-s1-forms).\n","readingTimeMins":3,"url":"https://ai.thesatyajit.com/snippets/two-ways-to-read-a-decision","lastUpdated":"2026-09-18"},{"title":"PyTorch channels-last + AMP training loop","description":"Memory-format + mixed-precision combo that speeds up conv nets on tensor cores.","date":"2026-05-25","lang":"python","tags":["pytorch","performance","training"],"kind":"snippets","slug":"pytorch-channels-last-amp","body":"For conv-heavy models on NVIDIA tensor cores, two cheap changes compound: move the\nmodel and inputs to `channels_last` memory format, and wrap the forward pass in\n`autocast` with a `GradScaler`. The memory format keeps activations in a layout the\ntensor cores prefer; AMP runs the math in bf16/fp16 while keeping a fp32 master copy.\n\n```python\nmodel = model.to(device, memory_format=torch.channels_last)\nscaler = torch.amp.GradScaler(\"cuda\")\n\nfor x, y in loader:\n    x = x.to(device, memory_format=torch.channels_last, non_blocking=True)\n    y = y.to(device, non_blocking=True)\n\n    optimizer.zero_grad(set_to_none=True)\n    with torch.autocast(\"cuda\", dtype=torch.float16):\n        loss = criterion(model(x), y)\n\n    scaler.scale(loss).backward()\n    scaler.step(optimizer)\n    scaler.update()\n```\n","readingTimeMins":1,"url":"https://ai.thesatyajit.com/snippets/pytorch-channels-last-amp","lastUpdated":"2026-05-25"},{"title":"CUDA grid-stride loop for vector add","description":"The grid-stride pattern that lets one kernel handle any input size.","date":"2026-05-22","lang":"cuda","tags":["cuda","kernels","performance"],"kind":"snippets","slug":"cuda-vector-add-grid-stride","body":"A grid-stride loop decouples the launch configuration from the problem size: each\nthread processes multiple elements, striding by the total number of threads. This\nkeeps the kernel correct for any `n`, plays nicely with occupancy tuning, and lets\nyou reuse the same launch geometry across input sizes.\n\n```cuda\n__global__ void vadd(const float *a, const float *b, float *c, int n) {\n  int stride = blockDim.x * gridDim.x;\n  for (int i = blockIdx.x * blockDim.x + threadIdx.x; i < n; i += stride) {\n    c[i] = a[i] + b[i];\n  }\n}\n\n// Launch: size the grid to the device, not to n.\nint block = 256;\nint grid = (n + block - 1) / block;        // cap this for very large n\nvadd<<<grid, block>>>(d_a, d_b, d_c, n);\n```\n","readingTimeMins":1,"url":"https://ai.thesatyajit.com/snippets/cuda-vector-add-grid-stride","lastUpdated":"2026-05-22"}]