Halo's 2.8× over TRL is not a kernel, and its own benchmark proves it

Halo is White Circle's open-source post-training framework, released 20 August 2026 under Apache-2.0. The claim on the README is specific:

On 8× B300, Halo delivers up to ~2.8× the training throughput of stock TRL (2.7× at 25% less peak memory when both sides shard ZeRO-3), with larger margins over the other frameworks benchmarked.

Two things about that sentence before anything else. It states its hardware. And it volunteers, in parentheses, the matched-configuration number — which is lower — without being asked. That is not how a framework comparison usually opens.

The reflex when reading "2.8× stock TRL" is that TRL's defaults are not tuned, and beating an untuned baseline is a weaker claim than beating a tuned one. That reflex is right often enough to be worth applying every time. So: what was stock TRL configured as, where does the gap actually come from, and does the training loop do what the docs say?

I cloned the repository at 4c1e6c6 — 83,866 lines of Python across src/ — read the EP layer and the benchmark runner, and checked the one claim Halo makes about somebody else's code against that code.

What stock TRL was configured as

This turns out to be the easiest question in the piece, because the answer is a committed file and a table headed "What differs between the two sides".

Held constant on both sides: bf16, FlashAttention-4, Liger, learning_rate=2e-5, seed 42, 10 steps with 3 warmup, gradient_accumulation_steps=1, drop_last, global batch = 8 × per-device, gradient checkpointing on, 8× B300 SXM6. The dataset is deliberately synthetic — create_benchmark_dataset builds 64 filler examples tokenised to exactly seq tokens — which removes packing as a variable, and the doc says so: "It is a shape fixture, not real data."

The baseline runner is in the tree, with a docstring that pre-empts the complaint:

# tests/gpu/profiling/benchmark_trl_baseline.py
# Expert compute is transformers' OWN grouped GEMM (torch.nn.functional.grouped_mm,
# the default experts_implementation for gpt-oss) — the same grouped-GEMM CLASS as
# Halo, NOT an eager loop. So this is a grouped-vs-grouped baseline on the expert
# kernel; Halo's lead comes from EP token distribution + AdamWBF16 + FSDP2, not
# from "grouped GEMM vs loop".

And the doc states outright that the baseline gets the stronger option where the two differ: "The baseline gets the strongest stock options — ZeRO-3 and Liger's FLCE, which is why TRL fits 128k/256k." Halo's default is plain Liger CE, which materialises logits and is worse at long context; on gpt-oss's 201k vocab that costs real memory, and the doc benchmarks it separately rather than quietly enabling it.

So the honest verdict on the baseline is: tuned, current, documented, and runnable. That is a higher bar than most framework comparisons clear. One cell is missing, and it is the one that matters.

The asymmetry: ZeRO-2 against ZeRO-3

TRL runs FSDP2 full_shard, which is ZeRO-3 — it re-gathers all 20.7B parameters every micro-step. Halo's default is ZeRO-2, reshard_after_forward=False, which keeps them resident. That is a memory-for-speed trade, and it is a large one on a short step: the doc measures Halo's own EP1 at ZeRO-3 as −38% against its ZeRO-2 at 4k·b1, falling to −5% at 16k·b1, because a fixed re-gather cost is a bigger share of a shorter step.

Which means the headline 2.8× is comparing two different sharding strategies as well as two frameworks. Halo handles this the right way: it publishes an EP1 ZeRO-3 column expressly to isolate the framework gap — "both sides shard every param 8-way with the same kernel" — and that column reads 1.43×, 1.25×, 1.49× at 4k and 2.68×, 2.51× at 16k, at lower peak memory in every row.

The README's own sentence reproduces exactly from the 16k·b1 row: 17,464 ÷ 6,513 = 2.68, at 37.9 GB against 50.6 GB, which is 25.1% less. "2.7× at 25% less peak memory when both sides shard ZeRO-3." Four for four.

So there are two true statements and the shorter one travelled:

• 2.3–2.8×, Halo's recommended configuration against TRL's recommended configuration. A legitimate "what should I run" comparison, and the one a user actually faces.
• 1.25–2.68×, matched at ZeRO-3, at less memory. The framework-versus-framework number, and a strong one at 16k.

The cell I would still like is the third: TRL at ZeRO-2. The benchmark script already takes --fsdp_sharding shard_grad_op for exactly that, and no ZeRO-2 TRL row is published. It would probably narrow the 4k gap and barely touch the 16k one — the doc's own reasoning about re-gather cost predicts that — but "probably" is doing work there, and one torchrun would replace it.

It is not the kernel, and Halo's own numbers prove it

The natural hypothesis for a 2.8× is fused kernels. Halo is loaded with them — FlashAttention-4, Liger, DeepGEMM, grouped GEMM, a bf16 Adam in Triton — so it would be an easy story to tell. The repository declines to tell it.

Both frameworks default to a grouped-GEMM expert kernel, and both expose the slow per-expert loop as an opt-in, so the comparison is a clean 2×2. The decisive cell is bottom-left: Halo running the loop at EP1 does 5,545 tok/s/GPU, against TRL's 3,885 with the fast kernel. Handicap Halo with the worse kernel and it is still 1.43× ahead. Whatever the gap is, it survives the kernel being removed.

The kernel uplift itself is interesting for a separate reason. TRL gains +100% from grouped GEMM, Halo EP1 +62%, EP2 +79% — and EP8 only +14%, because at four experts per rank each expert sees a larger M and the loop's per-shape tile already fits. The more you shard experts, the less a grouped kernel buys you. That is a useful rule of thumb and it is not in any paper I know of.

Where the gap actually is

Halo names it: "Most of the gap is structural rather than kernel-level." Three things, in descending order.

1. Token routing. This is the one Halo makes a claim about somebody else's code, so it is the one worth verifying. The assertion is that transformers v5's expert-parallel path never moves tokens: every rank holds the whole batch, zeroes the routing scores of experts it does not own, and all-reduces the full [tokens, hidden] MoE output.

That is exactly what transformers says about itself, in the docstring of the class in question:

# transformers/distributed/tensor_parallel.py — EpRouterParallel
"""Expert-parallel router: forward-only slicing of router outputs to local experts.
...
- zeroes scores for non-local experts
- remaps surviving global indices to local indices (`fmod` after masking non-local slots)
- sets dropped slots to sentinel `num_local_experts` (skipped by grouped_gemm experts forward)
 
Downstream `moe_tp_experts` allreduce-sums partial per-rank expert outputs.
"""

and MoeExpertsParallel.transform_output_post_forward ends in _AllReduceForward.apply(output, process_group) over the whole output tensor. Halo's characterisation of the baseline is accurate, checked against the baseline's own source rather than taken on trust.

expand

The forward path in src/distributed/expert_parallel/base_layer.py is the diagram, almost line for line — fp32 router logits, route_tokens_to_experts, then a single _dispatch_compute_combine_shared that hands the flattened tokens to a DeepEP V2 dispatcher, runs the local experts, and combines back.

2. The optimiser. AdamWBF16 keeps weights and both Adam moments in bf16 with stochastic rounding — 6 bytes per parameter against adamw_torch_fused's 12 — and Halo prices it at "a 17% shorter step". The stochastic rounding is not decoration: the kernel's docstring gives the reason, that nearest rounding would truncate a sub-ULP lr*step update to zero and inflate the non-negative second moment. Its RNG is seeded identically on every rank so replicated parameters round the same way, which is the sort of detail that only gets written down after it has gone wrong once.

3. The sharding default, discussed above, which is a choice rather than an advantage.

The check that makes the rest credible

A throughput benchmark that does not also show convergence is measuring how fast you can compute the wrong thing.

expand

expand

The small-hardware end

The headline is a datacentre number, and the framing that Halo is about small hardware is not quite right — 8× B300 SXM6 is 2.3 TB of HBM. But the repository ships an explicit small end, and it is the more interesting part for anyone who is not White Circle.

The runnable config is 42 lines and reads like a normal TRL YAML, which is the whole design argument:

# examples/sft/lfm2/lfm2.5-8b-a1b-ultrachat-ep2.yaml
model_name_or_path: LiquidAI/LFM2.5-8B-A1B
expert_parallel_size: 2              # 32 routed experts → 16/rank
moe_balancing: bias_update           # LFM2 has no router auxiliary loss
save_sharded_ep: false               # gather a standard HF checkpoint on save
use_grouped_gemm: true
fp32_router: true
attn_implementation: flash_attention_2
packing: true
max_length: 8192
per_device_train_batch_size: 1
gradient_accumulation_steps: 8
optim: adamw_torch_fused
learning_rate: 5.0e-06

Three lines in there are the entire product. expert_parallel_size: 2 wraps the MoE blocks in place rather than swapping in a separate distributed model. save_sharded_ep: false gathers a standard SafeTensors checkpoint on save, so from_pretrained still works afterwards. And moe_balancing: bias_update handles a family-specific fact — LFM2 has no router auxiliary loss, so balancing has to go through the expert-selection bias instead — which is the kind of thing that decides whether a framework actually supports a model or merely lists it.

The cookbook is also honest about what does not work: CP is unsupported for LFM2, because the short-convolution layers operate across the sequence axis and cannot take a Ulysses split. A support matrix with a "No" in it is worth more than one without.

What I would actually ship

• Read the performance doc before the README. human-docs/performance.md has a table headed "Levers that do not help here" — fp8/fp4 compute buys nothing because fine-grained experts are weight-bandwidth-bound, native DeepGEMM runs at 0.05–0.17× of bf16 at production shapes, torch_compile adds 2% over what Liger already fuses. A framework that tells you which of its own features to leave off is unusual and the list is worth more than the benchmark.
• Pick the lowest EP that fits, not the highest. "ep1 runs about 2× ep8 on the same model." Expert parallelism trades local parameters for all-to-all traffic. Sharding buys capacity and you pay in tokens per second — which is the opposite of how EP is usually pitched.
• Get M = per_device_batch_size × sequence_length above ~8k before judging anything. Below that the step is latency-bound and you are measuring your launch overhead. Batch 1 → 4 is worth 1.5–2.1× on MoE by itself.
• The 2.8× is a real number about a real choice; the 1.25–2.68× is the framework. Quote whichever answers your question, and say which one you are quoting.
• If you are publishing a framework benchmark, copy this structure. Baseline runner committed, differences tabulated in advance, convergence shown, the matched-configuration number volunteered in the README's own headline sentence. The one improvement is the cell that is missing, and it is one torchrun away.

No B300s were involved. Every throughput and memory figure is Reported: read out of whitecircle/halo at commit 4c1e6c6, cloned rather than summarised, Apache-2.0 — principally agent-docs/optimization/halo-vs-stock-trl.md, human-docs/performance.md and the six run receipts under agent-docs/assets/benchmarks/runs/. The three figures are the repository's own. What I read rather than reported: src/distributed/expert_parallel/base_layer.py and dispatcher.py for the forward path, src/optimizers/adamw_bf16.py for the optimiser, tests/gpu/profiling/benchmark_trl_baseline.py for the baseline's configuration, and examples/sft/lfm2/lfm2.5-8b-a1b-ultrachat-ep2.yaml with human-docs/cookbooks/halo-lfm2-moe-cookbook.md for the MoE recipes. The claim about the baseline's routing was checked against huggingface/transformers on main, in src/transformers/distributed/tensor_parallel.py. For what the expert-parallel mechanism is from first principles: Mixture of Experts, from scratch.