ORPO: the odds ratio is the good part

The version of ORPO that travels is four sentences long. Why run SFT and RLHF as two stages? ORPO merges them. It penalises the rejected response directly during SFT, with no reference model. Massive compute and VRAM savings from a single objective.

The middle two sentences are exactly right. The last one is the only claim in the set with a magnitude in it, and it is the only one nobody has attached a number to. This piece attaches one — and derives the objective properly on the way, because the interesting part of ORPO is not that it removes a stage (plenty of methods do) but which ratio it contrasts with, and why. That is an argument about gradients, the paper makes it carefully, and almost nobody repeats it.

Short version of where this lands. The odds ratio is a genuinely good idea and the derivation rewards reading. The reference-model saving is real, and it is about 11% of model-state memory and 25% of per-step arithmetic — and it is absent entirely in the one setup where it would have mattered most. And ORPO's best result is in a table the paper never printed.

The premise: SFT teaches the rejected response too

Everything follows from one observation in §3. Cross-entropy on the chosen response raises the probability of the chosen tokens and says nothing at all about the rejected ones — the label indicator $y_i$ is zero for every non-answer token, so there is no term in the loss that pushes anything down. What actually happens, empirically, is that fine-tuning on chosen-only data drags the rejected responses up alongside them, because the two share a style, a domain and most of their vocabulary.

expand

Read the axis before you accept the claim. The whole chart spans 0.35 nats, the two curves separate by roughly 0.02 nats at convergence, and the ribbons overlap for most of the run. The paper's own summary — "the absence of a penalty for unwanted generations results in rejected responses sometimes having even higher log probabilities than the chosen ones" — is stronger than this figure supports for this model; after step 2000 the chosen curve is above the rejected one throughout. What the figure does show cleanly is the thing that matters: the rejected curve goes up, by about the same amount as the chosen one, and nothing in the SFT objective was ever going to stop it.

So the fix is not exotic. Add a term that pushes the rejected response down. The interesting question is which term.

The objective, in five lines

Start with the quantity the whole method is built on. ORPO's $P_\theta(y \mid x)$ is not the joint probability of the sequence. Equation 3 defines it through an average:

which makes $P_\theta(y\mid x)$ the geometric mean of the per-token probabilities:

This is load-bearing and it is the detail most summaries drop. Hold onto it; in two sections it will be the only thing keeping the odds ratio from collapsing into the probability ratio.

Now the odds of generating $y$, and the ratio of two of them:

Wrap the log of that in a log-sigmoid so that minimising it means increasing the log odds ratio, and you have the penalty term:

Add it to the ordinary negative log-likelihood with a weight, and that is the entire method:

expand

Two things are worth noticing about that objective before moving on. First, the SFT term never leaves — ORPO is not a replacement for supervised fine-tuning, it is supervised fine-tuning with a second term bolted on, which is why the paper calls it monolithic rather than reference-free alignment. Second, there is no $\pi_{\text{ref}}$ anywhere. DPO's objective is built from the ratio $\pi_\theta/\pi_{\text{ref}}$, so it structurally requires a second set of weights; ORPO's is built only from $\pi_\theta$ evaluated on two sequences.

Why an odds ratio and not a probability ratio

This is the part of the paper that earns it. §7.1 asks the obvious question — DPO and IPO contrast a probability ratio, so why not

— and answers it with a distributional argument. Since the log-sigmoid is what the ratio gets fed into, what matters is the scale of the thing going in. A loss of the form $-\log\sigma(z)$ is small when $z$ is a few units positive; how hard the optimiser has to push to get there depends entirely on how wide $z$'s natural range is.

expand

The paper makes this point with 50,000 Monte Carlo samples and leaves it qualitative. It has an exact answer, and the exact answer is nicer. For $X \sim \mathrm{Unif}(0,1)$, $-\log X$ is $\mathrm{Exp}(1)$, so $\log X_1 - \log X_2$ is standard Laplace with standard deviation $\sqrt{2}$; and $\mathrm{logit}(X)$ is standard logistic with variance $\pi^2/3$, so the difference of two of them has standard deviation $\sqrt{2\pi^2/3}$. That gives:

(My arithmetic and my simulation, reproducing the paper's Figure 6 setup.) The odds ratio is 1.81× wider than the probability ratio at $\beta = 1.0$ and 9.07× wider at $\beta = 0.2$ — and 0.1 to 0.2 is the range these methods actually run at; the ORPO paper set DPO's $\beta$ to 0.1 for every one of its own comparisons.

Why that matters, concretely. Hold both formulations to the same length-normalised $P$ — which is what the paper's own ablation does — fix the chosen response at a geometric-mean per-token probability of 0.6, and ask how far the rejected response has to fall before each loss is satisfied:

(My arithmetic.) To get the loss down to 0.01, the odds-ratio version needs the rejected response's average token probability to reach about 1.5%. The probability-ratio version at $\beta = 0.2$ needs it to reach 6.2e-11. That is not a penalty, it is an erasure — and erasing the rejected response is catastrophic when you are simultaneously doing domain adaptation from a base checkpoint, because the rejected response is still mostly fluent English in the target domain. The paper's Appendix B shows exactly this: trained with the probability ratio, the rejected log-probabilities crash below −4 early; with the odds ratio the same thing only happens after overfitting sets in.

The gradient, and where the penalty actually bites

The paper gives the gradient as a product of two factors (Eq. 8–10), which is the clearest way to see what the term does:

$\delta(d) = \sigma(-\log \mathbf{OR})$ is a gate: near 1 when the model prefers the rejected response, near 0 once it prefers the chosen one by a comfortable margin. $h(d)$ is a contrast of the two per-sequence gradients, each divided by its own $1 - P$. The division is not decoration — it falls out of the algebra, because $1 + \mathbf{odds}_\theta(y\mid x) = 1/(1 - P_\theta(y\mid x))$ exactly, which is how Appendix A's Eq. 32 becomes Eq. 33.

Here is what those two factors do when you multiply them out. The quantity to watch is the coefficient the loss puts on $\nabla_\theta \log P_\theta(y_l\mid x)$ — how hard it pulls the rejected response down, as a function of how likely the model currently thinks the rejected response is.

The shape is the argument. When the model already assigns the rejected response a low probability, the gate is nearly shut and the whole term contributes about 0.036 — the SFT loss runs the batch unopposed. When the model prefers the rejected response, the coefficient reaches 8.57, a factor of 240 higher. "Weak penalty, strong adaptation" turns out not to be a claim about the size of the penalty but about which examples get one. On a pair the model already has right, ORPO is just SFT.

That also explains the $\lambda$ ablation in Appendix E, which is the paper's most useful and least quoted result. At $\lambda = 0.1$ the rejected log-probabilities do not fall at all — the chosen ones simply rise to satisfy the term. At $\lambda = 1.0$ both fall while the margin widens, and MT-Bench gets worse at extraction, math and reasoning while getting better at STEM, humanities and roleplay. The authors say so plainly: a bigger margin means "overly adapting to the chosen responses set in the training dataset". The knob that makes the LLM-judged benchmark go up makes the hard-answer categories go down.

The reference model, priced

Here is the claim worth checking, because it is the one the tweet version rests on. §7.3 argues two savings: memory, because there is no frozen $\pi_{SFT}$ resident; and compute, because "in theory, two forward passes should be calculated for each model… four forward passes happen in total for a single batch" while ORPO needs "half the number of forward passes".

The paper gives no GiB and no tokens per second. So:

Thirteen and a half gibibytes, against 107.91 of trainable state. 11.1% of model-state memory. The reference model is cheap for the same reason it is convenient: it is frozen, so it carries bf16 weights and nothing else — no gradients, no fp32 master copy, no Adam moments, and no stored activations, because its forward runs under no_grad. The expensive copy of the model is the one you are training, and ORPO still has that.

The compute claim needs the same treatment:

Counting forwards counts the cheap third of the work. Both methods run the policy forward and backward over the chosen and the rejected sequence; DPO adds two backward-free forwards. 12P against 16P is a 25% saving. The closest thing to an independent measurement comes from SimPO, which is reference-free in exactly the same structural way and reports "roughly 20%" less run time and "about 10%" less GPU memory against a vanilla DPO implementation on 8×H100 — which lines up with the arithmetic above almost exactly.

And then there is the part that the word vanilla is carrying. From TRL's DPOTrainer source, the reference model is not instantiated at all in two common cases:

# Reference model
if ref_model is None:
    if is_peft_model(self.model) or args.precompute_ref_log_probs:
        # If PEFT is used, the reference model is not needed since the adapter
        # can be disabled to revert to the initial model. If precompute_ref_log_probs
        # is True, the reference model does not need to be kept in memory during training.
        self.ref_model = None

Under LoRA, DPO recovers the reference by switching the adapter off — the base weights are $\pi_{\text{ref}}$. With precompute_ref_log_probs=True, the reference logprobs are computed once in a pass over the dataset and cached as scalars, and per-step cost drops to ORPO's. SimPO's own footnote concedes the point: "DPO can be as memory efficient as SimPO if it were implemented to separate the forward passes of the reference model from the actual preference optimization. However, this implementation is not standard practice."

This is the honest shape of the saving. Full fine-tuning at 7B in bf16: ORPO saves 11% of memory and 25% of step FLOPs, and that is worth having. Under LoRA — where the second copy would have nearly doubled resident memory, since the trainable state is about 0.6 GiB against 13.49 for the frozen base — DPO already does not pay it. The saving is smallest where memory is abundant and absent where memory is tight.

The stage it genuinely removes

There is a real win underneath the overstated one, and it is operational rather than arithmetic. SFT→DPO is two training runs, two sets of hyperparameters, an intermediate checkpoint to store and choose, and a second chance to get the learning rate wrong. ORPO is one run. If you have ever explained to someone why the DPO job reads its reference weights from a directory produced by a job that finished last Tuesday, that is the saving.

Except: count the epochs. Appendix C says ORPO was "trained for 10 epochs" with best-checkpoint selection by eval loss for the OPT series, Phi-2 and Llama-2, against a baseline of one epoch of SFT plus three epochs of DPO. In the units above that is 10 × 12P = 120P against 6P + 3 × 16P = 54P — the Llama-2 headline result used roughly 2.2× the arithmetic of the pipeline it is compared against. (My arithmetic, from the paper's stated epoch counts.) Mistral is conspicuously absent from that sentence and Figure 1's caption says "a single epoch", which would make it 12P against 54P and a 4.5× saving in the other direction. Both are in the same paper. "One stage" and "less compute" are separate claims and only one of them is structural.

The results, and their denominators

The roster, so the numbers have somewhere to sit. Controlled experiments: OPT at 125M, 350M and 1.3B, four methods each, on Anthropic HH-RLHF and on binarised UltraFeedback, judged by a 1.3B reward model. Headline experiments: Phi-2 (2.7B) at $\lambda = 0.25$, Llama-2 (7B) at $\lambda = 0.2$, and Mistral (7B) at $\lambda = 0.1$, all trained on UltraFeedback alone and scored on AlpacaEval 1.0 and 2.0, with MT-Bench and IFEval for the two Mistral checkpoints. Phi-2 + ORPO reaches 71.80% and 6.35%, against Phi-2 + SFT + DPO at 50.63% and 0.78% — the only matched SFT→DPO comparison above 1.3B anywhere in the paper, and a convincing one.

expand

The 7B numbers are the ones that travelled, so start with what they are measured against. Every non-ORPO bar in that chart is an entry someone else put on a leaderboard. The controlled comparison at 7B did not survive: "Llama-2 + SFT and Llama-2 + SFT + DPO yielded models with outputs that could not be evaluated." Publishing that sentence is to the authors' credit and it also means the 7B column has no matched DPO baseline in it at all.

Pulled from the leaderboard's own committed data rather than from the paper, so both rows come from one evaluator run:

Mistral-ORPO-β beats Zephyr-β by 1.57 raw points, which is 17 extra head-to-head wins out of 805 — 95 against 78. The standard errors are ±0.99 and ±0.96, so the gap is about 1.1 standard errors of itself: real, directionally consistent, and not separable from noise on this benchmark alone.

Two things the paper could not have known and one it could. Length-controlled win rate arrived after publication, and ORPO survives it: 14.72 against 13.20, a 1.51-point gap against the raw 1.57, so the advantage is not verbosity even though ORPO's responses average 1636 characters to Zephyr's 1444. Good. On the other hand Mistral-7B-Instruct-v0.2 — the instruct model Mistral shipped, with no published preference recipe and vastly more data — sits above both at 17.11 LC. That does not make the algorithm comparison wrong, but it is the number anyone deciding what to actually deploy should see.

And the one it could have known: on MT-Bench, the ORPO model's own card prints Zephyr-β at 7.34 against Mistral-ORPO-β's 7.32. The abstract headlines 7.32 as an achievement; on the paper's main DPO comparator the method loses by 0.02, which is noise, and the honest word is "matches". IFEval is the clean win — 66.19% instruction-level loose against Zephyr-β's 57.67% — although the authors' own repository table shows Mixtral-8×7B-Instruct at 68.23% on that column.

The controlled experiment, read carefully

The comparison that was matched is the OPT series, 125M to 1.3B, all four methods on the same data, judged by a 1.3B reward model. Win rate of ORPO over each baseline:

Against SFT-only and against PPO, ORPO wins decisively everywhere. Against DPO, below 1B, it is 41.7, 49.4, 48.8 and 50.5 — one loss and three coin flips — and it wins at 1.3B. The paper's framing, "the win rate over DPO was correlated to the model's size", is a three-point trend in each dataset and is offered as the reason to believe the 7B results. It is a reasonable hope. It is not a measurement.

Two caveats the authors supply themselves, which is worth saying because a less careful paper would not have: the judge RM-1.3B was trained on the same datasets the models were tuned on, and PPO was optimised against RM-350M and then scored by RM-1.3B — "the instability and reward mismatch problem of RLHF", in their words. The PPO column is measuring a reward-model mismatch as much as an algorithm.

That pattern repeats often enough to be worth naming. §6.4 measures lexical diversity against DPO and prints a column ORPO loses: per-input cosine similarity, where lower means more varied, comes out 0.8909 for Phi-2 + ORPO against 0.8012 for Phi-2 + SFT + DPO, and 0.9008 against 0.8889 on Llama-2. ORPO produces less varied output for a given prompt, in both families. The authors read it as ORPO "assigns high probabilities to the desired tokens" — which is a fair reading and also exactly what you would expect from a loss whose SFT term never turns off — and then publish the number either way. Together with the Llama-2 baseline that failed to produce evaluable output and Appendix E's finding that the $\lambda$ knob trades math for chattiness, that is three results reported against the paper's own interest, which is more than most.

Where ORPO sits among the reference-free crowd

ORPO was not the first method to drop $\pi_{\text{ref}}$, and it landed in the middle of a cluster. Stripped to their essentials:

• CPO — SFT NLL plus $-\log\sigma(\beta\log\pi_\theta(y_w) - \beta\log\pi_\theta(y_l))$. The probability-ratio version of the same idea, un-normalised for length. This is precisely the design §7.1 argues against, which makes it the most interesting neighbour.
• SimPO — $-\log\sigma\!\big(\tfrac{\beta}{|y_w|}\log\pi_\theta(y_w) - \tfrac{\beta}{|y_l|}\log\pi_\theta(y_l) - \gamma\big)$. Length-normalised like ORPO, but with a target margin $\gamma$ and no SFT term.
• KTO — still reference-dependent, but learns from unpaired thumbs-up/thumbs-down data, which is a different data-collection problem rather than a different optimiser.
• DPO — reference-dependent, the incumbent, and the thing everyone is trying to beat.

SimPO's Table 4 is the only matched comparison that includes ORPO — every objective trained on the same data from the same checkpoints, each with its hyperparameters tuned:

ORPO loses to DPO on length-controlled win rate in three of four settings and essentially ties in the fourth (14.7 against 15.1). It loses to SimPO everywhere, which is SimPO's paper, so read it with that in mind.

The head-to-head §7.1 predicted — odds ratio against probability ratio, since CPO is otherwise the same shape — comes out only half in ORPO's favour. On length-controlled win rate ORPO leads CPO in three settings of four and trails by 0.4 in the last. On Arena-Hard the same pairs split two apiece: ORPO 7.0 against 6.9 and 10.8 against 5.8, CPO 22.6 against 20.8 and 28.8 against 25.8. A theoretical argument about gradient scale predicting a 3–1 and a 2–2 is weak support, not none.

The footnote is the sharp bit. SimPO ran ORPO from the SFT checkpoint, like every other baseline, and explains why: doing so "yields better results than starting from base checkpoints." An independent group tried ORPO's headline claim — skip the SFT stage — found it worse, and put the SFT stage back. That is one reproduction, on their data, with their tuning, and it should not be the last word. It is also the single most load-bearing counter-observation in this piece.

The result the paper never reported

Which makes the next table doubly strange. It holds ORPO's best result by a distance, and it is in someone else's appendix — in the paper arguing against ORPO, no less.

In that setting ORPO takes GSM8K from 28.13 to 42.15, the highest in the table, while DPO takes it down to 21.76. It also posts the highest MMLU of any row, 63.20, above the SFT model it began from, in a column where DPO loses 1.6 points.

One setting is not a finding, so here is the same subtraction across all four of SimPO's settings — each objective's GSM8K minus the SFT checkpoint it was trained from:

ORPO is first, by 1.2 points over the next row and by more than ten over DPO. And the grouping is the interesting part: every objective with a positive mean carries an SFT term, and every objective without one is negative. (The converse fails — RRHF has an SFT term and sits fifth, sunk by a single catastrophic −23.35 on Mistral-Base and otherwise mid-pack.) SimPO's own explanation is the obvious one: "except for ORPO, almost all approaches lead to consistent drops in one or more settings. We hypothesize that ORPO retains performance largely due to its supervised fine-tuning loss for regulation."

Their sentence is slightly too kind — ORPO does drop in both Instruct settings, by 3.64 and 4.10 — but the mechanism they name is right, and the grouping in the table is the evidence for it. Which makes the strongest case for ORPO nearly the opposite of the case the paper makes. The selling point is not that it deletes a stage; it is that keeping the SFT term inside the preference objective, live on every step, stops preference training from eating the model's arithmetic. Alignment tax paid in GSM8K points is a well-known cost of DPO. ORPO mostly does not pay it — and the paper reports AlpacaEval, MT-Bench and IFEval, all three LLM-judged or instruction-format, and not one multiple-choice or exact-match task that would have shown this.

In the library

ORPO is in TRL, and where it is in TRL is itself the adoption story:

from trl.experimental.orpo import ORPOConfig, ORPOTrainer

Not trl.trainer. ORPOTrainer sits in the experimental namespace, whose stability contract reads: "Anything under trl.experimental may change or be removed in any release (including patch versions) without prior deprecation. Do not rely on these APIs for production workloads." CPO and KTO are in there with it. That is not a verdict on the method, but it is a data point about what a maintainer with usage telemetry thinks is load-bearing.

Five differences between the paper and the code, all checkable in the source:

The weight has three names and a tenfold gap between its defaults. The paper calls it $\lambda$ and uses 0.1, 0.2 and 0.25 for its three headline models. The reference implementation calls it alpha and defaults it to 1.0. TRL calls it beta and defaults it to 0.1 — and says so, in the config docstring: "Parameter controlling the relative ratio loss weight in the ORPO loss. In the paper, it is denoted by λ. In the code, it is denoted by alpha." Given Appendix E's finding that this knob trades open-ended quality against math and extraction, a 10× default difference between the two implementations is not cosmetic — and the reference repo's default is the one the paper never used.

TRL's SFT term includes the prompt. From concatenated_forward:

# orpo chosen nll loss is computed over the full prompt and response
chosen_nll_loss = cross_entropy_loss(all_logits[:len_chosen], labels[:len_chosen])

The labels there are the full concatenated input with only padding masked, so the NLL is charged on the prompt tokens as well as the completion. The odds-ratio term is not — it uses concatenated_labels, which masks the prompt with −100. The reference repo has a disable_prompt_loss flag for exactly this, and it is action='store_true', so both default to including the prompt. TRL simply has no way to turn it off.

One forward pass instead of two. The reference implementation runs model(...) separately on the positive and negative sequences; TRL concatenates them into one batch, "because it's faster for FSDP." Same arithmetic, better utilisation.

The numerics needed fixing. The loss needs $\log(1 - P)$, and $P$ here is a geometric mean that climbs toward 1 as the model memorises the chosen response. The reference repo's final commit is literally Merge pull request #29 from xfactlab/add-log1p; TRL goes further with a branched implementation:

def log1mexp(x: torch.FloatTensor) -> torch.FloatTensor:
    """Numerically stable computation of log(1-exp(x))."""
    # branch at -ln 2 ~ -0.693 to avoid cancellation
    t = -0.6931471805599453
    return torch.where(x < t, torch.log1p(-torch.exp(x)), torch.log(-torch.expm1(x)))

Worth knowing what this is defending against: at large $\lambda$, or many epochs, $P_\theta(y_w\mid x) \to 1$ and $\log(1-P) \to -\infty$. The stable form gets you much closer to the wall before you hit it; it does not move the wall.

The "reward" in the training curves is not an odds-ratio reward, and two of the metrics are named backwards. TRL logs chosen_rewards = self.beta * policy_chosen_logps and the rejected counterpart, so rewards/margins — the line the ORPO docs tell you should trend upward — is $\beta$ times the probability-ratio margin of two length-normalised log-likelihoods, not anything to do with odds. Fine as a monitoring signal; just not the quantity being optimised. Worse, the two metrics that are:

losses, chosen_rewards, rejected_rewards, log_odds_ratio, log_odds_chosen = self.odds_ratio_loss(...)
#   ...which returns:   losses, chosen_rewards, rejected_rewards, mean(ratio), mean(log_odds)
#   where ratio = F.logsigmoid(log_odds)

The metric called log_odds_ratio is $\log\sigma(\log \mathbf{OR})$ — that is, $-\mathcal{L}_{OR}$ — and the metric called log_odds_chosen is the actual log odds ratio. If you are watching a run and want to know whether the penalty term is doing anything, the curve you want is the one named after the wrong thing.

So: should you use it

If you are full-fine-tuning a base model on a paired preference set and you want one job instead of two, ORPO is a reasonable choice and the objective is sound. If you are LoRA-tuning, the headline saving does not exist — TRL's DPO already keeps no second copy — and DPO or SimPO score better in the only matched comparison available. If what you care about is not regressing on math and knowledge while aligning, ORPO leads that table — though SLiC-HF and CPO sit 1.2 and 1.9 points behind it and share the mechanism, so the honest recommendation is "an objective with an SFT term in it", and ORPO is the best-performing member of that family.

The thing worth taking from the paper regardless of what you train with is §7.1. "Use the odds ratio because the probability ratio over-suppresses in a length-normalised, joint-SFT setting" is a specific, falsifiable, correct argument about gradient scale, and it generalises past this method. That part is excellent.