2026-07-24 · 10 min · optimization · prompt-optimization · evolutionary · agents · explainer
Most ways to make an LLM system better are expensive. Reinforcement learning like GRPO needs thousands of rollouts to squeeze a scalar reward into the weights; hand-tuning a prompt needs a human in the loop. GEPA (Genetic-Pareto) makes a different bet: language is a richer learning signal than a number. Instead of a policy gradient from a sparse reward, GEPA reads the system's own execution traces, reflects on them in natural language to work out what went wrong, and writes a better prompt — keeping a Pareto frontier of candidates so specialist wins are never averaged away. The result, per its paper, is a prompt optimizer that turns a few rollouts into large gains.
That core has now been generalized twice. optimize_anything drops the "prompt" assumption entirely — it
optimizes any text artifact you can score and describe — and the new omni release composes GEPA
with agent-based optimizers into a single meta-optimizer. This is a walk through the mechanism, then what
each layer adds, honestly labelled.
The core: reflect, don't reward
GEPA treats a prompt (or a whole multi-prompt system) as the thing to evolve. One turn of its loop is small and legible — sample a candidate, run it, read what happened, and let an LLM rewrite it:
5. LLM reflects — One reflective LLM call reads the trace and the feedback and writes a better prompt — diagnosing the failure in words, not in a scalar reward or a policy gradient.
The move that makes this work is stage 4. A GRPO update collapses an entire trajectory into one scalar advantage and nudges billions of weights; almost all of the information in why the run failed is thrown away. GEPA does the opposite: it hands a reflective LLM call the full trace — the reasoning, the tool calls, the tool outputs — together with the evaluator's textual feedback, and asks it to diagnose the failure and propose a fix in words. "You mis-parsed the date format on the third call" is a far denser lesson than "reward = 0.2", and it applies after a single rollout instead of thousands.
Because the feedback is language, the same loop works on any system with one or more LLM prompts, and the gains show up fast. Across six tasks the paper reports GEPA outperforming GRPO by 6% on average and by up to 20%, while using up to 35x fewer rollouts — and beating MIPROv2, the previous leading prompt optimizer, by over 10% (for example, +12% accuracy on AIME-2025). The claim isn't that reflection is magic; it's that natural language is a higher-bandwidth channel than a reward scalar when your optimizer is itself a language model.
Why a Pareto frontier, not the best-so-far
The "Genetic" half of the name is evolution — mutate, evaluate, keep the good ones. The subtle part is which ones you keep. A greedy search keeps the single best-scoring candidate and mutates that. GEPA keeps the whole Pareto frontier: every candidate that is the best found so far on at least one task instance. Toggle between the two policies and watch what greedy throws away:
GEPA keeps every non-dominated candidate — the four on the frontier, including both specialists α and δ — and samples parents from that set. A prompt that wins even one instance stays in the gene pool.
Averaging is lossy. A prompt that nails a hard sub-case but is middling overall looks worthless to a greedy optimizer and gets discarded — along with the one trick it had figured out. By keeping the frontier and sampling parents from it, GEPA preserves those specialists so their lessons can be recombined later, and it keeps the search from collapsing into a single lineage that stalls in a local optimum. It's the same instinct as maintaining a diverse population in a genetic algorithm, made concrete on per-instance scores.
Optimize anything: score + describe
Once you notice that nothing in the loop actually requires the artifact to be a prompt, the interface
generalizes. optimize_anything asks for exactly two things: a seed text artifact, and an
evaluator that returns a score and some feedback. Everything else — objective, background context — is
plain English the reflective LLM reads.
from gepa.optimize_anything import optimize_anything, OptimizeAnythingConfig
def evaluate(candidate: str) -> tuple[float, dict]:
score, feedback = run_judge(candidate) # your metric + any text you want the LLM to see
return score, {"Feedback": feedback}
seed = open("seed_solution.py").read()
task = dict(
evaluator=evaluate,
objective="Maximize the score for this competitive-programming problem.",
background="A sandboxed judge runs hidden tests and returns a 0-100 score.",
)
result = optimize_anything(seed, **task, config=OptimizeAnythingConfig(engine="gepa"))The artifact can be a prompt, a code file, a system message, a config, a plan — anything expressible as
text and gradeable by a function. The feedback_dict is the whole trick: whatever diagnostic text you put
in it (a failing test's stderr, a judge's rationale, a lint report) becomes the material the reflective LLM
reasons over. Score tells the search whether a candidate is better; feedback tells it why, which is what
makes the next mutation informed instead of random.
Omni: no single optimizer wins, so compose them
The July 2026 post starts from an inconvenient observation: on hard, open-ended problems, no single
optimizer dominates. GEPA's reflective mutation, an autonomous coding agent, and an agent-based proposer
each win on different problems — and each one eventually plateaus. The engine= argument turns this
into a lever: the same optimize_anything task can be dispatched to any of three families.
| engine | family | how it proposes |
|---|---|---|
gepa | LLM-based optimizer | one reflective LLM call mutates a parent drawn from the Pareto frontier; the framework owns the loop |
meta_harness | agent-based | a coding-agent proposer mutates the candidate; the framework still owns the loop |
autoresearch | autonomous agent | a long-horizon agent session owns the entire loop — selection, proposal, and orchestration |
Across ten problems the winner is unpredictable — the paper's per-problem tally is GEPA 3, AutoResearch 3, Meta-Harness 4 — so betting on one engine is betting wrong 60–70% of the time. Worse, when an engine plateaus, seeding a different engine from the stuck candidate usually breaks through: on one problem GEPA stalled at 54.4 after about $1.3 of budget and switching to AutoResearch lifted it to 62.7; on another, AutoResearch stalled at 50.0 and both other engines climbed from there to a perfect 100.
omni turns that into a strategy. It splits a fixed budget in two phases: explore, running all three engines in parallel on a small slice (about $5 each) and keeping the best candidate; then continue, seeding a fresh optimizer instance with that winner and spending the rest (about $5) to push past the plateau — all capped at $20 total per problem.
The composition itself is exposed as a small kit of primitives — optimize_best_of (parallel, keep the
top), optimize_sequential (chain engines), optimize_vote (fair cross-engine comparison), and
optimize_adaptive_sequential (auto-switch on plateau detection) — so omni is one policy you can write, not
a hardcoded pipeline.
Results on Frontier-CS
The benchmark is Frontier-CS: ten open-ended competitive-programming problems, a $20 budget each, using Claude Sonnet 4.6 with medium thinking. A single zero-shot LLM call averages 7.72, so there is real headroom. Under a matched $20 budget, omni tops every standalone optimizer:
The per-engine story is that omni lifts every base optimizer, not just the strongest — the biggest jump is GEPA's, which nearly doubles once it stops having to break through on its own:
| optimizer | standalone | as omni | lift |
|---|---|---|---|
| GEPA | 43.8 | 61.8 | +18.0 (+41%) |
| AutoResearch | 55.4 | 63.2 | +7.8 (+14%) |
| Meta-Harness | 50.9 | 59.3 | +8.4 (+16%) |
The mechanism behind those lifts is the plateau-break — a stuck candidate handed to a different optimizer keeps climbing:
The take
GEPA is a clean idea executed in layers. The core is that language is a denser training signal than a
reward when the optimizer is an LLM: reflect on the trace, keep a Pareto frontier so specialists survive,
and a few rollouts go a long way — up to 20% over GRPO at up to 35x fewer rollouts, on the paper's tasks.
optimize_anything strips away the "prompt" assumption and leaves a genuinely general interface: any text
artifact you can score and describe is now optimizable, with the feedback string doing the heavy lifting.
And omni's contribution is an honest one — since no single optimizer wins and each one plateaus, explore
across engines, then continue from the best, which on Frontier-CS beat every standalone optimizer at a
matched budget. The caveats are the usual ones for a fresh result: small benchmark, LLM judge, provider
numbers. But the shape is compelling, and because the whole thing is pip install and a scoring function,
it's unusually easy for others to check on their own artifacts.
Sources: GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning (Agrawal et al., 2025) for the mechanism and the RL/MIPROv2 comparisons, and the GEPA team's optimize_anything goes omni post (Tan, Agrawal, Lee, Zhang, Klein, Sen, Dimakis, Zaharia, 2026) for the interface and Frontier-CS results. GEPA is developed in the open-source repo and integrates with DSPy. Figures are reproduced from the post for commentary; the interactive diagrams are mine. All benchmark numbers are author-reported.