Skip to main content
For a guided workflow, see Agent Optimizer. relai optimize improves the agent against selected agent tests and registered benchmarks.
Optimization can take a while depending on agent complexity, selected agent tests, and --total-rollouts. RELAI uses 1/3 of total rollouts for the optimization loop and reserves 2/3 for final before/after evaluation.

​
Select behavior

Optimization applies to one target at a time. In a multi-target repository, pass --agent-target {target} rather than applying the selected agent tests or benchmarks across every agent.

​
Runtime

Optimization uses the Harbor runtime. Harbor uses registered task packages and CSV benchmark rows for all optimization and final before/after rollouts, including evaluator scores and native ATIF trajectory evidence. The optimizer runs the current agent revision in its worktree and keeps task, evaluator, and benchmark packages outside its editable code snapshot. Omitting selectors includes all registered agent tests and benchmark samples. Optimization runs simulations, so configure required runtime values as described in Environment variables.

​
Optimizer controls

​
Sizing a run

Every selected sample is evaluated at least once before RELAI’s scheduler starts prioritizing the weakest ones, so a large benchmark spends much of a small rollout budget on its first full pass. For quick iteration against a large suite, register a subsample — especially the cases you already know fail — and keep the full benchmark for occasional complete passes. When most samples are genuinely challenging, there is no good subsample: give the run a larger rollout budget and let it fix as much as the budget allows. Prefer a smaller --batch-size when single runs of your agent produce long trajectories — long conversations, deep tool-call or reasoning chains — since each optimizer turn has to reason over every trajectory in the batch. When each simulation is short and fast, a larger batch size is more efficient.

​
Optimizer scope

Before optimization starts, interactive runs ask you to review the optimizer scope: the guardrails that define what kinds of changes RELAI may make to your agent. RELAI shows the current scope and lets you proceed or edit it. The confirmed scope is saved to .relai/.internal/optimizer-scope.json and reused by later non-interactive runs. That path applies when .relai/.internal/layout.json is present; legacy projects without the marker use .relai/optimizer-scope.json. The scope controls:
  • Allowed changes: choose Prompt/config only when RELAI should stay limited to prompts, instructions, configuration values, and model settings. Choose Structural changes allowed when RELAI may also change code, workflow structure, and tools.
  • Model changes: choose whether RELAI may change the AI models currently used by the agent. By default, model changes are not allowed.
  • Allowed models: when model changes are allowed, list the exact replacement model candidates RELAI may use. At least one model is required before RELAI can proceed with model changes enabled.
  • API-key-backed tools: choose whether RELAI may add new tools or integrations that require external credentials or may incur cost. By default, these are not allowed.
  • Cost optimization: choose whether RELAI should additionally prefer lower cost and token usage. The default is Disabled. The Conservative, Balanced, and Aggressive presets progressively increase how strongly RELAI should treat cost and token usage as optimization goals while preserving evaluation score and intended behavior.
  • Custom instructions: optionally provide free-text instructions for this optimizer run. RELAI tries hard to follow them when they are compatible with the confirmed scope, evaluator intent, safety, correctness, and required behavior.
The default scope allows structural changes, keeps the current models fixed, does not allow new API-key-backed or potentially paid tools, disables cost optimization, and has no custom instructions. Edit the scope when a run should be narrower, such as a prompt-only pass, or broader, such as a run where model changes are acceptable within a specific candidate list. Non-interactive optimization requires a saved optimizer scope. If .relai/.internal/optimizer-scope.json does not exist yet, run relai optimize interactively once to confirm it.

​
Evaluation during optimization

You do not need to run relai simulate before optimizing. Each optimizer run evaluates the agent itself: the reserved 2/3 of --total-rollouts pays for a “before” evaluation pass at the start and an “after” pass that verifies the final changes. Earlier simulation results are not reused. Optimizer evaluations run through the simulator, so each rollout scores the selected agent test’s or benchmark’s own evaluators plus every global evaluator whose scope matches the simulation. When an evaluator already scores perfectly in the “before” pass, there is nothing for the optimizer to fix for it, and the agent is left unchanged in that respect. Candidate changes are accepted only when the overall evaluation result improves. RELAI weighs evaluators against each other — a large gain on one with a slight regression on another can pass, while trading equal gains for equal losses cannot — and reviews whether the change is sensible overall rather than applying a purely numerical threshold. A candidate that causes a regression is rejected, and the optimizer records the lesson and tries a different approach. When no candidate survives, the run reports that it could not improve the agent and finishes with no accepted changes and no PR.

​
Where optimization happens

relai optimize never edits your checkout directly. Each run prepares a separate git worktree under ~/.relai/worktrees, copies the current checkout into it, and creates a local branch named relai/optimizer/{subject}-{timestamp}. The copy includes staged, unstaged, deleted, renamed, untracked, and Git-ignored files so optimization runs against the code and local runtime inputs you are actually using. Your source index, files, and HEAD remain unchanged. When the source checkout has publishable code changes, RELAI saves them in a separate Save source changes before optimization commit before dependency setup or optimization. A second commit contains only accepted optimizer changes. Any pull request therefore includes pre-existing code changes as its baseline; RELAI warns about this before the optimizer starts. Local .relai state, runtime env files, credentials, dependency-setup outputs, generated reports, and other Git-ignored private inputs remain uncommitted. A run with no accepted publishable optimizer diff keeps the baseline branch local and does not open a pull request. When a run succeeds, RELAI merges what its scheduler learned (which tests and tags it ran and their recent scores) back into the source checkout’s local optimizer state. Runs that overlap on one checkout keep each other’s learning, whichever finishes first. CLI versions before this change replace that state instead of merging it, so avoid overlapping runs across mixed CLI versions. To review accepted changes locally:
The optimizer branch stays checked out in its worktree under ~/.relai/worktrees, so git checkout relai/optimizer/{branch} in your main checkout fails with “already checked out”. Inspect the files in the worktree directly, or detach the branch first with git worktree remove {worktree-path} and then check it out normally.

​
Pull requests

When GitHub CLI (gh) is available and authenticated, relai optimize pushes the optimizer branch and opens a pull request for accepted changes so they can be reviewed in GitHub. With --no-pr, accepted changes stay on the local optimizer branch; review them with the commands above and merge or open a PR yourself.

​
Early stopping

​
Examples

​
What to expect

  • optimizer progress and rollout results.
  • generated or updated optimizer scope at .relai/.internal/optimizer-scope.json.
  • accepted changes committed to a local relai/optimizer/… branch.
  • optional PR output when GitHub CLI (gh) is available and authenticated, unless --no-pr is set.
Optimizer reports remain local under the project run directory. --total-rollouts should be set high enough to leave sufficient optimization budget after the 1/3 split. A run needs at least 6 * batch-size total rollouts to start one optimizer turn. Larger values allow more optimization turns before the reserved final evaluation pass. There is no fixed maximum. An external coding agent runs relai optimize with the rollout count you ask for, and confirms with you first only when it decides to optimize without your request. Optimization, including its final evaluation pass, keeps simulation results and artifacts local. Like relai simulate, it prints a Harbor trajectory warning and keeps running when the project recorder did not record a turn’s user input. If a selected agent test or benchmark has an unfinished update session, optimization stops before preparing its worktree and shows the resume command. If a simulation starts a repair that pauses, is cancelled, or fails, optimization stops further rollouts and reports the original failure, repair outcome, session path when available, and next step. Accepted changes remain on the optimizer branch; their final validation is incomplete. Resume a remaining session in the reported worktree, or fix the reported cause if there is no session. Then start a fresh optimization run from that repaired project.
This workflow can write code changes. Review git status, selected agent tests, credentials, and rollout budget before running it.
Common failures: missing simulator, missing runtime credentials, total rollouts below 6 * batch-size, backend optimizer failure, or missing GitHub CLI (gh) for automatic PR flows.