> ## Documentation Index
> Fetch the complete documentation index at: https://cli-docs.relai.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Agent Optimizer

> Use RELAI Agent Optimizer to improve selected behavior and compare quality and cost.

RELAI measures the current agent, explores scoped harness changes, and accepts only revisions whose overall evaluation improves before putting them on a reviewable branch.

## Optimization includes validation

<Steps>
  <Step title="Evaluate before">
    Measure the selected behavior
  </Step>

  <Step title="Explore changes">
    Improve prompts, tools, workflow, memory, or code within scope
  </Step>

  <Step title="Compare evidence">
    Discard candidates whose weighted overall evaluation does not improve
  </Step>

  <Step title="Evaluate after">
    Commit accepted changes with measured evidence
  </Step>
</Steps>

You do not need to simulate first or add a separate “verify” stage. Agent Optimizer runs fresh before-and-after evaluation; earlier simulation results are not reused. Your job is to review the result and decide whether to adopt it.

## Start with a bounded request

<Tabs>
  <Tab title="Ask your coding agent">
    <Prompt description="Use RELAI Agent Optimizer on <failing test> in Balanced quality/cost mode. Show the scope and rollout budget first, keep changes local, and report before-and-after evidence." actions={["copy"]}>
      Use RELAI Agent Optimizer on \<failing test> in Balanced quality/cost mode. Show the scope and rollout budget first, keep changes local, and report before-and-after evidence.
    </Prompt>
  </Tab>

  <Tab title="Run in terminal">
    ```sh theme={"system"}
    relai optimize --tests <agent-test-name> --no-pr
    ```
  </Tab>
</Tabs>

<Warning>
  **Select the behavior deliberately**

  Without `--tests`, `--tags`, or `--benchmarks`, Agent Optimizer includes every available agent test and registered benchmark sample. Start with the failures you intend to improve, and select one `--agent-target` in a multi-target project.
</Warning>

## Confirm the optimizer scope

Before provider-backed work begins, review the scope that controls what RELAI may change:

| Scope choice | What it controls |
| - | - |
| **Allowed changes** | Prompt/config only, or structural changes to code, workflow, and tools |
| **Model changes** | Whether the model may change, and the exact allowed candidates |
| **API-key-backed tools** | Whether new paid or credentialed integrations are allowed |
| **Cost optimization** | How strongly RELAI should prefer lower cost and token usage while preserving behavior |
| **Custom instructions** | Additional constraints for this optimizer run |

The default scope allows structural changes, keeps the current models fixed, blocks new paid or credentialed tools, and disables cost optimization. Your confirmed scope is saved under `.relai/.internal/optimizer-scope.json` in current projects; legacy layouts use `.relai/optimizer-scope.json`. A non-interactive run needs that saved decision.

## Choose the quality/cost objective

| Mode | Intent |
| - | - |
| **Disabled** | Optimize evaluation quality only. |
| **Conservative** | Preserve quality with cautious pressure on token usage and cost. |
| **Balanced** | Treat quality as primary and cost as a meaningful secondary objective. |
| **Aggressive** | Apply stronger cost pressure while preserving the selected behavior. |

<Note>
  **A cost objective is not proof of savings**

  Report token or dollar reductions only when the before-and-after telemetry is comparable. Otherwise report **Cost: not measured**.
</Note>

## Know what decides acceptance

Every rollout scores the selected agent test or benchmark evaluators plus all matching global evaluators. RELAI keeps those evaluation packages outside the optimizer’s editable source snapshot, so the target evidence is not part of the code it may change.

A candidate is accepted only when the weighted overall evaluation improves. A large gain on one evaluator can outweigh a slight regression on another; an equal gain-for-loss trade does not pass. Review the individual scores as well as the aggregate result.

## Size the rollout budget

RELAI uses one third of `--total-rollouts` for candidate exploration and reserves two thirds for final before-and-after evaluation. The default is the larger of 30 or `6 × batch-size`; the minimum is `6 × batch-size`, and batch size defaults to 1.

* Keep the first run focused on one or a few known failures.
* Every selected sample is evaluated once before scheduling favors the weakest cases, so large benchmarks need more budget or a focused subsample.
* Use a smaller `--batch-size` when one simulation produces a long conversation or tool trajectory.
* Interactive runs ask for confirmation above 50 total rollouts; non-interactive runs warn.
* Use the [optimizer sizing reference](/cli/optimize#sizing-a-run) for advanced tuning.

## Where the optimized agent lives

RELAI creates a separate worktree under `~/.relai/worktrees` and a local `relai/optimizer/…` branch. Your main checkout, index, and `HEAD` remain unchanged.

The worktree starts from your current source state, including staged, unstaged, untracked, deleted, renamed, and Git-ignored files. Publishable pre-existing changes are saved in a baseline commit before accepted optimizer changes, so review `git status` first: an automatic pull request includes that baseline.

<Tabs>
  <Tab title="Ask your coding agent">
    <Prompt description="Show me the accepted optimizer branch and diff. Explain what changed and connect each change to the before-and-after evaluation." actions={["copy"]}>
      Show me the accepted optimizer branch and diff. Explain what changed and connect each change to the before-and-after evaluation.
    </Prompt>
  </Tab>

  <Tab title="Run in terminal">
    ```sh theme={"system"}
    git branch --list 'relai/optimizer/*'
    git worktree list
    git show --stat <optimizer-branch>
    git diff <base-branch>...<optimizer-branch>
    ```
  </Tab>
</Tabs>

The optimizer branch is already checked out in its worktree. Inspect that worktree directly instead of trying to check out the same branch in your main checkout.

When authenticated GitHub CLI is available, RELAI pushes accepted changes and opens a pull request. Use `--no-pr` to keep the result on the local optimizer branch.

## Read the optimizer report

| Report section | Question it answers |
| - | - |
| **Quality** | How did the selected evaluator scores change? |
| **Cost** | Is there comparable token or dollar evidence? |
| **Accepted changes** | What files and behavior did RELAI retain? |
| **Rejected attempts** | Which ideas failed to improve the selected evals? |
| **Remaining gaps** | What still scores below the intended behavior? |
| **Location** | Where are the branch, worktree, commit, and local report? |

## Allow time and recover safely

The whole optimizer run defaults to a six-hour timeout. Within it, RELAI waits at most 30 minutes for one backend optimizer task; each Harbor trial’s run phase may use its task’s agent and verifier timeouts plus five minutes, capped at two hours.

<Warning>
  **Interrupted validation is not a validated result**

  On timeout, RELAI stops at a safe step and prints a resume command when progress can continue. If evaluator repair pauses, is cancelled, or fails, accepted changes remain on the branch but final validation is incomplete. Use the exact reported recovery step, then start a fresh optimization after the project is repaired; do not blind-retry or describe the branch as validated.
</Warning>

<Card title="Continue the learning loop" href="/agent-tests">
  Review or adopt the accepted branch, then turn a remaining gap or harder risk into the next agent test.
</Card>

Run incomplete? [Separate evaluator and runtime failures from agent behavior](/troubleshooting) before retrying.

## Command reference

See [`relai optimize`](/cli/optimize) for all flags, defaults, and recovery details.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.