The core objects
Five objects do the work in RELAI. Knowing what each one is — and where it lives on disk — makes every command read clearly.
| Object | What it is | Lives at |
|---|
| Learning environment | A repeatable scenario RELAI can simulate and optimize against — behavior you want the agent to learn or preserve. | .relai/learning-envs/ |
| Evaluator | Scores agent behavior. Can be scoped to an environment, a benchmark, or registered globally. | .relai/evaluators/ |
| Benchmark | A reusable evaluation suite (from CSV) you can reference across simulation and optimization. | .relai/benchmarks/ |
| Tag | A label that groups learning environments into a reusable suite you can select by name. | metadata |
| Memory | Markdown guidance RELAI considers during future generation and optimization, scoped to agent / tag / component / evaluator. | Markdown files |
Reading paths in this doc: anything under .relai/ is created and owned by RELAI — those example paths show where files actually land. Any other path (like a --log-file or --csv you pass in) is your file and can live anywhere; examples written as {path/to/…} are placeholders to replace.
Simulate, then optimize
- Simulate measures how the agent behaves today against the environments you select and reports evaluator scores. It doesn’t intentionally edit agent source — though if simulator execution fails, RELAI may ask the backend to repair the environment or simulator code and retry.
- Optimize consumes evaluator signals to propose or apply improvements to prompts, configs, and structure, then validates them with a before/after evaluation pass. Accepted improvements can land as repository changes or a GitHub PR.
The rollout budget split
Optimization is governed by --total-rollouts. RELAI spends one third on the optimization loop and reserves two thirds for the final before/after evaluation. A run needs at least 6 × batch-size rollouts to start a single optimizer turn — so larger budgets buy more optimization turns before that reserved evaluation pass.
Optimize can change your code. Review git status, selected environments, credentials, and rollout budget before running it. See relai optimize for every control. Run artifacts
Simulations and optimizer runs write local artifacts under .relai/runs/, and selected artifacts upload to the backend when configured. You review uploaded results on the matching tab — Simulations, Optimizations, Learning Environments, Benchmarks, or Evaluators — after opening your agent from the Agents page.
