Skip to main content
Harbor is an open-source framework for evaluating agents in containers. The Terminal-Bench team built it, and Terminal-Bench 2.0 runs on it. RELAI writes your agent tests, benchmarks, and evaluators as Harbor task packages in your repository, then runs them with a pinned Harbor release. You do not need to write Harbor tasks by hand. Describe the behavior, point RELAI at a failed run, or register a benchmark, and RELAI creates and validates the package.

Create Harbor tasks with RELAI

Use RELAI to create one agent test for this behavior: “…”. Check EvalsWiki and existing tests first, then summarize the scenario and evaluator criteria.

Where the packages live

Each agent test has the files Harbor expects, such as task.toml, instruction.md, its environment, and its tests, plus RELAI’s scenario and evaluator configuration. How an agent test is stored describes every file. Commit reviewed packages together with .relai/harbor/runtime/ so collaborators run the same tests. Keep run evidence and caches local.

How RELAI runs Harbor tasks

relai simulate and relai optimize run the selected tasks with the Harbor version pinned to your RELAI CLI release, which RELAI launches through uv. Each trial runs in a Docker container, so you need a running Docker daemon that can run Linux containers. RELAI adds what your agent needs to each run:
  • A fresh, read-only snapshot of your project source at /workspace. It leaves out sensitive and host-only paths such as .git, .relai, .env files, .venv, and node_modules.
  • The runtime values you configured, passed only to the agent phase.
RELAI records each run as an Agent Trajectory Interchange Format (ATIF) trajectory, which evaluators and RELAI’s reports read. Because of these additions, run RELAI’s packages through relai simulate rather than calling Harbor directly. To run one package during development, pass its folder:
See Simulation for selectors and results, and Compatibility for what the agent image needs.