> ## Documentation Index
> Fetch the complete documentation index at: https://cli-docs.relai.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Harbor format

> Create Harbor-format agent tests, benchmarks, and evaluators with RELAI, and see how RELAI runs them.

[Harbor](https://github.com/harbor-framework/harbor) is an open-source framework for evaluating agents in containers. The Terminal-Bench team built it, and Terminal-Bench 2.0 runs on it. RELAI writes your agent tests, benchmarks, and evaluators as Harbor task packages in your repository, then runs them with a pinned Harbor release.

You do not need to write Harbor tasks by hand. Describe the behavior, point RELAI at a failed run, or register a benchmark, and RELAI creates and validates the package.

## Create Harbor tasks with RELAI

| You have | Create | Guide |
| - | - | - |
| A behavior, an EvalsWiki requirement or risk, or one failed run | An agent test | [Agent tests](https://cli-docs.relai.ai/agent-tests.md) |
| A CSV of evaluation cases | A benchmark | [Evaluation suites](https://cli-docs.relai.ai/evaluation-suites.md#connect-an-existing-benchmark) |
| A check that should apply across tests | A global evaluator | [Evaluation suites](https://cli-docs.relai.ai/evaluation-suites.md#create-a-global-evaluator) |

> [tab] Ask your coding agent
>
> Use RELAI to create one agent test for this behavior: “…”. Check EvalsWiki and existing tests first, then summarize the scenario and evaluator criteria.
>
> Use RELAI to create one agent test for this behavior: “…”. Check EvalsWiki and existing tests first, then summarize the scenario and evaluator criteria.

> [tab] Run in terminal
>
> ```sh theme={"system"}
> relai test create --prompt "<behavior to test>"
> ```

## Where the packages live

| Package | Location |
| - | - |
| Agent tests | `.relai/harbor/agent-tests/<test-id>/` |
| Benchmarks and their datasets | `.relai/harbor/benchmarks/` |
| Global evaluators | `.relai/harbor/evaluators/` |
| Shared runtime: adapter, dependencies, launcher, and shared mocks | `.relai/harbor/runtime/` |

Each agent test has the files Harbor expects, such as `task.toml`, `instruction.md`, its environment, and its tests, plus RELAI's scenario and evaluator configuration. [How an agent test is stored](https://cli-docs.relai.ai/agent-tests.md#how-an-agent-test-is-stored) describes every file.

Commit reviewed packages together with `.relai/harbor/runtime/` so collaborators run the same tests. Keep run evidence and caches local.

## How RELAI runs Harbor tasks

`relai simulate` and `relai optimize` run the selected tasks with the Harbor version pinned to your RELAI CLI release, which RELAI launches through `uv`. Each trial runs in a Docker container, so you need a running Docker daemon that can run Linux containers.

RELAI adds what your agent needs to each run:

* A fresh, read-only snapshot of your project source at `/workspace`. It leaves out sensitive and host-only paths such as `.git`, `.relai`, `.env` files, `.venv`, and `node_modules`.
* The runtime values you configured, passed only to the agent phase.

RELAI records each run as an Agent Trajectory Interchange Format (ATIF) trajectory, which evaluators and RELAI's reports read.

Because of these additions, run RELAI's packages through `relai simulate` rather than calling Harbor directly. To run one package during development, pass its folder:

```sh theme={"system"}
relai simulate --harbor-task .relai/harbor/agent-tests/<test-id>
```

See [Simulation](https://cli-docs.relai.ai/simulation.md) for selectors and results, and [Compatibility](https://cli-docs.relai.ai/compatibility.md) for what the agent image needs.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.