> ## Documentation Index
> Fetch the complete documentation index at: https://cli-docs.relai.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# benchmark

> Reference for relai benchmark commands.

Use `relai benchmark` to register and list benchmark suites.

## `register`

Register a benchmark from CSV data.

```sh theme={"system"}
relai benchmark register --csv {PATH}
```

<Warning>
  Benchmark registration can take a while when RELAI needs to generate benchmark support from larger CSV data or evaluator context.
</Warning>

| Option                    | Description                                                                                                  |
| ------------------------- | ------------------------------------------------------------------------------------------------------------ |
| `--csv {PATH}`            | Benchmark CSV file.                                                                                          |
| `--name {NAME}`           | Optional benchmark name, id, and output filename.                                                            |
| `--evaluator-ref {PATH}`  | Optional evaluator reference or context file. May be repeated.                                               |
| `--prompt {TEXT}`         | Optional inline generation context.                                                                          |
| `--agent-target {TARGET}` | Registered target for the entire benchmark. Required non-interactively when multiple targets are registered. |

```sh theme={"system"}
relai benchmark register --csv {path/to/data.csv} --name {name}
relai benchmark register --csv {path/to/data.csv} --agent-target {target}
relai benchmark register \
  --csv {path/to/data.csv} \
  --evaluator-ref {path/to/evaluator.md} \
  --prompt "{scoring guidance}"
```

Creates benchmark files under `.relai/benchmarks/` and may upload benchmark artifacts depending on the flow.

A benchmark is the batch way to define many scenarios at once: registering a
CSV produces the same kind of objects as creating learning environments one at
a time with [`relai learning-env`](/cli/learning-env), one scenario per row.
Use whichever fits — a handful of known failures work fine as individual
learning environments; a reusable suite of cases belongs in a benchmark.

A benchmark has one agent target for all of its samples. When multiple targets
are registered, interactive runs present a single-target picker; pass
`--agent-target {target}` in non-interactive runs.

### CSV expectations

There is no fixed CSV schema. RELAI reads your columns as they are and infers
what kind of benchmark to generate from them — the required columns recorded
inside a generated benchmark file describe *that benchmark's* dataset; they
are not a format your CSV must match up front.

Multi-turn scenarios are supported without one-column-per-turn. A CSV can
carry just the first user message per row while later turns are simulated
adaptively — indicate in the CSV itself or in `--prompt` that the samples are
conversations, and RELAI asks when the intent is ambiguous.

### `--prompt` vs `--evaluator-ref`

Both supply generation context; they differ in focus and form.

`--prompt` is a free-form string for anything your CSV alone does not say:

* these samples are multi-turn conversations.
* what an ambiguously named column means, or that a column should be ignored.
* which aspects to measure, such as groundedness, token cost, or latency.

`--evaluator-ref` is a path (repeatable) to files that describe evaluation
specifically:

* existing evaluator code you ran this benchmark with before, so RELAI can
  reproduce your evaluation protocol as closely as possible.
* a Markdown description of an evaluation harness, including how to invoke it,
  when the code alone does not explain the workflow.

Evaluation guidance can also go in `--prompt`; use `--evaluator-ref` when it
already lives in files.

Uploaded benchmarks appear on the Benchmarks tab after you open your agent from the [Agents page](https://platform.relai.ai/agent).

Common failures: invalid CSV, missing required columns, generation ambiguity, validation failure, dirty generated paths, or backend upload failure.

## `list`

List registered benchmarks.

```sh theme={"system"}
relai benchmark list
```

Read-only. Common failures are malformed benchmark files or missing simulator dependencies.