Use relai benchmark to register and list benchmark suites.
register
Register a benchmark from CSV data.
Benchmark registration can take a while when RELAI needs to generate benchmark support from larger CSV data or evaluator context.
.relai/benchmarks/ and may upload benchmark artifacts depending on the flow.
A benchmark is the batch way to define many scenarios at once: registering a
CSV produces the same kind of objects as creating learning environments one at
a time with relai learning-env, one scenario per row.
Use whichever fits — a handful of known failures work fine as individual
learning environments; a reusable suite of cases belongs in a benchmark.
A benchmark has one agent target for all of its samples. When multiple targets
are registered, interactive runs present a single-target picker; pass
--agent-target {target} in non-interactive runs.
CSV expectations
There is no fixed CSV schema. RELAI reads your columns as they are and infers
what kind of benchmark to generate from them — the required columns recorded
inside a generated benchmark file describe that benchmark’s dataset; they
are not a format your CSV must match up front.
Multi-turn scenarios are supported without one-column-per-turn. A CSV can
carry just the first user message per row while later turns are simulated
adaptively — indicate in the CSV itself or in --prompt that the samples are
conversations, and RELAI asks when the intent is ambiguous.
--prompt vs --evaluator-ref
Both supply generation context; they differ in focus and form.
--prompt is a free-form string for anything your CSV alone does not say:
- these samples are multi-turn conversations.
- what an ambiguously named column means, or that a column should be ignored.
- which aspects to measure, such as groundedness, token cost, or latency.
--evaluator-ref is a path (repeatable) to files that describe evaluation
specifically:
- existing evaluator code you ran this benchmark with before, so RELAI can
reproduce your evaluation protocol as closely as possible.
- a Markdown description of an evaluation harness, including how to invoke it,
when the code alone does not explain the workflow.
Evaluation guidance can also go in --prompt; use --evaluator-ref when it
already lives in files.
Uploaded benchmarks appear on the Benchmarks tab after you open your agent from the Agents page.
Common failures: invalid CSV, missing required columns, generation ambiguity, validation failure, dirty generated paths, or backend upload failure.
list
List registered benchmarks.
Read-only. Common failures are malformed benchmark files or missing simulator dependencies. 
