> ## Documentation Index
> Fetch the complete documentation index at: https://cli-docs.relai.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Custom adapters

> Create a project-owned task catalog and runner for RELAI simulation and optimization.

A custom adapter lets an existing task runner supply RELAI with repeatable tasks,
traces, and evaluator results without packaging those tasks for Harbor. The
adapter belongs to your project: RELAI selects tasks and invokes its commands,
while your runner executes the target agent and grades one task at a time.

Use `relai init --bare` for a new project that needs only a custom adapter. It
creates RELAI project configuration without a Harbor runtime or an agent target.
Run `relai setup` to authenticate first, then run bare init from a nonempty Git
repository containing your project files. Interactive bare init offers setup
when credentials are missing; noninteractive bare init requires them already.
If the project already uses Harbor, keep that initialization and add the adapter
beside it. Custom selectors can be combined with explicit Harbor selectors.

## Files and catalog

Put the catalog at `.relai/custom-adaptor/<adapter-name>/catalog.json` and keep
the runner and any task fixtures in the project. The adapter name uses lowercase
ASCII letters, digits, `-`, or `_`. Commit the catalog and runner with the
project; keep credentials and generated run results out of Git.

```json theme={"system"}
{
  "schema_version": "relai.custom_adapter.v1",
  "prepare": ["python3", ".relai/custom-adaptor/example/prepare.py"],
  "runner": ["python3", ".relai/custom-adaptor/example/run.py"],
  "tasks": [
    {
      "id": "request/create",
      "tags": ["request.create", "request-workflow", "smoke"],
      "description": "Create the requested item",
      "input_summary": "The target agent receives a request and may use project tools.",
      "evaluators": [
        {"id": "correctness", "description": "Checks the requested result"},
        {"id": "quality", "description": "Rates answer quality", "min_score": 0, "max_score": 5}
      ],
      "metadata": {"suite": "example"}
    },
    {
      "id": "request/update",
      "tags": ["request.update", "request-workflow"],
      "description": "Update an existing item",
      "evaluators": [
        {"id": "correctness", "description": "Checks the requested update"}
      ]
    }
  ]
}
```

`runner` is a required, nonempty argument array. `prepare` is optional; RELAI
runs it once per selected adapter before the tasks. Both commands run from the
project checkout and receive the simulator environment. They are executed as
argument arrays, without a shell. RELAI appends `--task-id ID --output PATH` to
the runner command for each selected task. The runner must write one UTF-8 JSON
object at `PATH` and exit successfully. Use a nonzero exit for a command failure.

The Python commands above are only an example. A runner may be a script or a
compiled executable that can run on the host. For a Rust project, the
catalog can build the binary during `prepare` and invoke it directly:

```json theme={"system"}
{
  "prepare": ["cargo", "build", "--release", "--bin", "adapter-runner"],
  "runner": ["./target/release/adapter-runner"]
}
```

Provide the required compiler or toolchain on the host. RELAI runs `prepare`
again in the optimizer worktree, so the binary is built against that candidate
revision; do not rely on an ignored local build artifact being copied there.
Use the platform's executable path and implement the same runner arguments and
JSON result contract regardless of language.

Each task requires a unique `id` and at least one tag. IDs may contain `/` for
logical groups, but cannot be absolute, contain empty, `.` or `..` path segments,
or contain `::`, a comma, a backslash, `*` as the entire ID, or control
characters. Tags use ASCII letters, digits, `.`, `_`, and `-`, excluding `.` and
`..` as complete tags. `description`, `input_summary`, `metadata`, and
`agent_target` are optional. If `agent_target` is set, optimization with
`--agent-target` requires the same target.

Give each task or sample one meaningful, task-specific tag, such as
`request.create` or `request.update`. Add shared tags, such as
`request-workflow`, to tasks that exercise related behavior. A task-specific
tag lets you select a focused task and attach focused tag memory; a shared tag
selects a related suite and lets those tasks share guidance. You can also select
one task directly with `--custom-tests ADAPTER::ID`. Keep tag names stable as
you revise the catalog so saved selectors and memory still refer to the same
behavior. Avoid reusing a task-specific tag for an unrelated task, and use broad
tags such as `smoke` only for tasks you intend to run together. These are naming
recommendations; RELAI requires at least one tag but does not require unique
tags.

An `evaluators` list gives the optimizer each check's ID and description. IDs
must be unique within a task, and descriptions must be nonempty. Each evaluator
may declare **both** `min_score` and `max_score`; omit both for the 0–1 default.
Bounds must be finite, with `min_score < max_score`. Catalogs without an
`evaluators` list use one legacy evaluator named `custom-grade`; the optional
`evaluator_description` supplies its description. Existing 0–1 adapters do not
need to add range fields.

## Runner result

For a completed task, write `relai.custom_run.v1` JSON. The example below
matches the catalog above:

```json theme={"system"}
{
  "schema_version": "relai.custom_run.v1",
  "task_id": "request/create",
  "trace": {"steps": [{"step_id": 1, "source": "agent", "message": "Completed the request"}]},
  "final_output": "Completed the request",
  "evaluations": [
    {"evaluator_id": "correctness", "score": 1, "feedback": "Requested result was present"},
    {"evaluator_id": "quality", "score": 4, "min_score": 0, "max_score": 5, "feedback": "Clear response"}
  ]
}
```

The returned `task_id` must match the requested ID, and `trace.steps` must be
an array of trajectory steps. RELAI writes them into an
[Agent Trajectory Interchange Format (ATIF)](https://github.com/harbor-framework/harbor/blob/main/rfcs/0001-trajectory-format.md)
trajectory and includes it in the simulation result. Each scored evaluation
needs a catalog evaluator ID, a finite score within its declared inclusive
range, and nonempty feedback. The result's optional min/max fields must match
the catalog; RELAI fills them from the catalog when absent. A completed run
without errors must score every declared evaluator exactly once. RELAI
normalizes each score to 0–1 and averages the normalized values for
optimization.

If execution or grading fails, return a non-null `failure` object. Include
`evaluation_errors` with a nonempty `evaluator_id` and `message` for individual
grader diagnostics. Diagnostic error IDs may name the grading system instead
of a catalog evaluator. A failed result may retain partial scored evaluations
as evidence, but the optimizer scores the task as failed. `evaluation_errors`
without `failure` are invalid. A runner command that exits unsuccessfully is
reported as a task failure; it does not need to write a result first. During
optimization, that exit produces a failed, zero-score rollout, while a missing
runtime requirement stops the run for correction.

## Run and inspect

```sh theme={"system"}
relai simulate --custom-tests 'example::request/create' --result-json .relai/runs/custom-suite.json
relai simulate --custom-tags 'example::request.create'
relai simulate --custom-tags 'example::request-workflow'
relai simulate --custom-tests 'example::*'
relai optimize --custom-tags 'example::request-workflow' --no-pr
```

Only an exact `*` selects every task or tag in one adapter. Quote it to prevent
shell expansion. Explicit `--tests`, `--tags`, or `--benchmarks` may be supplied
alongside custom selectors. With any custom selector, optimization does not
add the default Harbor pool. `--max-simulations` limits the combined selection;
Harbor tasks run first when both kinds are selected.

RELAI loads runtime values from the process environment, then
`.relai/simulator.env`, then each `--env-file`, then each `--env` override. Keep
secrets in local files or the command environment, not in the catalog. The
runner sees the candidate agent revision in the optimizer worktree; adapter
files and fixtures are copied there but excluded from editable agent code.

### Optional runtime requirements

`relai init --bare` does not create or require `.relai/runtime-manifest.json`.
If your project has that file, its requirements apply to **all** custom adapter
commands as well as Harbor runs. RELAI checks the merged runtime environment
before running a custom adapter's `prepare` or runner command, including during
optimization. A required variable must have a nonempty value. A required file
must exist at the path named by its `path_env` variable; relative paths are
resolved from the project checkout. In a project using both Harbor and custom
adapters, a requirement blocks either workflow even if one runner does not use
it. Make every declared requirement available when running either workflow.

```json theme={"system"}
{
  "schema_version": "relai.runtime_manifest.v1",
  "env": [
    {"name": "AGENT_API_KEY", "required": true, "sensitive": true, "reason": "target agent API access"}
  ],
  "files": [
    {"path_env": "AGENT_DATA_PATH", "required": true, "reason": "target agent input data"}
  ]
}
```

The manifest declares variable names and reasons, never their values. Set the
values through the environment sources above. Mark secret variables
`sensitive: true` so RELAI includes their values in trajectory redaction,
including when they came from the process environment. RELAI also considers
values from `.relai/simulator.env`, `--env-file`, and `--env` for redaction.
If a custom command fails, RELAI redacts those values from its stderr before
showing or saving the failure. When stderr capture is truncated, RELAI omits
the captured text and retains the exit status.

Each custom run keeps the raw adapter JSON and trajectory in the local run
directory. The trajectory in a saved simulation result or optimizer input is
redacted; the raw local artifact is not rewritten. Keep secrets out of other
runner result fields, such as `final_output` and evaluator feedback.
`relai simulate` also saves each completed simulation result in its
default per-run result store. In a `--result-json` suite, a runner-reported
failure appears under `failures` with its validated result in `result`; it
counts as a failed task even if partial evaluator scores are high. The suite
also retains completed runs and task failures if a later task fails. If custom
preparation fails in a mixed suite, RELAI keeps completed Harbor runs, records
the selected custom tasks as failures, and writes the combined report. Custom
runs do not support `--outcome-json`, `--retry-from-outcome`, or `--harbor-task`.
The per-run path is `<runs>/results/custom/<SHA-256 of ADAPTER::ID>/<run-id>.json`;
the JSON retains the readable `ADAPTER::ID`. `<runs>` is `.relai/.internal/runs`
in new projects or `.relai/runs` in older projects.

Custom tags can have memory: use `relai memory read tag example::request.create`
for one task's guidance, or
`relai memory write tag example::request-workflow --text "..."` for guidance
shared by related tasks. `relai memory list` omits
custom adapter tags; use the catalog to find their names.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.