Skip to main content
A custom adapter lets an existing task runner supply RELAI with repeatable tasks, traces, and evaluator results without packaging those tasks for Harbor. The adapter belongs to your project: RELAI selects tasks and invokes its commands, while your runner executes the target agent and grades one task at a time. Use relai init --bare for a new project that needs only a custom adapter. It creates RELAI project configuration without a Harbor runtime or an agent target. Run relai setup to authenticate first, then run bare init from a nonempty Git repository containing your project files. Interactive bare init offers setup when credentials are missing; noninteractive bare init requires them already. If the project already uses Harbor, keep that initialization and add the adapter beside it. Custom selectors can be combined with explicit Harbor selectors.

​
Files and catalog

Put the catalog at .relai/custom-adaptor/<adapter-name>/catalog.json and keep the runner and any task fixtures in the project. The adapter name uses lowercase ASCII letters, digits, -, or _. Commit the catalog and runner with the project; keep credentials and generated run results out of Git.
runner is a required, nonempty argument array. prepare is optional; RELAI runs it once per selected adapter before the tasks. Both commands run from the project checkout and receive the simulator environment. They are executed as argument arrays, without a shell. RELAI appends --task-id ID --output PATH to the runner command for each selected task. The runner must write one UTF-8 JSON object at PATH and exit successfully. Use a nonzero exit for a command failure. The Python commands above are only an example. A runner may be a script or a compiled executable that can run on the host. For a Rust project, the catalog can build the binary during prepare and invoke it directly:
Provide the required compiler or toolchain on the host. RELAI runs prepare again in the optimizer worktree, so the binary is built against that candidate revision; do not rely on an ignored local build artifact being copied there. Use the platform’s executable path and implement the same runner arguments and JSON result contract regardless of language. Each task requires a unique id and at least one tag. IDs may contain / for logical groups, but cannot be absolute, contain empty, . or .. path segments, or contain ::, a comma, a backslash, * as the entire ID, or control characters. Tags use ASCII letters, digits, ., _, and -, excluding . and .. as complete tags. description, input_summary, metadata, and agent_target are optional. If agent_target is set, optimization with --agent-target requires the same target. Give each task or sample one meaningful, task-specific tag, such as request.create or request.update. Add shared tags, such as request-workflow, to tasks that exercise related behavior. A task-specific tag lets you select a focused task and attach focused tag memory; a shared tag selects a related suite and lets those tasks share guidance. You can also select one task directly with --custom-tests ADAPTER::ID. Keep tag names stable as you revise the catalog so saved selectors and memory still refer to the same behavior. Avoid reusing a task-specific tag for an unrelated task, and use broad tags such as smoke only for tasks you intend to run together. These are naming recommendations; RELAI requires at least one tag but does not require unique tags. An evaluators list gives the optimizer each check’s ID and description. IDs must be unique within a task, and descriptions must be nonempty. Each evaluator may declare both min_score and max_score; omit both for the 0–1 default. Bounds must be finite, with min_score < max_score. Catalogs without an evaluators list use one legacy evaluator named custom-grade; the optional evaluator_description supplies its description. Existing 0–1 adapters do not need to add range fields.

​
Runner result

For a completed task, write relai.custom_run.v1 JSON. The example below matches the catalog above:
The returned task_id must match the requested ID, and trace.steps must be an array of trajectory steps. RELAI writes them into an Agent Trajectory Interchange Format (ATIF) trajectory and includes it in the simulation result. Each scored evaluation needs a catalog evaluator ID, a finite score within its declared inclusive range, and nonempty feedback. The result’s optional min/max fields must match the catalog; RELAI fills them from the catalog when absent. A completed run without errors must score every declared evaluator exactly once. RELAI normalizes each score to 0–1 and averages the normalized values for optimization. If execution or grading fails, return a non-null failure object. Include evaluation_errors with a nonempty evaluator_id and message for individual grader diagnostics. Diagnostic error IDs may name the grading system instead of a catalog evaluator. A failed result may retain partial scored evaluations as evidence, but the optimizer scores the task as failed. evaluation_errors without failure are invalid. A runner command that exits unsuccessfully is reported as a task failure; it does not need to write a result first. During optimization, that exit produces a failed, zero-score rollout, while a missing runtime requirement stops the run for correction.

​
Run and inspect

Only an exact * selects every task or tag in one adapter. Quote it to prevent shell expansion. Explicit --tests, --tags, or --benchmarks may be supplied alongside custom selectors. With any custom selector, optimization does not add the default Harbor pool. --max-simulations limits the combined selection; Harbor tasks run first when both kinds are selected. RELAI loads runtime values from the process environment, then .relai/simulator.env, then each --env-file, then each --env override. Keep secrets in local files or the command environment, not in the catalog. The runner sees the candidate agent revision in the optimizer worktree; adapter files and fixtures are copied there but excluded from editable agent code.

​
Optional runtime requirements

relai init --bare does not create or require .relai/runtime-manifest.json. If your project has that file, its requirements apply to all custom adapter commands as well as Harbor runs. RELAI checks the merged runtime environment before running a custom adapter’s prepare or runner command, including during optimization. A required variable must have a nonempty value. A required file must exist at the path named by its path_env variable; relative paths are resolved from the project checkout. In a project using both Harbor and custom adapters, a requirement blocks either workflow even if one runner does not use it. Make every declared requirement available when running either workflow.
The manifest declares variable names and reasons, never their values. Set the values through the environment sources above. Mark secret variables sensitive: true so RELAI includes their values in trajectory redaction, including when they came from the process environment. RELAI also considers values from .relai/simulator.env, --env-file, and --env for redaction. If a custom command fails, RELAI redacts those values from its stderr before showing or saving the failure. When stderr capture is truncated, RELAI omits the captured text and retains the exit status. Each custom run keeps the raw adapter JSON and trajectory in the local run directory. The trajectory in a saved simulation result or optimizer input is redacted; the raw local artifact is not rewritten. Keep secrets out of other runner result fields, such as final_output and evaluator feedback. relai simulate also saves each completed simulation result in its default per-run result store. In a --result-json suite, a runner-reported failure appears under failures with its validated result in result; it counts as a failed task even if partial evaluator scores are high. The suite also retains completed runs and task failures if a later task fails. If custom preparation fails in a mixed suite, RELAI keeps completed Harbor runs, records the selected custom tasks as failures, and writes the combined report. Custom runs do not support --outcome-json, --retry-from-outcome, or --harbor-task. The per-run path is <runs>/results/custom/<SHA-256 of ADAPTER::ID>/<run-id>.json; the JSON retains the readable ADAPTER::ID. <runs> is .relai/.internal/runs in new projects or .relai/runs in older projects. Custom tags can have memory: use relai memory read tag example::request.create for one task’s guidance, or relai memory write tag example::request-workflow --text "..." for guidance shared by related tasks. relai memory list omits custom adapter tags; use the catalog to find their names.