relai evaluator to create, upload, or remove global evaluators.
create
Create a global evaluator from a prompt.
Creates an evaluator under
.relai/evaluators/ using the project’s language. Interactive successful flows may offer to commit and upload it.
Evaluators are scoped to the selected agent target. When multiple targets are
registered, interactive runs present a single-target picker; pass
--agent-target {target} in non-interactive runs.
Uploaded evaluators appear on the Evaluators tab after you open your agent from the Agents page.
Common failures: prompt over 8 KB, backend generation failure, evaluator validation failure, duplicate explicit name, or dirty generated evaluator paths.
How global evaluators are applied
Evaluators created here are global: they run in every simulation of the agent whose scope they cover, alongside the local evaluators a learning environment or benchmark defines for itself. Because optimization evaluates through the simulator, both kinds also score every optimizer run — see Evaluation during optimization. There is no per-run subset selection: all global evaluators with a matching scope apply whenever you simulate or optimize. When an evaluator should stop applying, remove it. Scope is inferred when the evaluator is created, and evaluators run only for simulations with the same scope. Most evaluators target end-to-end agent behavior and run for every end-to-end simulation. An evaluator whose prompt targets one component — for example, scoring the relevance of retrieved documents — runs only for simulations of that component, such as component-targeted learning environments; it does not run in end-to-end simulations, even when the end-to-end run exercises that component. When the intended scope is ambiguous, RELAI asks during creation; state it in the prompt to avoid the question.upload
Upload an existing committed evaluator.
--agent-target {target} only to upload a legacy evaluator that does not
declare its target in source; a declared target always takes precedence.
Interactive create runs can finish with a prompt that commits and uploads
the evaluator in one flow. Non-interactive runs — and interactive runs where
you decline that prompt — skip the upload and print the exact git add,
git commit, and relai evaluator upload commands to run; the evaluator is
not registered with the backend until you do. upload is also the recovery
path when an earlier upload failed, instead of recreating the evaluator from
scratch.
Common failures: file missing, validation failure, not committed at current HEAD, uncommitted changes, or backend branch conflict.