Skip to content

ADR 0020 — AI Lab eval workbench

Accepted · 2026-05-20 · amended 2026-10-31

Restura already ships an AI assistant — a request-aware chat panel talking to OpenAI, Anthropic, and OpenRouter through a provider-agnostic core (shared/protocol/ai). That core streams a single chat against a known-safe cloud endpoint. It does not cover the other thing developers want from a model: systematically testing a prompt — comparing models, running it over a dataset, and grading the outputs — including against local runtimes (Ollama, LM Studio, vLLM) on localhost.

Two things make this more than “reuse the chat panel”:

  1. Local runtimes need localhost. The SSRF guard (ADR 0004) blocks loopback on every existing path — correctly, since the cloud assistant never hits 127.0.0.1. A prompt-testing tool targeting Ollama must reach loopback, without thereby opening a hole to LAN, private ranges, or cloud-metadata.
  2. Evals are a fan-out, not a stream. Grading needs a non-streaming completion per (case × model) cell, structured-output calls (judge, dataset generation), and scorers that run untrusted user code — a shape the streaming chat orchestrator doesn’t provide.

Add AI Lab as a separate, Electron-only feature (src/features/ai-lab) that reuses the AI provider core but layers its own engine, stores, and a provider-kind-aware security carve-out.

  • Provider model — widen the Provider union into CloudProvider (openai/anthropic/openrouter) and LocalProvider (ollama/openai-compatible), with isLocalProvider() as the single predicate. Ollama and OpenAI-compatible share one route — the OpenAI wire shape — differing only in that auth is optional and the base URL is user-supplied. The OpenAI decoder is reused unchanged.
  • Security — the Electron AI Lab handler sets allowLocalhost = isLocalProvider(provider) and passes it into the same shared SSRF guard everything else uses. Local providers reach 127.0.0.1 / ::1 and nothing else; LAN, RFC 1918/6598, link-local, IPv6 unique-local, and metadata stay blocked for everyone, across redirects and DNS rebind. Cloud providers get no carve-out. No second guard to drift. (Loopback-only scope and the LAN block are superseded by Amendment (a).)
  • Eval engine — ai-complete.ts drains the provider stream to one CompletionResult. A bounded-concurrency runner sweeps (case × model) cells with AbortSignal cancel. Only the model call crosses IPC; scorers run in the renderer — the json-schema (Ajv) and script scorers included, the latter on the QuickJS sandbox. The judge and dataset-generation paths use structured output. Cost is null (unknown) for unpriced gateways rather than coerced to $0.
  • Persistence — new aiLab / evalRuns (and later arenaRuns) Dexie tables with Zod validators; evalRuns uses the shared debouncedStorage wrapper. API keys are SecretRef handles, never plaintext.
  • Capabilities — aiLab.* rows added to capabilities.ts (the single source of truth), all desktop-only.

Positive

  • Local-model testing without weakening the guard: one predicate, one guard, loopback-only.
  • Adding a local runtime is a base URL, not new code — anything OpenAI-compatible already works.
  • Deterministic scorers stay pure and unit-testable; judge/script capabilities are injected, not imported.
  • The provider-union split is reusable by a future Worker AI path.

Negative

  • Desktop-only — like the AI assistant’s web gap, there is no Worker route (recorded in the capability matrix, not hidden).
  • A new feature surface (six tabs, three stores, an engine) to maintain alongside the assistant.
  • script scorers run user code; safety rests entirely on the QuickJS sandbox boundary (ADR 0015), now exercised by a second caller.

Addendum (2026-10-30): held-out split and a judge that is never the model under test

Section titled “Addendum (2026-10-30): held-out split and a judge that is never the model under test”

Two gaps against Anthropic’s eval-design guidance, closed in the eval workbench (not the Agents tab):

  • Held-out cases. A dataset case carries an optional split: 'train' | 'test' (untagged counts as train; JSONL and a reserved CSV split column round-trip it; the structured editor has a per-case control). The report shows per-model pass rates for train and test when a dataset holds any case out, with the rule beside it: tune a prompt against train, read test only to confirm, and treat “train up, test flat” as overfitting. The runner itself is unchanged: every case still runs; the split is a reporting and discipline boundary, not a filter, because a user hillclimbing in the Playground needs to see both numbers.
  • No self-judging. A judge or pairwise scorer whose judge model is one of the models under test grades its own output. The builder refuses to start such a run with one message, and the runner fails the scorer closed per cell (judge model … is the model under test) for headless callers. Same model id on a different provider config is a different endpoint and stays allowed.

From the AI Lab review (#1057).

  • (a) Local-runtime address policy. Local kinds may reach loopback and RFC 1918, for literals and resolved hostnames alike, pinned to the checked address. Link-local, cloud metadata, CGNAT, IPv6 unique-local and this-network ranges stay refused; public addresses pass. The policy is independent of the Settings → Security toggles. Cloud kinds and other protocols are unchanged.
  • (b) The outgoing secret check is decided in Electron main from the pinned address. The renderer’s rawMode is only the explicit override; the handler skips the check for a local kind only when the pinned address is loopback or RFC 1918. A hostname resolving to RFC 1918 counts as local; a local kind pointed at a public host keeps the check. The one exception is the rs.judge script assertion (src/lib/shared/judgeBridge.ts): it always sends with rawMode: true and does its own redaction, controlled by the redactBeforeJudge judge setting, so with that setting off the candidate output reaches the judge unredacted. Provider error bodies have the resolved key and secret patterns removed before they surface (ADR 0007).
  • (c) The override is per suite, install-local and never portable. It is a map keyed by suite id in the AI Lab store, shown beside Run while on; migrateAgentSuite drops a metadata.sendUnredactedToCloud flag, so a shared suite cannot disable the check elsewhere.
  • (d) Eval reports are capped: newest 50 within 50 MiB; agent reports newest 20, 2 MiB each, 20 MiB total.
  • (e) Invalid persisted records are quarantined, not reset. Offenders are dropped, saved under a sibling __invalid-<timestamp> key, and offered through an “Export backup” action (a recovery aid via export, not an in-app restore).
  • (f) Telemetry export is DNS-guarded and rate-limited (120 calls per minute per window); OTLP is pinned to the validated address.
  • (g) Desktop telemetry credentials are secret handles only (2026-11-07). Electron main refuses a source: "env" credential ref, since the config is renderer-supplied; env refs stay CLI-only.
  • (h) Scorers no longer use Ajv (2026-11-07). json-schema and tool-call validate with the eval-free shared/protocol/json-schema-check.ts (Ajv’s new Function is blocked by the renderer CSP, ADR 0026); it enforces a keyword subset and fails closed on an over-complex schema. The “(Ajv)” above is superseded.