ADR 0020 — AI Lab eval workbench
Context
Section titled “Context”Restura already ships an AI assistant — a request-aware chat panel talking to OpenAI, Anthropic, and OpenRouter through a provider-agnostic core (shared/protocol/ai). That core streams a single chat against a known-safe cloud endpoint. It does not cover the other thing developers want from a model: systematically testing a prompt — comparing models, running it over a dataset, and grading the outputs — including against local runtimes (Ollama, LM Studio, vLLM) on localhost.
Two things make this more than “reuse the chat panel”:
- Local runtimes need localhost. The SSRF guard (ADR 0004) blocks loopback on every existing path — correctly, since the cloud assistant never hits
127.0.0.1. A prompt-testing tool targeting Ollama must reach loopback, without thereby opening a hole to LAN, private ranges, or cloud-metadata. - Evals are a fan-out, not a stream. Grading needs a non-streaming completion per (case × model) cell, structured-output calls (judge, dataset generation), and scorers that run untrusted user code — a shape the streaming chat orchestrator doesn’t provide.
Decision
Section titled “Decision”Add AI Lab as a separate, Electron-only feature (src/features/ai-lab) that reuses the AI provider core but layers its own engine, stores, and a provider-kind-aware security carve-out.
- Provider model — widen the
Providerunion intoCloudProvider(openai/anthropic/openrouter) andLocalProvider(ollama/openai-compatible), withisLocalProvider()as the single predicate. Ollama and OpenAI-compatible share one route — the OpenAI wire shape — differing only in that auth is optional and the base URL is user-supplied. The OpenAI decoder is reused unchanged. - Security — the Electron AI Lab handler sets
allowLocalhost = isLocalProvider(provider)and passes it into the same shared SSRF guard everything else uses. Local providers reach127.0.0.1/::1and nothing else; LAN, RFC 1918/6598, link-local, IPv6 unique-local, and metadata stay blocked for everyone, across redirects and DNS rebind. Cloud providers get no carve-out. No second guard to drift. (Loopback-only scope and the LAN block are superseded by Amendment (a).) - Eval engine —
ai-complete.tsdrains the provider stream to oneCompletionResult. A bounded-concurrency runner sweeps (case × model) cells withAbortSignalcancel. Only the model call crosses IPC; scorers run in the renderer — thejson-schema(Ajv) andscriptscorers included, the latter on the QuickJS sandbox. Thejudgeand dataset-generation paths use structured output. Cost isnull(unknown) for unpriced gateways rather than coerced to$0. - Persistence — new
aiLab/evalRuns(and laterarenaRuns) Dexie tables with Zod validators;evalRunsuses the shareddebouncedStoragewrapper. API keys areSecretRefhandles, never plaintext. - Capabilities —
aiLab.*rows added tocapabilities.ts(the single source of truth), all desktop-only.
Consequences
Section titled “Consequences”Positive
- Local-model testing without weakening the guard: one predicate, one guard, loopback-only.
- Adding a local runtime is a base URL, not new code — anything OpenAI-compatible already works.
- Deterministic scorers stay pure and unit-testable; judge/script capabilities are injected, not imported.
- The provider-union split is reusable by a future Worker AI path.
Negative
- Desktop-only — like the AI assistant’s web gap, there is no Worker route (recorded in the capability matrix, not hidden).
- A new feature surface (six tabs, three stores, an engine) to maintain alongside the assistant.
scriptscorers run user code; safety rests entirely on the QuickJS sandbox boundary (ADR 0015), now exercised by a second caller.
Addendum (2026-10-30): held-out split and a judge that is never the model under test
Section titled “Addendum (2026-10-30): held-out split and a judge that is never the model under test”Two gaps against Anthropic’s eval-design guidance, closed in the eval workbench (not the Agents tab):
- Held-out cases. A dataset case carries an optional
split: 'train' | 'test'(untagged counts as train; JSONL and a reserved CSVsplitcolumn round-trip it; the structured editor has a per-case control). The report shows per-model pass rates for train and test when a dataset holds any case out, with the rule beside it: tune a prompt against train, read test only to confirm, and treat “train up, test flat” as overfitting. The runner itself is unchanged: every case still runs; the split is a reporting and discipline boundary, not a filter, because a user hillclimbing in the Playground needs to see both numbers. - No self-judging. A
judgeorpairwisescorer whose judge model is one of the models under test grades its own output. The builder refuses to start such a run with one message, and the runner fails the scorer closed per cell (judge model … is the model under test) for headless callers. Same model id on a different provider config is a different endpoint and stays allowed.
Amendments (2026-10-31)
Section titled “Amendments (2026-10-31)”From the AI Lab review (#1057).
- (a) Local-runtime address policy. Local kinds may reach loopback and RFC 1918, for literals and resolved hostnames alike, pinned to the checked address. Link-local, cloud metadata, CGNAT, IPv6 unique-local and this-network ranges stay refused; public addresses pass. The policy is independent of the Settings → Security toggles. Cloud kinds and other protocols are unchanged.
- (b) The outgoing secret check is decided in Electron main from the pinned address. The renderer’s
rawModeis only the explicit override; the handler skips the check for a local kind only when the pinned address is loopback or RFC 1918. A hostname resolving to RFC 1918 counts as local; a local kind pointed at a public host keeps the check. The one exception is thers.judgescript assertion (src/lib/shared/judgeBridge.ts): it always sends withrawMode: trueand does its own redaction, controlled by theredactBeforeJudgejudge setting, so with that setting off the candidate output reaches the judge unredacted. Provider error bodies have the resolved key and secret patterns removed before they surface (ADR 0007). - (c) The override is per suite, install-local and never portable. It is a map keyed by suite id in the AI Lab store, shown beside Run while on;
migrateAgentSuitedrops ametadata.sendUnredactedToCloudflag, so a shared suite cannot disable the check elsewhere. - (d) Eval reports are capped: newest 50 within 50 MiB; agent reports newest 20, 2 MiB each, 20 MiB total.
- (e) Invalid persisted records are quarantined, not reset. Offenders are dropped, saved under a sibling
__invalid-<timestamp>key, and offered through an “Export backup” action (a recovery aid via export, not an in-app restore). - (f) Telemetry export is DNS-guarded and rate-limited (120 calls per minute per window); OTLP is pinned to the validated address.
- (g) Desktop telemetry credentials are secret handles only (2026-11-07). Electron main refuses a
source: "env"credential ref, since the config is renderer-supplied; env refs stay CLI-only. - (h) Scorers no longer use Ajv (2026-11-07).
json-schemaandtool-callvalidate with the eval-freeshared/protocol/json-schema-check.ts(Ajv’snew Functionis blocked by the renderer CSP, ADR 0026); it enforces a keyword subset and fails closed on an over-complex schema. The “(Ajv)” above is superseded.
Related
Section titled “Related”- Guide: AI Lab — the workbench in context.
- ADR 0010 — AI assistant architecture — the provider core AI Lab reuses.
- ADR 0004 — Security Hardening — the SSRF guard the localhost carve-out plugs into.
- ADR 0007 — SecretRef Pattern — how provider API keys avoid plaintext storage.
- ADR 0015 — QuickJS script sandbox — the boundary the
scriptscorer runs inside.