AI Lab
AI Lab is a workbench for testing prompts and models. Add a provider (a local Ollama, an OpenAI-compatible gateway, or a cloud key), compare model outputs side by side in the Playground, then run a prompt over a dataset and score every output with deterministic checks, sandboxed scripts, or an LLM-as-judge.
It is reachable at /ai-lab (the flask icon in the top bar) and is Electron-only — testing local LLMs and arbitrary OpenAI-compatible endpoints needs direct network access the browser can’t provide. On the web build the route renders a desktop-only state.
Providers
Section titled “Providers”A provider config is one row per endpoint you add. AI Lab splits them into two kinds:
| Kind | Providers | Base URL | API key | Pricing |
|---|---|---|---|---|
| Cloud | OpenAI, Anthropic, OpenRouter | Hardcoded (overridable) | Required | Known — cost estimated |
| Local | Ollama, OpenAI-compatible | User-supplied | Optional | Unknown — shown as “free”/“unknown” |
- Ollama defaults to
http://localhost:11434and needs no key. - OpenAI-compatible covers LM Studio, vLLM, llama.cpp, Together, Groq, and any gateway that speaks the OpenAI
/v1/chat/completionswire shape — you must supply the base URL.
API keys are stored as a SecretRef handle, never in plaintext in the renderer; the main process resolves the key only at the moment it signs the outbound request. The Providers tab also lets you test the connection and discover models (/api/tags for Ollama, /v1/models for OpenAI-compatible).
Playground
Section titled “Playground”Pick one or more model references (provider config + model id) and stream a prompt to all of them at once. Outputs render side by side as they arrive, so you can eyeball quality, speed, and tone differences between models or providers in one shot. Streaming uses the same subscribe-before-invoke IPC contract as the AI assistant, so no early tokens are dropped.
Datasets
Section titled “Datasets”A dataset is a list of cases. Each case carries:
vars— template variables substituted into the prompt’s{{placeholders}}.expected(optional) — an exact expected output, for exact-match scoring.reference(optional) — a gold/reference answer, for reference-based scorers, the judge, and the pairwise scorer.turns(optional) — a multi-turn conversation ({ role: 'user' | 'assistant'; content }[]). When present, the runner replays these turns as the model input instead of the prompt’s single user message ({{vars}}still resolve inside each turn).split(optional) —trainortest. Tag the cases you will not look at while tuning a prompt astest; untagged cases count as train. The report then shows pass rates per split. Tune against train and read test only to confirm: train going up while test stays flat means the prompt overfit the cases you looked at.
Cases are edited as a compact JSON array, and you can import / export a dataset as CSV or JSONL (JSONL preserves turns; flat CSV does not; both carry split). Beyond hand-authoring, four generators seed a dataset:
Generate from an OpenAPI spec
Section titled “Generate from an OpenAPI spec”The Datasets tab can seed a dataset from an OpenAPI 3 / Swagger 2 document. AI Lab extracts a compact operation summary (method, path, summary, parameter names — no $ref dereferencing) and asks a model to emit diverse test cases as a structured tool call. Review and edit the generated cases before saving.
Import from request history / collections
Section titled “Import from request history / collections”Turn real traffic into eval cases: From history lists your saved HTTP requests (request history and saved collection requests), and each selected request becomes a case — its method, URL, headers, and body become vars, and the captured response body becomes the reference. Secrets are redacted (credential headers, Bearer/JWT/key=value tokens, recognizable provider keys, URL userinfo and query values, and the values of your environment, collection, and folder variables) before anything reaches a model. Review redactions shows what changed before you save.
Generate adversarial / red-team cases
Section titled “Generate adversarial / red-team cases”Red-team asks a model to generate jailbreak, prompt-injection, or boundary/abuse inputs to probe a prompt’s robustness. Describe the system under test and pick a focus; each generated case’s reference describes the safe expected behavior, so you can pair it with a judge scorer.
Save Playground outputs as a dataset
Section titled “Save Playground outputs as a dataset”After a Playground run, Save outputs as dataset captures the current vars plus each finished model output (as the case reference) into a new dataset — a quick way to bootstrap a regression set from a good run.
An eval is the cross-product of dataset cases × selected models, each cell scored by your configured scorers. The runner is a bounded-concurrency sweep (like the load-test runner): render the prompt → call the model → score → emit progress. Model calls cross IPC to the main process; scorers run in the renderer.
- Pick a prompt template, a dataset, and one or more models.
- Add one or more scorers (below).
- Set a concurrency cap and run. Progress streams in per cell; you can cancel mid-run (in-flight cells finish).
Scorers
Section titled “Scorers”A cell passes only when its model call succeeded and every scorer passed. Each scorer fails closed.
| Scorer | Passes when |
|---|---|
exact-match | Output equals the case’s expected / reference (optional case-insensitive trim). |
contains | Output contains a substring. |
regex | Output matches a pattern (invalid pattern → fail). |
json-valid | Output parses as JSON. |
json-schema | Output is JSON valid against a supplied JSON Schema (eval-free subset: format, not, if/then/else, uniqueItems, multipleOf, patternProperties, tuple items and remote $ref are not enforced; a schema too complex to check fails the scorer). |
latency | Round-trip ≤ maxMs. |
cost | Estimated USD ≤ maxUSD. Unknown cost fails — an unpriced gateway can’t satisfy a budget. |
script | A QuickJS test script passes (the output is exposed as pm.response.text()). |
judge | An LLM-as-judge scores the output ≥ a pass threshold. |
tool-call | The model called the expected tool, and its JSON arguments validate against a JSON Schema (same subset as json-schema) and/or match the case’s expected / reference args. |
pairwise | A preference judge prefers the cell output over the case’s reference (head-to-head A/B, with optional position-bias swap). |
LLM-as-judge
Section titled “LLM-as-judge”The judge scorer calls a model of your choosing with the output, the case’s reference, and a rubric, and asks it to return a structured judgement (score 0–1, reasoning, pass) via a tool call. Use any provider for the judge, including a cheaper or local model than the one under test, but never the model under test itself: a judge grading its own output inflates agreement, so the eval refuses to start when a judge or pairwise scorer’s judge is one of the selected models. The judge engine supports multi-criteria weighted rubrics (each criterion scored independently; a gate criterion fails the cell regardless of the weighted score), self-consistency (run the judge N≤5 times and aggregate by median, reporting score variance), and calibration anchors (reference-scored examples that pin the 0–1 scale).
Tool-call & pairwise scorers
Section titled “Tool-call & pairwise scorers”tool-callgrades function-calling: expose tool definitions to the model and assert it called the right tool with arguments that validate against a schema (or match the case’s expected args). Useful for agent/MCP-style prompts.pairwiseis preference judging — instead of an absolute score, a judge picks a winner between the cell output and the casereference. Enable position swap to run both A/B orderings and cancel position bias (a flip-flop is scored as a tie). For model-vs-model ranking across a whole dataset, use the Arena instead.
Execute-and-score (http-exec target)
Section titled “Execute-and-score (http-exec target)”By default a cell scores the model’s text. Switch the eval’s target to Execute as HTTP request (or GraphQL) and the cell instead: calls the model → parses an HTTP/GraphQL request out of the output (a JSON object, or a fenced ```json block) → executes it through Restura’s real request executor → scores the upstream response. This answers “did the model produce a request that actually works”, not just “did it read correctly”. The executed request goes through the same SSRF guard, redirect policy, and cookie jar as any request you’d send by hand.
Reports
Section titled “Reports”The Reports tab shows a leaderboard across a run (pass rate, p50/p95 latency, cost per model) and a regression compare against an earlier run of the same eval config. Eval-run ids are stable per config so re-running and comparing works across sessions. Pass rate excludes “not evaluated” cells (a run with no scorers), so a misconfigured eval reads as neither 0% nor 100%.
Cost is only shown when it is known. A model missing from its provider’s price table (including Gemini and Bedrock, which have none) shows unknown cost — never $0 — so a cost scorer fails rather than passing on a fake zero. Dated snapshot ids (claude-…-20250929, gpt-…-2024-08-06) are priced from their base model’s entry. A stream that ends before the provider’s end marker is reported as a truncated-stream error, not as a completed reply.
Each eval has a pass threshold, 100% of scored cells by default. A run below it reads failed; errored cells count as not passed, and a run with nothing scored reads error. Reports saved before thresholds existed keep their stored status. A run interrupted by quitting the app reads interrupted.
AI Lab keeps the newest 50 eval reports, within a 50 MiB total. If a saved record fails validation on load, only that record is dropped: its raw data is backed up locally and you’re offered an export. A failed report save can be retried or discarded, so it no longer blocks new runs.
Each run can be exported as CSV, JSON, or Markdown. A per-case drill-down opens any case to see every model’s full output, per-scorer detail, and judge per-criterion reasoning side by side — a cross-model diff for that case.
The Arena ranks models head-to-head. Pick a dataset and two or more contestant models plus a judge model; the Arena runs every model pair against every case as a round-robin pairwise comparison (with position-bias swap) and folds the results into an Elo leaderboard and a win-rate matrix. It’s the model-vs-model counterpart to the per-cell pairwise scorer, and the right tool when you want a ranking across many models rather than a pass/fail per cell. Runs are bounded-concurrency and cancellable, and persist to their own history (the arenaRuns store).
A match where a contestant or the judge errors shows as an error and is left out of the Elo and win-rate results, never counted as a tie. A run with some errored matches ends partial and keeps its results; one where every match failed ends error. Stop cancels in-flight calls and is not an error.
Agent suites
Section titled “Agent suites”An agent suite evaluates a multi-step agent against repeatable tasks rather than a single completion. It records the trajectory and outcome for each trial, applies task-aware grading, and can repeat trials to show reliability rather than treating one run as conclusive. Suites have run-wide step, time, tool, token, cost, and output budgets; cancelling a suite stops provider, grader, and tool work, and late results cannot overwrite a cancelled run.
Providers and capabilities
Section titled “Providers and capabilities”Desktop suites use OpenAI Chat, Anthropic Messages, OpenRouter, Ollama, Hugging Face, generic OpenAI-compatible, Azure OpenAI, Google Gemini, and Amazon Bedrock providers through the encrypted Electron bridge. Model capabilities start conservative; Restura uses trusted catalog data where available and otherwise requires an explicit user assertion before it enables a capability such as tool calling.
- Azure OpenAI uses the v1 API at your resource URL (
https://{resource}.openai.azure.com) with anapi-key; the model is your deployment name. - Google Gemini uses the Gemini API with an API key sent as a header, never in the URL. Fetch models lists the chat-capable models.
- Amazon Bedrock uses the Converse API. Store either a Bedrock API key, or IAM credentials as JSON (
{"accessKeyId":…,"secretAccessKey":…,"sessionToken":…}), which are signed with SigV4 in the main process. Pick a region by setting the base URL tohttps://bedrock-runtime.{region}.amazonaws.com. Enter model ids by hand; the runtime endpoint has no model list.
These three have no static price table, so their cost shows as unknown.
Tools, grounding, and approval
Section titled “Tools, grounding, and approval”- Saved HTTP requests run through Restura’s normal request executor and its SSRF, auth, and cancellation boundaries. Only unscripted
GET,HEAD, andOPTIONSrequests are read-only; all other requests, and any request with executable scripts, require explicit per-call approval. - MCP tools reuse a saved MCP profile with fresh desktop-owned sessions, existing SSRF/DNS protections, tool allowlists, cancellation, and approval.
- Grounding is selected explicitly and recorded with provenance in the trace. Saved reports are bounded and sanitized before persistence or export.
- Code sandbox. Add
{ "kind": "sandbox", "sandboxId": "quickjs" }to give the agent arun_code_quickjstool. It runs JavaScript in a fresh QuickJS WASM runtime per call with the script memory and time caps, and no network, filesystem, timers orpm. Output is bounded. Because it cannot reach the host, calls need no approval. A Docker/Podman provider is planned. - Remote agents (A2A).
{ "kind": "a2a", "endpoint": "https://…" }exposes a remote Agent2Agent agent as one tool, described from its agent card. Calls go through the request executor and its SSRF guard, and each call needs approval. The card’s RPC URL must stay on the endpoint’s origin.
CI boundary
Section titled “CI boundary”restura agent eval runs versioned suites and portable Agent Bundles in CI. It supports stateless OpenAI Responses, Anthropic Messages, Azure OpenAI, Gemini and Bedrock with environment credentials, deterministic fixture tools, the QuickJS sandbox, and fail-closed baseline gates. Live tools must be explicitly listed in a runtime manifest: HTTP is limited to manifest-listed GET / HEAD / OPTIONS requests, and MCP and A2A sources must be declared read-only (MCP also needs a tool allowlist). The CLI refuses desktop secret handles, suite base-URL overrides and agent judge graders. See CLI — Agent eval for the command contract.
Dataset evals in CI
Section titled “Dataset evals in CI”Evals → Export for CLI downloads the current eval as a .eval.json file for restura eval run. The file holds the prompt, dataset, scorers and models, but no API keys: each endpoint names the environment variable the CLI reads its key from. The CLI scores with the same code as the desktop, judges included, and exits 0 only when the share of passing cells meets the eval’s pass threshold. The http-exec target and the Arena stay desktop-only.
Agent telemetry export
Section titled “Agent telemetry export”Agent telemetry is optional and off by default. It is available only in the desktop AI Lab and the restura agent eval CLI; the web app and Worker never export it. When you turn it on, Restura sends traces to Langfuse (via its OpenTelemetry processor) or to any OTLP/HTTP collector.
- Metadata only. Exported spans carry allowlisted fields: suite, task and agent identifiers, timing, model and provider names, token and cost totals, tool names and outcomes, and aggregate evaluation state. Prompts, model responses, tool arguments and output, URLs, request bodies, headers, credentials and error text are never exported.
- Desktop. Configure the destination in the AI Lab telemetry panel. It accepts HTTPS endpoints and local loopback HTTP collectors, and rejects endpoints with query strings, fragments or embedded credentials. Langfuse and OTLP credentials are stored as keychain handles and resolved only in the Electron main process. The desktop app refuses environment-variable (
source: "env") credential references; those work only in the CLI. - CI. Pass
--telemetry-config telemetry.json; in the CLI, credentials must be environment references ({ "source": "env", "name": "LANGFUSE_SECRET_KEY" }), never inline values. See the CLI reference. - OpenInference spans. Each trace is a root
AGENTspan withLLM(model call) andTOOLchild spans, usingopeninference.span.kindand the OpenTelemetrygen_ai.*attributes. See the full attribute list indocs/AGENT_TELEMETRY.md. - Best-effort. A failed export is reported locally and never changes a suite’s pass/fail exit code.
The full contract is in docs/AGENT_TELEMETRY.md.
Security & SSRF
Section titled “Security & SSRF”Every outbound model call goes through the Electron AI Lab handler, which applies the same shared SSRF guard as the rest of Restura, with one provider-kind-aware twist: allowLocalhost is enabled only for local providers (Ollama, OpenAI-compatible).
- Loopback and your LAN. Local providers may reach loopback and RFC 1918 addresses (literals and resolved hostnames alike, pinned to the checked address), independent of the Settings → Security toggles. Link-local
169.254/16, IPv6 link-local, CGNAT, IPv6 unique-local and cloud-metadata endpoints remain blocked, including across redirects and DNS rebind. - Secret check. Calls are refused if the prompt still contains a secret. Electron main skips that check only for a local provider on loopback or your LAN; a local-kind provider pointed at a public host is checked like a cloud one. “Send unredacted” overrides it per suite on this machine and is never stored in the suite JSON, so an imported suite cannot enable it. The one exception is the
rs.judgescript assertion (src/lib/shared/judgeBridge.ts): it always sends withrawMode: trueand does its own redaction, controlled by theredactBeforeJudgejudge setting, so with that setting off the candidate output reaches the judge unredacted. - Cloud providers get no carve-out — they can only reach their public endpoints.
- The handler also enforces a hard concurrency ceiling independent of the renderer-side cap.
- http-exec (the execute-and-score target) adds no new outbound path: AI-generated requests run through the standard request executor and the same SSRF guard as user-issued requests. The residual risk is that the model picks the destination — see the caution above and ADR 0023.
Where it runs
Section titled “Where it runs”- Desktop — fully wired. The renderer drives model calls over the
window.electron.aiLabIPC bridge →electron/main/handlers/ai-lab-handler.ts→shared/protocol/ai. primary path - Web / self-hosted — not available; the route shows a desktop-only message. There is no Worker route for AI Lab.
Related
Section titled “Related”- AI assistant — the request-aware chat panel (shares the provider core).
- Scripts — the same QuickJS sandbox the
scriptscorer uses. - ADR 0020 — AI Lab eval workbench — the architecture decision behind this feature.
- ADR 0023 — AI Lab http-exec — scoring AI-generated requests through the real executor.
- Capability matrix — what’s desktop-only and why.