Skip to content

AI Lab

Desktop only

AI Lab is a workbench for testing prompts and models. Add a provider (a local Ollama, an OpenAI-compatible gateway, or a cloud key), compare model outputs side by side in the Playground, then run a prompt over a dataset and score every output with deterministic checks, sandboxed scripts, or an LLM-as-judge.

It is reachable at /ai-lab (the flask icon in the top bar) and is Electron-only — testing local LLMs and arbitrary OpenAI-compatible endpoints needs direct network access the browser can’t provide. On the web build the route renders a desktop-only state.

A provider config is one row per endpoint you add. AI Lab splits them into two kinds:

KindProvidersBase URLAPI keyPricing
CloudOpenAI, Anthropic, OpenRouterHardcoded (overridable)RequiredKnown — cost estimated
LocalOllama, OpenAI-compatibleUser-suppliedOptionalUnknown — shown as “free”/“unknown”
  • Ollama defaults to http://localhost:11434 and needs no key.
  • OpenAI-compatible covers LM Studio, vLLM, llama.cpp, Together, Groq, and any gateway that speaks the OpenAI /v1/chat/completions wire shape — you must supply the base URL.

API keys are stored as a SecretRef handle, never in plaintext in the renderer; the main process resolves the key only at the moment it signs the outbound request. The Providers tab also lets you test the connection and discover models (/api/tags for Ollama, /v1/models for OpenAI-compatible).

Pick one or more model references (provider config + model id) and stream a prompt to all of them at once. Outputs render side by side as they arrive, so you can eyeball quality, speed, and tone differences between models or providers in one shot. Streaming uses the same subscribe-before-invoke IPC contract as the AI assistant, so no early tokens are dropped.

A dataset is a list of cases. Each case carries:

  • vars — template variables substituted into the prompt’s {{placeholders}}.
  • expected (optional) — an exact expected output, for exact-match scoring.
  • reference (optional) — a gold/reference answer, for reference-based scorers, the judge, and the pairwise scorer.
  • turns (optional) — a multi-turn conversation ({ role: 'user' | 'assistant'; content }[]). When present, the runner replays these turns as the model input instead of the prompt’s single user message ({{vars}} still resolve inside each turn).
  • split (optional) — train or test. Tag the cases you will not look at while tuning a prompt as test; untagged cases count as train. The report then shows pass rates per split. Tune against train and read test only to confirm: train going up while test stays flat means the prompt overfit the cases you looked at.

Cases are edited as a compact JSON array, and you can import / export a dataset as CSV or JSONL (JSONL preserves turns; flat CSV does not; both carry split). Beyond hand-authoring, four generators seed a dataset:

The Datasets tab can seed a dataset from an OpenAPI 3 / Swagger 2 document. AI Lab extracts a compact operation summary (method, path, summary, parameter names — no $ref dereferencing) and asks a model to emit diverse test cases as a structured tool call. Review and edit the generated cases before saving.

Turn real traffic into eval cases: From history lists your saved HTTP requests (request history and saved collection requests), and each selected request becomes a case — its method, URL, headers, and body become vars, and the captured response body becomes the reference. Secrets are redacted (credential headers, Bearer/JWT/key=value tokens, recognizable provider keys, URL userinfo and query values, and the values of your environment, collection, and folder variables) before anything reaches a model. Review redactions shows what changed before you save.

Red-team asks a model to generate jailbreak, prompt-injection, or boundary/abuse inputs to probe a prompt’s robustness. Describe the system under test and pick a focus; each generated case’s reference describes the safe expected behavior, so you can pair it with a judge scorer.

After a Playground run, Save outputs as dataset captures the current vars plus each finished model output (as the case reference) into a new dataset — a quick way to bootstrap a regression set from a good run.

An eval is the cross-product of dataset cases × selected models, each cell scored by your configured scorers. The runner is a bounded-concurrency sweep (like the load-test runner): render the prompt → call the model → score → emit progress. Model calls cross IPC to the main process; scorers run in the renderer.

  1. Pick a prompt template, a dataset, and one or more models.
  2. Add one or more scorers (below).
  3. Set a concurrency cap and run. Progress streams in per cell; you can cancel mid-run (in-flight cells finish).

A cell passes only when its model call succeeded and every scorer passed. Each scorer fails closed.

ScorerPasses when
exact-matchOutput equals the case’s expected / reference (optional case-insensitive trim).
containsOutput contains a substring.
regexOutput matches a pattern (invalid pattern → fail).
json-validOutput parses as JSON.
json-schemaOutput is JSON valid against a supplied JSON Schema (eval-free subset: format, not, if/then/else, uniqueItems, multipleOf, patternProperties, tuple items and remote $ref are not enforced; a schema too complex to check fails the scorer).
latencyRound-trip ≤ maxMs.
costEstimated USD ≤ maxUSD. Unknown cost fails — an unpriced gateway can’t satisfy a budget.
scriptA QuickJS test script passes (the output is exposed as pm.response.text()).
judgeAn LLM-as-judge scores the output ≥ a pass threshold.
tool-callThe model called the expected tool, and its JSON arguments validate against a JSON Schema (same subset as json-schema) and/or match the case’s expected / reference args.
pairwiseA preference judge prefers the cell output over the case’s reference (head-to-head A/B, with optional position-bias swap).

The judge scorer calls a model of your choosing with the output, the case’s reference, and a rubric, and asks it to return a structured judgement (score 0–1, reasoning, pass) via a tool call. Use any provider for the judge, including a cheaper or local model than the one under test, but never the model under test itself: a judge grading its own output inflates agreement, so the eval refuses to start when a judge or pairwise scorer’s judge is one of the selected models. The judge engine supports multi-criteria weighted rubrics (each criterion scored independently; a gate criterion fails the cell regardless of the weighted score), self-consistency (run the judge N≤5 times and aggregate by median, reporting score variance), and calibration anchors (reference-scored examples that pin the 0–1 scale).

  • tool-call grades function-calling: expose tool definitions to the model and assert it called the right tool with arguments that validate against a schema (or match the case’s expected args). Useful for agent/MCP-style prompts.
  • pairwise is preference judging — instead of an absolute score, a judge picks a winner between the cell output and the case reference. Enable position swap to run both A/B orderings and cancel position bias (a flip-flop is scored as a tie). For model-vs-model ranking across a whole dataset, use the Arena instead.

By default a cell scores the model’s text. Switch the eval’s target to Execute as HTTP request (or GraphQL) and the cell instead: calls the model → parses an HTTP/GraphQL request out of the output (a JSON object, or a fenced ```json block) → executes it through Restura’s real request executor → scores the upstream response. This answers “did the model produce a request that actually works”, not just “did it read correctly”. The executed request goes through the same SSRF guard, redirect policy, and cookie jar as any request you’d send by hand.

The Reports tab shows a leaderboard across a run (pass rate, p50/p95 latency, cost per model) and a regression compare against an earlier run of the same eval config. Eval-run ids are stable per config so re-running and comparing works across sessions. Pass rate excludes “not evaluated” cells (a run with no scorers), so a misconfigured eval reads as neither 0% nor 100%.

Cost is only shown when it is known. A model missing from its provider’s price table (including Gemini and Bedrock, which have none) shows unknown cost — never $0 — so a cost scorer fails rather than passing on a fake zero. Dated snapshot ids (claude-…-20250929, gpt-…-2024-08-06) are priced from their base model’s entry. A stream that ends before the provider’s end marker is reported as a truncated-stream error, not as a completed reply.

Each eval has a pass threshold, 100% of scored cells by default. A run below it reads failed; errored cells count as not passed, and a run with nothing scored reads error. Reports saved before thresholds existed keep their stored status. A run interrupted by quitting the app reads interrupted.

AI Lab keeps the newest 50 eval reports, within a 50 MiB total. If a saved record fails validation on load, only that record is dropped: its raw data is backed up locally and you’re offered an export. A failed report save can be retried or discarded, so it no longer blocks new runs.

Each run can be exported as CSV, JSON, or Markdown. A per-case drill-down opens any case to see every model’s full output, per-scorer detail, and judge per-criterion reasoning side by side — a cross-model diff for that case.

The Arena ranks models head-to-head. Pick a dataset and two or more contestant models plus a judge model; the Arena runs every model pair against every case as a round-robin pairwise comparison (with position-bias swap) and folds the results into an Elo leaderboard and a win-rate matrix. It’s the model-vs-model counterpart to the per-cell pairwise scorer, and the right tool when you want a ranking across many models rather than a pass/fail per cell. Runs are bounded-concurrency and cancellable, and persist to their own history (the arenaRuns store).

A match where a contestant or the judge errors shows as an error and is left out of the Elo and win-rate results, never counted as a tie. A run with some errored matches ends partial and keeps its results; one where every match failed ends error. Stop cancels in-flight calls and is not an error.

An agent suite evaluates a multi-step agent against repeatable tasks rather than a single completion. It records the trajectory and outcome for each trial, applies task-aware grading, and can repeat trials to show reliability rather than treating one run as conclusive. Suites have run-wide step, time, tool, token, cost, and output budgets; cancelling a suite stops provider, grader, and tool work, and late results cannot overwrite a cancelled run.

Desktop suites use OpenAI Chat, Anthropic Messages, OpenRouter, Ollama, Hugging Face, generic OpenAI-compatible, Azure OpenAI, Google Gemini, and Amazon Bedrock providers through the encrypted Electron bridge. Model capabilities start conservative; Restura uses trusted catalog data where available and otherwise requires an explicit user assertion before it enables a capability such as tool calling.

  • Azure OpenAI uses the v1 API at your resource URL (https://{resource}.openai.azure.com) with an api-key; the model is your deployment name.
  • Google Gemini uses the Gemini API with an API key sent as a header, never in the URL. Fetch models lists the chat-capable models.
  • Amazon Bedrock uses the Converse API. Store either a Bedrock API key, or IAM credentials as JSON ({"accessKeyId":…,"secretAccessKey":…,"sessionToken":…}), which are signed with SigV4 in the main process. Pick a region by setting the base URL to https://bedrock-runtime.{region}.amazonaws.com. Enter model ids by hand; the runtime endpoint has no model list.

These three have no static price table, so their cost shows as unknown.

  • Saved HTTP requests run through Restura’s normal request executor and its SSRF, auth, and cancellation boundaries. Only unscripted GET, HEAD, and OPTIONS requests are read-only; all other requests, and any request with executable scripts, require explicit per-call approval.
  • MCP tools reuse a saved MCP profile with fresh desktop-owned sessions, existing SSRF/DNS protections, tool allowlists, cancellation, and approval.
  • Grounding is selected explicitly and recorded with provenance in the trace. Saved reports are bounded and sanitized before persistence or export.
  • Code sandbox. Add { "kind": "sandbox", "sandboxId": "quickjs" } to give the agent a run_code_quickjs tool. It runs JavaScript in a fresh QuickJS WASM runtime per call with the script memory and time caps, and no network, filesystem, timers or pm. Output is bounded. Because it cannot reach the host, calls need no approval. A Docker/Podman provider is planned.
  • Remote agents (A2A). { "kind": "a2a", "endpoint": "https://…" } exposes a remote Agent2Agent agent as one tool, described from its agent card. Calls go through the request executor and its SSRF guard, and each call needs approval. The card’s RPC URL must stay on the endpoint’s origin.

restura agent eval runs versioned suites and portable Agent Bundles in CI. It supports stateless OpenAI Responses, Anthropic Messages, Azure OpenAI, Gemini and Bedrock with environment credentials, deterministic fixture tools, the QuickJS sandbox, and fail-closed baseline gates. Live tools must be explicitly listed in a runtime manifest: HTTP is limited to manifest-listed GET / HEAD / OPTIONS requests, and MCP and A2A sources must be declared read-only (MCP also needs a tool allowlist). The CLI refuses desktop secret handles, suite base-URL overrides and agent judge graders. See CLI — Agent eval for the command contract.

Evals → Export for CLI downloads the current eval as a .eval.json file for restura eval run. The file holds the prompt, dataset, scorers and models, but no API keys: each endpoint names the environment variable the CLI reads its key from. The CLI scores with the same code as the desktop, judges included, and exits 0 only when the share of passing cells meets the eval’s pass threshold. The http-exec target and the Arena stay desktop-only.

Agent telemetry is optional and off by default. It is available only in the desktop AI Lab and the restura agent eval CLI; the web app and Worker never export it. When you turn it on, Restura sends traces to Langfuse (via its OpenTelemetry processor) or to any OTLP/HTTP collector.

  • Metadata only. Exported spans carry allowlisted fields: suite, task and agent identifiers, timing, model and provider names, token and cost totals, tool names and outcomes, and aggregate evaluation state. Prompts, model responses, tool arguments and output, URLs, request bodies, headers, credentials and error text are never exported.
  • Desktop. Configure the destination in the AI Lab telemetry panel. It accepts HTTPS endpoints and local loopback HTTP collectors, and rejects endpoints with query strings, fragments or embedded credentials. Langfuse and OTLP credentials are stored as keychain handles and resolved only in the Electron main process. The desktop app refuses environment-variable (source: "env") credential references; those work only in the CLI.
  • CI. Pass --telemetry-config telemetry.json; in the CLI, credentials must be environment references ({ "source": "env", "name": "LANGFUSE_SECRET_KEY" }), never inline values. See the CLI reference.
  • OpenInference spans. Each trace is a root AGENT span with LLM (model call) and TOOL child spans, using openinference.span.kind and the OpenTelemetry gen_ai.* attributes. See the full attribute list in docs/AGENT_TELEMETRY.md.
  • Best-effort. A failed export is reported locally and never changes a suite’s pass/fail exit code.

The full contract is in docs/AGENT_TELEMETRY.md.

Every outbound model call goes through the Electron AI Lab handler, which applies the same shared SSRF guard as the rest of Restura, with one provider-kind-aware twist: allowLocalhost is enabled only for local providers (Ollama, OpenAI-compatible).

  • Loopback and your LAN. Local providers may reach loopback and RFC 1918 addresses (literals and resolved hostnames alike, pinned to the checked address), independent of the Settings → Security toggles. Link-local 169.254/16, IPv6 link-local, CGNAT, IPv6 unique-local and cloud-metadata endpoints remain blocked, including across redirects and DNS rebind.
  • Secret check. Calls are refused if the prompt still contains a secret. Electron main skips that check only for a local provider on loopback or your LAN; a local-kind provider pointed at a public host is checked like a cloud one. “Send unredacted” overrides it per suite on this machine and is never stored in the suite JSON, so an imported suite cannot enable it. The one exception is the rs.judge script assertion (src/lib/shared/judgeBridge.ts): it always sends with rawMode: true and does its own redaction, controlled by the redactBeforeJudge judge setting, so with that setting off the candidate output reaches the judge unredacted.
  • Cloud providers get no carve-out — they can only reach their public endpoints.
  • The handler also enforces a hard concurrency ceiling independent of the renderer-side cap.
  • http-exec (the execute-and-score target) adds no new outbound path: AI-generated requests run through the standard request executor and the same SSRF guard as user-issued requests. The residual risk is that the model picks the destination — see the caution above and ADR 0023.
  • Desktop — fully wired. The renderer drives model calls over the window.electron.aiLab IPC bridge → electron/main/handlers/ai-lab-handler.ts → shared/protocol/ai. primary path
  • Web / self-hosted — not available; the route shows a desktop-only message. There is no Worker route for AI Lab.