ADR 0032 — Agent harness: evals first, hook-enforced rules
Accepted · 2026-10-25
Context
Section titled “Context”ADR 0021 gave Restura an agent harness of skills, three reviewer subagents, slash commands and one measurement hook. Two weeks of agent-driven delivery (72 PRs) then showed where it fell short:
- Seam bugs got through. PRs that were green individually broke
maintogether: two MCP PRs collapsed a panel, a second auth path overwrote the first, two branches both added{{var}}resolution, and the CLI shipped without parity. Each was invisible in a per-branch review. - Rules were prose. “Never
git add -A”, “subagents don’t push”, “--adminis emergencies only” and “inline snapshots only” lived in CLAUDE.md and agent briefs. A straynode_modulessymlink reachedmain, and--adminbecame routine. - The harness was never measured.
skill-usage.logrecorded invocations, not outcomes. The structured loops (/batch-ship,/ship-check,/fix-until-green) did not appear in the log at all, and each/ship-checkstarted four review agents per branch (about 20 for a five-PR batch). - Context cost. CLAUDE.md had grown to 310 lines, mostly subsystem detail loaded on every turn, and AGENTS.md mirrored it.
- Worktrees accumulated: 73 linked worktrees by 2026-10-25.
Anthropic’s guidance on agent design (“when to fold subagents into your main agent loop”) recommends one loop with progressive disclosure, hard limits in tools and hooks rather than prompts, subagents only where a fresh context pays (parallel work, black-box verification), and task-completion evals before restructuring agents.
Decision
Section titled “Decision”- Evals first.
evals/harness/holds about 10 seam-bug cases, most rebuilt from real incidents as minimal patch fixtures, plus a runner (npm run eval:harness) that drives the review entry point headlessly in throwaway worktrees. Each case runs at least twice and is graded by deterministic checks plus a rubric judge. The output is a per-case flip report, not just an aggregate score. Runs are on demand only, never per-PR CI. No reviewer subagent is folded or removed without a before/after score on this set. - Hard rules are hooks. A
PreToolUseBash guard (.claude/hooks/guard-bash.mjs) denies, with a readable reason:git add -A/./--all;gh pr merge --adminunlessRESTURA_ALLOW_ADMIN=1; andgit push/gh pr createfrom inside.claude/worktrees/. The generated-files guard also blocks*.snapfiles. Worker reports are checked by a script (npm run verify:agent-report) against git, red-first run included, instead of being trusted. Metrics hooks log outcomes (subagent stops, stop-check verdicts), and log names and verdicts only, never content. - Progressive disclosure. CLAUDE.md and AGENTS.md keep only invariants, a routing table and the short playbook (about 100 lines each). Detail loads on demand from
docs/agents/architecture.mdanddocs/agents/delivery.mdand from skills. The old.claude/commands/*became skills with descriptions that state when to use them. - Review the combined result. For multiple branches, correctness, parity and docs review run once on the
integration-checkcombined diff. The security review stays per branch. A/ship-checkREADY marker bound to the head sha letsclaude-code-review.ymlskip its duplicate automatic run. - Worktree lifecycle.
npm run worktree:prune(dry run by default) removes only clean, merged worktrees. It runs at the end of/batch-shipand in/triage-maintenance.
Alternatives considered
Section titled “Alternatives considered”- Fold the parity and docs reviewers into main-loop skills now. Deferred. The decision depends on the eval baseline; folding without one is the regression the guidance warns about.
- Keep rules as prose and add more emphasis. Rejected. The rules were already explicit and were still broken.
- A shared
@restura/coreor an external eval service. Out of scope. The eval runner reusesclaude -pand local worktrees.
Consequences
Section titled “Consequences”- The harness now has a measurable quality signal. Folding a reviewer means comparing the current configuration with the folded one on the same cases.
- This partly supersedes ADR 0021’s “prompt/skill content, not executable gates” consequence: the delivery rules above are now executable hooks for agent sessions. CI remains the gate for humans.
- Eval runs cost model usage, so they run on demand only.
- More tooling to maintain (hooks, three scripts, fixtures). Fixtures are pinned to historical base SHAs, so they don’t drift with
main. --adminmerges now need an explicit, visible override.
Amendment 2026-10-25
Section titled “Amendment 2026-10-25”The eval tooling shipped as follows; the Decision above is unchanged.
- On demand for the gate. The 3-run harness regression starts from a manual dispatch or the
eval-harnesslabel on a same-repo PR (.github/workflows/eval-harness.yml), never on push. Routing is dispatch-only (suite: routing). A same-repo, non-draft, non-bot PR that touchesREVIEW.md,eval-harness.yml, or thecode-reviewskill also starts a non-blocking observe run (one test-split pass, or no model call when no baseline JSON is committed). The workflow is not a required check; it uploads.claude/metrics/evals/. See ADR 0033. - History-free, overlaid harness. The harness under test (
--harness-ref) is overlaid onto each case in a fresh one-commit repo, so the model cannot read git history for the fix. - Method. Cases carry a train/test split. Results report pass@1, pass@k and pass^k with confidence intervals and can be compared against committed baselines (
--save-baseline,--compare). The case count is no longer fixed at about 10. A routing suite (evals/routing/,npm run eval:routing) checks that realistic prompts load the right skill,npm run eval:new-casescaffolds a case from a fix PR, and a build-loop suite (evals/build/,node scripts/build-eval.mjs) covers implementation rather than review. - Governance of changes to the harness is decided in ADR 0033.
References
Section titled “References”- Code:
.claude/hooks/guard-bash.mjs,.claude/hooks/log-subagent.mjs,.claude/settings.json,scripts/verify-agent-report.mjs,scripts/worktree-prune.mjs,scripts/harness-eval.mjs,scripts/routing-eval.mjs,scripts/eval-new-case.mjs,scripts/build-eval.mjs,evals/harness/,evals/routing/,evals/build/,docs/agents/,.claude/skills/{batch-ship,ship-check,triage-maintenance,code-review}/SKILL.md,.github/workflows/claude-code-review.yml,.github/workflows/eval-harness.yml - Related: ADR 0021 (maintenance harness), ADR 0028 (architecture boundaries and ratchets), ADR 0033 (evals gate harness changes)