AIDiveTL;DR and verdict
- Shopify's Helix puts every checkpoint through four gates in order: behaviour tests, UI review, two independent context-isolated reviewers, an engineer's sign-off s1. The post gives no pass rate. We rebuilt gates 1, 3 and 4 as a Claude Code Stop hook on a real repo and counted.
- Our matrix: 50 runs, five tinydb tasks, five configurations, headless Sonnet 5.5. With no gate, 35 of 50 hidden requirements passed (70%). With the acceptance-test gate alone, 42 of 50 (84%).
- The two-reviewer gate as written did not move the pass rate: 34 of 50 alone, 41 of 50 after the tests. A reviewer plus a critic reached 39 of 50.
- Verdict: keep the test gate, with a writer that is not the implementer and tests the agent cannot edit. Expect one false block in six, and give the loop a counter.
- Keep the two-reviewer gate, if you keep it, as advice in the feedback, not as a bar.
- Limits: one repo, one worker model, five tasks, two repetitions per cell. A gap of two points between rows is noise. The 14 points between no gate and the test gate is not.
What the sources say
Helix names the rule itself: if a gate fails, the agent uses the feedback to fix the implementation and runs the check again, and it cannot override a failed check because it thinks the result is good enough s1. Gate 3 is two independent, context-isolated reviewer agents, every finding has to be fixed, and the loop repeats until both approve s1. A logic-only change simply skips the UI review gate s1, which is why our CLI-style library repo has no gate 2. An independent write-up classes the Helix evidence as case-study, first-party and unmeasured, and notes that nothing validates the judge: no false-blocker rate is reported s6.
Anthropic's own guide gives the problem: Claude stops when the work looks done, and without a check it can run, "looks done" is the only signal available s3. The deterministic version is a Stop hook that runs your check as a script and blocks the turn from ending until it passes s3. The guide adds that a reviewer in a fresh subagent context sees only the diff and the criteria, not the reasoning behind the change s3, and subagents do start with a fresh, isolated context window s22.
The hook contract: exit 2 on Stop prevents Claude from stopping s2, and the stderr message is what Claude receives as the reason s2. Any other exit code, exit 1 included, does not block on its own s2. The JSON form is decision: "block" with a required reason s2.
The gate has three exits in its own documentation:
- The cap. After stop hooks have continued the turn eight times in a row, Claude Code overrides the next block and ends the turn. The count resets each time Claude calls a tool, and
CLAUDE_CODE_STOP_HOOK_BLOCK_CAPraises it s2 s19. - The timeout. Command hooks default to 600 seconds, and a hook that reaches its timeout is cancelled, its output discarded, so on most events it renders no decision s2.
- The flag.
stop_hook_activeis true when Claude Code is already continuing because of a stop hook, so a script can avoid blocking on a condition that will never resolve s2.
One practitioner warns that a cheap, low-context auditor "gets things wrong a lot" and blocks good sessions, so it should not get the final say s7. A measured Stop gate on 100 labeled stops blocked 8 turns in 100, 7 of them rightly, with the caveat that the labels came from a model, not by hand s8. We report blocks the same way: right and false.
The scripting flags we relied on: --bare skips hooks, skills, plugins, MCP servers, auto memory and CLAUDE.md s20, --max-turns limits agentic turns in print mode s20, --json-schema returns validated output in the structured_output field s21, and --output-format json carries total_cost_usd, a client-side estimate that can differ from your bill s21.
On gate 3, a 2026 workshop paper measured the shape. On LiveCodeBench: zero-shot 77%, single reviewer 77%, two reviewers 75% on 105 problems, and 34 of 57 on the hard subset s5. The paper calls the two-reviewer result still within noise of single-reviewer, self-refine and zero-shot s5. Its adversarial shape, a reviewer plus a critic that audits the review before the main agent edits, reached 87% s5. On SWE-bench Verified it reached 75.2% against 71.6% zero-shot, at about 4.5x the tokens s5. Paper: s23.
The kit
The bench is tinydb at commit 18d73a1 with 226 passing tests. Five requests typed the way a developer types them (push and pull operations, a between query, a table rename, sorted search, a write cache size), each with five hidden requirements the agent never sees: 25 in all. The worker is headless Sonnet 5.5. Everything below is the harness and the gates. The tinydb checkout, the hidden tests and the task prompts are not included: point it at your own repo and write your own oracle.
The gate:
gate.sh: the Stop hook entry point, a one-line wrapper that runslib/hook.py.gate.sh
#!/usr/bin/env bash
# Stop hook entry point (bash 3.2 OK). All logic lives in lib/hook.py.
# Called as: GATE_RUN=<abs run dir> GATE_CONFIG=<cfg> bash <bench>/gate.sh (stdin = hook input JSON)
HERE="$(cd "$(dirname "$0")" && pwd)"
exec "$HERE/.venv/bin/python" -I "$HERE/lib/hook.py"
lib/hook.py
"""Stop hook: gate 1 (behaviour tests) and/or gate 3 (adversarial review), per GATE_CONFIG.
Env: GATE_RUN (abs run dir, holds meta.json + prompt.txt), GATE_CONFIG (key of configs.json).
stdin: the Stop hook input JSON. Block = exit 2 with the reason on stderr. Allow = exit 0, no output.
Every invocation appends ONE line to $GATE_RUN/gate.jsonl. Nothing but the reason ever reaches the agent.
"""
import json
import os
import sys
import time
import traceback
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from common import BENCH, run_hidden, run_suite, tests_touched, work_diff # noqa: E402
from testwriter import run_gate_tests # noqa: E402
GIVE_UP_AFTER = 6
def tail40(out, room=1400):
t = "\n".join(out.rstrip().splitlines()[-40:])
if len(t) > room:
t = "...\n" + t[-room:]
return t
def main():
t0 = time.time()
run = os.environ["GATE_RUN"]
cfg_name = os.environ["GATE_CONFIG"]
cfg = json.load(open(os.path.join(BENCH, "configs.json")))[cfg_name]
meta = json.load(open(os.path.join(run, "meta.json")))
work, task = meta["work"], meta["task"]
raw = sys.stdin.read()
try:
hin = json.loads(raw) if raw.strip() else {}
except Exception:
hin = {"_unparsed": raw[:500]}
log_path = os.path.join(run, "gate.jsonl")
prev = []
if os.path.exists(log_path):
prev = [json.loads(l) for l in open(log_path) if l.strip()]
n = len(prev) + 1
blocks_before = sum(1 for p in prev if p.get("decision") == "block")
# Snapshot of the work at this stop, before gate 1 runs.
open(os.path.join(run, "snap-%d.diff" % n), "w").write(work_diff(work))
# Ground truth at this moment (silent).
_, hout, hp, hf, hfailed = run_hidden(work, task)
open(os.path.join(run, "hidden-%d.txt" % n), "w").write(hout)
rec = {"ts": time.strftime("%Y-%m-%dT%H:%M:%S"), "n": n, "stop_hook_active": hin.get("stop_hook_active"),
"hook_input_keys": sorted(hin.keys()),
"gate1": {"ran": False}, "gate3": {"ran": False},
"hidden": {"pass": hp, "fail": hf, "failed_reqs": hfailed}}
reason, blocked_by = None, None
if cfg["gate1"]:
# Gate 1 = the repo suite AND the acceptance tests written before the run by the independent test writer
# (run dir gate_tests/, outside the work copy). A writer failure leaves the suite alone (logged).
rc, out, sp, sf = run_suite(work)
open(os.path.join(run, "gate1-%d.txt" % n), "w").write(out)
suite_ok = rc == 0 and sf == 0
gdir = os.path.join(run, "gate_tests")
has_gt = os.path.exists(os.path.join(gdir, "test_%s.py" % task))
g = {"ran": True, "suite_pass": sp, "suite_fail": sf, "tests_touched": tests_touched(work),
"gate_tests_present": has_gt}
gt_ok, gout = True, ""
if has_gt:
grc, gout, gp, gf, gnames = run_gate_tests(gdir, task, work)
open(os.path.join(run, "gatetests-%d.txt" % n), "w").write(gout)
gt_ok = grc == 0 and gf == 0
g.update(gate_tests_pass=gp, gate_tests_fail=gf, gate_tests_failed=gnames)
g["pass"] = suite_ok and gt_ok
rec["gate1"] = g
if not g["pass"]:
blocked_by = "g1"
parts = []
if not gt_ok:
parts.append("The acceptance tests for this change fail:\n" + tail40(gout, 1400 if suite_ok else 800))
if not suite_ok:
parts.append("The repo's test suite is red (python -m pytest -q -x):\n" + tail40(out, 1400 if gt_ok else 550))
reason = "\n\n".join(parts)
if cfg["gate3"] and blocked_by is None:
from review import gate3, reason_text
g3 = gate3(work, open(os.path.join(run, "task.md")).read(), cfg["gate3"])
rec["gate3"] = g3
if not g3["pass"]:
blocked_by = "g3"
reason = reason_text(g3)
if blocked_by is None:
decision = "allow"
elif blocks_before >= GIVE_UP_AFTER:
decision = "give_up"
else:
decision = "block"
rec["blocked_by"] = blocked_by
rec["decision"] = decision
rec["reason"] = (reason or "")[:500]
rec["reason_chars"] = len(reason or "")
rec["wall_s"] = round(time.time() - t0, 1)
with open(log_path, "a") as fh:
fh.write(json.dumps(rec) + "\n")
if n == 1:
json.dump(hin, open(os.path.join(run, "hook-input-1.json"), "w"), indent=2)
if decision == "block":
sys.stderr.write(reason)
sys.exit(2)
sys.exit(0)
if __name__ == "__main__":
try:
main()
except SystemExit:
raise
except Exception:
# A crashed gate must never deadlock a run: log and allow.
run = os.environ.get("GATE_RUN", "/tmp")
with open(os.path.join(run, "gate-errors.txt"), "a") as fh:
fh.write(traceback.format_exc() + "\n")
sys.exit(0)
lib/testwriter.py
"""Gate 1 test writer: an independent headless Sonnet 5.5 that sees ONLY the task prompt (plus read-only access to a
pristine upstream copy) and writes integration-style pytest cases. The file lives OUTSIDE the work copy, in
<out dir>/gate_tests/ next to a copy of hidden/conftest.py (TINYDB_TREE) and hidden/pytest.ini.
CLI: testwriter.py <task id> <out dir> -> writes <out dir>/gate_tests/test_<T>.py and <out dir>/testwriter.json
"""
import json
import os
import shutil
import subprocess
import sys
import tempfile
import time
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from common import BENCH, HIDDEN, PY, counts, env_clean, sh # noqa: E402
MODEL = "claude-sonnet-5-5"
CALL_TIMEOUT = 600
SCHEMA = {"type": "object", "properties": {"file": {"type": "string"}}, "required": ["file"]}
FRAMING = ("You are a senior engineer writing acceptance tests for a change request to tinydb, a small Python document "
"database. You can read the CURRENT repository (before the change) with Read, Grep and Glob to learn the API; "
"you cannot run code. The change is not implemented yet: your tests must describe the requested behaviour.")
ASK = ("Write integration-style pytest cases for this request against the tinydb API, one test per behaviour a careful "
"senior would expect, pushing for edge cases beyond the happy path (missing fields, wrong types, boundaries, "
"persistence, cache, existing behaviour unchanged). Tests only, no implementation. Return the file content.\n"
"Constraints: one self-contained pytest module (field `file`); `tinydb` is importable (the tree under test is put "
"first on sys.path); do not import anything from the repo's tests/ directory or its fixtures; use "
"tinydb.storages.MemoryStorage or pytest's tmp_path for files; import the new API inside each test, not at module "
"level, so one missing name fails its tests instead of the whole module; give every test a descriptive name.")
def write(task, out_dir, attempts=2):
gt = os.path.join(out_dir, "gate_tests")
os.makedirs(gt, exist_ok=True)
shutil.copy(os.path.join(HIDDEN, "conftest.py"), gt)
shutil.copy(os.path.join(HIDDEN, "pytest.ini"), gt)
task_prompt = open(os.path.join(BENCH, "tasks", task + ".md")).read().strip()
prompt = "%s\n\n## The request (as the developer typed it)\n%s\n\n%s" % (FRAMING, task_prompt, ASK)
rec = {"task": task, "attempts": [], "cost_usd": 0.0, "ok": False, "file": None}
pristine = tempfile.mkdtemp(prefix="gb-writer-")
try:
shutil.copytree(os.path.join(BENCH, "upstream"), os.path.join(pristine, "tinydb-repo"), symlinks=True)
cwd = os.path.join(pristine, "tinydb-repo")
for i in range(attempts):
t0 = time.time()
a = {"error": None}
args = ["claude", "-p", prompt, "--model", MODEL, "--tools", "Read,Grep,Glob", "--max-turns", "12",
"--output-format", "json", "--json-schema", json.dumps(SCHEMA), "--permission-mode", "dontAsk",
"--setting-sources", "", "--settings", '{"disableAllHooks":true}', "--strict-mcp-config"]
env = env_clean({"CLAUDE_CODE_DISABLE_AUTO_MEMORY": "1", "CLAUDE_CODE_DISABLE_CLAUDE_MDS": "1"})
try:
p = subprocess.run(args, cwd=cwd, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
stdin=subprocess.DEVNULL, timeout=CALL_TIMEOUT, env=env)
r = json.loads(p.stdout.decode("utf-8", "replace"))
a.update(cost_usd=r.get("total_cost_usd") or 0.0, turns=r.get("num_turns"),
models=sorted((r.get("modelUsage") or {}).keys()), subtype=r.get("subtype"))
so = r.get("structured_output")
src = so.get("file") if isinstance(so, dict) else None
if r.get("is_error") or not src:
a["error"] = "is_error=%s subtype=%s terminal=%s" % (r.get("is_error"), r.get("subtype"),
r.get("terminal_reason"))
else:
compile(src, "test_%s.py" % task, "exec")
if "def test_" not in src:
raise ValueError("no test functions")
open(os.path.join(gt, "test_%s.py" % task), "w").write(src)
rec["ok"], rec["file"] = True, os.path.join(gt, "test_%s.py" % task)
except subprocess.TimeoutExpired:
a["error"] = "timeout after %ss" % CALL_TIMEOUT
except Exception as e:
a["error"] = "%s: %s" % (type(e).__name__, str(e)[:300])
a["wall_s"] = round(time.time() - t0, 1)
rec["attempts"].append(a)
rec["cost_usd"] += a.get("cost_usd") or 0.0
if rec["ok"]:
break
finally:
shutil.rmtree(pristine, ignore_errors=True)
rec["cost_usd"] = round(rec["cost_usd"], 6)
rec["wall_s"] = round(sum(a["wall_s"] for a in rec["attempts"]), 1)
rec["models"] = sorted({m for a in rec["attempts"] for m in a.get("models") or []})
json.dump(rec, open(os.path.join(out_dir, "testwriter.json"), "w"), indent=2)
return rec
def run_gate_tests(gate_dir, task, tree, timeout=300):
"""Generated tests on `tree` (TINYDB_TREE = tree, cwd = tree). Returns (rc, out, passed, failed, failed_names).
Paths to the gate_tests dir are rewritten to `acceptance/` so the agent never learns where the file is."""
f = os.path.join(gate_dir, "test_%s.py" % task)
rc, out = sh([PY, "-m", "pytest", "-q", "--tb=short", "-rfE", "--no-header", "-c", os.path.join(gate_dir, "pytest.ini"),
"--rootdir", gate_dir, f], tree, timeout=timeout, env=env_clean({"TINYDB_TREE": tree}))
out = out.replace(os.path.realpath(gate_dir), "acceptance").replace(gate_dir, "acceptance")
rel = os.path.relpath(gate_dir, tree)
out = out.replace(rel, "acceptance")
import re
names = sorted(set(re.findall(r"(?:FAILED|ERROR) \S*::(test_\w+)", out)))
p, fl = counts(out)
if rc != 0 and fl == 0:
fl = max(1, len(names))
return rc, out, p, fl, names
if __name__ == "__main__":
r = write(sys.argv[1], os.path.abspath(sys.argv[2]))
print(json.dumps({k: r[k] for k in ("task", "ok", "cost_usd", "wall_s", "file")}))
lib/review.py
"""Gate 3 reviewers: headless `claude -p` calls returning schema-validated JSON.
CLI (validation step 4): review.py <work dir> <task id> [indep|adv] -> prints the gate3 record as JSON
"""
import json
import subprocess
import os
import sys
import time
from concurrent.futures import ThreadPoolExecutor
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from common import BENCH, env_clean, sh, work_diff # noqa: E402
MODEL = "claude-sonnet-5-5"
DIFF_CAP = 60000
CALL_TIMEOUT = 600 # seconds per reviewer call; the hook timeout is 1500
FINDING = {"type": "object",
"properties": {"severity": {"type": "string", "enum": ["blocking", "minor"]},
"file": {"type": "string"}, "line": {"type": "integer"}, "text": {"type": "string"}},
"required": ["severity", "file", "line", "text"]}
SCHEMA_REVIEW = {"type": "object",
"properties": {"approve": {"type": "boolean"}, "findings": {"type": "array", "items": FINDING}},
"required": ["approve", "findings"]}
SCHEMA_CHALLENGE = {"type": "object",
"properties": {
"verdicts": {"type": "array", "items": {
"type": "object",
"properties": {"id": {"type": "integer"},
"verdict": {"type": "string", "enum": ["upheld", "rejected"]},
"reason": {"type": "string"}},
"required": ["id", "verdict", "reason"]}},
"added_findings": {"type": "array", "items": FINDING},
"approve": {"type": "boolean"}},
"required": ["verdicts", "added_findings", "approve"]}
FRAMING = ("You are a senior engineer doing a pre-merge code review of a change to tinydb, a small Python "
"document database. You can read the repository with Read, Grep and Glob (cwd = the repo with the "
"change applied). You cannot run code. Below: the task the developer was given, then the full diff "
"(tracked and new files).")
ASK = ("Find defects: requirements not met, edge cases wrong, behaviour broken, tests that do not test the "
"requirement. Report only real defects with file and line. approve=true only when nothing blocks.")
def capped_diff(work):
d = work_diff(work)
if len(d) > DIFF_CAP:
d = d[:DIFF_CAP] + "\n[diff truncated at %d chars]\n" % DIFF_CAP
return d
def call(work, prompt, schema, rid):
"""One headless reviewer. Never raises: an error is recorded and counts as approve (fail open)."""
# Equivalent of --bare that still authenticates with the user's OAuth login (--bare reads
# ANTHROPIC_API_KEY only): no settings sources, hooks disabled, no MCP, no CLAUDE.md, no auto-memory.
args = ["claude", "-p", prompt, "--model", MODEL, "--tools", "Read,Grep,Glob", "--max-turns", "15",
"--output-format", "json", "--json-schema", json.dumps(schema), "--permission-mode", "dontAsk",
"--setting-sources", "", "--settings", '{"disableAllHooks":true}', "--strict-mcp-config"]
env = env_clean({"CLAUDE_CODE_DISABLE_AUTO_MEMORY": "1", "CLAUDE_CODE_DISABLE_CLAUDE_MDS": "1"})
t0 = time.time()
rec = {"id": rid, "approve": True, "findings": [], "cost_usd": 0.0, "turns": None, "wall_s": None, "error": None}
try:
try:
p = subprocess.run(args, cwd=work, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.DEVNULL,
timeout=CALL_TIMEOUT, env=env)
except OSError as e1: # seen once in a smoke run: transient FileNotFoundError on spawn -> one retry
rec["retry_after"] = "%r filename=%s" % (e1, getattr(e1, "filename", None))
time.sleep(3)
p = subprocess.run(args, cwd=work, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.DEVNULL,
timeout=CALL_TIMEOUT, env=env)
out = p.stdout.decode("utf-8", "replace")
try:
r = json.loads(out)
except Exception:
r = None
if not isinstance(r, dict):
rec["error"] = "no json (rc=%s): %s %s" % (p.returncode, out[-300:], p.stderr.decode("utf-8", "replace")[-300:])
else:
rec["cost_usd"] = r.get("total_cost_usd") or 0.0
rec["turns"] = r.get("num_turns")
rec["models"] = sorted((r.get("modelUsage") or {}).keys())
so = r.get("structured_output")
if r.get("is_error") or not isinstance(so, dict):
rec["error"] = "is_error=%s subtype=%s terminal=%s structured_output=%s" % (
r.get("is_error"), r.get("subtype"), r.get("terminal_reason"), type(so).__name__)
else:
rec["raw"] = so
except subprocess.TimeoutExpired:
rec["error"] = "timeout after %ss" % CALL_TIMEOUT
except Exception as e: # pragma: no cover
rec["error"] = "exception: %r filename=%s path_head=%s" % (e, getattr(e, "filename", None), os.environ.get("PATH", "")[:200])
rec["wall_s"] = round(time.time() - t0, 1)
return rec
def norm_findings(lst):
out = []
for f in lst or []:
if isinstance(f, dict):
out.append({"severity": f.get("severity") if f.get("severity") in ("blocking", "minor") else "minor",
"file": str(f.get("file", "")), "line": f.get("line"), "text": str(f.get("text", ""))})
return out
def review_prompt(task_prompt, diff):
return "%s\n\n## Task given to the developer\n%s\n\n## Diff (git diff HEAD, new files included)\n```diff\n%s\n```\n\n%s" % (
FRAMING, task_prompt.strip(), diff, ASK)
def challenge_prompt(task_prompt, diff, a_findings):
listing = "\n".join("[%d] (%s) %s:%s %s" % (i + 1, f["severity"], f["file"], f["line"], f["text"])
for i, f in enumerate(a_findings)) or "(reviewer A reported no findings)"
return ("%s\n\nA first reviewer (A) already reviewed this diff. You are the challenger: for EACH of A's findings "
"decide whether it is a real defect (upheld) or not (rejected: wrong, not required by the task, already "
"handled, or speculative), with a one-sentence reason grounded in the code; check the code before you "
"decide. Then add any real blocking defect A missed in added_findings (file and line). approve=true only "
"when no upheld or added finding blocks.\n\n## Task given to the developer\n%s\n\n## Diff (git diff HEAD, "
"new files included)\n```diff\n%s\n```\n\n## Reviewer A's findings (id in brackets)\n%s\n\n%s" % (
FRAMING, task_prompt.strip(), diff, listing,
"Answer every id in verdicts. Report only real defects."))
def gate3(work, task_prompt, variant):
diff = capped_diff(work)
reviewers, blocking = [], []
if variant == "indep":
prompt = review_prompt(task_prompt, diff)
with ThreadPoolExecutor(max_workers=2) as ex:
recs = list(ex.map(lambda rid: call(work, prompt, SCHEMA_REVIEW, rid), ["A", "B"]))
for rec in recs:
raw = rec.pop("raw", None) or {}
rec["findings"] = norm_findings(raw.get("findings"))
rec["approve_raw"] = raw.get("approve", True) if not rec["error"] else None
rb = [f for f in rec["findings"] if f["severity"] == "blocking"]
rec["approve"] = not rb # effective verdict = no blocking finding (an errored call approves)
blocking += [dict(f, by=rec["id"]) for f in rb]
reviewers.append(rec)
elif variant == "adv":
a = call(work, review_prompt(task_prompt, diff), SCHEMA_REVIEW, "A")
raw = a.pop("raw", None) or {}
a["findings"] = norm_findings(raw.get("findings"))
a["approve_raw"] = raw.get("approve", True) if not a["error"] else None
a["approve"] = not [f for f in a["findings"] if f["severity"] == "blocking"]
b = call(work, challenge_prompt(task_prompt, diff, a["findings"]), SCHEMA_CHALLENGE, "B")
rawb = b.pop("raw", None) or {}
verdicts = {}
for v in rawb.get("verdicts") or []:
if isinstance(v, dict) and isinstance(v.get("id"), int):
verdicts[v["id"]] = {"verdict": v.get("verdict"), "reason": str(v.get("reason", ""))}
b["verdicts"] = [dict(id=k, **verdicts[k]) for k in sorted(verdicts)]
b["findings"] = norm_findings(rawb.get("added_findings"))
b["approve_raw"] = rawb.get("approve", True) if not b["error"] else None
# Only upheld findings of A block. An errored challenger upholds nothing (fail open, logged).
for i, f in enumerate(a["findings"]):
v = verdicts.get(i + 1)
f["challenge"] = v["verdict"] if v else ("error" if b["error"] else "unanswered")
if f["severity"] == "blocking" and v and v["verdict"] == "upheld":
blocking.append(dict(f, by="A", upheld_by="B"))
blocking += [dict(f, by="B") for f in b["findings"] if f["severity"] == "blocking"]
b["approve"] = not blocking
reviewers = [a, b]
else:
raise ValueError(variant)
return {"ran": True, "variant": variant, "reviewers": reviewers, "blocking_findings": len(blocking),
"blocking": blocking, "pass": not blocking,
"cost_usd": round(sum(r.get("cost_usd") or 0 for r in reviewers), 6), "diff_chars": len(diff)}
def reason_text(g3):
lines = ["Code review found %d blocking defect(s):" % g3["blocking_findings"]]
for i, f in enumerate(g3["blocking"]):
lines.append("%d. %s:%s %s" % (i + 1, f["file"], f["line"], " ".join(f["text"].split())))
tail = "Fix every finding, re-run the tests, then stop."
body = "\n".join(lines)
room = 1500 - len(tail) - 2
if len(body) > room:
body = body[:room - 4] + " ..."
return body + "\n" + tail
if __name__ == "__main__":
work, task = sys.argv[1], sys.argv[2]
variant = sys.argv[3] if len(sys.argv) > 3 else "indep"
tp = open(os.path.join(BENCH, "tasks", task + ".md")).read()
g3 = gate3(os.path.abspath(work), tp, variant)
print(json.dumps(g3, indent=2))
if not g3["pass"]:
print("--- reason ---\n" + reason_text(g3))
lib/common.py
"""Shared helpers: work-copy diff, pytest runs and parsing. Used by hook.py, review.py, finalize.py."""
import os
import re
import subprocess
BENCH = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
PY = os.path.join(BENCH, ".venv", "bin", "python")
HIDDEN = os.path.join(BENCH, "hidden")
def sh(args, cwd, timeout=None, env=None):
"""Run a command, capture stdout+stderr together. Returns (rc, text). rc=124 on timeout."""
try:
p = subprocess.run(args, cwd=cwd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
stdin=subprocess.DEVNULL, timeout=timeout, env=env)
return p.returncode, p.stdout.decode("utf-8", "replace")
except subprocess.TimeoutExpired as e:
out = (e.stdout or b"").decode("utf-8", "replace")
return 124, out + "\n[timed out after %ss]" % timeout
def work_diff(work):
"""`git diff HEAD` of tracked files plus every untracked (non-ignored) file as a new-file diff.
Never touches the index (no `git add -N`), so the agent's `git status` is unaffected."""
_, d = sh(["git", "diff", "HEAD", "--no-color", "--", ".", ":(exclude).coverage"], work)
_, others = sh(["git", "ls-files", "--others", "--exclude-standard"], work)
for f in [l for l in others.splitlines() if l.strip()]:
_, nd = sh(["git", "diff", "--no-color", "--no-index", "--", "/dev/null", f], work)
d += nd
return d
def tests_touched(work):
_, mod = sh(["git", "diff", "HEAD", "--name-only", "--", "tests"], work)
_, new = sh(["git", "ls-files", "--others", "--exclude-standard", "--", "tests"], work)
return bool(mod.strip() or new.strip())
def counts(txt):
"""(passed, failed+errors) from a pytest summary line."""
p = re.findall(r"(\d+) passed", txt)
f = re.findall(r"(\d+) failed", txt)
e = re.findall(r"(\d+) errors?\b", txt)
return (int(p[-1]) if p else 0), (int(f[-1]) if f else 0) + (int(e[-1]) if e else 0)
def env_clean(extra=None):
env = {k: v for k, v in os.environ.items() if not k.startswith("CLAUDE") and not k.startswith("GATE_")}
env["PATH"] = os.path.join(BENCH, ".venv", "bin") + os.pathsep + env.get("PATH", "/usr/bin:/bin")
env["PYTHONDONTWRITEBYTECODE"] = "1"
env.update(extra or {})
return env
def run_suite(work, timeout=300):
"""Gate 1 / final suite: the repo's own pytest run (226 tests + whatever the agent wrote).
--no-cov: the repo's pytest.ini adds --cov; the coverage table would fill the 40-line tail."""
rc, out = sh([PY, "-m", "pytest", "-q", "-x", "--no-header", "-p", "no:cacheprovider", "--no-cov"],
work, timeout=timeout, env=env_clean())
p, f = counts(out)
if rc != 0 and f == 0:
f = 1 # collection error, timeout, crash: count as red
return rc, out, p, f
def run_hidden(work, task, timeout=300):
"""Ground truth: hidden tests on the work copy (cwd = work copy, TINYDB_TREE set). Output never reaches the agent."""
rc, out = sh([PY, "-m", "pytest", "-q", "-rA", "-c", os.path.join(HIDDEN, "pytest.ini"), "--rootdir", HIDDEN,
os.path.join(HIDDEN, "test_%s.py" % task)],
work, timeout=timeout, env=env_clean({"TINYDB_TREE": work}))
# Requirement level (T5 R4 is parametrized: one req = several test ids). A req passes iff none of its
# test ids failed or errored; a run that collected nothing counts every req as failed.
reqs = sorted(set(re.findall(r"^def (test_req\d+)", open(os.path.join(HIDDEN, "test_%s.py" % task)).read(), re.M)))
failed = sorted(set(re.findall(r"(?:FAILED|ERROR) \S+::(test_req\d+)", out)))
p_tests, f_tests = counts(out)
if rc != 0 and not failed:
failed = list(reqs)
if p_tests == 0 and f_tests == 0:
failed = list(reqs)
return rc, out, len(reqs) - len(failed), len(failed), failed
lib/finalize.py
"""After the worker exits: final diff, suite, hidden run, result.json and score.json. Prints one line.
Usage: finalize.py <run dir> (re-runnable: rescoring an existing run only reads/rewrites files in it)"""
import json
import os
import re
import sys
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from common import PY, env_clean, run_hidden, sh, work_diff # noqa: E402
from testwriter import run_gate_tests # noqa: E402
MODEL = "claude-sonnet-5-5"
out = os.path.abspath(sys.argv[1])
meta = json.load(open(os.path.join(out, "meta.json")))
work, task, cfg, rep = meta["work"], meta["task"], meta["cfg"], meta["rep"]
def w(name, txt):
open(os.path.join(out, name), "w").write(txt)
def rd(name, default=""):
try:
return open(os.path.join(out, name)).read().strip()
except Exception:
return default
# Final state of the work copy
w("post.diff", work_diff(work))
_, ns = sh(["git", "diff", "HEAD", "--numstat", "--", ".", ":(exclude).coverage"], work)
_, others = sh(["git", "ls-files", "--others", "--exclude-standard"], work)
for f in [l for l in others.splitlines() if l.strip()]:
try:
ns += "%d\t0\t%s\n" % (sum(1 for _ in open(os.path.join(work, f), errors="replace")), f)
except Exception:
pass
w("numstat.txt", ns)
rc, suite = sh([PY, "-m", "pytest", "-q", "--no-header", "-p", "no:cacheprovider", "--no-cov"], work,
timeout=300, env=env_clean())
w("suite.txt", suite)
sp = int((re.findall(r"(\d+) passed", suite) or [0])[-1])
sf = int((re.findall(r"(\d+) failed", suite) or [0])[-1]) + int((re.findall(r"(\d+) errors?\b", suite) or [0])[-1])
if rc != 0 and sf == 0:
sf = 1
_, hout, hp, hf, hfailed = run_hidden(work, task)
w("hidden.txt", hout)
# Worker result line
lines = []
for raw in open(os.path.join(out, "stream.jsonl"), errors="replace") if os.path.exists(os.path.join(out, "stream.jsonl")) else []:
raw = raw.strip()
if raw.startswith("{"):
try:
lines.append(json.loads(raw))
except Exception:
pass
result = next((o for o in reversed(lines) if o.get("type") == "result"), None) or {}
init = next((o for o in lines if o.get("type") == "system" and o.get("subtype") == "init"), {})
res = {k: result.get(k) for k in ("total_cost_usd", "num_turns", "usage", "modelUsage", "is_error", "subtype",
"stop_reason", "terminal_reason", "duration_ms")}
res["final_text"] = (result.get("result") or "")[-3000:]
res["has_result_line"] = bool(result)
json.dump(res, open(os.path.join(out, "result.json"), "w"), indent=2)
# Hook events seen in the stream (which hooks ran)
hook_events = sorted({"%s:%s" % (o.get("hook_event") or o.get("hook_event_name") or "?", o.get("hook_name") or "")
for o in lines if o.get("type") == "system" and str(o.get("subtype", "")).startswith("hook")})
# Gate log
gl = [json.loads(l) for l in open(os.path.join(out, "gate.jsonl")) if l.strip()] if os.path.exists(os.path.join(out, "gate.jsonl")) else []
blocks = [g for g in gl if g.get("decision") == "block"]
by_gate = {"g1": sum(1 for g in blocks if g.get("blocked_by") == "g1"),
"g3": sum(1 for g in blocks if g.get("blocked_by") == "g3")}
right = {k: sum(1 for g in blocks if g.get("blocked_by") == k and g["hidden"]["fail"] > 0) for k in ("g1", "g3")}
false = {k: sum(1 for g in blocks if g.get("blocked_by") == k and g["hidden"]["fail"] == 0) for k in ("g1", "g3")}
rev_cost, rev_findings, rev_errors, rev_models = 0.0, 0, 0, set()
for g in gl:
for r in (g.get("gate3") or {}).get("reviewers") or []:
rev_cost += r.get("cost_usd") or 0
rev_findings += len(r.get("findings") or [])
rev_errors += 1 if r.get("error") else 0
rev_models |= set(r.get("models") or [])
# Gate-1 test writer (gated-1 configs only): cost and the final state of its tests on the work copy
tw = {}
try:
tw = json.load(open(os.path.join(out, "testwriter.json")))
except Exception:
pass
cost_tw = tw.get("cost_usd") or 0.0
rev_models |= set(tw.get("models") or [])
gt_total = gt_fail = None
gdir = os.path.join(out, "gate_tests")
if os.path.exists(os.path.join(gdir, "test_%s.py" % task)):
_, gout, gp, gf, _ = run_gate_tests(gdir, task, work)
w("gatetests-final.txt", gout)
gt_total, gt_fail = gp + gf, gf
mu = result.get("modelUsage") or {}
cw = result.get("total_cost_usd") or 0.0
s = {
"task": task, "cfg": cfg, "rep": rep, "run": meta["name"],
"hidden_pass": hp, "hidden_fail": hf, "failed_reqs": hfailed,
"suite_pass": sp, "suite_fail": sf,
"hook_calls": len(gl), "blocks_total": len(blocks), "blocks_by_gate": by_gate,
"right_blocks": sum(right.values()), "false_blocks": sum(false.values()),
"right_blocks_by_gate": right, "false_blocks_by_gate": false,
"give_up": any(g.get("decision") == "give_up" for g in gl),
"ended_blocked": bool(gl) and gl[-1].get("decision") == "block",
"cost_worker_usd": round(cw, 6), "cost_reviewers_usd": round(rev_cost, 6), "cost_testwriter_usd": round(cost_tw, 6),
"cost_total_usd": round(cw + rev_cost + cost_tw, 6),
"testwriter_ok": tw.get("ok") if tw else None, "gate_tests_total": gt_total, "gate_tests_fail_final": gt_fail,
"turns": result.get("num_turns"), "wall_s": int(rd("wall_s.txt", "0") or 0),
"reviewer_findings_total": rev_findings, "reviewer_errors": rev_errors,
"gate_crashes": rd("gate-errors.txt").count("Traceback"),
"model_ok": init.get("model") == MODEL and bool(mu) and set(mu) <= {MODEL} and rev_models <= {MODEL},
"worker_models": sorted(mu), "reviewer_models": sorted(rev_models),
"is_error": result.get("is_error"), "subtype": result.get("subtype"), "exit_code": rd("exit_code.txt"),
"has_result_line": bool(result), "claude_code_version": init.get("claude_code_version"),
"hook_events_in_stream": hook_events,
"hidden_first_stop": gl[0]["hidden"] if gl else None,
}
json.dump(s, open(os.path.join(out, "score.json"), "w"), indent=2)
print("%s: hidden %d/%d suite %d/%d blocks=%d (g1 %d, g3 %d; right %d, false %d) give_up=%s ended_blocked=%s "
"gate_tests_fail=%s/%s cost=$%.3f (worker %.3f + rev %.3f + writer %.3f) turns=%s wall=%ss model_ok=%s" % (
s["run"], hp, hp + hf, sp, sp + sf, len(blocks), by_gate["g1"], by_gate["g3"], s["right_blocks"],
s["false_blocks"], s["give_up"], s["ended_blocked"], gt_fail, gt_total, s["cost_total_usd"], cw, rev_cost, cost_tw, s["turns"],
s["wall_s"], s["model_ok"]))
The .claude settings are described in HARNESS.md: a settings.json with one Stop command hook (timeout 1500) that calls a one-line stop.sh, plus a permissions.deny list blocking reads of the hook, the settings and the hidden tests. Reviewers and the writer run with an empty settings source and hooks disabled, so the gate never recurses.
HARNESS.md
# gate-bench
What does each of Shopify's Helix gates catch when rebuilt as a Claude Code **Stop hook**, and what does it
cost? Headless Sonnet 5.5 workers implement five tinydb features (T1..T5); a Stop hook runs gate 1
(behaviour tests) and/or gate 3 (adversarial review by headless Sonnet 5.5 reviewers) every time the worker
tries to stop; hidden tests (the oracle, never visible to the agent) record the ground truth at every stop
and at the end. Built 2026-10-08 from `SPEC.md`; upstream, hidden tests, tasks and reference patches copied
from `../verify-loops-bench` (nothing there or in `../haiku-routing-bench` is modified).
## Commands (from this dir)
```bash
./validate.sh # steps 1,3,4 (+ step 2 once its run exists) -> validation.log
INJECT=$PWD/inject/break.patch RUN_TAG=val2 ./run.sh g1 T1 1 # validation step 2 (red start state)
RUN_TAG=smoke ./run.sh g13adv T1 1 # smoke (tagged runs are ignored by analyze.py)
./validate_writer.sh # gate-1 test writer on T1..T5: tests on upstream vs reference (REWRITE=1 to redo)
./gen_matrix.py # matrix.txt (50 runs, already generated)
nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 & # 3 runs in parallel (MATRIX_JOBS=3)
SKIP_DONE=1 nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 & # resume
.venv/bin/python analyze.py # results.json + results.md
.venv/bin/python lib/finalize.py runs/<run> # rescore one run from its work copy
.venv/bin/python lib/review.py work/<run> T1 adv # one gate-3 round by hand
```
Single run: `./run.sh <cfg> <task> <rep>` -> `runs/<task>-<cfg>-r<rep>/`, work copy `work/<same name>/`.
## Layout
| path | what |
|---|---|
| `SPEC.md` | the spec this harness implements |
| `upstream/` | tinydb @ 18d73a1 (git repo) |
| `hidden/` | `test_T1..T6.py` (T6 unused), `conftest.py` (`$TINYDB_TREE` or cwd first on sys.path), `pytest.ini` |
| `tasks/T1..T5.md` + `tasks/README.md` | the task prompts (verbatim) and what each hidden req checks |
| `reference/T1..T5.patch` | reference solutions (validation step 1) |
| `inject/break.patch` | `>=` -> `>` in `Query.__ge__` (one repo test red; validation steps 2-3) |
| `inject/test_pushpull.py` | one thin test, makes the T1 reference copy "green with tests touched" (step 3) |
| `configs.json` | `bare` (no hook), `g1`, `g3` (indep), `g13` (g1 then g3 indep), `g13adv` (g1 then g3 adv) |
| `env.sh` | scrubs inherited `CLAUDE*` vars, default `~/.claude` auth, `CLAUDE_CODE_DISABLE_AUTO_MEMORY=1`, venv on PATH |
| `prep.sh <cfg> <work> <run>` | fresh copy of upstream, `.claude/` in `.git/info/exclude`, hook settings for gated configs |
| `run.sh <cfg> <task> <rep>` | one run (worker + hook + `lib/finalize.py`) |
| `gate.sh` | the Stop hook (bash 3.2 wrapper: `exec .venv/bin/python -I lib/hook.py`) |
| `lib/hook.py` | gate dispatch, snapshot, hidden run, gate 1, gate 3, decision, `gate.jsonl` line |
| `lib/review.py` | gate-3 reviewers (indep / adv), schemas, prompts, reason text; CLI for a manual round |
| `lib/testwriter.py` | gate-1 test writer (before the worker) + `run_gate_tests` (path rewritten to `acceptance/`) |
| `validate_writer.sh`, `validation/writer/<T>/` | writer validation files and their runs on upstream / reference |
| `tasks/` (terse prompts, used), `tasks-explicit/` + `runs-explicit/` | the matrix uses terse prompts; see `tasks/README.md` |
| `lib/common.py` | diff incl. untracked files, suite run, hidden run (requirement level) |
| `lib/finalize.py` | post-run diff/suite/hidden, `result.json`, `score.json` |
| `gen_matrix.py`, `matrix.txt`, `matrix.sh`, `matrix-one.sh`, `matrix.log` | 50-run matrix, 3 in parallel (xargs -P), resumable |
| `analyze.py` -> `results.json`, `results.md` | per config, per task, the five questions |
| `validate.sh`, `VALIDATION.md`, `validation.log` | pre-matrix checks |
| `.venv/` | python 3.14.7, pytest 9.1.1, pytest-cov 7.1.0, pyyaml, ruff 0.16.10 (same pins as haiku-routing-bench) |
### Worker (run.sh)
`claude -p "<T-file verbatim>\n\nWhen you are done, stop and summarise what you changed." --model claude-sonnet-5-5
--output-format stream-json --verbose --include-hook-events --max-turns 60 --permission-mode acceptEdits
--allowedTools <haiku-routing-bench list> --setting-sources project --strict-mcp-config`, cwd = `work/<run>/`, stdin /dev/null.
Gated configs get `work/<run>/.claude/settings.json`:
`{"permissions":{"deny":[...see traps...]},"hooks":{"Stop":[{"hooks":[{"type":"command","command":"bash \"$CLAUDE_PROJECT_DIR/.claude/hooks/stop.sh\"","timeout":1500}]}]}}`
and `.claude/hooks/stop.sh` = `GATE_RUN=<abs run dir> GATE_CONFIG=<cfg> bash <bench>/gate.sh` (the SPEC command, one level down: see traps).
Copies of both land in the run dir (`settings.json`, `stop.sh`).
### Hook (lib/hook.py), one invocation
1. read stdin JSON; n = lines already in `gate.jsonl` + 1; blocks so far from the same file
2. `snap-<n>.diff` = `git diff HEAD` + untracked files (index never touched)
3. hidden tests on the work copy (`TINYDB_TREE` = work copy, cwd = work copy) -> `hidden-<n>.txt`; never shown to the agent
4. gate 1 (if configured) = the repo suite `python -m pytest -q -x --no-header -p no:cacheprovider --no-cov` (300 s)
-> `gate1-<n>.txt` AND the writer's acceptance tests `runs/<run>/gate_tests/test_<T>.py` run with `TINYDB_TREE` = work
copy -> `gatetests-<n>.txt`. Block when either is red. Reason: "The acceptance tests for this change fail:" + last 40
lines (failing test names + assertion output, the gate_tests path rewritten to `acceptance/`), then the suite tail if the
suite is red too; <= 1500 chars. `tests_touched` is still logged but no longer blocks. No writer file (writer failed
twice) -> suite only, logged as `gate_tests_present: false`.
5. gate 3 (if configured and gate 1 passed or absent): see below
6. decision: allow / block / give_up (a would-be block when 6 blocks already happened -> allow, logged `give_up`).
`stop_hook_active` is logged, never used. Block = exit 2 + reason on stderr. A crash of the hook itself is
logged to `gate-errors.txt` and allows (counted as `gate_crashes` in score.json).
### Gate 1 test writer (lib/testwriter.py, run.sh before the worker, configs g1/g13/g13adv)
`claude -p` Sonnet 5.5 with the reviewer isolation flags, `--tools Read,Grep,Glob --max-turns 12 --json-schema
{"file": string}`, cwd = a pristine temp copy of upstream (deleted after). Prompt = framing + `tasks/<T>.md` + the
coordinator's instruction ("Write integration-style pytest cases for this request against the tinydb API, one test per
behaviour a careful senior would expect, pushing for edge cases beyond the happy path (missing fields, wrong types,
boundaries, persistence, cache, existing behaviour unchanged). Tests only, no implementation. Return the file content.")
+ mechanics (self-contained module, imports of the new API inside each test, MemoryStorage/tmp_path). The file must
compile and contain `def test_`, else one retry. Output `runs/<run>/gate_tests/` (+ hidden's conftest.py with
`TINYDB_TREE`, pytest.ini) and `testwriter.json` (cost, turns, attempts). Fixed per-run cost, outside the worker's wall.
### Gate 3 (lib/review.py)
Each reviewer: `claude -p "<prompt>" --model claude-sonnet-5-5 --tools Read,Grep,Glob --max-turns 15 --output-format json
--json-schema <schema> --permission-mode dontAsk --setting-sources "" --settings '{"disableAllHooks":true}' --strict-mcp-config`
with `CLAUDE_CODE_DISABLE_CLAUDE_MDS=1 CLAUDE_CODE_DISABLE_AUTO_MEMORY=1`, all inherited `CLAUDE*` vars dropped, cwd = work copy,
600 s timeout. Prompt = framing + task prompt (T-file) + diff (tracked + untracked, cap 60k chars) + the SPEC's "Find defects..."
sentence. `structured_output` is read from the JSON result; `total_cost_usd`, `num_turns`, wall and models are recorded.
- `indep`: reviewers A and B, same prompt, run in parallel, no shared context. Blocking findings = union. Pass when both approve.
- `adv`: A as above; challenger B gets A's numbered findings + the diff, answers `upheld|rejected` + reason per id, may add
findings (`added_findings`). Blocking = A's blocking findings B upheld + B's added blocking findings. B always runs (same
2-call budget as indep).
- Effective `approve` = "no blocking finding" (the raw `approve` is logged as `approve_raw`; a reviewer saying approve=false with
only minor findings does not block). A reviewer call that errors approves (fail open) with `error` recorded; an errored
challenger upholds nothing.
- Reason = "Code review found N blocking defect(s):" + numbered `file:line text` + "Fix every finding, re-run the tests, then stop.", <= 1500 chars.
### Per run (`runs/<task>-<cfg>-r<rep>/`)
`stream.jsonl`, `stderr.txt`, `exit_code.txt`, `wall_s.txt` (includes hook time: the hook runs inside the session),
`gate.jsonl`, `snap-<n>.diff`, `hidden-<n>.txt`, `gate1-<n>.txt`, `hook-input-1.json`, `post.diff`, `numstat.txt`,
`suite.txt` (full suite, no -x), `hidden.txt`, `gate_tests/`, `testwriter.json`, `gatetests-<n>.txt`, `gatetests-final.txt`, `result.json`, `score.json`, `meta.json`, `task.md`, `prompt.txt`,
`settings.json` + `stop.sh` (gated configs).
`score.json`: task, cfg, rep, hidden_pass/hidden_fail/failed_reqs (**requirement level**: 5 reqs per task), suite_pass/fail,
hook_calls, blocks_total, blocks_by_gate {g1,g3}, right_blocks / false_blocks (+ `_by_gate`), give_up, ended_blocked,
cost_worker_usd, cost_reviewers_usd, cost_testwriter_usd, cost_total_usd (all three), testwriter_ok, gate_tests_total,
gate_tests_fail_final, turns, wall_s, reviewer_findings_total, reviewer_errors, gate_crashes,
model_ok (worker init model + every modelUsage key + every reviewer model == claude-sonnet-5-5), hook_events_in_stream,
hidden_first_stop.
## Matrix
5 tasks x 5 configs x 2 reps = 50 runs. Rep 1 config blocks bare, g1, g3, g13, g13adv (tasks rotated per block), rep 2 the
reverse block order. Up to 3 runs at once, each in its own work copy (`matrix-one.sh` via `xargs -P 3`, bash 3.2 has no
`wait -n`). `matrix.log`: one `start` and one result line per run.
## Traps found
- **Gate 1 as first built passed by construction** (it only re-ran the implementer's own tests: `T2-g1-r1` one allow, hidden
R4 missed). Redesigned around an independent test writer (above). The writer hedges where the terse prompt is silent
(accepts "raises or no-op"), so it cannot recover unstated requirements; per-task overlap is in VALIDATION.md.
- **A blocked agent hunts for the tests.** After a gate-1 block the smoke agent ran `cat .claude/hooks/stop.sh; find / -name
test_T1.py` (denied only because `find` is not on the allow-list; `cat` is, and `stop.sh` names the bench dir, from where
`hidden/` is one `cat` away). Gated configs' `.claude/settings.json` now carries `permissions.deny`: Read of `./.claude/**`,
the bench's `hidden/ runs/ validation/ reference/`, the two sibling benches; Bash patterns containing `.claude/hooks`,
`.claude/settings`, `../hidden`, `../runs`, `gate_tests`, the bench sub-paths, `find *`. Probed with a Haiku session: all
five escape attempts (cat, Read, ls, python open) denied, in-tree reads fine. Bash rules match command text, not a security
boundary. Bare runs have no settings file (bare r1 ran before this change; bare has no motive to hunt).
Reviewers and the writer run with `--setting-sources ""`, so these rules do not apply to them.
- **Transient `FileNotFoundError` spawning a reviewer** inside the hook (once, both reviewers of one call, wall 0.0 s;
never reproduced by hand). Reviewer calls now retry once after 3 s; the error message records `filename`.
- **`--bare` cannot authenticate here.** It reads `ANTHROPIC_API_KEY` / `apiKeyHelper` only, never OAuth or the keychain;
this account is OAuth (`~/.claude`), so a `--bare` reviewer returns `terminal_reason: api_error`, cost 0. Used the
documented equivalents instead: `--setting-sources ""` + `--settings '{"disableAllHooks":true}'` + `--strict-mcp-config` +
`CLAUDE_CODE_DISABLE_CLAUDE_MDS=1` + `CLAUDE_CODE_DISABLE_AUTO_MEMORY=1` + `--tools Read,Grep,Glob`. The reviewers' cwd is the
work copy, whose `.claude/settings.json` carries the Stop hook: without `--setting-sources ""`/`disableAllHooks` a reviewer
would run the gate recursively. Verified: a dummy Stop hook in the probe copy never fired.
- **Inherited session env.** A matrix launched from inside Claude Code inherits `CLAUDE_EFFORT=medium`,
`CLAUDE_CODE_SUBAGENT_MODEL=opus`, `CLAUDE_CODE_SESSION_ID`, `CLAUDECODE`, `CLAUDE_CODE_CHILD_SESSION`, messaging tokens.
`env.sh` unsets every `CLAUDE*` var before setting its own; reviewers get a scrubbed env too (the hook process itself
inherits the worker's `CLAUDE_*` vars, e.g. the worker's session id).
- **The feedback line leaks the hook command.** Claude Code prefixes each block with the command line:
`Stop hook feedback:\n[<command>]: <reason>`. With the SPEC command inline the agent would read `GATE_CONFIG=g13adv` and
`gate-bench/gate.sh`. The settings command is therefore `bash "$CLAUDE_PROJECT_DIR/.claude/hooks/stop.sh"`, a one-line
wrapper holding the SPEC command. The agent can still `cat` it (it lives in the work copy) but it is not pushed at it.
- **Stop hook input keys (2.1.293):** `background_tasks, cwd, effort ({"level":"medium"}), hook_event_name,
last_assistant_message, permission_mode, prompt_id, session_crons, session_id, stop_hook_active, transcript_path`.
- **8-cap resets on tool calls.** Per the docs the consecutive-continuation count resets each time Claude calls a tool, so the
built-in cap only bites an agent that answers blocks with text alone. Our give-up (6 blocks per run, counted from
`gate.jsonl`, tool calls or not) fires first in every case.
- **Red start state: the agent refuses to fix code it did not write.** Validation step 2 (T1 + an uncommitted `>=` -> `>`
edit) shows Sonnet 5.5 implementing T1, then answering 6 consecutive gate-1 blocks with "the red test is your pre-existing
uncommitted edit, not my change" without touching it, until give_up. The hook mechanics are validated; the behaviour is a
finding in itself (gate 1 cannot force a fix the agent considers out of scope).
- **pytest-cov.** The repo's `pytest.ini` adds `--cov tinydb`; gate 1 adds `--no-cov` so the 40-line tail holds the failure,
not the coverage table (and no `.coverage` file is written). `.coverage` is gitignored anyway.
- **T5 R4 is parametrized** (several test ids, one requirement). hidden counts are per requirement (5 per task), a req
passes only when all its ids pass. `validate.sh` step 1: T5 upstream passes 1 req (R3, the regression guard) as designed.
- **Diff without touching the index.** `git add -N` would change what the agent sees in `git status`; untracked files are
diffed with `git diff --no-index /dev/null <f>` instead. `.claude/` is in `.git/info/exclude`, so the hook config never
shows in any diff or in `git status`.
- **Worker Bash denials.** The 2.1.293 permission layer denies some agent commands with "Contains brace with quote character
(expansion obfuscation)" even under acceptEdits + allow-list; agents route around it (visible as `permission_denied` events).
- **Reviewers on a correct solution still flag "blocking".** On the T1 reference patch, reviewer A marked "pull does not check
the field is a list" (not required by the task) as blocking; indep therefore blocks, the adv challenger rejected it. This is
exactly what false_blocks measures (hidden all green at that stop).
- **Shared plan meter.** Workers and reviewers bill the user's plan quota; `total_cost_usd` is the list-price equivalent the
CLI reports, not a bill. Nothing else heavy should run on the account during the matrix.
- **rtk rewrites interactive `grep`/`diff`** in the launching Claude session only; scripts here call binaries directly
(`/usr/bin/diff`/`cmp` for verification diffs).
- **CLI version:** `claude --version` = 2.1.293 (Claude Code); flags checked in `claude --help`: `--json-schema`, `--bare`, `--tools`,
`--permission-mode` (choices acceptEdits, auto, bypassPermissions, manual, dontAsk, plan), `--setting-sources`,
`--include-hook-events`. `--max-turns` is accepted though not listed in `--help`.
## Matrix launch (2026-10-08 16:33:54, after the gate-1 redesign)
`cd /Users/thomas/dev/gate-bench && SKIP_DONE=1 nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 &` -> PID 71317, 3 lanes.
The 5 bare r1 runs (terse prompts) are kept and skipped; 45 runs to go. Expected wall ~40-70 min: gated runs so far take
~30-70 s of worker wall + ~35-40 s of test writer (before the worker) + 1-35 s per hook call, more with blocks.
Then `.venv/bin/python analyze.py`.
The runner:
run.sh: one run, worker plus hook plus scoring.run.sh
#!/usr/bin/env bash
# One run: fresh work copy, headless Sonnet 5.5 worker, Stop hook per config, final scoring.
# Usage: run.sh <cfg> <task> <rep> cfg: bare|g1|g3|g13|g13adv task: T1..T5
# RUN_TAG=x -> runs/<task>-<cfg>-r<rep>-x (smoke runs stay out of the matrix)
# INJECT=<patch> -> git apply it to the work copy before the worker starts (validation: red start state)
set -uo pipefail
CFG="$1"; TASK="$2"; REP="${3:-1}"
HERE="$(cd "$(dirname "$0")" && pwd)"
. "$HERE/env.sh"
PY="$HERE/.venv/bin/python"
MAX_TURNS="${BENCH_MAX_TURNS:-60}"
NAME="${TASK}-${CFG}-r${REP}${RUN_TAG:+-$RUN_TAG}"
OUT="$HERE/runs/$NAME"; W="$HERE/work/$NAME"
rm -rf "$OUT"; mkdir -p "$OUT"
"$HERE/prep.sh" "$CFG" "$W" "$OUT" > "$OUT/prep.txt" 2>&1 || { echo "$NAME: prep failed"; cat "$OUT/prep.txt"; exit 2; }
if [ -n "${INJECT:-}" ]; then ( cd "$W" && git apply "$INJECT" ) || { echo "$NAME: inject failed"; exit 2; }; fi
cp "$HERE/tasks/$TASK.md" "$OUT/task.md"
printf '%s\n\nWhen you are done, stop and summarise what you changed.\n' "$(cat "$HERE/tasks/$TASK.md")" > "$OUT/prompt.txt"
printf '{"task":"%s","cfg":"%s","rep":%s,"work":"%s","name":"%s"}\n' "$TASK" "$CFG" "$REP" "$W" "$NAME" > "$OUT/meta.json"
# Gate 1 configs: the independent test writer runs BEFORE the worker (fixed per-run cost), sees only tasks/<T>.md,
# writes $OUT/gate_tests/test_<T>.py outside the work copy.
if "$PY" -c "import json,sys; sys.exit(0 if json.load(open('$HERE/configs.json'))['$CFG']['gate1'] else 1)"; then
"$PY" "$HERE/lib/testwriter.py" "$TASK" "$OUT" > "$OUT/testwriter.out" 2>&1
fi
[ -f "$W/.claude/settings.json" ] && cp "$W/.claude/settings.json" "$OUT/settings.json" && cp "$W/.claude/hooks/stop.sh" "$OUT/stop.sh"
# Same allow-list as haiku-routing-bench/run.sh (verify-loops-bench's + Bash(ruff:*)).
TOOLS="Read,Edit,Write,MultiEdit,Glob,Grep,Skill,TodoWrite,Bash(uv:*),Bash(python:*),Bash(python3:*),Bash(pytest:*),Bash(ruff:*),Bash(git status:*),Bash(git diff:*),Bash(git log:*),Bash(ls:*),Bash(cat:*),Bash(mkdir:*),Bash(mktemp:*),Bash(head:*),Bash(tail:*),Bash(grep:*),Bash(wc:*)"
cd "$W"
START=$(date +%s)
claude -p "$(cat "$OUT/prompt.txt")" --model claude-sonnet-5-5 \
--output-format stream-json --verbose --include-hook-events --max-turns "$MAX_TURNS" \
--permission-mode acceptEdits --allowedTools "$TOOLS" \
--setting-sources project --strict-mcp-config \
< /dev/null > "$OUT/stream.jsonl" 2> "$OUT/stderr.txt"
echo "$?" > "$OUT/exit_code.txt"
END=$(date +%s)
echo "$((END-START))" > "$OUT/wall_s.txt"
"$PY" "$HERE/lib/finalize.py" "$OUT"
prep.sh
#!/usr/bin/env bash
# Fresh work copy of upstream tinydb (git clean at HEAD) + the Stop hook settings for gated configs.
# Usage: prep.sh <cfg> <work dir> <run dir>
set -euo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"
CFG="$1"; W="$2"; RUN="$3"
rm -rf "$W"; mkdir -p "$(dirname "$W")"
cp -R "$HERE/upstream" "$W"
cd "$W"
/usr/bin/find . -name __pycache__ -type d -prune -exec rm -rf {} +
rm -f .coverage
# .claude/ is excluded locally (not committed, not in .gitignore): `git status` stays clean,
# the hook config never shows up in the diffs the reviewers and the scorer read.
echo ".claude/" >> .git/info/exclude
git update-index -q --refresh || true
if [ "$CFG" != "bare" ]; then
# The settings command is a neutral wrapper: Claude Code prefixes every "Stop hook feedback" with the
# hook's command line, so the SPEC command (GATE_RUN=... GATE_CONFIG=<cfg> bash <bench>/gate.sh) would
# show the agent the config name and the bench path. The wrapper holds that exact command instead.
mkdir -p .claude/hooks
printf '#!/bin/bash\nGATE_RUN=%s GATE_CONFIG=%s bash %s/gate.sh\n' "$RUN" "$CFG" "$HERE" > .claude/hooks/stop.sh
"$HERE/.venv/bin/python" - "$HERE" > .claude/settings.json <<'PY'
import json
import sys
bench = sys.argv[1]
cmd = 'bash "$CLAUDE_PROJECT_DIR/.claude/hooks/stop.sh"'
# Once a gate blocks, the agent hunts for the tests behind it (seen in a smoke: `cat .claude/hooks/stop.sh;
# find / -name test_T1.py`). The oracle and the gate tests live under the bench: deny the usual forms.
# Read rules cover Read/Grep/Glob; Bash rules match the command text (not a security boundary).
deny = ["Read(./.claude/**)", "Read(/%s/hidden/**)" % bench, "Read(/%s/runs/**)" % bench,
"Read(/%s/validation/**)" % bench, "Read(/%s/reference/**)" % bench,
"Read(//Users/thomas/dev/verify-loops-bench/**)", "Read(//Users/thomas/dev/haiku-routing-bench/**)",
"Bash(*.claude/hooks*)", "Bash(*.claude/settings*)", "Bash(*gate-bench/hidden*)", "Bash(*../hidden*)",
"Bash(*gate-bench/runs*)", "Bash(*../runs*)", "Bash(*gate_tests*)", "Bash(*gate-bench/reference*)",
"Bash(*gate-bench/validation*)", "Bash(*verify-loops-bench*)", "Bash(*haiku-routing-bench*)", "Bash(find *)"]
print(json.dumps({"permissions": {"deny": deny},
"hooks": {"Stop": [{"hooks": [{"type": "command", "command": cmd, "timeout": 1500}]}]}}, indent=2))
PY
fi
test -z "$(git status --porcelain)" || { echo "work copy not clean"; git status --porcelain; exit 1; }
env.sh
# Sourced by run.sh, gate.sh and validate.sh.
# 1. Scrub every CLAUDE* variable inherited from a launching Claude Code session
# (CLAUDE_EFFORT, CLAUDE_CODE_SUBAGENT_MODEL, CLAUDE_CODE_SESSION_ID, CLAUDECODE,
# CLAUDE_CODE_CHILD_SESSION, ...): they would silently change the worker's effort,
# subagent model and session bookkeeping.
for _v in $(env | sed -n 's/^\(CLAUDE[A-Za-z0-9_]*\)=.*/\1/p'); do unset "$_v"; done
unset _v
# 2. Auth: the user's default config dir (~/.claude, OAuth). See README traps (--bare cannot auth).
unset CLAUDE_CONFIG_DIR
# 3. Fresh work copies at the same path must not share auto-memory across reps.
export CLAUDE_CODE_DISABLE_AUTO_MEMORY=1
# 4. No bytecode in the shared hidden/ dir when 3 runs execute hidden tests in parallel.
export PYTHONDONTWRITEBYTECODE=1
# 5. The bench venv first: `python -m pytest` inside the work copy uses it.
GATE_BENCH="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
export GATE_BENCH
export PATH="$GATE_BENCH/.venv/bin:$PATH"
configs.json
{
"bare": {"gate1": false, "gate3": null, "desc": "no hook"},
"g1": {"gate1": true, "gate3": null, "desc": "gate 1 (behaviour tests) only"},
"g3": {"gate1": false, "gate3": "indep", "desc": "gate 3, two independent reviewers"},
"g13": {"gate1": true, "gate3": "indep", "desc": "gate 1 then gate 3 indep (Shopify order)"},
"g13adv": {"gate1": true, "gate3": "adv", "desc": "gate 1 then gate 3 reviewer + challenger"}
}
gen_matrix.py
#!/usr/bin/env python3
"""Write matrix.txt: "<cfg> <task> <rep>" lines. Rep 1: configs bare,g1,g3,g13,g13adv as contiguous blocks,
tasks rotated per block; rep 2: same blocks in reverse config order (ABBA over time)."""
CFGS = ["bare", "g1", "g3", "g13", "g13adv"]
TASKS = ["T1", "T2", "T3", "T4", "T5"]
lines = []
for rep, order in ((1, CFGS), (2, CFGS[::-1])):
lines.append("# rep %d" % rep)
for i, c in enumerate(order):
k = (i + rep - 1) % len(TASKS)
for t in TASKS[k:] + TASKS[:k]:
lines.append("%s %s %d" % (c, t, rep))
open("matrix.txt", "w").write("\n".join(lines) + "\n")
print(sum(1 for l in lines if not l.startswith("#")), "runs")
matrix.txt
# rep 1
bare T1 1
bare T2 1
bare T3 1
bare T4 1
bare T5 1
g1 T2 1
g1 T3 1
g1 T4 1
g1 T5 1
g1 T1 1
g3 T3 1
g3 T4 1
g3 T5 1
g3 T1 1
g3 T2 1
g13 T4 1
g13 T5 1
g13 T1 1
g13 T2 1
g13 T3 1
g13adv T5 1
g13adv T1 1
g13adv T2 1
g13adv T3 1
g13adv T4 1
# rep 2
g13adv T2 2
g13adv T3 2
g13adv T4 2
g13adv T5 2
g13adv T1 2
g13 T3 2
g13 T4 2
g13 T5 2
g13 T1 2
g13 T2 2
g3 T4 2
g3 T5 2
g3 T1 2
g3 T2 2
g3 T3 2
g1 T5 2
g1 T1 2
g1 T2 2
g1 T3 2
g1 T4 2
bare T1 2
bare T2 2
bare T3 2
bare T4 2
bare T5 2
matrix.sh
#!/usr/bin/env bash
# Run an ordered list of "<cfg> <task> <rep>" lines, up to MATRIX_JOBS (default 3) at a time, each run in
# its own work copy. Blank lines and #comments skipped. Appends to matrix.log. Resumable: SKIP_DONE=1.
# Usage: nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 &
set -uo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"; cd "$HERE"
LIST="${1:-matrix.txt}"; JOBS="${MATRIX_JOBS:-3}"
mkdir "$HERE/.matrix.lock" 2>/dev/null || { echo "another matrix.sh is running (remove .matrix.lock if stale)" >&2; exit 1; }
trap 'rmdir "$HERE/.matrix.lock"' EXIT
echo "=== $(date '+%F %T') start $LIST jobs=$JOBS pid=$$" >> matrix.log
grep -vE '^[[:space:]]*(#|$)' "$LIST" | xargs -P "$JOBS" -L 1 ./matrix-one.sh
echo "=== $(date '+%F %T') done $LIST" >> matrix.log
matrix-one.sh
#!/usr/bin/env bash
# One matrix cell (called by matrix.sh through xargs). Args: <cfg> <task> <rep>
HERE="$(cd "$(dirname "$0")" && pwd)"; cd "$HERE"
CFG="$1"; TASK="$2"; REP="$3"
if [ "${SKIP_DONE:-0}" = 1 ] && [ -f "runs/${TASK}-${CFG}-r${REP}/score.json" ]; then
echo "$(date '+%F %T') skip ${TASK}-${CFG}-r${REP}" >> matrix.log; exit 0
fi
echo "$(date '+%F %T') start ${TASK}-${CFG}-r${REP}" >> matrix.log
line=$(./run.sh "$CFG" "$TASK" "$REP" < /dev/null 2>&1 | tail -1)
echo "$(date '+%F %T') $line" >> matrix.log
echo "$line"
analyze.py
#!/usr/bin/env python3
"""Aggregate runs/*/score.json (matrix runs only: <task>-<cfg>-r<rep>, no tag) -> results.json + results.md."""
import glob
import json
import os
import re
HERE = os.path.dirname(os.path.abspath(__file__))
CFGS = ["bare", "g1", "g3", "g13", "g13adv"]
TASKS = ["T1", "T2", "T3", "T4", "T5"]
rows = []
for p in sorted(glob.glob(os.path.join(HERE, "runs", "*", "score.json"))):
if re.fullmatch(r"T\d-[a-z0-9]+-r\d+", os.path.basename(os.path.dirname(p))):
rows.append(json.load(open(p)))
def mean(xs):
xs = [x for x in xs if x is not None]
return round(sum(xs) / len(xs), 4) if xs else None
def agg(rs):
if not rs:
return None
n = len(rs)
hp = sum(r["hidden_pass"] for r in rs)
ht = sum(r["hidden_pass"] + r["hidden_fail"] for r in rs)
rb = {g: sum(r["right_blocks_by_gate"][g] for r in rs) for g in ("g1", "g3")}
fb = {g: sum(r["false_blocks_by_gate"][g] for r in rs) for g in ("g1", "g3")}
return {
"runs": n, "hidden_reqs_passed": hp, "hidden_reqs_total": ht, "hidden_rate": round(hp / ht, 4) if ht else None,
"runs_fully_passing": sum(1 for r in rs if r["hidden_fail"] == 0),
"suite_green_runs": sum(1 for r in rs if r["suite_fail"] == 0),
"mean_cost_total_usd": mean([r["cost_total_usd"] for r in rs]),
"mean_cost_worker_usd": mean([r["cost_worker_usd"] for r in rs]),
"mean_cost_reviewers_usd": mean([r["cost_reviewers_usd"] for r in rs]),
"mean_cost_testwriter_usd": mean([r.get("cost_testwriter_usd") or 0 for r in rs]),
# per-gate cost: gate 1 = the test writer (its tests then run locally for free), gate 3 = the reviewers
"mean_cost_gate1_usd": mean([r.get("cost_testwriter_usd") or 0 for r in rs]),
"mean_cost_gate3_usd": mean([r["cost_reviewers_usd"] for r in rs]),
"gate_tests_total": sum(r.get("gate_tests_total") or 0 for r in rs),
"gate_tests_fail_final": sum(r.get("gate_tests_fail_final") or 0 for r in rs),
"mean_wall_s": mean([r["wall_s"] for r in rs]), "mean_turns": mean([r["turns"] for r in rs]),
"blocks_per_run": round(sum(r["blocks_total"] for r in rs) / n, 3),
"blocks_by_gate": {g: sum(r["blocks_by_gate"][g] for r in rs) for g in ("g1", "g3")},
"right_blocks": rb, "false_blocks": fb,
"false_block_rate": {g: (round(fb[g] / (rb[g] + fb[g]), 3) if rb[g] + fb[g] else None) for g in ("g1", "g3")},
"give_ups": sum(1 for r in rs if r["give_up"]), "ended_blocked": sum(1 for r in rs if r["ended_blocked"]),
"reviewer_findings": sum(r["reviewer_findings_total"] for r in rs),
"reviewer_errors": sum(r.get("reviewer_errors", 0) for r in rs),
"model_ok": all(r["model_ok"] for r in rs),
"missed_reqs": sorted(set("%s:%s" % (r["task"], q) for r in rs for q in r["failed_reqs"])),
}
by_cfg = {c: agg([r for r in rows if r["cfg"] == c]) for c in CFGS}
by_task = {t: {c: agg([r for r in rows if r["task"] == t and r["cfg"] == c]) for c in CFGS} for t in TASKS}
by_task_all = {t: agg([r for r in rows if r["task"] == t]) for t in TASKS}
def rate(c):
return (by_cfg.get(c) or {}).get("hidden_rate")
def d(a, b):
return round(a - b, 4) if a is not None and b is not None else None
answers = {
"q1_gate1_vs_bare": {"bare": rate("bare"), "g1": rate("g1"), "delta": d(rate("g1"), rate("bare"))},
"q2_gate3_over_g1": {"g1": rate("g1"), "g13": rate("g13"), "delta": d(rate("g13"), rate("g1")),
"g3_alone": rate("g3"), "g3_vs_bare": d(rate("g3"), rate("bare"))},
"q3_adv_vs_indep": {"g13": rate("g13"), "g13adv": rate("g13adv"), "delta": d(rate("g13adv"), rate("g13")),
"false_blocks_g3": {c: (by_cfg.get(c) or {}).get("false_blocks", {}).get("g3") for c in ("g13", "g13adv")}},
"q4_false_block_rate": {c: (by_cfg.get(c) or {}).get("false_block_rate") for c in CFGS},
"q4_gate1_false_blocks": {c: {"false": (by_cfg.get(c) or {}).get("false_blocks", {}).get("g1"),
"right": (by_cfg.get(c) or {}).get("right_blocks", {}).get("g1")}
for c in ("g1", "g13", "g13adv")},
"q5_cost": {c: {k: (by_cfg.get(c) or {}).get(k) for k in ("mean_cost_total_usd", "mean_cost_worker_usd",
"mean_cost_gate1_usd", "mean_cost_gate3_usd", "mean_wall_s")} for c in CFGS},
}
json.dump({"n_runs": len(rows), "by_cfg": by_cfg, "by_task": by_task, "by_task_all": by_task_all,
"answers": answers, "runs": rows}, open(os.path.join(HERE, "results.json"), "w"), indent=2)
L = ["# gate-bench results", "", "%d runs scored (matrix runs only)." % len(rows), "",
"## Per config", "",
"| cfg | runs | hidden reqs | fully passing | suite green | mean $ total (worker + g1 writer + g3 reviewers) | mean wall s | blocks/run | g1 right/false | g3 right/false | give-ups | ended blocked | model ok |",
"|---|---|---|---|---|---|---|---|---|---|---|---|---|"]
for c in CFGS:
a = by_cfg[c]
if not a:
L.append("| %s | 0 |" % c); continue
L.append("| %s | %d | %d/%d (%.0f%%) | %d | %d | %.3f (%.3f + %.3f + %.3f) | %.0f | %.2f | %d/%d | %d/%d | %d | %d | %s |" % (
c, a["runs"], a["hidden_reqs_passed"], a["hidden_reqs_total"], 100 * (a["hidden_rate"] or 0),
a["runs_fully_passing"], a["suite_green_runs"], a["mean_cost_total_usd"] or 0, a["mean_cost_worker_usd"] or 0,
a["mean_cost_testwriter_usd"] or 0, a["mean_cost_reviewers_usd"] or 0, a["mean_wall_s"] or 0, a["blocks_per_run"], a["right_blocks"]["g1"],
a["false_blocks"]["g1"], a["right_blocks"]["g3"], a["false_blocks"]["g3"], a["give_ups"], a["ended_blocked"],
a["model_ok"]))
L += ["", "## Per task (hidden reqs passed / total, per config)", "", "| task | " + " | ".join(CFGS) + " | missed reqs (all configs) |",
"|---|" + "---|" * (len(CFGS) + 1)]
for t in TASKS:
cells = []
for c in CFGS:
a = by_task[t][c]
cells.append("%d/%d, $%.2f, %.1f blk" % (a["hidden_reqs_passed"], a["hidden_reqs_total"], a["mean_cost_total_usd"] or 0,
a["blocks_per_run"]) if a else "-")
L.append("| %s | %s | %s |" % (t, " | ".join(cells), ", ".join((by_task_all[t] or {}).get("missed_reqs", []))))
L += ["", "## The video's questions", ""]
for k, v in answers.items():
L.append("- **%s**: `%s`" % (k, json.dumps(v)))
L += ["", "Gate 1 false blocks (gate 1 blocked while the hidden oracle was fully green): " + ", ".join(
"%s %s/%s" % (c, (by_cfg.get(c) or {}).get("false_blocks", {}).get("g1"),
((by_cfg.get(c) or {}).get("false_blocks", {}).get("g1") or 0) + ((by_cfg.get(c) or {}).get("right_blocks", {}).get("g1") or 0))
for c in ("g1", "g13", "g13adv")) + " (false / all g1 blocks)."]
L += ["", "Right block = the hidden oracle had at least one failing requirement at that stop; false block = the oracle "
"was fully green at that stop (the gate sent the agent back for nothing the user asked for)."]
open(os.path.join(HERE, "results.md"), "w").write("\n".join(L) + "\n")
print("\n".join(L))
results.md
# gate-bench results
50 runs scored (matrix runs only).
## Per config
| cfg | runs | hidden reqs | fully passing | suite green | mean $ total (worker + g1 writer + g3 reviewers) | mean wall s | blocks/run | g1 right/false | g3 right/false | give-ups | ended blocked | model ok |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bare | 10 | 35/50 (70%) | 3 | 10 | 0.085 (0.085 + 0.000 + 0.000) | 33 | 0.00 | 0/0 | 0/0 | 0 | 0 | True |
| g1 | 10 | 42/50 (84%) | 6 | 10 | 0.201 (0.130 + 0.071 + 0.000) | 60 | 0.60 | 3/3 | 0/0 | 0 | 0 | True |
| g3 | 10 | 34/50 (68%) | 3 | 10 | 0.151 (0.094 + 0.000 + 0.058) | 54 | 0.10 | 0/0 | 1/0 | 0 | 0 | True |
| g13 | 10 | 41/50 (82%) | 5 | 10 | 0.282 (0.153 + 0.071 + 0.057) | 86 | 1.70 | 14/1 | 2/0 | 2 | 0 | True |
| g13adv | 10 | 39/50 (78%) | 6 | 10 | 0.276 (0.157 + 0.072 + 0.048) | 86 | 1.20 | 11/1 | 0/0 | 1 | 0 | True |
## Per task (hidden reqs passed / total, per config)
| task | bare | g1 | g3 | g13 | g13adv | missed reqs (all configs) |
|---|---|---|---|---|---|---|
| T1 | 4/10, $0.08, 0.0 blk | 4/10, $0.22, 0.5 blk | 3/10, $0.16, 0.5 blk | 4/10, $0.36, 3.0 blk | 5/10, $0.31, 1.0 blk | T1:test_req2, T1:test_req3, T1:test_req4, T1:test_req5 |
| T2 | 8/10, $0.10, 0.0 blk | 8/10, $0.17, 0.0 blk | 8/10, $0.17, 0.0 blk | 8/10, $0.27, 3.0 blk | 9/10, $0.30, 3.5 blk | T2:test_req4 |
| T3 | 10/10, $0.09, 0.0 blk | 10/10, $0.21, 1.0 blk | 10/10, $0.14, 0.0 blk | 10/10, $0.26, 0.5 blk | 10/10, $0.31, 0.5 blk | |
| T4 | 5/10, $0.07, 0.0 blk | 10/10, $0.23, 0.5 blk | 5/10, $0.15, 0.0 blk | 10/10, $0.30, 1.0 blk | 5/10, $0.21, 0.0 blk | T4:test_req1, T4:test_req2, T4:test_req3, T4:test_req4, T4:test_req5 |
| T5 | 8/10, $0.08, 0.0 blk | 10/10, $0.18, 1.0 blk | 8/10, $0.12, 0.0 blk | 9/10, $0.23, 1.0 blk | 10/10, $0.25, 1.0 blk | T5:test_req4 |
## The video's questions
- **q1_gate1_vs_bare**: `{"bare": 0.7, "g1": 0.84, "delta": 0.14}`
- **q2_gate3_over_g1**: `{"g1": 0.84, "g13": 0.82, "delta": -0.02, "g3_alone": 0.68, "g3_vs_bare": -0.02}`
- **q3_adv_vs_indep**: `{"g13": 0.82, "g13adv": 0.78, "delta": -0.04, "false_blocks_g3": {"g13": 0, "g13adv": 0}}`
- **q4_false_block_rate**: `{"bare": {"g1": null, "g3": null}, "g1": {"g1": 0.5, "g3": null}, "g3": {"g1": null, "g3": 0.0}, "g13": {"g1": 0.067, "g3": 0.0}, "g13adv": {"g1": 0.083, "g3": null}}`
- **q4_gate1_false_blocks**: `{"g1": {"false": 3, "right": 3}, "g13": {"false": 1, "right": 14}, "g13adv": {"false": 1, "right": 11}}`
- **q5_cost**: `{"bare": {"mean_cost_total_usd": 0.0849, "mean_cost_worker_usd": 0.0849, "mean_cost_gate1_usd": 0.0, "mean_cost_gate3_usd": 0.0, "mean_wall_s": 32.7}, "g1": {"mean_cost_total_usd": 0.201, "mean_cost_worker_usd": 0.1302, "mean_cost_gate1_usd": 0.0709, "mean_cost_gate3_usd": 0.0, "mean_wall_s": 59.5}, "g3": {"mean_cost_total_usd": 0.1513, "mean_cost_worker_usd": 0.0938, "mean_cost_gate1_usd": 0.0, "mean_cost_gate3_usd": 0.0575, "mean_wall_s": 54.1}, "g13": {"mean_cost_total_usd": 0.2818, "mean_cost_worker_usd": 0.1534, "mean_cost_gate1_usd": 0.0712, "mean_cost_gate3_usd": 0.0573, "mean_wall_s": 86.4}, "g13adv": {"mean_cost_total_usd": 0.2763, "mean_cost_worker_usd": 0.1568, "mean_cost_gate1_usd": 0.0718, "mean_cost_gate3_usd": 0.0478, "mean_wall_s": 85.9}}`
Gate 1 false blocks (gate 1 blocked while the hidden oracle was fully green): g1 3/6, g13 1/15, g13adv 1/12 (false / all g1 blocks).
Right block = the hidden oracle had at least one failing requirement at that stop; false block = the oracle was fully green at that stop (the gate sent the agent back for nothing the user asked for).
Validation and demo:
validate.sh: checks that the oracle fails on upstream, the hook fires,gate.shblocks a red copy and a reviewer returns valid JSON.validate.sh
#!/usr/bin/env bash
# Validation before the matrix (SPEC § Validation). Steps 1, 3, 4 here; step 2 = the g1 red-start smoke run:
# INJECT=$PWD/inject/break.patch RUN_TAG=val2 ./run.sh g1 T1 1 (then: ./validate.sh step2)
# Usage: validate.sh [all|step1|step2|step3|step4] -> appends to validation.log
set -u
HERE="$(cd "$(dirname "$0")" && pwd)"; cd "$HERE"
. "$HERE/env.sh"
PY="$HERE/.venv/bin/python"
STEP="${1:-all}"
status=0
count() { grep -Eo "[0-9]+ $2" <<<"$1" | tail -1 | awk '{print $1}'; }
log() { echo "$*" | tee -a validation.log; }
log "=== $(date '+%F %T') validate.sh $STEP ($(claude --version))"
if [ "$STEP" = all ] || [ "$STEP" = step1 ]; then
# 1. upstream fails the hidden tests (<= 1 req passing: T5 R3 is a regression guard); reference passes hidden + suite
TMP="$(mktemp -d "${TMPDIR:-/tmp}/gb-validate.XXXXXX")"
for t in T1 T2 T3 T4 T5; do
tree="$TMP/$t"; git clone -q "$HERE/upstream" "$tree"
v=$(cd "$tree" && TINYDB_TREE="$tree" "$PY" -m pytest -v -c "$HERE/hidden/pytest.ini" --rootdir "$HERE/hidden" "$HERE/hidden/test_$t.py" 2>&1)
up_pass=$(grep -Eo "test_req[0-9]+[^ ]* PASSED" <<<"$v" | wc -l | tr -d ' ')
(cd "$tree" && git apply "$HERE/reference/$t.patch") || { log "$t: reference patch does not apply"; status=1; continue; }
h=$(cd "$tree" && TINYDB_TREE="$tree" "$PY" -m pytest -q -c "$HERE/hidden/pytest.ini" --rootdir "$HERE/hidden" "$HERE/hidden/test_$t.py" 2>&1); hrc=$?
s=$(cd "$tree" && "$PY" -m pytest -q -p no:cacheprovider --no-cov 2>&1); src=$?
ok=OK; { [ "$up_pass" -gt 1 ] || [ $hrc -ne 0 ] || [ $src -ne 0 ]; } && { ok=FAIL; status=1; }
log "step1 $t: upstream reqs passing=$up_pass reference hidden rc=$hrc ($(count "$h" passed) passed) suite rc=$src ($(count "$s" passed) passed) [$ok]"
done
rm -rf "$TMP"
fi
if [ "$STEP" = all ] || [ "$STEP" = step3 ]; then
# 3. gate.sh by hand, fake stdin. green = T1 reference + one new test (g1 passes) -> exit 0 allow;
# red = break.patch (one repo test fails) -> exit 2, pytest tail on stderr.
FAKE='{"session_id":"validate","transcript_path":"/dev/null","cwd":"x","hook_event_name":"Stop","stop_hook_active":false}'
for k in green red; do
R="$HERE/runs/_val3-$k"; W="$HERE/work/_val3-$k"; rm -rf "$R"; mkdir -p "$R"
./prep.sh g1 "$W" "$R" > /dev/null
if [ $k = green ]; then
(cd "$W" && git apply "$HERE/reference/T1.patch" && cp "$HERE/inject/test_pushpull.py" tests/)
else
(cd "$W" && git apply "$HERE/inject/break.patch")
fi
printf '{"task":"T1","cfg":"g1","rep":0,"work":"%s","name":"_val3-%s"}\n' "$W" "$k" > "$R/meta.json"; cp tasks/T1.md "$R/task.md"
echo "$FAKE" | GATE_RUN="$R" GATE_CONFIG=g1 bash gate.sh > "$R/stdout.txt" 2> "$R/stderr.txt"; rc=$?
dec=$("$PY" -c "import json;print(json.loads(open('$R/gate.jsonl').readline())['decision'])")
want=0; [ $k = red ] && want=2
ok=OK; [ $rc -ne $want ] && { ok=FAIL; status=1; }
[ $k = red ] && ! grep -q "short test summary" "$R/stderr.txt" && { ok=FAIL; status=1; }
[ $k = green ] && [ -s "$R/stderr.txt" ] && { ok=FAIL; status=1; }
log "step3 $k: exit=$rc decision=$dec stderr_bytes=$(wc -c < "$R/stderr.txt" | tr -d ' ') [$ok]"
done
fi
if [ "$STEP" = all ] || [ "$STEP" = step4 ]; then
# 4. one reviewer pair by hand on the green copy (needs step3's work/_val3-green): structured_output valid
W="$HERE/work/_val3-green"
[ -d "$W" ] || { log "step4: run step3 first"; exit 1; }
for v in indep adv; do
"$PY" lib/review.py "$W" T1 $v > "runs/_val3-green/review-$v.txt" 2>&1
s=$("$PY" - "runs/_val3-green/review-$v.txt" <<'PY'
import json, sys
t = open(sys.argv[1]).read(); i = t.find("--- reason"); d = json.loads(t[:i] if i > 0 else t)
ok = all(r["error"] is None and isinstance(r["approve_raw"], bool) for r in d["reviewers"])
ok = ok and all(f["severity"] in ("blocking", "minor") and isinstance(f["line"], int) for r in d["reviewers"] for f in r["findings"])
print("%s pass=%s blocking=%d findings=%s cost=$%.4f wall=%s" % ("OK" if ok else "FAIL", d["pass"], d["blocking_findings"],
[len(r["findings"]) for r in d["reviewers"]], d["cost_usd"], [r["wall_s"] for r in d["reviewers"]]))
PY
)
case "$s" in OK*) ;; *) status=1 ;; esac
log "step4 $v: $s"
done
fi
if [ "$STEP" = all ] || [ "$STEP" = step2 ]; then
# 2. hook fires on a red start: runs/T1-g1-r1-val2 must log a block and the stream must show the agent continuing
R="$HERE/runs/T1-g1-r1-val2"
if [ -f "$R/gate.jsonl" ]; then
s=$("$PY" - "$R" <<'PY'
import json, sys, os
r = sys.argv[1]
g = [json.loads(l) for l in open(os.path.join(r, "gate.jsonl")) if l.strip()]
st = [json.loads(l) for l in open(os.path.join(r, "stream.jsonl")) if l.startswith("{")]
blocks = [x for x in g if x["decision"] == "block"]
# index in the stream of the first hook response that blocked, then any assistant tool_use after it
txt = [json.dumps(o) for o in st]
fb = next((i for i, t in enumerate(txt) if "Stop hook feedback" in t or ('"hook_event": "Stop"' in t and '"exit_code": 2' in t)), None)
after = fb is not None and any(o.get("type") == "assistant" for o in st[fb + 1:])
print("%s blocks=%d hook_calls=%d feedback_idx=%s assistant_after=%s last=%s" % (
"OK" if blocks and after else "FAIL", len(blocks), len(g), fb, after, g[-1]["decision"] if g else None))
PY
)
case "$s" in OK*) ;; *) status=1 ;; esac
log "step2: $s"
else
log "step2: not run yet (INJECT=\$PWD/inject/break.patch RUN_TAG=val2 ./run.sh g1 T1 1)"
fi
fi
[ $status -eq 0 ] && log "VALIDATION OK" || log "VALIDATION FAILED"
exit $status
validate_writer.sh
#!/usr/bin/env bash
# Validation of the gate-1 test writer: one writer call per task (in parallel) into validation/writer/<T>/,
# then its tests on upstream and on the reference patch. Usage: validate_writer.sh [T1 ...] (reuses an existing
# file unless REWRITE=1). Appends to validation.log.
set -u
HERE="$(cd "$(dirname "$0")" && pwd)"; cd "$HERE"
. "$HERE/env.sh"
PY="$HERE/.venv/bin/python"
TASKS="${*:-T1 T2 T3 T4 T5}"
for t in $TASKS; do
D="$HERE/validation/writer/$t"; mkdir -p "$D"
if [ "${REWRITE:-0}" = 1 ] || [ ! -f "$D/gate_tests/test_$t.py" ]; then "$PY" lib/testwriter.py "$t" "$D" > "$D/writer.out" 2>&1 & fi
done
wait
TMP="$(mktemp -d "${TMPDIR:-/tmp}/gb-vw.XXXXXX")"; trap 'rm -rf "$TMP"' EXIT
echo "=== $(date '+%F %T') validate_writer.sh $TASKS" >> validation.log
for t in $TASKS; do
D="$HERE/validation/writer/$t"
git clone -q "$HERE/upstream" "$TMP/up-$t"; git clone -q "$HERE/upstream" "$TMP/ref-$t"
(cd "$TMP/ref-$t" && git apply "$HERE/reference/$t.patch")
line=$("$PY" - "$D" "$t" "$TMP/up-$t" "$TMP/ref-$t" <<'PY'
import json, os, sys
sys.path.insert(0, "lib")
from testwriter import run_gate_tests
d, t, up, ref = sys.argv[1:5]
w = json.load(open(os.path.join(d, "testwriter.json")))
if not w["ok"]:
print("%s: writer FAILED %s" % (t, [a["error"] for a in w["attempts"]])); sys.exit()
g = os.path.join(d, "gate_tests")
_, uo, up_p, up_f, _ = run_gate_tests(g, t, up)
_, ro, rp, rf, rnames = run_gate_tests(g, t, ref)
open(os.path.join(d, "on-upstream.txt"), "w").write(uo); open(os.path.join(d, "on-reference.txt"), "w").write(ro)
print("%s: writer tests: %d, pass on reference: %d, pass on upstream: %d | fail on reference: %s | writer $%.3f %ss turns=%s" % (
t, rp + rf, rp, up_p, rnames, w["cost_usd"], w["wall_s"], [a.get("turns") for a in w["attempts"]]))
PY
)
echo "$line" | tee -a validation.log
done
demo.sh
#!/usr/bin/env bash
# Interactive demo of the gate for a screen recording: a gated tinydb work copy (config g13: acceptance
# tests, then two reviewers) with T1's acceptance tests already written. Usage: ./demo.sh [T1..T5]
# Then: cd work/demo && claude --setting-sources project --permission-mode acceptEdits
# and paste the request printed below. When Claude stops, the Stop hook runs; a red acceptance test shows
# as "Stop hook feedback: ... The acceptance tests for this change fail:" and Claude keeps working.
set -euo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"; TASK="${1:-T1}"
. "$HERE/env.sh"
RUN="$HERE/runs/demo"; W="$HERE/work/demo"
rm -rf "$RUN"; mkdir -p "$RUN"
"$HERE/prep.sh" g13 "$W" "$RUN" > "$RUN/prep.txt" 2>&1
printf '{"task":"%s","cfg":"g13","rep":0,"work":"%s","name":"demo"}\n' "$TASK" "$W" > "$RUN/meta.json"
cp "$HERE/tasks/$TASK.md" "$RUN/task.md"
printf '%s\n\nWhen you are done, stop and summarise what you changed.\n' "$(cat "$HERE/tasks/$TASK.md")" > "$RUN/prompt.txt"
# Reuse the acceptance tests of a matrix run of the same task (no new writer call).
SRC=$(ls -d "$HERE"/runs/"$TASK"-g13-r1/gate_tests 2>/dev/null | head -1)
cp -R "$SRC" "$RUN/gate_tests"
echo "ready: cd $W && claude --setting-sources project --permission-mode acceptEdits"
echo "--- paste this request ---"; cat "$RUN/prompt.txt"
inject/break.patch
diff --git a/tinydb/queries.py b/tinydb/queries.py
index cdab9fb..db00a2a 100644
--- a/tinydb/queries.py
+++ b/tinydb/queries.py
@@ -322,7 +322,7 @@ class Query(QueryInstance):
:param rhs: The value to compare against
"""
return self._generate_test(
- lambda value: value >= rhs,
+ lambda value: value > rhs,
('>=', self._path, rhs)
)
inject/test_pushpull.py
from tinydb import TinyDB, Query
from tinydb.storages import MemoryStorage
from tinydb.operations import push, pull
def test_push_pull():
db = TinyDB(storage=MemoryStorage)
db.insert({'n': 1, 't': ['a']})
db.update(push('t', 'b'), Query().n == 1)
db.update(pull('t', 'a'), Query().n == 1)
assert db.all()[0]['t'] == ['b']
inject/test_pushpull.py on GitHub
Docs:
README.md: layout, commands and what the harness is.README.md
# gate-bench
What does each of Shopify's Helix gates catch when rebuilt as a Claude Code **Stop hook**, and what does it
cost? Headless Sonnet 5.5 workers implement five tinydb features (T1..T5); a Stop hook runs gate 1
(behaviour tests) and/or gate 3 (adversarial review by headless Sonnet 5.5 reviewers) every time the worker
tries to stop; hidden tests (the oracle, never visible to the agent) record the ground truth at every stop
and at the end. Built 2026-10-08 from `SPEC.md`; upstream, hidden tests, tasks and reference patches copied
from `../verify-loops-bench` (nothing there or in `../haiku-routing-bench` is modified).
## Commands (from this dir)
```bash
./validate.sh # steps 1,3,4 (+ step 2 once its run exists) -> validation.log
INJECT=$PWD/inject/break.patch RUN_TAG=val2 ./run.sh g1 T1 1 # validation step 2 (red start state)
RUN_TAG=smoke ./run.sh g13adv T1 1 # smoke (tagged runs are ignored by analyze.py)
./validate_writer.sh # gate-1 test writer on T1..T5: tests on upstream vs reference (REWRITE=1 to redo)
./gen_matrix.py # matrix.txt (50 runs, already generated)
nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 & # 3 runs in parallel (MATRIX_JOBS=3)
SKIP_DONE=1 nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 & # resume
.venv/bin/python analyze.py # results.json + results.md
.venv/bin/python lib/finalize.py runs/<run> # rescore one run from its work copy
.venv/bin/python lib/review.py work/<run> T1 adv # one gate-3 round by hand
```
Single run: `./run.sh <cfg> <task> <rep>` -> `runs/<task>-<cfg>-r<rep>/`, work copy `work/<same name>/`.
## Layout
| path | what |
|---|---|
| `SPEC.md` | the spec this harness implements |
| `upstream/` | tinydb @ 18d73a1 (git repo) |
| `hidden/` | `test_T1..T6.py` (T6 unused), `conftest.py` (`$TINYDB_TREE` or cwd first on sys.path), `pytest.ini` |
| `tasks/T1..T5.md` + `tasks/README.md` | the task prompts (verbatim) and what each hidden req checks |
| `reference/T1..T5.patch` | reference solutions (validation step 1) |
| `inject/break.patch` | `>=` -> `>` in `Query.__ge__` (one repo test red; validation steps 2-3) |
| `inject/test_pushpull.py` | one thin test, makes the T1 reference copy "green with tests touched" (step 3) |
| `configs.json` | `bare` (no hook), `g1`, `g3` (indep), `g13` (g1 then g3 indep), `g13adv` (g1 then g3 adv) |
| `env.sh` | scrubs inherited `CLAUDE*` vars, default `~/.claude` auth, `CLAUDE_CODE_DISABLE_AUTO_MEMORY=1`, venv on PATH |
| `prep.sh <cfg> <work> <run>` | fresh copy of upstream, `.claude/` in `.git/info/exclude`, hook settings for gated configs |
| `run.sh <cfg> <task> <rep>` | one run (worker + hook + `lib/finalize.py`) |
| `gate.sh` | the Stop hook (bash 3.2 wrapper: `exec .venv/bin/python -I lib/hook.py`) |
| `lib/hook.py` | gate dispatch, snapshot, hidden run, gate 1, gate 3, decision, `gate.jsonl` line |
| `lib/review.py` | gate-3 reviewers (indep / adv), schemas, prompts, reason text; CLI for a manual round |
| `lib/testwriter.py` | gate-1 test writer (before the worker) + `run_gate_tests` (path rewritten to `acceptance/`) |
| `validate_writer.sh`, `validation/writer/<T>/` | writer validation files and their runs on upstream / reference |
| `tasks/` (terse prompts, used), `tasks-explicit/` + `runs-explicit/` | the matrix uses terse prompts; see `tasks/README.md` |
| `lib/common.py` | diff incl. untracked files, suite run, hidden run (requirement level) |
| `lib/finalize.py` | post-run diff/suite/hidden, `result.json`, `score.json` |
| `gen_matrix.py`, `matrix.txt`, `matrix.sh`, `matrix-one.sh`, `matrix.log` | 50-run matrix, 3 in parallel (xargs -P), resumable |
| `analyze.py` -> `results.json`, `results.md` | per config, per task, the five questions |
| `validate.sh`, `VALIDATION.md`, `validation.log` | pre-matrix checks |
| `.venv/` | python 3.14.7, pytest 9.1.1, pytest-cov 7.1.0, pyyaml, ruff 0.16.10 (same pins as haiku-routing-bench) |
### Worker (run.sh)
`claude -p "<T-file verbatim>\n\nWhen you are done, stop and summarise what you changed." --model claude-sonnet-5-5
--output-format stream-json --verbose --include-hook-events --max-turns 60 --permission-mode acceptEdits
--allowedTools <haiku-routing-bench list> --setting-sources project --strict-mcp-config`, cwd = `work/<run>/`, stdin /dev/null.
Gated configs get `work/<run>/.claude/settings.json`:
`{"permissions":{"deny":[...see traps...]},"hooks":{"Stop":[{"hooks":[{"type":"command","command":"bash \"$CLAUDE_PROJECT_DIR/.claude/hooks/stop.sh\"","timeout":1500}]}]}}`
and `.claude/hooks/stop.sh` = `GATE_RUN=<abs run dir> GATE_CONFIG=<cfg> bash <bench>/gate.sh` (the SPEC command, one level down: see traps).
Copies of both land in the run dir (`settings.json`, `stop.sh`).
### Hook (lib/hook.py), one invocation
1. read stdin JSON; n = lines already in `gate.jsonl` + 1; blocks so far from the same file
2. `snap-<n>.diff` = `git diff HEAD` + untracked files (index never touched)
3. hidden tests on the work copy (`TINYDB_TREE` = work copy, cwd = work copy) -> `hidden-<n>.txt`; never shown to the agent
4. gate 1 (if configured) = the repo suite `python -m pytest -q -x --no-header -p no:cacheprovider --no-cov` (300 s)
-> `gate1-<n>.txt` AND the writer's acceptance tests `runs/<run>/gate_tests/test_<T>.py` run with `TINYDB_TREE` = work
copy -> `gatetests-<n>.txt`. Block when either is red. Reason: "The acceptance tests for this change fail:" + last 40
lines (failing test names + assertion output, the gate_tests path rewritten to `acceptance/`), then the suite tail if the
suite is red too; <= 1500 chars. `tests_touched` is still logged but no longer blocks. No writer file (writer failed
twice) -> suite only, logged as `gate_tests_present: false`.
5. gate 3 (if configured and gate 1 passed or absent): see below
6. decision: allow / block / give_up (a would-be block when 6 blocks already happened -> allow, logged `give_up`).
`stop_hook_active` is logged, never used. Block = exit 2 + reason on stderr. A crash of the hook itself is
logged to `gate-errors.txt` and allows (counted as `gate_crashes` in score.json).
### Gate 1 test writer (lib/testwriter.py, run.sh before the worker, configs g1/g13/g13adv)
`claude -p` Sonnet 5.5 with the reviewer isolation flags, `--tools Read,Grep,Glob --max-turns 12 --json-schema
{"file": string}`, cwd = a pristine temp copy of upstream (deleted after). Prompt = framing + `tasks/<T>.md` + the
coordinator's instruction ("Write integration-style pytest cases for this request against the tinydb API, one test per
behaviour a careful senior would expect, pushing for edge cases beyond the happy path (missing fields, wrong types,
boundaries, persistence, cache, existing behaviour unchanged). Tests only, no implementation. Return the file content.")
+ mechanics (self-contained module, imports of the new API inside each test, MemoryStorage/tmp_path). The file must
compile and contain `def test_`, else one retry. Output `runs/<run>/gate_tests/` (+ hidden's conftest.py with
`TINYDB_TREE`, pytest.ini) and `testwriter.json` (cost, turns, attempts). Fixed per-run cost, outside the worker's wall.
### Gate 3 (lib/review.py)
Each reviewer: `claude -p "<prompt>" --model claude-sonnet-5-5 --tools Read,Grep,Glob --max-turns 15 --output-format json
--json-schema <schema> --permission-mode dontAsk --setting-sources "" --settings '{"disableAllHooks":true}' --strict-mcp-config`
with `CLAUDE_CODE_DISABLE_CLAUDE_MDS=1 CLAUDE_CODE_DISABLE_AUTO_MEMORY=1`, all inherited `CLAUDE*` vars dropped, cwd = work copy,
600 s timeout. Prompt = framing + task prompt (T-file) + diff (tracked + untracked, cap 60k chars) + the SPEC's "Find defects..."
sentence. `structured_output` is read from the JSON result; `total_cost_usd`, `num_turns`, wall and models are recorded.
- `indep`: reviewers A and B, same prompt, run in parallel, no shared context. Blocking findings = union. Pass when both approve.
- `adv`: A as above; challenger B gets A's numbered findings + the diff, answers `upheld|rejected` + reason per id, may add
findings (`added_findings`). Blocking = A's blocking findings B upheld + B's added blocking findings. B always runs (same
2-call budget as indep).
- Effective `approve` = "no blocking finding" (the raw `approve` is logged as `approve_raw`; a reviewer saying approve=false with
only minor findings does not block). A reviewer call that errors approves (fail open) with `error` recorded; an errored
challenger upholds nothing.
- Reason = "Code review found N blocking defect(s):" + numbered `file:line text` + "Fix every finding, re-run the tests, then stop.", <= 1500 chars.
### Per run (`runs/<task>-<cfg>-r<rep>/`)
`stream.jsonl`, `stderr.txt`, `exit_code.txt`, `wall_s.txt` (includes hook time: the hook runs inside the session),
`gate.jsonl`, `snap-<n>.diff`, `hidden-<n>.txt`, `gate1-<n>.txt`, `hook-input-1.json`, `post.diff`, `numstat.txt`,
`suite.txt` (full suite, no -x), `hidden.txt`, `gate_tests/`, `testwriter.json`, `gatetests-<n>.txt`, `gatetests-final.txt`, `result.json`, `score.json`, `meta.json`, `task.md`, `prompt.txt`,
`settings.json` + `stop.sh` (gated configs).
`score.json`: task, cfg, rep, hidden_pass/hidden_fail/failed_reqs (**requirement level**: 5 reqs per task), suite_pass/fail,
hook_calls, blocks_total, blocks_by_gate {g1,g3}, right_blocks / false_blocks (+ `_by_gate`), give_up, ended_blocked,
cost_worker_usd, cost_reviewers_usd, cost_testwriter_usd, cost_total_usd (all three), testwriter_ok, gate_tests_total,
gate_tests_fail_final, turns, wall_s, reviewer_findings_total, reviewer_errors, gate_crashes,
model_ok (worker init model + every modelUsage key + every reviewer model == claude-sonnet-5-5), hook_events_in_stream,
hidden_first_stop.
## Matrix
5 tasks x 5 configs x 2 reps = 50 runs. Rep 1 config blocks bare, g1, g3, g13, g13adv (tasks rotated per block), rep 2 the
reverse block order. Up to 3 runs at once, each in its own work copy (`matrix-one.sh` via `xargs -P 3`, bash 3.2 has no
`wait -n`). `matrix.log`: one `start` and one result line per run.
## Traps found
- **Gate 1 as first built passed by construction** (it only re-ran the implementer's own tests: `T2-g1-r1` one allow, hidden
R4 missed). Redesigned around an independent test writer (above). The writer hedges where the terse prompt is silent
(accepts "raises or no-op"), so it cannot recover unstated requirements; per-task overlap is in VALIDATION.md.
- **A blocked agent hunts for the tests.** After a gate-1 block the smoke agent ran `cat .claude/hooks/stop.sh; find / -name
test_T1.py` (denied only because `find` is not on the allow-list; `cat` is, and `stop.sh` names the bench dir, from where
`hidden/` is one `cat` away). Gated configs' `.claude/settings.json` now carries `permissions.deny`: Read of `./.claude/**`,
the bench's `hidden/ runs/ validation/ reference/`, the two sibling benches; Bash patterns containing `.claude/hooks`,
`.claude/settings`, `../hidden`, `../runs`, `gate_tests`, the bench sub-paths, `find *`. Probed with a Haiku session: all
five escape attempts (cat, Read, ls, python open) denied, in-tree reads fine. Bash rules match command text, not a security
boundary. Bare runs have no settings file (bare r1 ran before this change; bare has no motive to hunt).
Reviewers and the writer run with `--setting-sources ""`, so these rules do not apply to them.
- **Transient `FileNotFoundError` spawning a reviewer** inside the hook (once, both reviewers of one call, wall 0.0 s;
never reproduced by hand). Reviewer calls now retry once after 3 s; the error message records `filename`.
- **`--bare` cannot authenticate here.** It reads `ANTHROPIC_API_KEY` / `apiKeyHelper` only, never OAuth or the keychain;
this account is OAuth (`~/.claude`), so a `--bare` reviewer returns `terminal_reason: api_error`, cost 0. Used the
documented equivalents instead: `--setting-sources ""` + `--settings '{"disableAllHooks":true}'` + `--strict-mcp-config` +
`CLAUDE_CODE_DISABLE_CLAUDE_MDS=1` + `CLAUDE_CODE_DISABLE_AUTO_MEMORY=1` + `--tools Read,Grep,Glob`. The reviewers' cwd is the
work copy, whose `.claude/settings.json` carries the Stop hook: without `--setting-sources ""`/`disableAllHooks` a reviewer
would run the gate recursively. Verified: a dummy Stop hook in the probe copy never fired.
- **Inherited session env.** A matrix launched from inside Claude Code inherits `CLAUDE_EFFORT=medium`,
`CLAUDE_CODE_SUBAGENT_MODEL=opus`, `CLAUDE_CODE_SESSION_ID`, `CLAUDECODE`, `CLAUDE_CODE_CHILD_SESSION`, messaging tokens.
`env.sh` unsets every `CLAUDE*` var before setting its own; reviewers get a scrubbed env too (the hook process itself
inherits the worker's `CLAUDE_*` vars, e.g. the worker's session id).
- **The feedback line leaks the hook command.** Claude Code prefixes each block with the command line:
`Stop hook feedback:\n[<command>]: <reason>`. With the SPEC command inline the agent would read `GATE_CONFIG=g13adv` and
`gate-bench/gate.sh`. The settings command is therefore `bash "$CLAUDE_PROJECT_DIR/.claude/hooks/stop.sh"`, a one-line
wrapper holding the SPEC command. The agent can still `cat` it (it lives in the work copy) but it is not pushed at it.
- **Stop hook input keys (2.1.293):** `background_tasks, cwd, effort ({"level":"medium"}), hook_event_name,
last_assistant_message, permission_mode, prompt_id, session_crons, session_id, stop_hook_active, transcript_path`.
- **8-cap resets on tool calls.** Per the docs the consecutive-continuation count resets each time Claude calls a tool, so the
built-in cap only bites an agent that answers blocks with text alone. Our give-up (6 blocks per run, counted from
`gate.jsonl`, tool calls or not) fires first in every case.
- **Red start state: the agent refuses to fix code it did not write.** Validation step 2 (T1 + an uncommitted `>=` -> `>`
edit) shows Sonnet 5.5 implementing T1, then answering 6 consecutive gate-1 blocks with "the red test is your pre-existing
uncommitted edit, not my change" without touching it, until give_up. The hook mechanics are validated; the behaviour is a
finding in itself (gate 1 cannot force a fix the agent considers out of scope).
- **pytest-cov.** The repo's `pytest.ini` adds `--cov tinydb`; gate 1 adds `--no-cov` so the 40-line tail holds the failure,
not the coverage table (and no `.coverage` file is written). `.coverage` is gitignored anyway.
- **T5 R4 is parametrized** (several test ids, one requirement). hidden counts are per requirement (5 per task), a req
passes only when all its ids pass. `validate.sh` step 1: T5 upstream passes 1 req (R3, the regression guard) as designed.
- **Diff without touching the index.** `git add -N` would change what the agent sees in `git status`; untracked files are
diffed with `git diff --no-index /dev/null <f>` instead. `.claude/` is in `.git/info/exclude`, so the hook config never
shows in any diff or in `git status`.
- **Worker Bash denials.** The 2.1.293 permission layer denies some agent commands with "Contains brace with quote character
(expansion obfuscation)" even under acceptEdits + allow-list; agents route around it (visible as `permission_denied` events).
- **Reviewers on a correct solution still flag "blocking".** On the T1 reference patch, reviewer A marked "pull does not check
the field is a list" (not required by the task) as blocking; indep therefore blocks, the adv challenger rejected it. This is
exactly what false_blocks measures (hidden all green at that stop).
- **Shared plan meter.** Workers and reviewers bill the user's plan quota; `total_cost_usd` is the list-price equivalent the
CLI reports, not a bill. Nothing else heavy should run on the account during the matrix.
- **rtk rewrites interactive `grep`/`diff`** in the launching Claude session only; scripts here call binaries directly
(`/usr/bin/diff`/`cmp` for verification diffs).
- **CLI version:** `claude --version` = 2.1.293 (Claude Code); flags checked in `claude --help`: `--json-schema`, `--bare`, `--tools`,
`--permission-mode` (choices acceptEdits, auto, bypassPermissions, manual, dontAsk, plan), `--setting-sources`,
`--include-hook-events`. `--max-turns` is accepted though not listed in `--help`.
## Matrix launch (2026-10-08 16:33:54, after the gate-1 redesign)
`cd /Users/thomas/dev/gate-bench && SKIP_DONE=1 nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 &` -> PID 71317, 3 lanes.
The 5 bare r1 runs (terse prompts) are kept and skipped; 45 runs to go. Expected wall ~40-70 min: gated runs so far take
~30-70 s of worker wall + ~35-40 s of test writer (before the worker) + 1-35 s per hook call, more with blocks.
Then `.venv/bin/python analyze.py`.
HARNESS.md
# gate-bench
What does each of Shopify's Helix gates catch when rebuilt as a Claude Code **Stop hook**, and what does it
cost? Headless Sonnet 5.5 workers implement five tinydb features (T1..T5); a Stop hook runs gate 1
(behaviour tests) and/or gate 3 (adversarial review by headless Sonnet 5.5 reviewers) every time the worker
tries to stop; hidden tests (the oracle, never visible to the agent) record the ground truth at every stop
and at the end. Built 2026-10-08 from `SPEC.md`; upstream, hidden tests, tasks and reference patches copied
from `../verify-loops-bench` (nothing there or in `../haiku-routing-bench` is modified).
## Commands (from this dir)
```bash
./validate.sh # steps 1,3,4 (+ step 2 once its run exists) -> validation.log
INJECT=$PWD/inject/break.patch RUN_TAG=val2 ./run.sh g1 T1 1 # validation step 2 (red start state)
RUN_TAG=smoke ./run.sh g13adv T1 1 # smoke (tagged runs are ignored by analyze.py)
./validate_writer.sh # gate-1 test writer on T1..T5: tests on upstream vs reference (REWRITE=1 to redo)
./gen_matrix.py # matrix.txt (50 runs, already generated)
nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 & # 3 runs in parallel (MATRIX_JOBS=3)
SKIP_DONE=1 nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 & # resume
.venv/bin/python analyze.py # results.json + results.md
.venv/bin/python lib/finalize.py runs/<run> # rescore one run from its work copy
.venv/bin/python lib/review.py work/<run> T1 adv # one gate-3 round by hand
```
Single run: `./run.sh <cfg> <task> <rep>` -> `runs/<task>-<cfg>-r<rep>/`, work copy `work/<same name>/`.
## Layout
| path | what |
|---|---|
| `SPEC.md` | the spec this harness implements |
| `upstream/` | tinydb @ 18d73a1 (git repo) |
| `hidden/` | `test_T1..T6.py` (T6 unused), `conftest.py` (`$TINYDB_TREE` or cwd first on sys.path), `pytest.ini` |
| `tasks/T1..T5.md` + `tasks/README.md` | the task prompts (verbatim) and what each hidden req checks |
| `reference/T1..T5.patch` | reference solutions (validation step 1) |
| `inject/break.patch` | `>=` -> `>` in `Query.__ge__` (one repo test red; validation steps 2-3) |
| `inject/test_pushpull.py` | one thin test, makes the T1 reference copy "green with tests touched" (step 3) |
| `configs.json` | `bare` (no hook), `g1`, `g3` (indep), `g13` (g1 then g3 indep), `g13adv` (g1 then g3 adv) |
| `env.sh` | scrubs inherited `CLAUDE*` vars, default `~/.claude` auth, `CLAUDE_CODE_DISABLE_AUTO_MEMORY=1`, venv on PATH |
| `prep.sh <cfg> <work> <run>` | fresh copy of upstream, `.claude/` in `.git/info/exclude`, hook settings for gated configs |
| `run.sh <cfg> <task> <rep>` | one run (worker + hook + `lib/finalize.py`) |
| `gate.sh` | the Stop hook (bash 3.2 wrapper: `exec .venv/bin/python -I lib/hook.py`) |
| `lib/hook.py` | gate dispatch, snapshot, hidden run, gate 1, gate 3, decision, `gate.jsonl` line |
| `lib/review.py` | gate-3 reviewers (indep / adv), schemas, prompts, reason text; CLI for a manual round |
| `lib/testwriter.py` | gate-1 test writer (before the worker) + `run_gate_tests` (path rewritten to `acceptance/`) |
| `validate_writer.sh`, `validation/writer/<T>/` | writer validation files and their runs on upstream / reference |
| `tasks/` (terse prompts, used), `tasks-explicit/` + `runs-explicit/` | the matrix uses terse prompts; see `tasks/README.md` |
| `lib/common.py` | diff incl. untracked files, suite run, hidden run (requirement level) |
| `lib/finalize.py` | post-run diff/suite/hidden, `result.json`, `score.json` |
| `gen_matrix.py`, `matrix.txt`, `matrix.sh`, `matrix-one.sh`, `matrix.log` | 50-run matrix, 3 in parallel (xargs -P), resumable |
| `analyze.py` -> `results.json`, `results.md` | per config, per task, the five questions |
| `validate.sh`, `VALIDATION.md`, `validation.log` | pre-matrix checks |
| `.venv/` | python 3.14.7, pytest 9.1.1, pytest-cov 7.1.0, pyyaml, ruff 0.16.10 (same pins as haiku-routing-bench) |
### Worker (run.sh)
`claude -p "<T-file verbatim>\n\nWhen you are done, stop and summarise what you changed." --model claude-sonnet-5-5
--output-format stream-json --verbose --include-hook-events --max-turns 60 --permission-mode acceptEdits
--allowedTools <haiku-routing-bench list> --setting-sources project --strict-mcp-config`, cwd = `work/<run>/`, stdin /dev/null.
Gated configs get `work/<run>/.claude/settings.json`:
`{"permissions":{"deny":[...see traps...]},"hooks":{"Stop":[{"hooks":[{"type":"command","command":"bash \"$CLAUDE_PROJECT_DIR/.claude/hooks/stop.sh\"","timeout":1500}]}]}}`
and `.claude/hooks/stop.sh` = `GATE_RUN=<abs run dir> GATE_CONFIG=<cfg> bash <bench>/gate.sh` (the SPEC command, one level down: see traps).
Copies of both land in the run dir (`settings.json`, `stop.sh`).
### Hook (lib/hook.py), one invocation
1. read stdin JSON; n = lines already in `gate.jsonl` + 1; blocks so far from the same file
2. `snap-<n>.diff` = `git diff HEAD` + untracked files (index never touched)
3. hidden tests on the work copy (`TINYDB_TREE` = work copy, cwd = work copy) -> `hidden-<n>.txt`; never shown to the agent
4. gate 1 (if configured) = the repo suite `python -m pytest -q -x --no-header -p no:cacheprovider --no-cov` (300 s)
-> `gate1-<n>.txt` AND the writer's acceptance tests `runs/<run>/gate_tests/test_<T>.py` run with `TINYDB_TREE` = work
copy -> `gatetests-<n>.txt`. Block when either is red. Reason: "The acceptance tests for this change fail:" + last 40
lines (failing test names + assertion output, the gate_tests path rewritten to `acceptance/`), then the suite tail if the
suite is red too; <= 1500 chars. `tests_touched` is still logged but no longer blocks. No writer file (writer failed
twice) -> suite only, logged as `gate_tests_present: false`.
5. gate 3 (if configured and gate 1 passed or absent): see below
6. decision: allow / block / give_up (a would-be block when 6 blocks already happened -> allow, logged `give_up`).
`stop_hook_active` is logged, never used. Block = exit 2 + reason on stderr. A crash of the hook itself is
logged to `gate-errors.txt` and allows (counted as `gate_crashes` in score.json).
### Gate 1 test writer (lib/testwriter.py, run.sh before the worker, configs g1/g13/g13adv)
`claude -p` Sonnet 5.5 with the reviewer isolation flags, `--tools Read,Grep,Glob --max-turns 12 --json-schema
{"file": string}`, cwd = a pristine temp copy of upstream (deleted after). Prompt = framing + `tasks/<T>.md` + the
coordinator's instruction ("Write integration-style pytest cases for this request against the tinydb API, one test per
behaviour a careful senior would expect, pushing for edge cases beyond the happy path (missing fields, wrong types,
boundaries, persistence, cache, existing behaviour unchanged). Tests only, no implementation. Return the file content.")
+ mechanics (self-contained module, imports of the new API inside each test, MemoryStorage/tmp_path). The file must
compile and contain `def test_`, else one retry. Output `runs/<run>/gate_tests/` (+ hidden's conftest.py with
`TINYDB_TREE`, pytest.ini) and `testwriter.json` (cost, turns, attempts). Fixed per-run cost, outside the worker's wall.
### Gate 3 (lib/review.py)
Each reviewer: `claude -p "<prompt>" --model claude-sonnet-5-5 --tools Read,Grep,Glob --max-turns 15 --output-format json
--json-schema <schema> --permission-mode dontAsk --setting-sources "" --settings '{"disableAllHooks":true}' --strict-mcp-config`
with `CLAUDE_CODE_DISABLE_CLAUDE_MDS=1 CLAUDE_CODE_DISABLE_AUTO_MEMORY=1`, all inherited `CLAUDE*` vars dropped, cwd = work copy,
600 s timeout. Prompt = framing + task prompt (T-file) + diff (tracked + untracked, cap 60k chars) + the SPEC's "Find defects..."
sentence. `structured_output` is read from the JSON result; `total_cost_usd`, `num_turns`, wall and models are recorded.
- `indep`: reviewers A and B, same prompt, run in parallel, no shared context. Blocking findings = union. Pass when both approve.
- `adv`: A as above; challenger B gets A's numbered findings + the diff, answers `upheld|rejected` + reason per id, may add
findings (`added_findings`). Blocking = A's blocking findings B upheld + B's added blocking findings. B always runs (same
2-call budget as indep).
- Effective `approve` = "no blocking finding" (the raw `approve` is logged as `approve_raw`; a reviewer saying approve=false with
only minor findings does not block). A reviewer call that errors approves (fail open) with `error` recorded; an errored
challenger upholds nothing.
- Reason = "Code review found N blocking defect(s):" + numbered `file:line text` + "Fix every finding, re-run the tests, then stop.", <= 1500 chars.
### Per run (`runs/<task>-<cfg>-r<rep>/`)
`stream.jsonl`, `stderr.txt`, `exit_code.txt`, `wall_s.txt` (includes hook time: the hook runs inside the session),
`gate.jsonl`, `snap-<n>.diff`, `hidden-<n>.txt`, `gate1-<n>.txt`, `hook-input-1.json`, `post.diff`, `numstat.txt`,
`suite.txt` (full suite, no -x), `hidden.txt`, `gate_tests/`, `testwriter.json`, `gatetests-<n>.txt`, `gatetests-final.txt`, `result.json`, `score.json`, `meta.json`, `task.md`, `prompt.txt`,
`settings.json` + `stop.sh` (gated configs).
`score.json`: task, cfg, rep, hidden_pass/hidden_fail/failed_reqs (**requirement level**: 5 reqs per task), suite_pass/fail,
hook_calls, blocks_total, blocks_by_gate {g1,g3}, right_blocks / false_blocks (+ `_by_gate`), give_up, ended_blocked,
cost_worker_usd, cost_reviewers_usd, cost_testwriter_usd, cost_total_usd (all three), testwriter_ok, gate_tests_total,
gate_tests_fail_final, turns, wall_s, reviewer_findings_total, reviewer_errors, gate_crashes,
model_ok (worker init model + every modelUsage key + every reviewer model == claude-sonnet-5-5), hook_events_in_stream,
hidden_first_stop.
## Matrix
5 tasks x 5 configs x 2 reps = 50 runs. Rep 1 config blocks bare, g1, g3, g13, g13adv (tasks rotated per block), rep 2 the
reverse block order. Up to 3 runs at once, each in its own work copy (`matrix-one.sh` via `xargs -P 3`, bash 3.2 has no
`wait -n`). `matrix.log`: one `start` and one result line per run.
## Traps found
- **Gate 1 as first built passed by construction** (it only re-ran the implementer's own tests: `T2-g1-r1` one allow, hidden
R4 missed). Redesigned around an independent test writer (above). The writer hedges where the terse prompt is silent
(accepts "raises or no-op"), so it cannot recover unstated requirements; per-task overlap is in VALIDATION.md.
- **A blocked agent hunts for the tests.** After a gate-1 block the smoke agent ran `cat .claude/hooks/stop.sh; find / -name
test_T1.py` (denied only because `find` is not on the allow-list; `cat` is, and `stop.sh` names the bench dir, from where
`hidden/` is one `cat` away). Gated configs' `.claude/settings.json` now carries `permissions.deny`: Read of `./.claude/**`,
the bench's `hidden/ runs/ validation/ reference/`, the two sibling benches; Bash patterns containing `.claude/hooks`,
`.claude/settings`, `../hidden`, `../runs`, `gate_tests`, the bench sub-paths, `find *`. Probed with a Haiku session: all
five escape attempts (cat, Read, ls, python open) denied, in-tree reads fine. Bash rules match command text, not a security
boundary. Bare runs have no settings file (bare r1 ran before this change; bare has no motive to hunt).
Reviewers and the writer run with `--setting-sources ""`, so these rules do not apply to them.
- **Transient `FileNotFoundError` spawning a reviewer** inside the hook (once, both reviewers of one call, wall 0.0 s;
never reproduced by hand). Reviewer calls now retry once after 3 s; the error message records `filename`.
- **`--bare` cannot authenticate here.** It reads `ANTHROPIC_API_KEY` / `apiKeyHelper` only, never OAuth or the keychain;
this account is OAuth (`~/.claude`), so a `--bare` reviewer returns `terminal_reason: api_error`, cost 0. Used the
documented equivalents instead: `--setting-sources ""` + `--settings '{"disableAllHooks":true}'` + `--strict-mcp-config` +
`CLAUDE_CODE_DISABLE_CLAUDE_MDS=1` + `CLAUDE_CODE_DISABLE_AUTO_MEMORY=1` + `--tools Read,Grep,Glob`. The reviewers' cwd is the
work copy, whose `.claude/settings.json` carries the Stop hook: without `--setting-sources ""`/`disableAllHooks` a reviewer
would run the gate recursively. Verified: a dummy Stop hook in the probe copy never fired.
- **Inherited session env.** A matrix launched from inside Claude Code inherits `CLAUDE_EFFORT=medium`,
`CLAUDE_CODE_SUBAGENT_MODEL=opus`, `CLAUDE_CODE_SESSION_ID`, `CLAUDECODE`, `CLAUDE_CODE_CHILD_SESSION`, messaging tokens.
`env.sh` unsets every `CLAUDE*` var before setting its own; reviewers get a scrubbed env too (the hook process itself
inherits the worker's `CLAUDE_*` vars, e.g. the worker's session id).
- **The feedback line leaks the hook command.** Claude Code prefixes each block with the command line:
`Stop hook feedback:\n[<command>]: <reason>`. With the SPEC command inline the agent would read `GATE_CONFIG=g13adv` and
`gate-bench/gate.sh`. The settings command is therefore `bash "$CLAUDE_PROJECT_DIR/.claude/hooks/stop.sh"`, a one-line
wrapper holding the SPEC command. The agent can still `cat` it (it lives in the work copy) but it is not pushed at it.
- **Stop hook input keys (2.1.293):** `background_tasks, cwd, effort ({"level":"medium"}), hook_event_name,
last_assistant_message, permission_mode, prompt_id, session_crons, session_id, stop_hook_active, transcript_path`.
- **8-cap resets on tool calls.** Per the docs the consecutive-continuation count resets each time Claude calls a tool, so the
built-in cap only bites an agent that answers blocks with text alone. Our give-up (6 blocks per run, counted from
`gate.jsonl`, tool calls or not) fires first in every case.
- **Red start state: the agent refuses to fix code it did not write.** Validation step 2 (T1 + an uncommitted `>=` -> `>`
edit) shows Sonnet 5.5 implementing T1, then answering 6 consecutive gate-1 blocks with "the red test is your pre-existing
uncommitted edit, not my change" without touching it, until give_up. The hook mechanics are validated; the behaviour is a
finding in itself (gate 1 cannot force a fix the agent considers out of scope).
- **pytest-cov.** The repo's `pytest.ini` adds `--cov tinydb`; gate 1 adds `--no-cov` so the 40-line tail holds the failure,
not the coverage table (and no `.coverage` file is written). `.coverage` is gitignored anyway.
- **T5 R4 is parametrized** (several test ids, one requirement). hidden counts are per requirement (5 per task), a req
passes only when all its ids pass. `validate.sh` step 1: T5 upstream passes 1 req (R3, the regression guard) as designed.
- **Diff without touching the index.** `git add -N` would change what the agent sees in `git status`; untracked files are
diffed with `git diff --no-index /dev/null <f>` instead. `.claude/` is in `.git/info/exclude`, so the hook config never
shows in any diff or in `git status`.
- **Worker Bash denials.** The 2.1.293 permission layer denies some agent commands with "Contains brace with quote character
(expansion obfuscation)" even under acceptEdits + allow-list; agents route around it (visible as `permission_denied` events).
- **Reviewers on a correct solution still flag "blocking".** On the T1 reference patch, reviewer A marked "pull does not check
the field is a list" (not required by the task) as blocking; indep therefore blocks, the adv challenger rejected it. This is
exactly what false_blocks measures (hidden all green at that stop).
- **Shared plan meter.** Workers and reviewers bill the user's plan quota; `total_cost_usd` is the list-price equivalent the
CLI reports, not a bill. Nothing else heavy should run on the account during the matrix.
- **rtk rewrites interactive `grep`/`diff`** in the launching Claude session only; scripts here call binaries directly
(`/usr/bin/diff`/`cmp` for verification diffs).
- **CLI version:** `claude --version` = 2.1.293 (Claude Code); flags checked in `claude --help`: `--json-schema`, `--bare`, `--tools`,
`--permission-mode` (choices acceptEdits, auto, bypassPermissions, manual, dontAsk, plan), `--setting-sources`,
`--include-hook-events`. `--max-turns` is accepted though not listed in `--help`.
## Matrix launch (2026-10-08 16:33:54, after the gate-1 redesign)
`cd /Users/thomas/dev/gate-bench && SKIP_DONE=1 nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 &` -> PID 71317, 3 lanes.
The 5 bare r1 runs (terse prompts) are kept and skipped; 45 runs to go. Expected wall ~40-70 min: gated runs so far take
~30-70 s of worker wall + ~35-40 s of test writer (before the worker) + 1-35 s per hook call, more with blocks.
Then `.venv/bin/python analyze.py`.
SPEC.md
# gate-bench — SPEC (written 2026-10-08 by the /plan stage of the AIDive pipeline)
Goal: measure, on a real repo, what each of Shopify's Helix gates catches when rebuilt as a Claude Code
Stop hook, and what it costs. Output feeds a YouTube video; every number must be reproducible from this dir.
## Reuse (read-only, never modify)
- `/Users/thomas/dev/verify-loops-bench/upstream/` : tinydb @ 18d73a1 (git repo). Copy it as `upstream/` here.
- `/Users/thomas/dev/verify-loops-bench/hidden/` : `test_T1.py..test_T6.py`, `conftest.py`, `pytest.ini`. Copy as `hidden/`.
Hidden tests are the ORACLE: one `test_reqN` per requirement of the task prompt. They live OUTSIDE every work copy
and the agent never sees them. Run: `cd <tree> && python -m pytest -c <bench>/hidden/pytest.ini --rootdir <bench>/hidden <bench>/hidden/test_Tn.py -q`
(conftest puts cwd or `$TINYDB_TREE` first on sys.path). Check `tasks/README.md` there for the exact invocation and validate with `validate.sh`.
- `/Users/thomas/dev/verify-loops-bench/tasks/T1.md..T5.md` : the five task prompts, typed as a developer would. Copy as `tasks/`. T6 is NOT used.
- `/Users/thomas/dev/haiku-routing-bench/{env.sh,run.sh,extract.py,prep.sh,matrix.sh}` : patterns to copy (auth via the
user's default config dir, `CLAUDE_CODE_DISABLE_AUTO_MEMORY=1`, `.venv` with pytest, stream-json extraction of
`total_cost_usd`, `num_turns`, `usage`, `modelUsage`, fresh work copy per run, `--setting-sources project --strict-mcp-config`).
- Claude Code CLI 2.1.293 is installed. Docs that define the hook contract: https://code.claude.com/docs/en/hooks
(fetch with `curl -s https://r.jina.ai/https://code.claude.com/docs/en/hooks`): a Stop hook that exits 2 blocks the
stop and its stderr is fed back; `{"decision":"block","reason":"..."}` on stdout does the same; input JSON on stdin
carries `stop_hook_active`, `session_id`, `transcript_path`; after 8 consecutive continuations Claude Code overrides
the next block (`CLAUDE_CODE_STOP_HOOK_BLOCK_CAP` raises it); a hook past `timeout` (seconds, default 600) renders no decision.
## Worker
`claude -p "<task prompt>" --model claude-sonnet-5-5 --output-format stream-json --verbose --max-turns 60
--permission-mode acceptEdits --allowedTools "<same list as haiku-routing-bench/run.sh>" --setting-sources project
--strict-mcp-config`, cwd = fresh work copy `work/<run>/` (copy of upstream, `git` clean at HEAD). The task prompt is the
T-file verbatim plus one line appended for every config (bare included): "When you are done, stop and summarise what
you changed." No mention of gates in the prompt: the gate is the hook, not the instruction.
Work copy carries `.claude/settings.json` with the Stop hook for gated configs (none for `bare`):
`{"hooks":{"Stop":[{"hooks":[{"type":"command","command":"GATE_RUN=<abs run dir> GATE_CONFIG=<cfg> bash <bench>/gate.sh","timeout":1500}]}]}}`
(check the exact settings shape in the hooks doc; `timeout` is in seconds; verify the hook fires with a smoke run before the matrix).
## Gates (gate.sh, one script, dispatches on GATE_CONFIG)
Reads stdin JSON. Logs ONE line per invocation to `$GATE_RUN/gate.jsonl`: {ts, n (invocation index), stop_hook_active,
gate1:{ran, suite_pass, suite_fail, tests_touched (bool: any file under tests/ added or modified vs HEAD), pass},
gate3:{ran, variant, reviewers:[{id, approve, findings:[{severity, file, line, text}], cost_usd, turns, wall_s}], blocking_findings, pass},
hidden:{pass, fail, failed_reqs} (ground truth at this moment, run SILENTLY on the work copy; never shown to the agent),
decision ("block"|"allow"|"give_up"), reason (the text sent back, truncated 500), wall_s}.
Give-up: after 6 blocks in one run the hook allows and logs decision "give_up" (the video needs to count it; Claude
Code's own 8-cap would otherwise hide it). Never use stop_hook_active to allow: log it only.
Order inside one invocation: gate 1 (when the config has it) first; gate 3 only when gate 1 passed or is absent.
- Gate 1 (behaviour tests): `python -m pytest -q -x --no-header -p no:cacheprovider` on the work copy (the repo's own 226 tests plus
whatever the agent wrote). Block when red (reason = last 40 lines of pytest), OR when `tests_touched` is false
(reason: "No test covers the new behaviour. Add tests for every requirement in the task, run them, then stop.").
- Gate 3 (adversarial reviews): headless reviewers, each `claude -p --bare --model claude-sonnet-5-5 --tools "Read,Grep,Glob"
--max-turns 15 --output-format json --json-schema '<schema>' --permission-mode dontAsk` (confirm each flag exists in
`claude --help`; drop `--permission-mode dontAsk` if it errors), cwd = work copy, prompt = system framing + the task
prompt + `git diff HEAD` (tracked + untracked files, cap 60k chars) + "Find defects: requirements not met, edge cases
wrong, behaviour broken, tests that do not test the requirement. Report only real defects with file and line. approve=true
only when nothing blocks." Schema: {approve: boolean, findings: [{severity: "blocking"|"minor", file, line, text}]}.
Read `structured_output` from the JSON result; record `total_cost_usd`. Set `CLAUDE_CODE_DISABLE_AUTO_MEMORY=1`.
- variant `indep` (Shopify as written): two reviewers, same prompt, independent, no shared context; union of findings;
block if any blocking finding from either; reason = the blocking findings, numbered, "Fix every finding, re-run the
tests, then stop." Pass when BOTH approve.
- variant `adv` (reviewer + challenger, arXiv 2608.18167 shape): reviewer A as above; then challenger B gets A's
findings plus the diff and must, per finding, answer upheld|rejected with a reason, and may add findings A missed;
only upheld (or B-added) blocking findings block. Pass when nothing upheld blocks.
- Reason text is what the worker sees next turn: keep it factual, under 1500 chars.
## Configs (configs.json)
- `bare`: no hook.
- `g1`: gate 1 only.
- `g3`: gate 3 indep only.
- `g13`: gate 1 then gate 3 indep (Shopify's order).
- `g13adv`: gate 1 then gate 3 adv.
Gate 2 (UI) and gate 4 (engineer) are out of scope by design: tinydb is a library, Shopify allows skipping UI review for
logic-only changes; gate 4 is the human, measured as "what reached the human": the hidden failures left at the end.
## Per run outputs (runs/<task>-<cfg>-r<rep>/)
stream.jsonl, stderr.txt, exit_code.txt, wall_s.txt, gate.jsonl, post.diff, numstat.txt, suite.txt (final repo suite),
hidden.txt (final hidden run), result.json (worker: total_cost_usd, num_turns, usage, modelUsage, is_error, subtype,
final text), score.json: {task, cfg, rep, hidden_pass, hidden_fail, failed_reqs, suite_pass, suite_fail, blocks_total,
blocks_by_gate {g1, g3}, right_blocks (hidden failing at that moment), false_blocks (hidden all passing at that moment),
give_up, ended_blocked (the LAST gate.jsonl line is a block, i.e. Claude Code's cap or the agent ended anyway),
cost_worker_usd, cost_reviewers_usd, cost_total_usd, turns, wall_s, reviewer_findings_total, model_ok}.
Snapshot at every hook call: before gate 1 runs, `git diff HEAD` of the work copy to `$GATE_RUN/snap-<n>.diff`.
## Matrix
5 tasks (T1..T5) × 5 configs × 2 reps = 50 runs. `matrix.txt` ordered: rep 1 of everything first (config-contiguous:
bare, g1, g3, g13, g13adv, tasks rotated), then rep 2 in reverse config order. `matrix.sh` runs up to 3 runs in parallel
(each in its own work copy), appends one line per run to `matrix.log`, resumable (`SKIP_DONE=1` skips runs with score.json).
`analyze.py` → `results.json` + `results.md`: per config: hidden requirements passed / total, runs fully passing, mean
cost, mean wall, blocks per run, right vs false blocks per gate, give-ups, ended_blocked; per task the same; and the
answer to the video's questions: (1) does gate 1 alone move hidden pass rate vs bare, (2) does gate 3 indep add anything
over g1, (3) does adv beat indep, (4) false-block rate per gate, (5) cost per gate.
## Validation before the matrix (VALIDATION.md + validate.sh)
1. upstream fails every hidden test file; the reference patches in `/Users/thomas/dev/verify-loops-bench/reference/`
pass hidden + suite (reuse their validate.sh logic).
2. The hook fires: one smoke run of T1 on `g1` with a deliberately failing state (apply a patch that breaks one test)
must show a block in gate.jsonl, and the stream must show the agent continuing after "Stop hook feedback".
3. gate.sh called by hand with a fake stdin on a clean work copy returns allow, on a red copy returns exit 2 with the
pytest tail on stderr.
4. One reviewer call by hand returns valid structured_output against the schema.
5. Record the exact `claude --version`, the settings.json used, and the hook input keys seen.
## Deliverables
Everything under /Users/thomas/dev/gate-bench with README.md (layout, commands, traps found). When the matrix is launched,
run it with `nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 &` and report the PID; do not wait for it to finish.
Copy the scripts (not runs/, not work/) to /Users/thomas/dev/faceless-video-pipeline/jobs/shopify-four-gates-claude-code-20261008-1529/research/code/
with a one-line README. Report: what was validated, the smoke run's gate.jsonl, the matrix PID, expected duration.
VALIDATION.md
# Validation (2026-10-08, claude 2.1.293) — raw output in validation.log
1. **Oracle.** `./validate.sh step1`: on upstream every hidden req fails (T5: 1 passing, R3 = regression guard, as designed);
each reference patch passes its hidden file (T1-T4 5 tests, T5 6 tests = 5 reqs) and the repo's 226 tests. OK.
2. **Hook fires.** `INJECT=$PWD/inject/break.patch RUN_TAG=val2 ./run.sh g1 T1 1` (T1 + an uncommitted `>=` -> `>` break):
gate.jsonl = 7 calls, 6 `block` (g1, suite red 1 failed / 33 passed under -x) then `give_up`; the stream shows
`Stop hook feedback:` user messages followed by new assistant turns (the agent continued each time; it kept
explaining the red test is the user's pre-existing edit and never touched it). `stop_hook_active` false on call 1, true after. OK.
3. **gate.sh by hand** (`./validate.sh step3`, fake stdin): green copy (T1 reference + one new test) -> exit 0, `allow`,
empty stderr; red copy (break.patch) -> exit 2, `block`, stderr = pytest tail incl. `short test summary info`. OK.
4. **Reviewer by hand** (`./validate.sh step4`, `lib/review.py work/_val3-green T1 indep|adv`): every call returned a
schema-valid `structured_output`, cost $0.03-0.05 per pair, 10-45 s per call. OK.
5. **Recorded.** `claude --version` 2.1.293. settings.json (gated configs):
`{"hooks":{"Stop":[{"hooks":[{"type":"command","command":"bash \"$CLAUDE_PROJECT_DIR/.claude/hooks/stop.sh\"","timeout":1500}]}]}}`,
`.claude/hooks/stop.sh` = `GATE_RUN=<run dir> GATE_CONFIG=<cfg> bash /Users/thomas/dev/gate-bench/gate.sh`.
Hook input keys seen: background_tasks, cwd, effort, hook_event_name, last_assistant_message, permission_mode, prompt_id,
session_crons, session_id, stop_hook_active, transcript_path. Stream hook events in the smoke run: exactly one
`hook_started`/`hook_response` pair, `hook_name: Stop` (no user/global hooks: `--setting-sources project`).
Smoke (`runs/T1-g13adv-r1-smoke`, wrapper hook): hidden 5/5, suite 238/238, 1 hook call, gate 1 green with tests touched,
adv reviewer A approve (0 findings), challenger approve, allow. Worker $0.084 + reviewers $0.034 = $0.117, 12 turns, 66 s.
`runs/T1-g13adv-r1-smoke0` = the first smoke with the inline SPEC command (same outcome, $0.195, 69 s), superseded because the
feedback prefix would leak the command line.
## Gate 1 redesign (2026-10-08, after the coordinator's review): independent test writer
Gate 1 no longer re-runs only the implementer's own tests (it passed by construction: `T2-g1-r1` one allow, hidden R4
missed, suite green). Before the worker starts, `lib/testwriter.py` (headless Sonnet 5.5, reviewer isolation flags,
`--tools Read,Grep,Glob --max-turns 12 --json-schema {file}`, cwd = a pristine temp copy of upstream) gets ONLY
`tasks/<T>.md` and writes `runs/<run>/gate_tests/test_<T>.py` (+ hidden's conftest/pytest.ini) outside the work copy.
Gate 1 = repo suite AND those tests (`TINYDB_TREE` = work copy). `tests_touched` is logged, no longer a rule.
`./validate_writer.sh` (one writer call per task, files in `validation/writer/<T>/`; reference = verify-loops-bench patches):
| task | writer tests | pass on reference | pass on upstream | fail on reference (possible false blocks, kept) | writer $ / wall |
|---|---|---|---|---|---|
| T1 | 52 | 51 | 1 | `test_pull_on_non_list_field_raises_and_leaves_data_unchanged` | 0.099 / 39 s |
| T2 | 42 | 42 | 5 | none | 0.094 / 35 s |
| T3 | 35 | 34 | 11 | `test_rename_table_non_string_target_does_not_corrupt_database` | 0.077 / 34 s |
| T4 | 48 | 47 | 7 | `test_reverse_without_sort_by_reverses_default_order` | 0.092 / 42 s |
| T5 | 26 | 25 | 7 | `test_non_integer_size_is_rejected` | 0.086 / 30 s |
Tests passing on upstream are the "existing behaviour unchanged" guards the instruction asks for (plus a few that
accept "raises" as an outcome); every task's file fails on upstream overall. Each matrix run writes its own file, so
these counts are one sample; per-run counts land in `score.json` (`gate_tests_total`, `gate_tests_fail_final`).
Overlap with the 5 hidden requirements (judgement from test names + spot reads of the assertions):
- T1: 2/5 strict (R1 order, R5 push non-list raises). R2/R4 hedged (`try: ... except (KeyError, AttributeError, TypeError)`: a raise is accepted), R3 hedged (`count('x') < 2`, first-occurrence removal passes).
- T2: 4/5 (R1, R2, R3, R5 cache inclusive vs exclusive and other bounds); R4 hedged (accepts a TypeError).
- T3: 3/5 strict (R1 ids, R2 persistence + `tables()`, R5 stale cache / old name empty / id sequence); R3/R4 use `pytest.raises(Exception)`, not KeyError/ValueError (R4 does check both tables intact).
- T4: 4/5 (R1, R3, R4 limit 0 / negative ValueError, R5 cache not polluted + Document/doc_id); R2 hedged (checks order of docs that have the field and that none are dropped, not "missing last in both directions").
- T5: 5/5 (R1, R2 restart, R3 class/subclass default, R4 non-positive ValueError at construction, R5 close flushes).
The writer hedges exactly where the terse prompt is silent: it encodes "a raise or a no-op are both fine", so gate 1
cannot recover a requirement the prompt never states (T1 R2/R4, T4 R2).
Smoke after the redesign (`runs/T1-g13adv-r1-smoke`, with the deny rules): writer 45 tests ($0.071), first stop gate 1
green (232 suite + 45/45 gate tests), adv A+B approve, allow; hidden 2/5 (R2, R4, R5 missed). Total $0.189, 68 s.
`runs/T1-g13adv-r1-smoke-a` (first redesign smoke): stop 1 blocked by gate 1 (5 gate tests red, reason starts
"The acceptance tests for this change fail:"), the agent fixed push/pull type checks, stop 2 gate 1 green, both reviewers
errored (transient FileNotFoundError on spawn, logged, fail-open allow; one retry added since). The agent also tried
`cat .claude/hooks/stop.sh; find / -name test_T1.py` after the block (denied by the allow-list) -> deny rules added.
`runs/T1-g13adv-r1-smoke-b`: second redesign smoke, clean (reviewers ran, $0.167).
The run logs: runs/<task>-<config>-<rep>/gate.jsonl is the gate's own log for each of the 50 matrix runs, one line per stop (what gate 1 ran, what the reviewers answered, the decision), and meta.json names the task, configuration and repetition; the ten bare runs have only meta.json, since no gate ran.
runs/T1-g1-r1/gate.jsonl
{"ts": "2026-10-08T16:36:49", "n": 1, "stop_hook_active": false, "hook_input_keys": ["background_tasks", "cwd", "effort", "hook_event_name", "last_assistant_message", "permission_mode", "prompt_id", "session_crons", "session_id", "stop_hook_active", "transcript_path"], "gate1": {"ran": true, "suite_pass": 232, "suite_fail": 0, "tests_touched": true, "gate_tests_present": true, "gate_tests_pass": 41, "gate_tests_fail": 2, "gate_tests_failed": ["test_push_does_not_mutate_value_shared_between_documents", "test_push_failure_on_one_document_does_not_persist_partial_changes"], "pass": false}, "gate3": {"ran": false}, "hidden": {"pass": 2, "fail": 3, "failed_reqs": ["test_req2", "test_req4", "test_req5"]}, "blocked_by": "g1", "decision": "block", "reason": "The acceptance tests for this change fail:\n.............F........F.................... [100%]\n=================================== FAILURES ===================================\n______ test_push_failure_on_one_document_does_not_persist_partial_changes ______\nacceptance/test_T1.py:120: in test_push_failure_on_one_document_does_not_persist_partial_changes\n assert db.get(doc_id=1)['tags'] == []\nE assert [1] == []\nE \nE Left contains one more item: 1\nE Use -", "reason_chars": 1137, "wall_s": 1.2}
{"ts": "2026-10-08T16:37:26", "n": 2, "stop_hook_active": true, "hook_input_keys": ["background_tasks", "cwd", "effort", "hook_event_name", "last_assistant_message", "permission_mode", "prompt_id", "session_crons", "session_id", "stop_hook_active", "transcript_path"], "gate1": {"ran": true, "suite_pass": 232, "suite_fail": 0, "tests_touched": true, "gate_tests_present": true, "gate_tests_pass": 43, "gate_tests_fail": 0, "gate_tests_failed": [], "pass": true}, "gate3": {"ran": false}, "hidden": {"pass": 2, "fail": 3, "failed_reqs": ["test_req2", "test_req4", "test_req5"]}, "blocked_by": null, "decision": "allow", "reason": "", "reason_chars": 0, "wall_s": 1.1}
runs/T1-g1-r1/gate.jsonl on GitHub
Do this Monday
- Write the check as a script that exits 2 with the failure on stderr, and wire it as a Stop command hook with a timeout longer than your suite.
- Have a separate agent write the acceptance tests from the request alone, store them outside the repo, and deny the implementer reads of the hook, the settings and those tests.
- Log every stop to a file with the decision and whether the gate was red, so a gate that let work through is visible.
- Add your own counter that lets the loop out after six blocks. Claude Code's cap is eight consecutive blocks and resets on every tool call.
- Run reviewers headless with
--tools Read,Grep,Glob, an empty settings source and hooks disabled, so they cannot trigger the gate recursively. - If you keep a review gate, send its findings back as advice in the feedback instead of making them a bar.
- Count right blocks and false blocks for two weeks before you trust the gate.
- Remember that
--barereads only an API key: on a subscription login it returns an error, so isolate reviewers with settings flags instead.
Measurements
From results.md: 50 runs, 10 per configuration, 5 tasks times 2 repetitions. Hidden requirements are counted out of 50 per configuration. Blocks are written right/false: a right block means the hidden oracle had a failing requirement at that stop, a false block means it was fully green.
| Configuration | Hidden requirements passed | Fully clean runs | Mean cost per run | Mean wall seconds | Gate 1 blocks right/false | Gate 3 blocks right/false | Give-ups |
|---|---|---|---|---|---|---|---|
| bare (no hook) | 35/50 (70%) | 3 | $0.085 | 33 | 0/0 | 0/0 | 0 |
| g1 (acceptance tests) | 42/50 (84%) | 6 | $0.201 | 60 | 3/3 | 0/0 | 0 |
| g3 (two reviewers) | 34/50 (68%) | 3 | $0.151 | 54 | 0/0 | 1/0 | 0 |
| g13 (tests, then two reviewers) | 41/50 (82%) | 5 | $0.282 | 86 | 14/1 | 2/0 | 2 |
| g13adv (tests, then reviewer plus critic) | 39/50 (78%) | 6 | $0.276 | 86 | 11/1 | 0/0 | 1 |
Cost is the sum of the worker, the gate-1 writer and the gate-3 reviewers, as total_cost_usd reports it, a list-price equivalent and not a bill. Every run carried the configured model. Across the gated runs, gate 1 sent the agent back 28 times with a real requirement failing and 5 times with every requirement passing. Gate 3 blocked 3 times and never wrongly. Three loops gave up at six blocks, and Claude Code's own cap never fired.
Left for the human out of 50 requirements: 15 with no gate, 8 with the tests, 16 with the reviewers alone, 9 with tests and reviewers, 11 with tests and the reviewer plus critic. Where the tests moved a task: the sorted search went from 5 of 10 to 10 of 10 and the cache size task from 8 to 10. The push and pull task stayed at 4 of 10, because the request never stated what to do with a document missing the field.
Go further
- The UI gate. Shopify's gate 2 has a second model act as a design reviewer on a front-end repo s1. We skipped it because the repo is a library. A later test on a front-end repo is the place to measure it.
- The Ralph loop. It also keeps an agent going until a condition holds, with the exit living in a phrase the agent can write. Our exit is external: a script and a counter outside the agent's reach.
- The adversarial-review paper. Read the tables, not the summary: the reviewer plus critic shape s5 is the one that earns its tokens on paper, and our run did not reproduce a gain on this repo, so test it on yours before adopting it. Abstract page: s23.
- The block cap and timeout. Read the Stop section of the hooks reference before you rely on a hook as a hard stop s2, and the troubleshooting section of the hooks guide s19.
- A measured gate to compare your reporting with: s8.
FAQ
Can a Stop hook really stop Claude Code from finishing?
Mostly. Exit 2 prevents the stop and feeds stderr back s2, but after eight consecutive continuations Claude Code overrides the next block, and a hook that times out renders no decision s2. Log every stop so you can see when it let work through.
Why is the test writer a separate agent?
Our first version of gate 1 re-ran the implementer's own tests. It allowed every run on its first stop while a hidden requirement failed, because an agent's own tests test what it built. The redesigned writer sees only the request and the pre-change repo.
Why not use --bare for the reviewers?
It skips hooks and memory s20, but on a subscription login it reads only an API key and returned an api_error here. We isolated reviewers with an empty settings source, hooks disabled, a strict MCP config and read-only tools.
Did the reviewers ever block good work?
Not in our 50 runs: 3 blocks, none wrong. The cost was what they waved through. In two runs both reviewers saw the same crash on a missing sort field and rated it minor, and both runs ended at zero of five.
Does this prove reviewers never help?
No. It is one repo, one worker model, five tasks and two repetitions per cell. It says the reviewers did not move the pass rate here and the test gate did.