AIDiveTL;DR i werdykt
- Helix od Shopify przepuszcza każdy checkpoint przez cztery bramki po kolei: testy zachowania, przegląd UI, dwóch niezależnych recenzentów z izolowanym kontekstem i akceptację inżyniera s1. Post nie podaje odsetka zaliczeń. Odtworzyliśmy bramki 1, 3 i 4 jako hook Stop w Claude Code na prawdziwym repo i policzyliśmy.
- Nasza macierz: 50 runów, pięć zadań tinydb, pięć konfiguracji, Sonnet 5.5 w trybie headless. Bez bramki przeszło 35 z 50 ukrytych wymagań (70%). Z samą bramką testów akceptacyjnych 42 z 50 (84%).
- Bramka dwóch recenzentów w opisanej formie nie ruszyła odsetka zaliczeń: 34 z 50 samodzielnie, 41 z 50 po testach. Recenzent plus krytyk osiągnął 39 z 50.
- Werdykt: zachowaj bramkę testów, z piszącym, który nie jest implementującym, i testami, których agent nie może edytować. Spodziewaj się jednej fałszywej blokady na sześć i daj pętli licznik.
- Bramkę dwóch recenzentów, jeśli ją zachowasz, traktuj jako radę w informacji zwrotnej, nie jako poprzeczkę.
- Ograniczenia: jedno repo, jeden model workera, pięć zadań, dwa powtórzenia na komórkę. Różnica dwóch punktów między wierszami to szum. 14 punktów między brakiem bramki a bramką testów nim nie jest.
Co mówią źródła
Helix sam nazywa regułę: gdy bramka zawodzi, agent używa informacji zwrotnej, by poprawić implementację, i uruchamia kontrolę ponownie, a nie może obejść nieudanej kontroli dlatego, że uważa wynik za wystarczająco dobry s1. Bramka 3 to dwóch niezależnych agentów recenzentów z izolowanym kontekstem, każde ustalenie trzeba poprawić, a pętla powtarza się, aż obaj zatwierdzą s1. Zmiana czysto logiczna po prostu pomija bramkę przeglądu UI s1, dlatego nasze repo z biblioteką w stylu CLI nie ma bramki 2. Niezależne opracowanie klasyfikuje dowody dotyczące Helix jako studium przypadku, pochodzące od samego autora i niezmierzone, i zauważa, że nic nie waliduje sędziego: nie podano odsetka fałszywych blokad s6.
Własny poradnik Anthropic podaje problem: Claude kończy, gdy praca wygląda na skończoną, a bez kontroli, którą może uruchomić, „wygląda na skończoną” to jedyny dostępny sygnał s3. Wersją deterministyczną jest hook Stop, który uruchamia twoją kontrolę jako skrypt i blokuje zakończenie tury, dopóki nie przejdzie s3. Poradnik dodaje, że recenzent w świeżym kontekście subagenta widzi tylko diff i kryteria, a nie rozumowanie stojące za zmianą s3, a subagenty faktycznie startują ze świeżym, izolowanym oknem kontekstu s22.
Kontrakt hooka: exit 2 na Stop uniemożliwia Claude zakończenie s2, a komunikat ze stderr jest tym, co Claude dostaje jako powód s2. Każdy inny kod wyjścia, także exit 1, sam z siebie nie blokuje s2. Forma JSON to decision: "block" z wymaganym reason s2.
Bramka ma we własnej dokumentacji trzy wyjścia:
- Limit. Gdy hooki stop kontynuowały turę osiem razy z rzędu, Claude Code nadpisuje następną blokadę i kończy turę. Licznik zeruje się za każdym razem, gdy Claude wywoła narzędzie, a
CLAUDE_CODE_STOP_HOOK_BLOCK_CAPgo podnosi s2 s19. - Timeout. Hooki typu command mają domyślnie 600 sekund, a hook, który osiągnie timeout, jest anulowany, jego wyjście odrzucone, więc przy większości zdarzeń nie wydaje żadnej decyzji s2.
- Flaga.
stop_hook_activema wartość true, gdy Claude Code już kontynuuje z powodu hooka stop, więc skrypt może uniknąć blokowania z powodu warunku, który nigdy się nie spełni s2.
Jeden praktyk ostrzega, że tani audytor z małym kontekstem „gets things wrong a lot” i blokuje dobre sesje, więc nie powinien mieć ostatniego słowa s7. Zmierzona bramka Stop na 100 oznaczonych zatrzymaniach zablokowała 8 tur na 100, 7 z nich słusznie, z zastrzeżeniem, że etykiety pochodziły od modelu, a nie z ręcznej oceny s8. My raportujemy blokady tak samo: słuszne i fałszywe.
Flagi skryptowe, na których polegaliśmy: --bare pomija hooki, skille, pluginy, serwery MCP, auto memory i CLAUDE.md s20, --max-turns ogranicza liczbę tur agenta w trybie print s20, --json-schema zwraca zwalidowane wyjście w polu structured_output s21, a --output-format json niesie total_cost_usd, szacunek po stronie klienta, który może różnić się od twojego rachunku s21.
W sprawie bramki 3 publikacja warsztatowa z 2026 zmierzyła ten układ. Na LiveCodeBench: zero-shot 77%, pojedynczy recenzent 77%, dwóch recenzentów 75% na 105 zadaniach i 34 z 57 na trudnym podzbiorze s5. Publikacja nazywa wynik dwóch recenzentów wciąż mieszczącym się w szumie względem pojedynczego recenzenta, self-refine i zero-shot s5. Jej układ adwersaryjny, recenzent plus krytyk audytujący przegląd, zanim główny agent edytuje, osiągnął 87% s5. Na SWE-bench Verified osiągnął 75.2% wobec 71.6% zero-shot, przy około 4.5x większej liczbie tokenów s5. Publikacja: s23.
Zestaw
Stanowisko testowe to tinydb na commicie 18d73a1 z 226 przechodzącymi testami. Pięć zleceń napisanych tak, jak pisze je programista (operacje push i pull, zapytanie between, zmiana nazwy tabeli, wyszukiwanie z sortowaniem, rozmiar cache zapisu), każde z pięcioma ukrytymi wymaganiami, których agent nigdy nie widzi: razem 25. Workerem jest Sonnet 5.5 w trybie headless. Wszystko poniżej to uprząż i bramki. Checkout tinydb, ukryte testy i prompty zadań nie są dołączone: skieruj to na własne repo i napisz własną wyrocznię.
Bramka:
gate.sh: punkt wejścia hooka Stop, jednolinijkowy wrapper uruchamiającylib/hook.py.gate.sh
#!/usr/bin/env bash
# Stop hook entry point (bash 3.2 OK). All logic lives in lib/hook.py.
# Called as: GATE_RUN=<abs run dir> GATE_CONFIG=<cfg> bash <bench>/gate.sh (stdin = hook input JSON)
HERE="$(cd "$(dirname "$0")" && pwd)"
exec "$HERE/.venv/bin/python" -I "$HERE/lib/hook.py"
lib/hook.py
"""Stop hook: gate 1 (behaviour tests) and/or gate 3 (adversarial review), per GATE_CONFIG.
Env: GATE_RUN (abs run dir, holds meta.json + prompt.txt), GATE_CONFIG (key of configs.json).
stdin: the Stop hook input JSON. Block = exit 2 with the reason on stderr. Allow = exit 0, no output.
Every invocation appends ONE line to $GATE_RUN/gate.jsonl. Nothing but the reason ever reaches the agent.
"""
import json
import os
import sys
import time
import traceback
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from common import BENCH, run_hidden, run_suite, tests_touched, work_diff # noqa: E402
from testwriter import run_gate_tests # noqa: E402
GIVE_UP_AFTER = 6
def tail40(out, room=1400):
t = "\n".join(out.rstrip().splitlines()[-40:])
if len(t) > room:
t = "...\n" + t[-room:]
return t
def main():
t0 = time.time()
run = os.environ["GATE_RUN"]
cfg_name = os.environ["GATE_CONFIG"]
cfg = json.load(open(os.path.join(BENCH, "configs.json")))[cfg_name]
meta = json.load(open(os.path.join(run, "meta.json")))
work, task = meta["work"], meta["task"]
raw = sys.stdin.read()
try:
hin = json.loads(raw) if raw.strip() else {}
except Exception:
hin = {"_unparsed": raw[:500]}
log_path = os.path.join(run, "gate.jsonl")
prev = []
if os.path.exists(log_path):
prev = [json.loads(l) for l in open(log_path) if l.strip()]
n = len(prev) + 1
blocks_before = sum(1 for p in prev if p.get("decision") == "block")
# Snapshot of the work at this stop, before gate 1 runs.
open(os.path.join(run, "snap-%d.diff" % n), "w").write(work_diff(work))
# Ground truth at this moment (silent).
_, hout, hp, hf, hfailed = run_hidden(work, task)
open(os.path.join(run, "hidden-%d.txt" % n), "w").write(hout)
rec = {"ts": time.strftime("%Y-%m-%dT%H:%M:%S"), "n": n, "stop_hook_active": hin.get("stop_hook_active"),
"hook_input_keys": sorted(hin.keys()),
"gate1": {"ran": False}, "gate3": {"ran": False},
"hidden": {"pass": hp, "fail": hf, "failed_reqs": hfailed}}
reason, blocked_by = None, None
if cfg["gate1"]:
# Gate 1 = the repo suite AND the acceptance tests written before the run by the independent test writer
# (run dir gate_tests/, outside the work copy). A writer failure leaves the suite alone (logged).
rc, out, sp, sf = run_suite(work)
open(os.path.join(run, "gate1-%d.txt" % n), "w").write(out)
suite_ok = rc == 0 and sf == 0
gdir = os.path.join(run, "gate_tests")
has_gt = os.path.exists(os.path.join(gdir, "test_%s.py" % task))
g = {"ran": True, "suite_pass": sp, "suite_fail": sf, "tests_touched": tests_touched(work),
"gate_tests_present": has_gt}
gt_ok, gout = True, ""
if has_gt:
grc, gout, gp, gf, gnames = run_gate_tests(gdir, task, work)
open(os.path.join(run, "gatetests-%d.txt" % n), "w").write(gout)
gt_ok = grc == 0 and gf == 0
g.update(gate_tests_pass=gp, gate_tests_fail=gf, gate_tests_failed=gnames)
g["pass"] = suite_ok and gt_ok
rec["gate1"] = g
if not g["pass"]:
blocked_by = "g1"
parts = []
if not gt_ok:
parts.append("The acceptance tests for this change fail:\n" + tail40(gout, 1400 if suite_ok else 800))
if not suite_ok:
parts.append("The repo's test suite is red (python -m pytest -q -x):\n" + tail40(out, 1400 if gt_ok else 550))
reason = "\n\n".join(parts)
if cfg["gate3"] and blocked_by is None:
from review import gate3, reason_text
g3 = gate3(work, open(os.path.join(run, "task.md")).read(), cfg["gate3"])
rec["gate3"] = g3
if not g3["pass"]:
blocked_by = "g3"
reason = reason_text(g3)
if blocked_by is None:
decision = "allow"
elif blocks_before >= GIVE_UP_AFTER:
decision = "give_up"
else:
decision = "block"
rec["blocked_by"] = blocked_by
rec["decision"] = decision
rec["reason"] = (reason or "")[:500]
rec["reason_chars"] = len(reason or "")
rec["wall_s"] = round(time.time() - t0, 1)
with open(log_path, "a") as fh:
fh.write(json.dumps(rec) + "\n")
if n == 1:
json.dump(hin, open(os.path.join(run, "hook-input-1.json"), "w"), indent=2)
if decision == "block":
sys.stderr.write(reason)
sys.exit(2)
sys.exit(0)
if __name__ == "__main__":
try:
main()
except SystemExit:
raise
except Exception:
# A crashed gate must never deadlock a run: log and allow.
run = os.environ.get("GATE_RUN", "/tmp")
with open(os.path.join(run, "gate-errors.txt"), "a") as fh:
fh.write(traceback.format_exc() + "\n")
sys.exit(0)
lib/testwriter.py
"""Gate 1 test writer: an independent headless Sonnet 5.5 that sees ONLY the task prompt (plus read-only access to a
pristine upstream copy) and writes integration-style pytest cases. The file lives OUTSIDE the work copy, in
<out dir>/gate_tests/ next to a copy of hidden/conftest.py (TINYDB_TREE) and hidden/pytest.ini.
CLI: testwriter.py <task id> <out dir> -> writes <out dir>/gate_tests/test_<T>.py and <out dir>/testwriter.json
"""
import json
import os
import shutil
import subprocess
import sys
import tempfile
import time
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from common import BENCH, HIDDEN, PY, counts, env_clean, sh # noqa: E402
MODEL = "claude-sonnet-5-5"
CALL_TIMEOUT = 600
SCHEMA = {"type": "object", "properties": {"file": {"type": "string"}}, "required": ["file"]}
FRAMING = ("You are a senior engineer writing acceptance tests for a change request to tinydb, a small Python document "
"database. You can read the CURRENT repository (before the change) with Read, Grep and Glob to learn the API; "
"you cannot run code. The change is not implemented yet: your tests must describe the requested behaviour.")
ASK = ("Write integration-style pytest cases for this request against the tinydb API, one test per behaviour a careful "
"senior would expect, pushing for edge cases beyond the happy path (missing fields, wrong types, boundaries, "
"persistence, cache, existing behaviour unchanged). Tests only, no implementation. Return the file content.\n"
"Constraints: one self-contained pytest module (field `file`); `tinydb` is importable (the tree under test is put "
"first on sys.path); do not import anything from the repo's tests/ directory or its fixtures; use "
"tinydb.storages.MemoryStorage or pytest's tmp_path for files; import the new API inside each test, not at module "
"level, so one missing name fails its tests instead of the whole module; give every test a descriptive name.")
def write(task, out_dir, attempts=2):
gt = os.path.join(out_dir, "gate_tests")
os.makedirs(gt, exist_ok=True)
shutil.copy(os.path.join(HIDDEN, "conftest.py"), gt)
shutil.copy(os.path.join(HIDDEN, "pytest.ini"), gt)
task_prompt = open(os.path.join(BENCH, "tasks", task + ".md")).read().strip()
prompt = "%s\n\n## The request (as the developer typed it)\n%s\n\n%s" % (FRAMING, task_prompt, ASK)
rec = {"task": task, "attempts": [], "cost_usd": 0.0, "ok": False, "file": None}
pristine = tempfile.mkdtemp(prefix="gb-writer-")
try:
shutil.copytree(os.path.join(BENCH, "upstream"), os.path.join(pristine, "tinydb-repo"), symlinks=True)
cwd = os.path.join(pristine, "tinydb-repo")
for i in range(attempts):
t0 = time.time()
a = {"error": None}
args = ["claude", "-p", prompt, "--model", MODEL, "--tools", "Read,Grep,Glob", "--max-turns", "12",
"--output-format", "json", "--json-schema", json.dumps(SCHEMA), "--permission-mode", "dontAsk",
"--setting-sources", "", "--settings", '{"disableAllHooks":true}', "--strict-mcp-config"]
env = env_clean({"CLAUDE_CODE_DISABLE_AUTO_MEMORY": "1", "CLAUDE_CODE_DISABLE_CLAUDE_MDS": "1"})
try:
p = subprocess.run(args, cwd=cwd, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
stdin=subprocess.DEVNULL, timeout=CALL_TIMEOUT, env=env)
r = json.loads(p.stdout.decode("utf-8", "replace"))
a.update(cost_usd=r.get("total_cost_usd") or 0.0, turns=r.get("num_turns"),
models=sorted((r.get("modelUsage") or {}).keys()), subtype=r.get("subtype"))
so = r.get("structured_output")
src = so.get("file") if isinstance(so, dict) else None
if r.get("is_error") or not src:
a["error"] = "is_error=%s subtype=%s terminal=%s" % (r.get("is_error"), r.get("subtype"),
r.get("terminal_reason"))
else:
compile(src, "test_%s.py" % task, "exec")
if "def test_" not in src:
raise ValueError("no test functions")
open(os.path.join(gt, "test_%s.py" % task), "w").write(src)
rec["ok"], rec["file"] = True, os.path.join(gt, "test_%s.py" % task)
except subprocess.TimeoutExpired:
a["error"] = "timeout after %ss" % CALL_TIMEOUT
except Exception as e:
a["error"] = "%s: %s" % (type(e).__name__, str(e)[:300])
a["wall_s"] = round(time.time() - t0, 1)
rec["attempts"].append(a)
rec["cost_usd"] += a.get("cost_usd") or 0.0
if rec["ok"]:
break
finally:
shutil.rmtree(pristine, ignore_errors=True)
rec["cost_usd"] = round(rec["cost_usd"], 6)
rec["wall_s"] = round(sum(a["wall_s"] for a in rec["attempts"]), 1)
rec["models"] = sorted({m for a in rec["attempts"] for m in a.get("models") or []})
json.dump(rec, open(os.path.join(out_dir, "testwriter.json"), "w"), indent=2)
return rec
def run_gate_tests(gate_dir, task, tree, timeout=300):
"""Generated tests on `tree` (TINYDB_TREE = tree, cwd = tree). Returns (rc, out, passed, failed, failed_names).
Paths to the gate_tests dir are rewritten to `acceptance/` so the agent never learns where the file is."""
f = os.path.join(gate_dir, "test_%s.py" % task)
rc, out = sh([PY, "-m", "pytest", "-q", "--tb=short", "-rfE", "--no-header", "-c", os.path.join(gate_dir, "pytest.ini"),
"--rootdir", gate_dir, f], tree, timeout=timeout, env=env_clean({"TINYDB_TREE": tree}))
out = out.replace(os.path.realpath(gate_dir), "acceptance").replace(gate_dir, "acceptance")
rel = os.path.relpath(gate_dir, tree)
out = out.replace(rel, "acceptance")
import re
names = sorted(set(re.findall(r"(?:FAILED|ERROR) \S*::(test_\w+)", out)))
p, fl = counts(out)
if rc != 0 and fl == 0:
fl = max(1, len(names))
return rc, out, p, fl, names
if __name__ == "__main__":
r = write(sys.argv[1], os.path.abspath(sys.argv[2]))
print(json.dumps({k: r[k] for k in ("task", "ok", "cost_usd", "wall_s", "file")}))
lib/review.py
"""Gate 3 reviewers: headless `claude -p` calls returning schema-validated JSON.
CLI (validation step 4): review.py <work dir> <task id> [indep|adv] -> prints the gate3 record as JSON
"""
import json
import subprocess
import os
import sys
import time
from concurrent.futures import ThreadPoolExecutor
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from common import BENCH, env_clean, sh, work_diff # noqa: E402
MODEL = "claude-sonnet-5-5"
DIFF_CAP = 60000
CALL_TIMEOUT = 600 # seconds per reviewer call; the hook timeout is 1500
FINDING = {"type": "object",
"properties": {"severity": {"type": "string", "enum": ["blocking", "minor"]},
"file": {"type": "string"}, "line": {"type": "integer"}, "text": {"type": "string"}},
"required": ["severity", "file", "line", "text"]}
SCHEMA_REVIEW = {"type": "object",
"properties": {"approve": {"type": "boolean"}, "findings": {"type": "array", "items": FINDING}},
"required": ["approve", "findings"]}
SCHEMA_CHALLENGE = {"type": "object",
"properties": {
"verdicts": {"type": "array", "items": {
"type": "object",
"properties": {"id": {"type": "integer"},
"verdict": {"type": "string", "enum": ["upheld", "rejected"]},
"reason": {"type": "string"}},
"required": ["id", "verdict", "reason"]}},
"added_findings": {"type": "array", "items": FINDING},
"approve": {"type": "boolean"}},
"required": ["verdicts", "added_findings", "approve"]}
FRAMING = ("You are a senior engineer doing a pre-merge code review of a change to tinydb, a small Python "
"document database. You can read the repository with Read, Grep and Glob (cwd = the repo with the "
"change applied). You cannot run code. Below: the task the developer was given, then the full diff "
"(tracked and new files).")
ASK = ("Find defects: requirements not met, edge cases wrong, behaviour broken, tests that do not test the "
"requirement. Report only real defects with file and line. approve=true only when nothing blocks.")
def capped_diff(work):
d = work_diff(work)
if len(d) > DIFF_CAP:
d = d[:DIFF_CAP] + "\n[diff truncated at %d chars]\n" % DIFF_CAP
return d
def call(work, prompt, schema, rid):
"""One headless reviewer. Never raises: an error is recorded and counts as approve (fail open)."""
# Equivalent of --bare that still authenticates with the user's OAuth login (--bare reads
# ANTHROPIC_API_KEY only): no settings sources, hooks disabled, no MCP, no CLAUDE.md, no auto-memory.
args = ["claude", "-p", prompt, "--model", MODEL, "--tools", "Read,Grep,Glob", "--max-turns", "15",
"--output-format", "json", "--json-schema", json.dumps(schema), "--permission-mode", "dontAsk",
"--setting-sources", "", "--settings", '{"disableAllHooks":true}', "--strict-mcp-config"]
env = env_clean({"CLAUDE_CODE_DISABLE_AUTO_MEMORY": "1", "CLAUDE_CODE_DISABLE_CLAUDE_MDS": "1"})
t0 = time.time()
rec = {"id": rid, "approve": True, "findings": [], "cost_usd": 0.0, "turns": None, "wall_s": None, "error": None}
try:
try:
p = subprocess.run(args, cwd=work, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.DEVNULL,
timeout=CALL_TIMEOUT, env=env)
except OSError as e1: # seen once in a smoke run: transient FileNotFoundError on spawn -> one retry
rec["retry_after"] = "%r filename=%s" % (e1, getattr(e1, "filename", None))
time.sleep(3)
p = subprocess.run(args, cwd=work, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.DEVNULL,
timeout=CALL_TIMEOUT, env=env)
out = p.stdout.decode("utf-8", "replace")
try:
r = json.loads(out)
except Exception:
r = None
if not isinstance(r, dict):
rec["error"] = "no json (rc=%s): %s %s" % (p.returncode, out[-300:], p.stderr.decode("utf-8", "replace")[-300:])
else:
rec["cost_usd"] = r.get("total_cost_usd") or 0.0
rec["turns"] = r.get("num_turns")
rec["models"] = sorted((r.get("modelUsage") or {}).keys())
so = r.get("structured_output")
if r.get("is_error") or not isinstance(so, dict):
rec["error"] = "is_error=%s subtype=%s terminal=%s structured_output=%s" % (
r.get("is_error"), r.get("subtype"), r.get("terminal_reason"), type(so).__name__)
else:
rec["raw"] = so
except subprocess.TimeoutExpired:
rec["error"] = "timeout after %ss" % CALL_TIMEOUT
except Exception as e: # pragma: no cover
rec["error"] = "exception: %r filename=%s path_head=%s" % (e, getattr(e, "filename", None), os.environ.get("PATH", "")[:200])
rec["wall_s"] = round(time.time() - t0, 1)
return rec
def norm_findings(lst):
out = []
for f in lst or []:
if isinstance(f, dict):
out.append({"severity": f.get("severity") if f.get("severity") in ("blocking", "minor") else "minor",
"file": str(f.get("file", "")), "line": f.get("line"), "text": str(f.get("text", ""))})
return out
def review_prompt(task_prompt, diff):
return "%s\n\n## Task given to the developer\n%s\n\n## Diff (git diff HEAD, new files included)\n```diff\n%s\n```\n\n%s" % (
FRAMING, task_prompt.strip(), diff, ASK)
def challenge_prompt(task_prompt, diff, a_findings):
listing = "\n".join("[%d] (%s) %s:%s %s" % (i + 1, f["severity"], f["file"], f["line"], f["text"])
for i, f in enumerate(a_findings)) or "(reviewer A reported no findings)"
return ("%s\n\nA first reviewer (A) already reviewed this diff. You are the challenger: for EACH of A's findings "
"decide whether it is a real defect (upheld) or not (rejected: wrong, not required by the task, already "
"handled, or speculative), with a one-sentence reason grounded in the code; check the code before you "
"decide. Then add any real blocking defect A missed in added_findings (file and line). approve=true only "
"when no upheld or added finding blocks.\n\n## Task given to the developer\n%s\n\n## Diff (git diff HEAD, "
"new files included)\n```diff\n%s\n```\n\n## Reviewer A's findings (id in brackets)\n%s\n\n%s" % (
FRAMING, task_prompt.strip(), diff, listing,
"Answer every id in verdicts. Report only real defects."))
def gate3(work, task_prompt, variant):
diff = capped_diff(work)
reviewers, blocking = [], []
if variant == "indep":
prompt = review_prompt(task_prompt, diff)
with ThreadPoolExecutor(max_workers=2) as ex:
recs = list(ex.map(lambda rid: call(work, prompt, SCHEMA_REVIEW, rid), ["A", "B"]))
for rec in recs:
raw = rec.pop("raw", None) or {}
rec["findings"] = norm_findings(raw.get("findings"))
rec["approve_raw"] = raw.get("approve", True) if not rec["error"] else None
rb = [f for f in rec["findings"] if f["severity"] == "blocking"]
rec["approve"] = not rb # effective verdict = no blocking finding (an errored call approves)
blocking += [dict(f, by=rec["id"]) for f in rb]
reviewers.append(rec)
elif variant == "adv":
a = call(work, review_prompt(task_prompt, diff), SCHEMA_REVIEW, "A")
raw = a.pop("raw", None) or {}
a["findings"] = norm_findings(raw.get("findings"))
a["approve_raw"] = raw.get("approve", True) if not a["error"] else None
a["approve"] = not [f for f in a["findings"] if f["severity"] == "blocking"]
b = call(work, challenge_prompt(task_prompt, diff, a["findings"]), SCHEMA_CHALLENGE, "B")
rawb = b.pop("raw", None) or {}
verdicts = {}
for v in rawb.get("verdicts") or []:
if isinstance(v, dict) and isinstance(v.get("id"), int):
verdicts[v["id"]] = {"verdict": v.get("verdict"), "reason": str(v.get("reason", ""))}
b["verdicts"] = [dict(id=k, **verdicts[k]) for k in sorted(verdicts)]
b["findings"] = norm_findings(rawb.get("added_findings"))
b["approve_raw"] = rawb.get("approve", True) if not b["error"] else None
# Only upheld findings of A block. An errored challenger upholds nothing (fail open, logged).
for i, f in enumerate(a["findings"]):
v = verdicts.get(i + 1)
f["challenge"] = v["verdict"] if v else ("error" if b["error"] else "unanswered")
if f["severity"] == "blocking" and v and v["verdict"] == "upheld":
blocking.append(dict(f, by="A", upheld_by="B"))
blocking += [dict(f, by="B") for f in b["findings"] if f["severity"] == "blocking"]
b["approve"] = not blocking
reviewers = [a, b]
else:
raise ValueError(variant)
return {"ran": True, "variant": variant, "reviewers": reviewers, "blocking_findings": len(blocking),
"blocking": blocking, "pass": not blocking,
"cost_usd": round(sum(r.get("cost_usd") or 0 for r in reviewers), 6), "diff_chars": len(diff)}
def reason_text(g3):
lines = ["Code review found %d blocking defect(s):" % g3["blocking_findings"]]
for i, f in enumerate(g3["blocking"]):
lines.append("%d. %s:%s %s" % (i + 1, f["file"], f["line"], " ".join(f["text"].split())))
tail = "Fix every finding, re-run the tests, then stop."
body = "\n".join(lines)
room = 1500 - len(tail) - 2
if len(body) > room:
body = body[:room - 4] + " ..."
return body + "\n" + tail
if __name__ == "__main__":
work, task = sys.argv[1], sys.argv[2]
variant = sys.argv[3] if len(sys.argv) > 3 else "indep"
tp = open(os.path.join(BENCH, "tasks", task + ".md")).read()
g3 = gate3(os.path.abspath(work), tp, variant)
print(json.dumps(g3, indent=2))
if not g3["pass"]:
print("--- reason ---\n" + reason_text(g3))
lib/common.py
"""Shared helpers: work-copy diff, pytest runs and parsing. Used by hook.py, review.py, finalize.py."""
import os
import re
import subprocess
BENCH = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
PY = os.path.join(BENCH, ".venv", "bin", "python")
HIDDEN = os.path.join(BENCH, "hidden")
def sh(args, cwd, timeout=None, env=None):
"""Run a command, capture stdout+stderr together. Returns (rc, text). rc=124 on timeout."""
try:
p = subprocess.run(args, cwd=cwd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
stdin=subprocess.DEVNULL, timeout=timeout, env=env)
return p.returncode, p.stdout.decode("utf-8", "replace")
except subprocess.TimeoutExpired as e:
out = (e.stdout or b"").decode("utf-8", "replace")
return 124, out + "\n[timed out after %ss]" % timeout
def work_diff(work):
"""`git diff HEAD` of tracked files plus every untracked (non-ignored) file as a new-file diff.
Never touches the index (no `git add -N`), so the agent's `git status` is unaffected."""
_, d = sh(["git", "diff", "HEAD", "--no-color", "--", ".", ":(exclude).coverage"], work)
_, others = sh(["git", "ls-files", "--others", "--exclude-standard"], work)
for f in [l for l in others.splitlines() if l.strip()]:
_, nd = sh(["git", "diff", "--no-color", "--no-index", "--", "/dev/null", f], work)
d += nd
return d
def tests_touched(work):
_, mod = sh(["git", "diff", "HEAD", "--name-only", "--", "tests"], work)
_, new = sh(["git", "ls-files", "--others", "--exclude-standard", "--", "tests"], work)
return bool(mod.strip() or new.strip())
def counts(txt):
"""(passed, failed+errors) from a pytest summary line."""
p = re.findall(r"(\d+) passed", txt)
f = re.findall(r"(\d+) failed", txt)
e = re.findall(r"(\d+) errors?\b", txt)
return (int(p[-1]) if p else 0), (int(f[-1]) if f else 0) + (int(e[-1]) if e else 0)
def env_clean(extra=None):
env = {k: v for k, v in os.environ.items() if not k.startswith("CLAUDE") and not k.startswith("GATE_")}
env["PATH"] = os.path.join(BENCH, ".venv", "bin") + os.pathsep + env.get("PATH", "/usr/bin:/bin")
env["PYTHONDONTWRITEBYTECODE"] = "1"
env.update(extra or {})
return env
def run_suite(work, timeout=300):
"""Gate 1 / final suite: the repo's own pytest run (226 tests + whatever the agent wrote).
--no-cov: the repo's pytest.ini adds --cov; the coverage table would fill the 40-line tail."""
rc, out = sh([PY, "-m", "pytest", "-q", "-x", "--no-header", "-p", "no:cacheprovider", "--no-cov"],
work, timeout=timeout, env=env_clean())
p, f = counts(out)
if rc != 0 and f == 0:
f = 1 # collection error, timeout, crash: count as red
return rc, out, p, f
def run_hidden(work, task, timeout=300):
"""Ground truth: hidden tests on the work copy (cwd = work copy, TINYDB_TREE set). Output never reaches the agent."""
rc, out = sh([PY, "-m", "pytest", "-q", "-rA", "-c", os.path.join(HIDDEN, "pytest.ini"), "--rootdir", HIDDEN,
os.path.join(HIDDEN, "test_%s.py" % task)],
work, timeout=timeout, env=env_clean({"TINYDB_TREE": work}))
# Requirement level (T5 R4 is parametrized: one req = several test ids). A req passes iff none of its
# test ids failed or errored; a run that collected nothing counts every req as failed.
reqs = sorted(set(re.findall(r"^def (test_req\d+)", open(os.path.join(HIDDEN, "test_%s.py" % task)).read(), re.M)))
failed = sorted(set(re.findall(r"(?:FAILED|ERROR) \S+::(test_req\d+)", out)))
p_tests, f_tests = counts(out)
if rc != 0 and not failed:
failed = list(reqs)
if p_tests == 0 and f_tests == 0:
failed = list(reqs)
return rc, out, len(reqs) - len(failed), len(failed), failed
lib/finalize.py
"""After the worker exits: final diff, suite, hidden run, result.json and score.json. Prints one line.
Usage: finalize.py <run dir> (re-runnable: rescoring an existing run only reads/rewrites files in it)"""
import json
import os
import re
import sys
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from common import PY, env_clean, run_hidden, sh, work_diff # noqa: E402
from testwriter import run_gate_tests # noqa: E402
MODEL = "claude-sonnet-5-5"
out = os.path.abspath(sys.argv[1])
meta = json.load(open(os.path.join(out, "meta.json")))
work, task, cfg, rep = meta["work"], meta["task"], meta["cfg"], meta["rep"]
def w(name, txt):
open(os.path.join(out, name), "w").write(txt)
def rd(name, default=""):
try:
return open(os.path.join(out, name)).read().strip()
except Exception:
return default
# Final state of the work copy
w("post.diff", work_diff(work))
_, ns = sh(["git", "diff", "HEAD", "--numstat", "--", ".", ":(exclude).coverage"], work)
_, others = sh(["git", "ls-files", "--others", "--exclude-standard"], work)
for f in [l for l in others.splitlines() if l.strip()]:
try:
ns += "%d\t0\t%s\n" % (sum(1 for _ in open(os.path.join(work, f), errors="replace")), f)
except Exception:
pass
w("numstat.txt", ns)
rc, suite = sh([PY, "-m", "pytest", "-q", "--no-header", "-p", "no:cacheprovider", "--no-cov"], work,
timeout=300, env=env_clean())
w("suite.txt", suite)
sp = int((re.findall(r"(\d+) passed", suite) or [0])[-1])
sf = int((re.findall(r"(\d+) failed", suite) or [0])[-1]) + int((re.findall(r"(\d+) errors?\b", suite) or [0])[-1])
if rc != 0 and sf == 0:
sf = 1
_, hout, hp, hf, hfailed = run_hidden(work, task)
w("hidden.txt", hout)
# Worker result line
lines = []
for raw in open(os.path.join(out, "stream.jsonl"), errors="replace") if os.path.exists(os.path.join(out, "stream.jsonl")) else []:
raw = raw.strip()
if raw.startswith("{"):
try:
lines.append(json.loads(raw))
except Exception:
pass
result = next((o for o in reversed(lines) if o.get("type") == "result"), None) or {}
init = next((o for o in lines if o.get("type") == "system" and o.get("subtype") == "init"), {})
res = {k: result.get(k) for k in ("total_cost_usd", "num_turns", "usage", "modelUsage", "is_error", "subtype",
"stop_reason", "terminal_reason", "duration_ms")}
res["final_text"] = (result.get("result") or "")[-3000:]
res["has_result_line"] = bool(result)
json.dump(res, open(os.path.join(out, "result.json"), "w"), indent=2)
# Hook events seen in the stream (which hooks ran)
hook_events = sorted({"%s:%s" % (o.get("hook_event") or o.get("hook_event_name") or "?", o.get("hook_name") or "")
for o in lines if o.get("type") == "system" and str(o.get("subtype", "")).startswith("hook")})
# Gate log
gl = [json.loads(l) for l in open(os.path.join(out, "gate.jsonl")) if l.strip()] if os.path.exists(os.path.join(out, "gate.jsonl")) else []
blocks = [g for g in gl if g.get("decision") == "block"]
by_gate = {"g1": sum(1 for g in blocks if g.get("blocked_by") == "g1"),
"g3": sum(1 for g in blocks if g.get("blocked_by") == "g3")}
right = {k: sum(1 for g in blocks if g.get("blocked_by") == k and g["hidden"]["fail"] > 0) for k in ("g1", "g3")}
false = {k: sum(1 for g in blocks if g.get("blocked_by") == k and g["hidden"]["fail"] == 0) for k in ("g1", "g3")}
rev_cost, rev_findings, rev_errors, rev_models = 0.0, 0, 0, set()
for g in gl:
for r in (g.get("gate3") or {}).get("reviewers") or []:
rev_cost += r.get("cost_usd") or 0
rev_findings += len(r.get("findings") or [])
rev_errors += 1 if r.get("error") else 0
rev_models |= set(r.get("models") or [])
# Gate-1 test writer (gated-1 configs only): cost and the final state of its tests on the work copy
tw = {}
try:
tw = json.load(open(os.path.join(out, "testwriter.json")))
except Exception:
pass
cost_tw = tw.get("cost_usd") or 0.0
rev_models |= set(tw.get("models") or [])
gt_total = gt_fail = None
gdir = os.path.join(out, "gate_tests")
if os.path.exists(os.path.join(gdir, "test_%s.py" % task)):
_, gout, gp, gf, _ = run_gate_tests(gdir, task, work)
w("gatetests-final.txt", gout)
gt_total, gt_fail = gp + gf, gf
mu = result.get("modelUsage") or {}
cw = result.get("total_cost_usd") or 0.0
s = {
"task": task, "cfg": cfg, "rep": rep, "run": meta["name"],
"hidden_pass": hp, "hidden_fail": hf, "failed_reqs": hfailed,
"suite_pass": sp, "suite_fail": sf,
"hook_calls": len(gl), "blocks_total": len(blocks), "blocks_by_gate": by_gate,
"right_blocks": sum(right.values()), "false_blocks": sum(false.values()),
"right_blocks_by_gate": right, "false_blocks_by_gate": false,
"give_up": any(g.get("decision") == "give_up" for g in gl),
"ended_blocked": bool(gl) and gl[-1].get("decision") == "block",
"cost_worker_usd": round(cw, 6), "cost_reviewers_usd": round(rev_cost, 6), "cost_testwriter_usd": round(cost_tw, 6),
"cost_total_usd": round(cw + rev_cost + cost_tw, 6),
"testwriter_ok": tw.get("ok") if tw else None, "gate_tests_total": gt_total, "gate_tests_fail_final": gt_fail,
"turns": result.get("num_turns"), "wall_s": int(rd("wall_s.txt", "0") or 0),
"reviewer_findings_total": rev_findings, "reviewer_errors": rev_errors,
"gate_crashes": rd("gate-errors.txt").count("Traceback"),
"model_ok": init.get("model") == MODEL and bool(mu) and set(mu) <= {MODEL} and rev_models <= {MODEL},
"worker_models": sorted(mu), "reviewer_models": sorted(rev_models),
"is_error": result.get("is_error"), "subtype": result.get("subtype"), "exit_code": rd("exit_code.txt"),
"has_result_line": bool(result), "claude_code_version": init.get("claude_code_version"),
"hook_events_in_stream": hook_events,
"hidden_first_stop": gl[0]["hidden"] if gl else None,
}
json.dump(s, open(os.path.join(out, "score.json"), "w"), indent=2)
print("%s: hidden %d/%d suite %d/%d blocks=%d (g1 %d, g3 %d; right %d, false %d) give_up=%s ended_blocked=%s "
"gate_tests_fail=%s/%s cost=$%.3f (worker %.3f + rev %.3f + writer %.3f) turns=%s wall=%ss model_ok=%s" % (
s["run"], hp, hp + hf, sp, sp + sf, len(blocks), by_gate["g1"], by_gate["g3"], s["right_blocks"],
s["false_blocks"], s["give_up"], s["ended_blocked"], gt_fail, gt_total, s["cost_total_usd"], cw, rev_cost, cost_tw, s["turns"],
s["wall_s"], s["model_ok"]))
Ustawienia .claude opisuje HARNESS.md: settings.json z jednym poleceniem hooka Stop (timeout 1500), które wywołuje jednolinijkowy stop.sh, oraz lista permissions.deny blokująca odczyt hooka, ustawień i ukrytych testów. Recenzenci i piszący działają z pustym źródłem ustawień i wyłączonymi hookami, więc bramka nigdy nie wpada w rekurencję.
HARNESS.md
# gate-bench
What does each of Shopify's Helix gates catch when rebuilt as a Claude Code **Stop hook**, and what does it
cost? Headless Sonnet 5.5 workers implement five tinydb features (T1..T5); a Stop hook runs gate 1
(behaviour tests) and/or gate 3 (adversarial review by headless Sonnet 5.5 reviewers) every time the worker
tries to stop; hidden tests (the oracle, never visible to the agent) record the ground truth at every stop
and at the end. Built 2026-10-08 from `SPEC.md`; upstream, hidden tests, tasks and reference patches copied
from `../verify-loops-bench` (nothing there or in `../haiku-routing-bench` is modified).
## Commands (from this dir)
```bash
./validate.sh # steps 1,3,4 (+ step 2 once its run exists) -> validation.log
INJECT=$PWD/inject/break.patch RUN_TAG=val2 ./run.sh g1 T1 1 # validation step 2 (red start state)
RUN_TAG=smoke ./run.sh g13adv T1 1 # smoke (tagged runs are ignored by analyze.py)
./validate_writer.sh # gate-1 test writer on T1..T5: tests on upstream vs reference (REWRITE=1 to redo)
./gen_matrix.py # matrix.txt (50 runs, already generated)
nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 & # 3 runs in parallel (MATRIX_JOBS=3)
SKIP_DONE=1 nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 & # resume
.venv/bin/python analyze.py # results.json + results.md
.venv/bin/python lib/finalize.py runs/<run> # rescore one run from its work copy
.venv/bin/python lib/review.py work/<run> T1 adv # one gate-3 round by hand
```
Single run: `./run.sh <cfg> <task> <rep>` -> `runs/<task>-<cfg>-r<rep>/`, work copy `work/<same name>/`.
## Layout
| path | what |
|---|---|
| `SPEC.md` | the spec this harness implements |
| `upstream/` | tinydb @ 18d73a1 (git repo) |
| `hidden/` | `test_T1..T6.py` (T6 unused), `conftest.py` (`$TINYDB_TREE` or cwd first on sys.path), `pytest.ini` |
| `tasks/T1..T5.md` + `tasks/README.md` | the task prompts (verbatim) and what each hidden req checks |
| `reference/T1..T5.patch` | reference solutions (validation step 1) |
| `inject/break.patch` | `>=` -> `>` in `Query.__ge__` (one repo test red; validation steps 2-3) |
| `inject/test_pushpull.py` | one thin test, makes the T1 reference copy "green with tests touched" (step 3) |
| `configs.json` | `bare` (no hook), `g1`, `g3` (indep), `g13` (g1 then g3 indep), `g13adv` (g1 then g3 adv) |
| `env.sh` | scrubs inherited `CLAUDE*` vars, default `~/.claude` auth, `CLAUDE_CODE_DISABLE_AUTO_MEMORY=1`, venv on PATH |
| `prep.sh <cfg> <work> <run>` | fresh copy of upstream, `.claude/` in `.git/info/exclude`, hook settings for gated configs |
| `run.sh <cfg> <task> <rep>` | one run (worker + hook + `lib/finalize.py`) |
| `gate.sh` | the Stop hook (bash 3.2 wrapper: `exec .venv/bin/python -I lib/hook.py`) |
| `lib/hook.py` | gate dispatch, snapshot, hidden run, gate 1, gate 3, decision, `gate.jsonl` line |
| `lib/review.py` | gate-3 reviewers (indep / adv), schemas, prompts, reason text; CLI for a manual round |
| `lib/testwriter.py` | gate-1 test writer (before the worker) + `run_gate_tests` (path rewritten to `acceptance/`) |
| `validate_writer.sh`, `validation/writer/<T>/` | writer validation files and their runs on upstream / reference |
| `tasks/` (terse prompts, used), `tasks-explicit/` + `runs-explicit/` | the matrix uses terse prompts; see `tasks/README.md` |
| `lib/common.py` | diff incl. untracked files, suite run, hidden run (requirement level) |
| `lib/finalize.py` | post-run diff/suite/hidden, `result.json`, `score.json` |
| `gen_matrix.py`, `matrix.txt`, `matrix.sh`, `matrix-one.sh`, `matrix.log` | 50-run matrix, 3 in parallel (xargs -P), resumable |
| `analyze.py` -> `results.json`, `results.md` | per config, per task, the five questions |
| `validate.sh`, `VALIDATION.md`, `validation.log` | pre-matrix checks |
| `.venv/` | python 3.14.7, pytest 9.1.1, pytest-cov 7.1.0, pyyaml, ruff 0.16.10 (same pins as haiku-routing-bench) |
### Worker (run.sh)
`claude -p "<T-file verbatim>\n\nWhen you are done, stop and summarise what you changed." --model claude-sonnet-5-5
--output-format stream-json --verbose --include-hook-events --max-turns 60 --permission-mode acceptEdits
--allowedTools <haiku-routing-bench list> --setting-sources project --strict-mcp-config`, cwd = `work/<run>/`, stdin /dev/null.
Gated configs get `work/<run>/.claude/settings.json`:
`{"permissions":{"deny":[...see traps...]},"hooks":{"Stop":[{"hooks":[{"type":"command","command":"bash \"$CLAUDE_PROJECT_DIR/.claude/hooks/stop.sh\"","timeout":1500}]}]}}`
and `.claude/hooks/stop.sh` = `GATE_RUN=<abs run dir> GATE_CONFIG=<cfg> bash <bench>/gate.sh` (the SPEC command, one level down: see traps).
Copies of both land in the run dir (`settings.json`, `stop.sh`).
### Hook (lib/hook.py), one invocation
1. read stdin JSON; n = lines already in `gate.jsonl` + 1; blocks so far from the same file
2. `snap-<n>.diff` = `git diff HEAD` + untracked files (index never touched)
3. hidden tests on the work copy (`TINYDB_TREE` = work copy, cwd = work copy) -> `hidden-<n>.txt`; never shown to the agent
4. gate 1 (if configured) = the repo suite `python -m pytest -q -x --no-header -p no:cacheprovider --no-cov` (300 s)
-> `gate1-<n>.txt` AND the writer's acceptance tests `runs/<run>/gate_tests/test_<T>.py` run with `TINYDB_TREE` = work
copy -> `gatetests-<n>.txt`. Block when either is red. Reason: "The acceptance tests for this change fail:" + last 40
lines (failing test names + assertion output, the gate_tests path rewritten to `acceptance/`), then the suite tail if the
suite is red too; <= 1500 chars. `tests_touched` is still logged but no longer blocks. No writer file (writer failed
twice) -> suite only, logged as `gate_tests_present: false`.
5. gate 3 (if configured and gate 1 passed or absent): see below
6. decision: allow / block / give_up (a would-be block when 6 blocks already happened -> allow, logged `give_up`).
`stop_hook_active` is logged, never used. Block = exit 2 + reason on stderr. A crash of the hook itself is
logged to `gate-errors.txt` and allows (counted as `gate_crashes` in score.json).
### Gate 1 test writer (lib/testwriter.py, run.sh before the worker, configs g1/g13/g13adv)
`claude -p` Sonnet 5.5 with the reviewer isolation flags, `--tools Read,Grep,Glob --max-turns 12 --json-schema
{"file": string}`, cwd = a pristine temp copy of upstream (deleted after). Prompt = framing + `tasks/<T>.md` + the
coordinator's instruction ("Write integration-style pytest cases for this request against the tinydb API, one test per
behaviour a careful senior would expect, pushing for edge cases beyond the happy path (missing fields, wrong types,
boundaries, persistence, cache, existing behaviour unchanged). Tests only, no implementation. Return the file content.")
+ mechanics (self-contained module, imports of the new API inside each test, MemoryStorage/tmp_path). The file must
compile and contain `def test_`, else one retry. Output `runs/<run>/gate_tests/` (+ hidden's conftest.py with
`TINYDB_TREE`, pytest.ini) and `testwriter.json` (cost, turns, attempts). Fixed per-run cost, outside the worker's wall.
### Gate 3 (lib/review.py)
Each reviewer: `claude -p "<prompt>" --model claude-sonnet-5-5 --tools Read,Grep,Glob --max-turns 15 --output-format json
--json-schema <schema> --permission-mode dontAsk --setting-sources "" --settings '{"disableAllHooks":true}' --strict-mcp-config`
with `CLAUDE_CODE_DISABLE_CLAUDE_MDS=1 CLAUDE_CODE_DISABLE_AUTO_MEMORY=1`, all inherited `CLAUDE*` vars dropped, cwd = work copy,
600 s timeout. Prompt = framing + task prompt (T-file) + diff (tracked + untracked, cap 60k chars) + the SPEC's "Find defects..."
sentence. `structured_output` is read from the JSON result; `total_cost_usd`, `num_turns`, wall and models are recorded.
- `indep`: reviewers A and B, same prompt, run in parallel, no shared context. Blocking findings = union. Pass when both approve.
- `adv`: A as above; challenger B gets A's numbered findings + the diff, answers `upheld|rejected` + reason per id, may add
findings (`added_findings`). Blocking = A's blocking findings B upheld + B's added blocking findings. B always runs (same
2-call budget as indep).
- Effective `approve` = "no blocking finding" (the raw `approve` is logged as `approve_raw`; a reviewer saying approve=false with
only minor findings does not block). A reviewer call that errors approves (fail open) with `error` recorded; an errored
challenger upholds nothing.
- Reason = "Code review found N blocking defect(s):" + numbered `file:line text` + "Fix every finding, re-run the tests, then stop.", <= 1500 chars.
### Per run (`runs/<task>-<cfg>-r<rep>/`)
`stream.jsonl`, `stderr.txt`, `exit_code.txt`, `wall_s.txt` (includes hook time: the hook runs inside the session),
`gate.jsonl`, `snap-<n>.diff`, `hidden-<n>.txt`, `gate1-<n>.txt`, `hook-input-1.json`, `post.diff`, `numstat.txt`,
`suite.txt` (full suite, no -x), `hidden.txt`, `gate_tests/`, `testwriter.json`, `gatetests-<n>.txt`, `gatetests-final.txt`, `result.json`, `score.json`, `meta.json`, `task.md`, `prompt.txt`,
`settings.json` + `stop.sh` (gated configs).
`score.json`: task, cfg, rep, hidden_pass/hidden_fail/failed_reqs (**requirement level**: 5 reqs per task), suite_pass/fail,
hook_calls, blocks_total, blocks_by_gate {g1,g3}, right_blocks / false_blocks (+ `_by_gate`), give_up, ended_blocked,
cost_worker_usd, cost_reviewers_usd, cost_testwriter_usd, cost_total_usd (all three), testwriter_ok, gate_tests_total,
gate_tests_fail_final, turns, wall_s, reviewer_findings_total, reviewer_errors, gate_crashes,
model_ok (worker init model + every modelUsage key + every reviewer model == claude-sonnet-5-5), hook_events_in_stream,
hidden_first_stop.
## Matrix
5 tasks x 5 configs x 2 reps = 50 runs. Rep 1 config blocks bare, g1, g3, g13, g13adv (tasks rotated per block), rep 2 the
reverse block order. Up to 3 runs at once, each in its own work copy (`matrix-one.sh` via `xargs -P 3`, bash 3.2 has no
`wait -n`). `matrix.log`: one `start` and one result line per run.
## Traps found
- **Gate 1 as first built passed by construction** (it only re-ran the implementer's own tests: `T2-g1-r1` one allow, hidden
R4 missed). Redesigned around an independent test writer (above). The writer hedges where the terse prompt is silent
(accepts "raises or no-op"), so it cannot recover unstated requirements; per-task overlap is in VALIDATION.md.
- **A blocked agent hunts for the tests.** After a gate-1 block the smoke agent ran `cat .claude/hooks/stop.sh; find / -name
test_T1.py` (denied only because `find` is not on the allow-list; `cat` is, and `stop.sh` names the bench dir, from where
`hidden/` is one `cat` away). Gated configs' `.claude/settings.json` now carries `permissions.deny`: Read of `./.claude/**`,
the bench's `hidden/ runs/ validation/ reference/`, the two sibling benches; Bash patterns containing `.claude/hooks`,
`.claude/settings`, `../hidden`, `../runs`, `gate_tests`, the bench sub-paths, `find *`. Probed with a Haiku session: all
five escape attempts (cat, Read, ls, python open) denied, in-tree reads fine. Bash rules match command text, not a security
boundary. Bare runs have no settings file (bare r1 ran before this change; bare has no motive to hunt).
Reviewers and the writer run with `--setting-sources ""`, so these rules do not apply to them.
- **Transient `FileNotFoundError` spawning a reviewer** inside the hook (once, both reviewers of one call, wall 0.0 s;
never reproduced by hand). Reviewer calls now retry once after 3 s; the error message records `filename`.
- **`--bare` cannot authenticate here.** It reads `ANTHROPIC_API_KEY` / `apiKeyHelper` only, never OAuth or the keychain;
this account is OAuth (`~/.claude`), so a `--bare` reviewer returns `terminal_reason: api_error`, cost 0. Used the
documented equivalents instead: `--setting-sources ""` + `--settings '{"disableAllHooks":true}'` + `--strict-mcp-config` +
`CLAUDE_CODE_DISABLE_CLAUDE_MDS=1` + `CLAUDE_CODE_DISABLE_AUTO_MEMORY=1` + `--tools Read,Grep,Glob`. The reviewers' cwd is the
work copy, whose `.claude/settings.json` carries the Stop hook: without `--setting-sources ""`/`disableAllHooks` a reviewer
would run the gate recursively. Verified: a dummy Stop hook in the probe copy never fired.
- **Inherited session env.** A matrix launched from inside Claude Code inherits `CLAUDE_EFFORT=medium`,
`CLAUDE_CODE_SUBAGENT_MODEL=opus`, `CLAUDE_CODE_SESSION_ID`, `CLAUDECODE`, `CLAUDE_CODE_CHILD_SESSION`, messaging tokens.
`env.sh` unsets every `CLAUDE*` var before setting its own; reviewers get a scrubbed env too (the hook process itself
inherits the worker's `CLAUDE_*` vars, e.g. the worker's session id).
- **The feedback line leaks the hook command.** Claude Code prefixes each block with the command line:
`Stop hook feedback:\n[<command>]: <reason>`. With the SPEC command inline the agent would read `GATE_CONFIG=g13adv` and
`gate-bench/gate.sh`. The settings command is therefore `bash "$CLAUDE_PROJECT_DIR/.claude/hooks/stop.sh"`, a one-line
wrapper holding the SPEC command. The agent can still `cat` it (it lives in the work copy) but it is not pushed at it.
- **Stop hook input keys (2.1.293):** `background_tasks, cwd, effort ({"level":"medium"}), hook_event_name,
last_assistant_message, permission_mode, prompt_id, session_crons, session_id, stop_hook_active, transcript_path`.
- **8-cap resets on tool calls.** Per the docs the consecutive-continuation count resets each time Claude calls a tool, so the
built-in cap only bites an agent that answers blocks with text alone. Our give-up (6 blocks per run, counted from
`gate.jsonl`, tool calls or not) fires first in every case.
- **Red start state: the agent refuses to fix code it did not write.** Validation step 2 (T1 + an uncommitted `>=` -> `>`
edit) shows Sonnet 5.5 implementing T1, then answering 6 consecutive gate-1 blocks with "the red test is your pre-existing
uncommitted edit, not my change" without touching it, until give_up. The hook mechanics are validated; the behaviour is a
finding in itself (gate 1 cannot force a fix the agent considers out of scope).
- **pytest-cov.** The repo's `pytest.ini` adds `--cov tinydb`; gate 1 adds `--no-cov` so the 40-line tail holds the failure,
not the coverage table (and no `.coverage` file is written). `.coverage` is gitignored anyway.
- **T5 R4 is parametrized** (several test ids, one requirement). hidden counts are per requirement (5 per task), a req
passes only when all its ids pass. `validate.sh` step 1: T5 upstream passes 1 req (R3, the regression guard) as designed.
- **Diff without touching the index.** `git add -N` would change what the agent sees in `git status`; untracked files are
diffed with `git diff --no-index /dev/null <f>` instead. `.claude/` is in `.git/info/exclude`, so the hook config never
shows in any diff or in `git status`.
- **Worker Bash denials.** The 2.1.293 permission layer denies some agent commands with "Contains brace with quote character
(expansion obfuscation)" even under acceptEdits + allow-list; agents route around it (visible as `permission_denied` events).
- **Reviewers on a correct solution still flag "blocking".** On the T1 reference patch, reviewer A marked "pull does not check
the field is a list" (not required by the task) as blocking; indep therefore blocks, the adv challenger rejected it. This is
exactly what false_blocks measures (hidden all green at that stop).
- **Shared plan meter.** Workers and reviewers bill the user's plan quota; `total_cost_usd` is the list-price equivalent the
CLI reports, not a bill. Nothing else heavy should run on the account during the matrix.
- **rtk rewrites interactive `grep`/`diff`** in the launching Claude session only; scripts here call binaries directly
(`/usr/bin/diff`/`cmp` for verification diffs).
- **CLI version:** `claude --version` = 2.1.293 (Claude Code); flags checked in `claude --help`: `--json-schema`, `--bare`, `--tools`,
`--permission-mode` (choices acceptEdits, auto, bypassPermissions, manual, dontAsk, plan), `--setting-sources`,
`--include-hook-events`. `--max-turns` is accepted though not listed in `--help`.
## Matrix launch (2026-10-08 16:33:54, after the gate-1 redesign)
`cd /Users/thomas/dev/gate-bench && SKIP_DONE=1 nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 &` -> PID 71317, 3 lanes.
The 5 bare r1 runs (terse prompts) are kept and skipped; 45 runs to go. Expected wall ~40-70 min: gated runs so far take
~30-70 s of worker wall + ~35-40 s of test writer (before the worker) + 1-35 s per hook call, more with blocks.
Then `.venv/bin/python analyze.py`.
Runner:
run.sh: jeden run, worker plus hook plus ocena.run.sh
#!/usr/bin/env bash
# One run: fresh work copy, headless Sonnet 5.5 worker, Stop hook per config, final scoring.
# Usage: run.sh <cfg> <task> <rep> cfg: bare|g1|g3|g13|g13adv task: T1..T5
# RUN_TAG=x -> runs/<task>-<cfg>-r<rep>-x (smoke runs stay out of the matrix)
# INJECT=<patch> -> git apply it to the work copy before the worker starts (validation: red start state)
set -uo pipefail
CFG="$1"; TASK="$2"; REP="${3:-1}"
HERE="$(cd "$(dirname "$0")" && pwd)"
. "$HERE/env.sh"
PY="$HERE/.venv/bin/python"
MAX_TURNS="${BENCH_MAX_TURNS:-60}"
NAME="${TASK}-${CFG}-r${REP}${RUN_TAG:+-$RUN_TAG}"
OUT="$HERE/runs/$NAME"; W="$HERE/work/$NAME"
rm -rf "$OUT"; mkdir -p "$OUT"
"$HERE/prep.sh" "$CFG" "$W" "$OUT" > "$OUT/prep.txt" 2>&1 || { echo "$NAME: prep failed"; cat "$OUT/prep.txt"; exit 2; }
if [ -n "${INJECT:-}" ]; then ( cd "$W" && git apply "$INJECT" ) || { echo "$NAME: inject failed"; exit 2; }; fi
cp "$HERE/tasks/$TASK.md" "$OUT/task.md"
printf '%s\n\nWhen you are done, stop and summarise what you changed.\n' "$(cat "$HERE/tasks/$TASK.md")" > "$OUT/prompt.txt"
printf '{"task":"%s","cfg":"%s","rep":%s,"work":"%s","name":"%s"}\n' "$TASK" "$CFG" "$REP" "$W" "$NAME" > "$OUT/meta.json"
# Gate 1 configs: the independent test writer runs BEFORE the worker (fixed per-run cost), sees only tasks/<T>.md,
# writes $OUT/gate_tests/test_<T>.py outside the work copy.
if "$PY" -c "import json,sys; sys.exit(0 if json.load(open('$HERE/configs.json'))['$CFG']['gate1'] else 1)"; then
"$PY" "$HERE/lib/testwriter.py" "$TASK" "$OUT" > "$OUT/testwriter.out" 2>&1
fi
[ -f "$W/.claude/settings.json" ] && cp "$W/.claude/settings.json" "$OUT/settings.json" && cp "$W/.claude/hooks/stop.sh" "$OUT/stop.sh"
# Same allow-list as haiku-routing-bench/run.sh (verify-loops-bench's + Bash(ruff:*)).
TOOLS="Read,Edit,Write,MultiEdit,Glob,Grep,Skill,TodoWrite,Bash(uv:*),Bash(python:*),Bash(python3:*),Bash(pytest:*),Bash(ruff:*),Bash(git status:*),Bash(git diff:*),Bash(git log:*),Bash(ls:*),Bash(cat:*),Bash(mkdir:*),Bash(mktemp:*),Bash(head:*),Bash(tail:*),Bash(grep:*),Bash(wc:*)"
cd "$W"
START=$(date +%s)
claude -p "$(cat "$OUT/prompt.txt")" --model claude-sonnet-5-5 \
--output-format stream-json --verbose --include-hook-events --max-turns "$MAX_TURNS" \
--permission-mode acceptEdits --allowedTools "$TOOLS" \
--setting-sources project --strict-mcp-config \
< /dev/null > "$OUT/stream.jsonl" 2> "$OUT/stderr.txt"
echo "$?" > "$OUT/exit_code.txt"
END=$(date +%s)
echo "$((END-START))" > "$OUT/wall_s.txt"
"$PY" "$HERE/lib/finalize.py" "$OUT"
prep.sh
#!/usr/bin/env bash
# Fresh work copy of upstream tinydb (git clean at HEAD) + the Stop hook settings for gated configs.
# Usage: prep.sh <cfg> <work dir> <run dir>
set -euo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"
CFG="$1"; W="$2"; RUN="$3"
rm -rf "$W"; mkdir -p "$(dirname "$W")"
cp -R "$HERE/upstream" "$W"
cd "$W"
/usr/bin/find . -name __pycache__ -type d -prune -exec rm -rf {} +
rm -f .coverage
# .claude/ is excluded locally (not committed, not in .gitignore): `git status` stays clean,
# the hook config never shows up in the diffs the reviewers and the scorer read.
echo ".claude/" >> .git/info/exclude
git update-index -q --refresh || true
if [ "$CFG" != "bare" ]; then
# The settings command is a neutral wrapper: Claude Code prefixes every "Stop hook feedback" with the
# hook's command line, so the SPEC command (GATE_RUN=... GATE_CONFIG=<cfg> bash <bench>/gate.sh) would
# show the agent the config name and the bench path. The wrapper holds that exact command instead.
mkdir -p .claude/hooks
printf '#!/bin/bash\nGATE_RUN=%s GATE_CONFIG=%s bash %s/gate.sh\n' "$RUN" "$CFG" "$HERE" > .claude/hooks/stop.sh
"$HERE/.venv/bin/python" - "$HERE" > .claude/settings.json <<'PY'
import json
import sys
bench = sys.argv[1]
cmd = 'bash "$CLAUDE_PROJECT_DIR/.claude/hooks/stop.sh"'
# Once a gate blocks, the agent hunts for the tests behind it (seen in a smoke: `cat .claude/hooks/stop.sh;
# find / -name test_T1.py`). The oracle and the gate tests live under the bench: deny the usual forms.
# Read rules cover Read/Grep/Glob; Bash rules match the command text (not a security boundary).
deny = ["Read(./.claude/**)", "Read(/%s/hidden/**)" % bench, "Read(/%s/runs/**)" % bench,
"Read(/%s/validation/**)" % bench, "Read(/%s/reference/**)" % bench,
"Read(//Users/thomas/dev/verify-loops-bench/**)", "Read(//Users/thomas/dev/haiku-routing-bench/**)",
"Bash(*.claude/hooks*)", "Bash(*.claude/settings*)", "Bash(*gate-bench/hidden*)", "Bash(*../hidden*)",
"Bash(*gate-bench/runs*)", "Bash(*../runs*)", "Bash(*gate_tests*)", "Bash(*gate-bench/reference*)",
"Bash(*gate-bench/validation*)", "Bash(*verify-loops-bench*)", "Bash(*haiku-routing-bench*)", "Bash(find *)"]
print(json.dumps({"permissions": {"deny": deny},
"hooks": {"Stop": [{"hooks": [{"type": "command", "command": cmd, "timeout": 1500}]}]}}, indent=2))
PY
fi
test -z "$(git status --porcelain)" || { echo "work copy not clean"; git status --porcelain; exit 1; }
env.sh
# Sourced by run.sh, gate.sh and validate.sh.
# 1. Scrub every CLAUDE* variable inherited from a launching Claude Code session
# (CLAUDE_EFFORT, CLAUDE_CODE_SUBAGENT_MODEL, CLAUDE_CODE_SESSION_ID, CLAUDECODE,
# CLAUDE_CODE_CHILD_SESSION, ...): they would silently change the worker's effort,
# subagent model and session bookkeeping.
for _v in $(env | sed -n 's/^\(CLAUDE[A-Za-z0-9_]*\)=.*/\1/p'); do unset "$_v"; done
unset _v
# 2. Auth: the user's default config dir (~/.claude, OAuth). See README traps (--bare cannot auth).
unset CLAUDE_CONFIG_DIR
# 3. Fresh work copies at the same path must not share auto-memory across reps.
export CLAUDE_CODE_DISABLE_AUTO_MEMORY=1
# 4. No bytecode in the shared hidden/ dir when 3 runs execute hidden tests in parallel.
export PYTHONDONTWRITEBYTECODE=1
# 5. The bench venv first: `python -m pytest` inside the work copy uses it.
GATE_BENCH="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
export GATE_BENCH
export PATH="$GATE_BENCH/.venv/bin:$PATH"
configs.json
{
"bare": {"gate1": false, "gate3": null, "desc": "no hook"},
"g1": {"gate1": true, "gate3": null, "desc": "gate 1 (behaviour tests) only"},
"g3": {"gate1": false, "gate3": "indep", "desc": "gate 3, two independent reviewers"},
"g13": {"gate1": true, "gate3": "indep", "desc": "gate 1 then gate 3 indep (Shopify order)"},
"g13adv": {"gate1": true, "gate3": "adv", "desc": "gate 1 then gate 3 reviewer + challenger"}
}
gen_matrix.py
#!/usr/bin/env python3
"""Write matrix.txt: "<cfg> <task> <rep>" lines. Rep 1: configs bare,g1,g3,g13,g13adv as contiguous blocks,
tasks rotated per block; rep 2: same blocks in reverse config order (ABBA over time)."""
CFGS = ["bare", "g1", "g3", "g13", "g13adv"]
TASKS = ["T1", "T2", "T3", "T4", "T5"]
lines = []
for rep, order in ((1, CFGS), (2, CFGS[::-1])):
lines.append("# rep %d" % rep)
for i, c in enumerate(order):
k = (i + rep - 1) % len(TASKS)
for t in TASKS[k:] + TASKS[:k]:
lines.append("%s %s %d" % (c, t, rep))
open("matrix.txt", "w").write("\n".join(lines) + "\n")
print(sum(1 for l in lines if not l.startswith("#")), "runs")
matrix.txt
# rep 1
bare T1 1
bare T2 1
bare T3 1
bare T4 1
bare T5 1
g1 T2 1
g1 T3 1
g1 T4 1
g1 T5 1
g1 T1 1
g3 T3 1
g3 T4 1
g3 T5 1
g3 T1 1
g3 T2 1
g13 T4 1
g13 T5 1
g13 T1 1
g13 T2 1
g13 T3 1
g13adv T5 1
g13adv T1 1
g13adv T2 1
g13adv T3 1
g13adv T4 1
# rep 2
g13adv T2 2
g13adv T3 2
g13adv T4 2
g13adv T5 2
g13adv T1 2
g13 T3 2
g13 T4 2
g13 T5 2
g13 T1 2
g13 T2 2
g3 T4 2
g3 T5 2
g3 T1 2
g3 T2 2
g3 T3 2
g1 T5 2
g1 T1 2
g1 T2 2
g1 T3 2
g1 T4 2
bare T1 2
bare T2 2
bare T3 2
bare T4 2
bare T5 2
matrix.sh
#!/usr/bin/env bash
# Run an ordered list of "<cfg> <task> <rep>" lines, up to MATRIX_JOBS (default 3) at a time, each run in
# its own work copy. Blank lines and #comments skipped. Appends to matrix.log. Resumable: SKIP_DONE=1.
# Usage: nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 &
set -uo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"; cd "$HERE"
LIST="${1:-matrix.txt}"; JOBS="${MATRIX_JOBS:-3}"
mkdir "$HERE/.matrix.lock" 2>/dev/null || { echo "another matrix.sh is running (remove .matrix.lock if stale)" >&2; exit 1; }
trap 'rmdir "$HERE/.matrix.lock"' EXIT
echo "=== $(date '+%F %T') start $LIST jobs=$JOBS pid=$$" >> matrix.log
grep -vE '^[[:space:]]*(#|$)' "$LIST" | xargs -P "$JOBS" -L 1 ./matrix-one.sh
echo "=== $(date '+%F %T') done $LIST" >> matrix.log
matrix-one.sh
#!/usr/bin/env bash
# One matrix cell (called by matrix.sh through xargs). Args: <cfg> <task> <rep>
HERE="$(cd "$(dirname "$0")" && pwd)"; cd "$HERE"
CFG="$1"; TASK="$2"; REP="$3"
if [ "${SKIP_DONE:-0}" = 1 ] && [ -f "runs/${TASK}-${CFG}-r${REP}/score.json" ]; then
echo "$(date '+%F %T') skip ${TASK}-${CFG}-r${REP}" >> matrix.log; exit 0
fi
echo "$(date '+%F %T') start ${TASK}-${CFG}-r${REP}" >> matrix.log
line=$(./run.sh "$CFG" "$TASK" "$REP" < /dev/null 2>&1 | tail -1)
echo "$(date '+%F %T') $line" >> matrix.log
echo "$line"
analyze.py
#!/usr/bin/env python3
"""Aggregate runs/*/score.json (matrix runs only: <task>-<cfg>-r<rep>, no tag) -> results.json + results.md."""
import glob
import json
import os
import re
HERE = os.path.dirname(os.path.abspath(__file__))
CFGS = ["bare", "g1", "g3", "g13", "g13adv"]
TASKS = ["T1", "T2", "T3", "T4", "T5"]
rows = []
for p in sorted(glob.glob(os.path.join(HERE, "runs", "*", "score.json"))):
if re.fullmatch(r"T\d-[a-z0-9]+-r\d+", os.path.basename(os.path.dirname(p))):
rows.append(json.load(open(p)))
def mean(xs):
xs = [x for x in xs if x is not None]
return round(sum(xs) / len(xs), 4) if xs else None
def agg(rs):
if not rs:
return None
n = len(rs)
hp = sum(r["hidden_pass"] for r in rs)
ht = sum(r["hidden_pass"] + r["hidden_fail"] for r in rs)
rb = {g: sum(r["right_blocks_by_gate"][g] for r in rs) for g in ("g1", "g3")}
fb = {g: sum(r["false_blocks_by_gate"][g] for r in rs) for g in ("g1", "g3")}
return {
"runs": n, "hidden_reqs_passed": hp, "hidden_reqs_total": ht, "hidden_rate": round(hp / ht, 4) if ht else None,
"runs_fully_passing": sum(1 for r in rs if r["hidden_fail"] == 0),
"suite_green_runs": sum(1 for r in rs if r["suite_fail"] == 0),
"mean_cost_total_usd": mean([r["cost_total_usd"] for r in rs]),
"mean_cost_worker_usd": mean([r["cost_worker_usd"] for r in rs]),
"mean_cost_reviewers_usd": mean([r["cost_reviewers_usd"] for r in rs]),
"mean_cost_testwriter_usd": mean([r.get("cost_testwriter_usd") or 0 for r in rs]),
# per-gate cost: gate 1 = the test writer (its tests then run locally for free), gate 3 = the reviewers
"mean_cost_gate1_usd": mean([r.get("cost_testwriter_usd") or 0 for r in rs]),
"mean_cost_gate3_usd": mean([r["cost_reviewers_usd"] for r in rs]),
"gate_tests_total": sum(r.get("gate_tests_total") or 0 for r in rs),
"gate_tests_fail_final": sum(r.get("gate_tests_fail_final") or 0 for r in rs),
"mean_wall_s": mean([r["wall_s"] for r in rs]), "mean_turns": mean([r["turns"] for r in rs]),
"blocks_per_run": round(sum(r["blocks_total"] for r in rs) / n, 3),
"blocks_by_gate": {g: sum(r["blocks_by_gate"][g] for r in rs) for g in ("g1", "g3")},
"right_blocks": rb, "false_blocks": fb,
"false_block_rate": {g: (round(fb[g] / (rb[g] + fb[g]), 3) if rb[g] + fb[g] else None) for g in ("g1", "g3")},
"give_ups": sum(1 for r in rs if r["give_up"]), "ended_blocked": sum(1 for r in rs if r["ended_blocked"]),
"reviewer_findings": sum(r["reviewer_findings_total"] for r in rs),
"reviewer_errors": sum(r.get("reviewer_errors", 0) for r in rs),
"model_ok": all(r["model_ok"] for r in rs),
"missed_reqs": sorted(set("%s:%s" % (r["task"], q) for r in rs for q in r["failed_reqs"])),
}
by_cfg = {c: agg([r for r in rows if r["cfg"] == c]) for c in CFGS}
by_task = {t: {c: agg([r for r in rows if r["task"] == t and r["cfg"] == c]) for c in CFGS} for t in TASKS}
by_task_all = {t: agg([r for r in rows if r["task"] == t]) for t in TASKS}
def rate(c):
return (by_cfg.get(c) or {}).get("hidden_rate")
def d(a, b):
return round(a - b, 4) if a is not None and b is not None else None
answers = {
"q1_gate1_vs_bare": {"bare": rate("bare"), "g1": rate("g1"), "delta": d(rate("g1"), rate("bare"))},
"q2_gate3_over_g1": {"g1": rate("g1"), "g13": rate("g13"), "delta": d(rate("g13"), rate("g1")),
"g3_alone": rate("g3"), "g3_vs_bare": d(rate("g3"), rate("bare"))},
"q3_adv_vs_indep": {"g13": rate("g13"), "g13adv": rate("g13adv"), "delta": d(rate("g13adv"), rate("g13")),
"false_blocks_g3": {c: (by_cfg.get(c) or {}).get("false_blocks", {}).get("g3") for c in ("g13", "g13adv")}},
"q4_false_block_rate": {c: (by_cfg.get(c) or {}).get("false_block_rate") for c in CFGS},
"q4_gate1_false_blocks": {c: {"false": (by_cfg.get(c) or {}).get("false_blocks", {}).get("g1"),
"right": (by_cfg.get(c) or {}).get("right_blocks", {}).get("g1")}
for c in ("g1", "g13", "g13adv")},
"q5_cost": {c: {k: (by_cfg.get(c) or {}).get(k) for k in ("mean_cost_total_usd", "mean_cost_worker_usd",
"mean_cost_gate1_usd", "mean_cost_gate3_usd", "mean_wall_s")} for c in CFGS},
}
json.dump({"n_runs": len(rows), "by_cfg": by_cfg, "by_task": by_task, "by_task_all": by_task_all,
"answers": answers, "runs": rows}, open(os.path.join(HERE, "results.json"), "w"), indent=2)
L = ["# gate-bench results", "", "%d runs scored (matrix runs only)." % len(rows), "",
"## Per config", "",
"| cfg | runs | hidden reqs | fully passing | suite green | mean $ total (worker + g1 writer + g3 reviewers) | mean wall s | blocks/run | g1 right/false | g3 right/false | give-ups | ended blocked | model ok |",
"|---|---|---|---|---|---|---|---|---|---|---|---|---|"]
for c in CFGS:
a = by_cfg[c]
if not a:
L.append("| %s | 0 |" % c); continue
L.append("| %s | %d | %d/%d (%.0f%%) | %d | %d | %.3f (%.3f + %.3f + %.3f) | %.0f | %.2f | %d/%d | %d/%d | %d | %d | %s |" % (
c, a["runs"], a["hidden_reqs_passed"], a["hidden_reqs_total"], 100 * (a["hidden_rate"] or 0),
a["runs_fully_passing"], a["suite_green_runs"], a["mean_cost_total_usd"] or 0, a["mean_cost_worker_usd"] or 0,
a["mean_cost_testwriter_usd"] or 0, a["mean_cost_reviewers_usd"] or 0, a["mean_wall_s"] or 0, a["blocks_per_run"], a["right_blocks"]["g1"],
a["false_blocks"]["g1"], a["right_blocks"]["g3"], a["false_blocks"]["g3"], a["give_ups"], a["ended_blocked"],
a["model_ok"]))
L += ["", "## Per task (hidden reqs passed / total, per config)", "", "| task | " + " | ".join(CFGS) + " | missed reqs (all configs) |",
"|---|" + "---|" * (len(CFGS) + 1)]
for t in TASKS:
cells = []
for c in CFGS:
a = by_task[t][c]
cells.append("%d/%d, $%.2f, %.1f blk" % (a["hidden_reqs_passed"], a["hidden_reqs_total"], a["mean_cost_total_usd"] or 0,
a["blocks_per_run"]) if a else "-")
L.append("| %s | %s | %s |" % (t, " | ".join(cells), ", ".join((by_task_all[t] or {}).get("missed_reqs", []))))
L += ["", "## The video's questions", ""]
for k, v in answers.items():
L.append("- **%s**: `%s`" % (k, json.dumps(v)))
L += ["", "Gate 1 false blocks (gate 1 blocked while the hidden oracle was fully green): " + ", ".join(
"%s %s/%s" % (c, (by_cfg.get(c) or {}).get("false_blocks", {}).get("g1"),
((by_cfg.get(c) or {}).get("false_blocks", {}).get("g1") or 0) + ((by_cfg.get(c) or {}).get("right_blocks", {}).get("g1") or 0))
for c in ("g1", "g13", "g13adv")) + " (false / all g1 blocks)."]
L += ["", "Right block = the hidden oracle had at least one failing requirement at that stop; false block = the oracle "
"was fully green at that stop (the gate sent the agent back for nothing the user asked for)."]
open(os.path.join(HERE, "results.md"), "w").write("\n".join(L) + "\n")
print("\n".join(L))
results.md
# gate-bench results
50 runs scored (matrix runs only).
## Per config
| cfg | runs | hidden reqs | fully passing | suite green | mean $ total (worker + g1 writer + g3 reviewers) | mean wall s | blocks/run | g1 right/false | g3 right/false | give-ups | ended blocked | model ok |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bare | 10 | 35/50 (70%) | 3 | 10 | 0.085 (0.085 + 0.000 + 0.000) | 33 | 0.00 | 0/0 | 0/0 | 0 | 0 | True |
| g1 | 10 | 42/50 (84%) | 6 | 10 | 0.201 (0.130 + 0.071 + 0.000) | 60 | 0.60 | 3/3 | 0/0 | 0 | 0 | True |
| g3 | 10 | 34/50 (68%) | 3 | 10 | 0.151 (0.094 + 0.000 + 0.058) | 54 | 0.10 | 0/0 | 1/0 | 0 | 0 | True |
| g13 | 10 | 41/50 (82%) | 5 | 10 | 0.282 (0.153 + 0.071 + 0.057) | 86 | 1.70 | 14/1 | 2/0 | 2 | 0 | True |
| g13adv | 10 | 39/50 (78%) | 6 | 10 | 0.276 (0.157 + 0.072 + 0.048) | 86 | 1.20 | 11/1 | 0/0 | 1 | 0 | True |
## Per task (hidden reqs passed / total, per config)
| task | bare | g1 | g3 | g13 | g13adv | missed reqs (all configs) |
|---|---|---|---|---|---|---|
| T1 | 4/10, $0.08, 0.0 blk | 4/10, $0.22, 0.5 blk | 3/10, $0.16, 0.5 blk | 4/10, $0.36, 3.0 blk | 5/10, $0.31, 1.0 blk | T1:test_req2, T1:test_req3, T1:test_req4, T1:test_req5 |
| T2 | 8/10, $0.10, 0.0 blk | 8/10, $0.17, 0.0 blk | 8/10, $0.17, 0.0 blk | 8/10, $0.27, 3.0 blk | 9/10, $0.30, 3.5 blk | T2:test_req4 |
| T3 | 10/10, $0.09, 0.0 blk | 10/10, $0.21, 1.0 blk | 10/10, $0.14, 0.0 blk | 10/10, $0.26, 0.5 blk | 10/10, $0.31, 0.5 blk | |
| T4 | 5/10, $0.07, 0.0 blk | 10/10, $0.23, 0.5 blk | 5/10, $0.15, 0.0 blk | 10/10, $0.30, 1.0 blk | 5/10, $0.21, 0.0 blk | T4:test_req1, T4:test_req2, T4:test_req3, T4:test_req4, T4:test_req5 |
| T5 | 8/10, $0.08, 0.0 blk | 10/10, $0.18, 1.0 blk | 8/10, $0.12, 0.0 blk | 9/10, $0.23, 1.0 blk | 10/10, $0.25, 1.0 blk | T5:test_req4 |
## The video's questions
- **q1_gate1_vs_bare**: `{"bare": 0.7, "g1": 0.84, "delta": 0.14}`
- **q2_gate3_over_g1**: `{"g1": 0.84, "g13": 0.82, "delta": -0.02, "g3_alone": 0.68, "g3_vs_bare": -0.02}`
- **q3_adv_vs_indep**: `{"g13": 0.82, "g13adv": 0.78, "delta": -0.04, "false_blocks_g3": {"g13": 0, "g13adv": 0}}`
- **q4_false_block_rate**: `{"bare": {"g1": null, "g3": null}, "g1": {"g1": 0.5, "g3": null}, "g3": {"g1": null, "g3": 0.0}, "g13": {"g1": 0.067, "g3": 0.0}, "g13adv": {"g1": 0.083, "g3": null}}`
- **q4_gate1_false_blocks**: `{"g1": {"false": 3, "right": 3}, "g13": {"false": 1, "right": 14}, "g13adv": {"false": 1, "right": 11}}`
- **q5_cost**: `{"bare": {"mean_cost_total_usd": 0.0849, "mean_cost_worker_usd": 0.0849, "mean_cost_gate1_usd": 0.0, "mean_cost_gate3_usd": 0.0, "mean_wall_s": 32.7}, "g1": {"mean_cost_total_usd": 0.201, "mean_cost_worker_usd": 0.1302, "mean_cost_gate1_usd": 0.0709, "mean_cost_gate3_usd": 0.0, "mean_wall_s": 59.5}, "g3": {"mean_cost_total_usd": 0.1513, "mean_cost_worker_usd": 0.0938, "mean_cost_gate1_usd": 0.0, "mean_cost_gate3_usd": 0.0575, "mean_wall_s": 54.1}, "g13": {"mean_cost_total_usd": 0.2818, "mean_cost_worker_usd": 0.1534, "mean_cost_gate1_usd": 0.0712, "mean_cost_gate3_usd": 0.0573, "mean_wall_s": 86.4}, "g13adv": {"mean_cost_total_usd": 0.2763, "mean_cost_worker_usd": 0.1568, "mean_cost_gate1_usd": 0.0718, "mean_cost_gate3_usd": 0.0478, "mean_wall_s": 85.9}}`
Gate 1 false blocks (gate 1 blocked while the hidden oracle was fully green): g1 3/6, g13 1/15, g13adv 1/12 (false / all g1 blocks).
Right block = the hidden oracle had at least one failing requirement at that stop; false block = the oracle was fully green at that stop (the gate sent the agent back for nothing the user asked for).
Walidacja i demo:
validate.sh: sprawdza, że wyrocznia pada na upstreamie, hook się odpala,gate.shblokuje czerwoną kopię, a recenzent zwraca poprawny JSON.validate.sh
#!/usr/bin/env bash
# Validation before the matrix (SPEC § Validation). Steps 1, 3, 4 here; step 2 = the g1 red-start smoke run:
# INJECT=$PWD/inject/break.patch RUN_TAG=val2 ./run.sh g1 T1 1 (then: ./validate.sh step2)
# Usage: validate.sh [all|step1|step2|step3|step4] -> appends to validation.log
set -u
HERE="$(cd "$(dirname "$0")" && pwd)"; cd "$HERE"
. "$HERE/env.sh"
PY="$HERE/.venv/bin/python"
STEP="${1:-all}"
status=0
count() { grep -Eo "[0-9]+ $2" <<<"$1" | tail -1 | awk '{print $1}'; }
log() { echo "$*" | tee -a validation.log; }
log "=== $(date '+%F %T') validate.sh $STEP ($(claude --version))"
if [ "$STEP" = all ] || [ "$STEP" = step1 ]; then
# 1. upstream fails the hidden tests (<= 1 req passing: T5 R3 is a regression guard); reference passes hidden + suite
TMP="$(mktemp -d "${TMPDIR:-/tmp}/gb-validate.XXXXXX")"
for t in T1 T2 T3 T4 T5; do
tree="$TMP/$t"; git clone -q "$HERE/upstream" "$tree"
v=$(cd "$tree" && TINYDB_TREE="$tree" "$PY" -m pytest -v -c "$HERE/hidden/pytest.ini" --rootdir "$HERE/hidden" "$HERE/hidden/test_$t.py" 2>&1)
up_pass=$(grep -Eo "test_req[0-9]+[^ ]* PASSED" <<<"$v" | wc -l | tr -d ' ')
(cd "$tree" && git apply "$HERE/reference/$t.patch") || { log "$t: reference patch does not apply"; status=1; continue; }
h=$(cd "$tree" && TINYDB_TREE="$tree" "$PY" -m pytest -q -c "$HERE/hidden/pytest.ini" --rootdir "$HERE/hidden" "$HERE/hidden/test_$t.py" 2>&1); hrc=$?
s=$(cd "$tree" && "$PY" -m pytest -q -p no:cacheprovider --no-cov 2>&1); src=$?
ok=OK; { [ "$up_pass" -gt 1 ] || [ $hrc -ne 0 ] || [ $src -ne 0 ]; } && { ok=FAIL; status=1; }
log "step1 $t: upstream reqs passing=$up_pass reference hidden rc=$hrc ($(count "$h" passed) passed) suite rc=$src ($(count "$s" passed) passed) [$ok]"
done
rm -rf "$TMP"
fi
if [ "$STEP" = all ] || [ "$STEP" = step3 ]; then
# 3. gate.sh by hand, fake stdin. green = T1 reference + one new test (g1 passes) -> exit 0 allow;
# red = break.patch (one repo test fails) -> exit 2, pytest tail on stderr.
FAKE='{"session_id":"validate","transcript_path":"/dev/null","cwd":"x","hook_event_name":"Stop","stop_hook_active":false}'
for k in green red; do
R="$HERE/runs/_val3-$k"; W="$HERE/work/_val3-$k"; rm -rf "$R"; mkdir -p "$R"
./prep.sh g1 "$W" "$R" > /dev/null
if [ $k = green ]; then
(cd "$W" && git apply "$HERE/reference/T1.patch" && cp "$HERE/inject/test_pushpull.py" tests/)
else
(cd "$W" && git apply "$HERE/inject/break.patch")
fi
printf '{"task":"T1","cfg":"g1","rep":0,"work":"%s","name":"_val3-%s"}\n' "$W" "$k" > "$R/meta.json"; cp tasks/T1.md "$R/task.md"
echo "$FAKE" | GATE_RUN="$R" GATE_CONFIG=g1 bash gate.sh > "$R/stdout.txt" 2> "$R/stderr.txt"; rc=$?
dec=$("$PY" -c "import json;print(json.loads(open('$R/gate.jsonl').readline())['decision'])")
want=0; [ $k = red ] && want=2
ok=OK; [ $rc -ne $want ] && { ok=FAIL; status=1; }
[ $k = red ] && ! grep -q "short test summary" "$R/stderr.txt" && { ok=FAIL; status=1; }
[ $k = green ] && [ -s "$R/stderr.txt" ] && { ok=FAIL; status=1; }
log "step3 $k: exit=$rc decision=$dec stderr_bytes=$(wc -c < "$R/stderr.txt" | tr -d ' ') [$ok]"
done
fi
if [ "$STEP" = all ] || [ "$STEP" = step4 ]; then
# 4. one reviewer pair by hand on the green copy (needs step3's work/_val3-green): structured_output valid
W="$HERE/work/_val3-green"
[ -d "$W" ] || { log "step4: run step3 first"; exit 1; }
for v in indep adv; do
"$PY" lib/review.py "$W" T1 $v > "runs/_val3-green/review-$v.txt" 2>&1
s=$("$PY" - "runs/_val3-green/review-$v.txt" <<'PY'
import json, sys
t = open(sys.argv[1]).read(); i = t.find("--- reason"); d = json.loads(t[:i] if i > 0 else t)
ok = all(r["error"] is None and isinstance(r["approve_raw"], bool) for r in d["reviewers"])
ok = ok and all(f["severity"] in ("blocking", "minor") and isinstance(f["line"], int) for r in d["reviewers"] for f in r["findings"])
print("%s pass=%s blocking=%d findings=%s cost=$%.4f wall=%s" % ("OK" if ok else "FAIL", d["pass"], d["blocking_findings"],
[len(r["findings"]) for r in d["reviewers"]], d["cost_usd"], [r["wall_s"] for r in d["reviewers"]]))
PY
)
case "$s" in OK*) ;; *) status=1 ;; esac
log "step4 $v: $s"
done
fi
if [ "$STEP" = all ] || [ "$STEP" = step2 ]; then
# 2. hook fires on a red start: runs/T1-g1-r1-val2 must log a block and the stream must show the agent continuing
R="$HERE/runs/T1-g1-r1-val2"
if [ -f "$R/gate.jsonl" ]; then
s=$("$PY" - "$R" <<'PY'
import json, sys, os
r = sys.argv[1]
g = [json.loads(l) for l in open(os.path.join(r, "gate.jsonl")) if l.strip()]
st = [json.loads(l) for l in open(os.path.join(r, "stream.jsonl")) if l.startswith("{")]
blocks = [x for x in g if x["decision"] == "block"]
# index in the stream of the first hook response that blocked, then any assistant tool_use after it
txt = [json.dumps(o) for o in st]
fb = next((i for i, t in enumerate(txt) if "Stop hook feedback" in t or ('"hook_event": "Stop"' in t and '"exit_code": 2' in t)), None)
after = fb is not None and any(o.get("type") == "assistant" for o in st[fb + 1:])
print("%s blocks=%d hook_calls=%d feedback_idx=%s assistant_after=%s last=%s" % (
"OK" if blocks and after else "FAIL", len(blocks), len(g), fb, after, g[-1]["decision"] if g else None))
PY
)
case "$s" in OK*) ;; *) status=1 ;; esac
log "step2: $s"
else
log "step2: not run yet (INJECT=\$PWD/inject/break.patch RUN_TAG=val2 ./run.sh g1 T1 1)"
fi
fi
[ $status -eq 0 ] && log "VALIDATION OK" || log "VALIDATION FAILED"
exit $status
validate_writer.sh
#!/usr/bin/env bash
# Validation of the gate-1 test writer: one writer call per task (in parallel) into validation/writer/<T>/,
# then its tests on upstream and on the reference patch. Usage: validate_writer.sh [T1 ...] (reuses an existing
# file unless REWRITE=1). Appends to validation.log.
set -u
HERE="$(cd "$(dirname "$0")" && pwd)"; cd "$HERE"
. "$HERE/env.sh"
PY="$HERE/.venv/bin/python"
TASKS="${*:-T1 T2 T3 T4 T5}"
for t in $TASKS; do
D="$HERE/validation/writer/$t"; mkdir -p "$D"
if [ "${REWRITE:-0}" = 1 ] || [ ! -f "$D/gate_tests/test_$t.py" ]; then "$PY" lib/testwriter.py "$t" "$D" > "$D/writer.out" 2>&1 & fi
done
wait
TMP="$(mktemp -d "${TMPDIR:-/tmp}/gb-vw.XXXXXX")"; trap 'rm -rf "$TMP"' EXIT
echo "=== $(date '+%F %T') validate_writer.sh $TASKS" >> validation.log
for t in $TASKS; do
D="$HERE/validation/writer/$t"
git clone -q "$HERE/upstream" "$TMP/up-$t"; git clone -q "$HERE/upstream" "$TMP/ref-$t"
(cd "$TMP/ref-$t" && git apply "$HERE/reference/$t.patch")
line=$("$PY" - "$D" "$t" "$TMP/up-$t" "$TMP/ref-$t" <<'PY'
import json, os, sys
sys.path.insert(0, "lib")
from testwriter import run_gate_tests
d, t, up, ref = sys.argv[1:5]
w = json.load(open(os.path.join(d, "testwriter.json")))
if not w["ok"]:
print("%s: writer FAILED %s" % (t, [a["error"] for a in w["attempts"]])); sys.exit()
g = os.path.join(d, "gate_tests")
_, uo, up_p, up_f, _ = run_gate_tests(g, t, up)
_, ro, rp, rf, rnames = run_gate_tests(g, t, ref)
open(os.path.join(d, "on-upstream.txt"), "w").write(uo); open(os.path.join(d, "on-reference.txt"), "w").write(ro)
print("%s: writer tests: %d, pass on reference: %d, pass on upstream: %d | fail on reference: %s | writer $%.3f %ss turns=%s" % (
t, rp + rf, rp, up_p, rnames, w["cost_usd"], w["wall_s"], [a.get("turns") for a in w["attempts"]]))
PY
)
echo "$line" | tee -a validation.log
done
demo.sh
#!/usr/bin/env bash
# Interactive demo of the gate for a screen recording: a gated tinydb work copy (config g13: acceptance
# tests, then two reviewers) with T1's acceptance tests already written. Usage: ./demo.sh [T1..T5]
# Then: cd work/demo && claude --setting-sources project --permission-mode acceptEdits
# and paste the request printed below. When Claude stops, the Stop hook runs; a red acceptance test shows
# as "Stop hook feedback: ... The acceptance tests for this change fail:" and Claude keeps working.
set -euo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"; TASK="${1:-T1}"
. "$HERE/env.sh"
RUN="$HERE/runs/demo"; W="$HERE/work/demo"
rm -rf "$RUN"; mkdir -p "$RUN"
"$HERE/prep.sh" g13 "$W" "$RUN" > "$RUN/prep.txt" 2>&1
printf '{"task":"%s","cfg":"g13","rep":0,"work":"%s","name":"demo"}\n' "$TASK" "$W" > "$RUN/meta.json"
cp "$HERE/tasks/$TASK.md" "$RUN/task.md"
printf '%s\n\nWhen you are done, stop and summarise what you changed.\n' "$(cat "$HERE/tasks/$TASK.md")" > "$RUN/prompt.txt"
# Reuse the acceptance tests of a matrix run of the same task (no new writer call).
SRC=$(ls -d "$HERE"/runs/"$TASK"-g13-r1/gate_tests 2>/dev/null | head -1)
cp -R "$SRC" "$RUN/gate_tests"
echo "ready: cd $W && claude --setting-sources project --permission-mode acceptEdits"
echo "--- paste this request ---"; cat "$RUN/prompt.txt"
inject/break.patch
diff --git a/tinydb/queries.py b/tinydb/queries.py
index cdab9fb..db00a2a 100644
--- a/tinydb/queries.py
+++ b/tinydb/queries.py
@@ -322,7 +322,7 @@ class Query(QueryInstance):
:param rhs: The value to compare against
"""
return self._generate_test(
- lambda value: value >= rhs,
+ lambda value: value > rhs,
('>=', self._path, rhs)
)
inject/test_pushpull.py
from tinydb import TinyDB, Query
from tinydb.storages import MemoryStorage
from tinydb.operations import push, pull
def test_push_pull():
db = TinyDB(storage=MemoryStorage)
db.insert({'n': 1, 't': ['a']})
db.update(push('t', 'b'), Query().n == 1)
db.update(pull('t', 'a'), Query().n == 1)
assert db.all()[0]['t'] == ['b']
inject/test_pushpull.py on GitHub
Dokumenty:
README.md: układ, polecenia i czym jest uprząż.README.md
# gate-bench
What does each of Shopify's Helix gates catch when rebuilt as a Claude Code **Stop hook**, and what does it
cost? Headless Sonnet 5.5 workers implement five tinydb features (T1..T5); a Stop hook runs gate 1
(behaviour tests) and/or gate 3 (adversarial review by headless Sonnet 5.5 reviewers) every time the worker
tries to stop; hidden tests (the oracle, never visible to the agent) record the ground truth at every stop
and at the end. Built 2026-10-08 from `SPEC.md`; upstream, hidden tests, tasks and reference patches copied
from `../verify-loops-bench` (nothing there or in `../haiku-routing-bench` is modified).
## Commands (from this dir)
```bash
./validate.sh # steps 1,3,4 (+ step 2 once its run exists) -> validation.log
INJECT=$PWD/inject/break.patch RUN_TAG=val2 ./run.sh g1 T1 1 # validation step 2 (red start state)
RUN_TAG=smoke ./run.sh g13adv T1 1 # smoke (tagged runs are ignored by analyze.py)
./validate_writer.sh # gate-1 test writer on T1..T5: tests on upstream vs reference (REWRITE=1 to redo)
./gen_matrix.py # matrix.txt (50 runs, already generated)
nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 & # 3 runs in parallel (MATRIX_JOBS=3)
SKIP_DONE=1 nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 & # resume
.venv/bin/python analyze.py # results.json + results.md
.venv/bin/python lib/finalize.py runs/<run> # rescore one run from its work copy
.venv/bin/python lib/review.py work/<run> T1 adv # one gate-3 round by hand
```
Single run: `./run.sh <cfg> <task> <rep>` -> `runs/<task>-<cfg>-r<rep>/`, work copy `work/<same name>/`.
## Layout
| path | what |
|---|---|
| `SPEC.md` | the spec this harness implements |
| `upstream/` | tinydb @ 18d73a1 (git repo) |
| `hidden/` | `test_T1..T6.py` (T6 unused), `conftest.py` (`$TINYDB_TREE` or cwd first on sys.path), `pytest.ini` |
| `tasks/T1..T5.md` + `tasks/README.md` | the task prompts (verbatim) and what each hidden req checks |
| `reference/T1..T5.patch` | reference solutions (validation step 1) |
| `inject/break.patch` | `>=` -> `>` in `Query.__ge__` (one repo test red; validation steps 2-3) |
| `inject/test_pushpull.py` | one thin test, makes the T1 reference copy "green with tests touched" (step 3) |
| `configs.json` | `bare` (no hook), `g1`, `g3` (indep), `g13` (g1 then g3 indep), `g13adv` (g1 then g3 adv) |
| `env.sh` | scrubs inherited `CLAUDE*` vars, default `~/.claude` auth, `CLAUDE_CODE_DISABLE_AUTO_MEMORY=1`, venv on PATH |
| `prep.sh <cfg> <work> <run>` | fresh copy of upstream, `.claude/` in `.git/info/exclude`, hook settings for gated configs |
| `run.sh <cfg> <task> <rep>` | one run (worker + hook + `lib/finalize.py`) |
| `gate.sh` | the Stop hook (bash 3.2 wrapper: `exec .venv/bin/python -I lib/hook.py`) |
| `lib/hook.py` | gate dispatch, snapshot, hidden run, gate 1, gate 3, decision, `gate.jsonl` line |
| `lib/review.py` | gate-3 reviewers (indep / adv), schemas, prompts, reason text; CLI for a manual round |
| `lib/testwriter.py` | gate-1 test writer (before the worker) + `run_gate_tests` (path rewritten to `acceptance/`) |
| `validate_writer.sh`, `validation/writer/<T>/` | writer validation files and their runs on upstream / reference |
| `tasks/` (terse prompts, used), `tasks-explicit/` + `runs-explicit/` | the matrix uses terse prompts; see `tasks/README.md` |
| `lib/common.py` | diff incl. untracked files, suite run, hidden run (requirement level) |
| `lib/finalize.py` | post-run diff/suite/hidden, `result.json`, `score.json` |
| `gen_matrix.py`, `matrix.txt`, `matrix.sh`, `matrix-one.sh`, `matrix.log` | 50-run matrix, 3 in parallel (xargs -P), resumable |
| `analyze.py` -> `results.json`, `results.md` | per config, per task, the five questions |
| `validate.sh`, `VALIDATION.md`, `validation.log` | pre-matrix checks |
| `.venv/` | python 3.14.7, pytest 9.1.1, pytest-cov 7.1.0, pyyaml, ruff 0.16.10 (same pins as haiku-routing-bench) |
### Worker (run.sh)
`claude -p "<T-file verbatim>\n\nWhen you are done, stop and summarise what you changed." --model claude-sonnet-5-5
--output-format stream-json --verbose --include-hook-events --max-turns 60 --permission-mode acceptEdits
--allowedTools <haiku-routing-bench list> --setting-sources project --strict-mcp-config`, cwd = `work/<run>/`, stdin /dev/null.
Gated configs get `work/<run>/.claude/settings.json`:
`{"permissions":{"deny":[...see traps...]},"hooks":{"Stop":[{"hooks":[{"type":"command","command":"bash \"$CLAUDE_PROJECT_DIR/.claude/hooks/stop.sh\"","timeout":1500}]}]}}`
and `.claude/hooks/stop.sh` = `GATE_RUN=<abs run dir> GATE_CONFIG=<cfg> bash <bench>/gate.sh` (the SPEC command, one level down: see traps).
Copies of both land in the run dir (`settings.json`, `stop.sh`).
### Hook (lib/hook.py), one invocation
1. read stdin JSON; n = lines already in `gate.jsonl` + 1; blocks so far from the same file
2. `snap-<n>.diff` = `git diff HEAD` + untracked files (index never touched)
3. hidden tests on the work copy (`TINYDB_TREE` = work copy, cwd = work copy) -> `hidden-<n>.txt`; never shown to the agent
4. gate 1 (if configured) = the repo suite `python -m pytest -q -x --no-header -p no:cacheprovider --no-cov` (300 s)
-> `gate1-<n>.txt` AND the writer's acceptance tests `runs/<run>/gate_tests/test_<T>.py` run with `TINYDB_TREE` = work
copy -> `gatetests-<n>.txt`. Block when either is red. Reason: "The acceptance tests for this change fail:" + last 40
lines (failing test names + assertion output, the gate_tests path rewritten to `acceptance/`), then the suite tail if the
suite is red too; <= 1500 chars. `tests_touched` is still logged but no longer blocks. No writer file (writer failed
twice) -> suite only, logged as `gate_tests_present: false`.
5. gate 3 (if configured and gate 1 passed or absent): see below
6. decision: allow / block / give_up (a would-be block when 6 blocks already happened -> allow, logged `give_up`).
`stop_hook_active` is logged, never used. Block = exit 2 + reason on stderr. A crash of the hook itself is
logged to `gate-errors.txt` and allows (counted as `gate_crashes` in score.json).
### Gate 1 test writer (lib/testwriter.py, run.sh before the worker, configs g1/g13/g13adv)
`claude -p` Sonnet 5.5 with the reviewer isolation flags, `--tools Read,Grep,Glob --max-turns 12 --json-schema
{"file": string}`, cwd = a pristine temp copy of upstream (deleted after). Prompt = framing + `tasks/<T>.md` + the
coordinator's instruction ("Write integration-style pytest cases for this request against the tinydb API, one test per
behaviour a careful senior would expect, pushing for edge cases beyond the happy path (missing fields, wrong types,
boundaries, persistence, cache, existing behaviour unchanged). Tests only, no implementation. Return the file content.")
+ mechanics (self-contained module, imports of the new API inside each test, MemoryStorage/tmp_path). The file must
compile and contain `def test_`, else one retry. Output `runs/<run>/gate_tests/` (+ hidden's conftest.py with
`TINYDB_TREE`, pytest.ini) and `testwriter.json` (cost, turns, attempts). Fixed per-run cost, outside the worker's wall.
### Gate 3 (lib/review.py)
Each reviewer: `claude -p "<prompt>" --model claude-sonnet-5-5 --tools Read,Grep,Glob --max-turns 15 --output-format json
--json-schema <schema> --permission-mode dontAsk --setting-sources "" --settings '{"disableAllHooks":true}' --strict-mcp-config`
with `CLAUDE_CODE_DISABLE_CLAUDE_MDS=1 CLAUDE_CODE_DISABLE_AUTO_MEMORY=1`, all inherited `CLAUDE*` vars dropped, cwd = work copy,
600 s timeout. Prompt = framing + task prompt (T-file) + diff (tracked + untracked, cap 60k chars) + the SPEC's "Find defects..."
sentence. `structured_output` is read from the JSON result; `total_cost_usd`, `num_turns`, wall and models are recorded.
- `indep`: reviewers A and B, same prompt, run in parallel, no shared context. Blocking findings = union. Pass when both approve.
- `adv`: A as above; challenger B gets A's numbered findings + the diff, answers `upheld|rejected` + reason per id, may add
findings (`added_findings`). Blocking = A's blocking findings B upheld + B's added blocking findings. B always runs (same
2-call budget as indep).
- Effective `approve` = "no blocking finding" (the raw `approve` is logged as `approve_raw`; a reviewer saying approve=false with
only minor findings does not block). A reviewer call that errors approves (fail open) with `error` recorded; an errored
challenger upholds nothing.
- Reason = "Code review found N blocking defect(s):" + numbered `file:line text` + "Fix every finding, re-run the tests, then stop.", <= 1500 chars.
### Per run (`runs/<task>-<cfg>-r<rep>/`)
`stream.jsonl`, `stderr.txt`, `exit_code.txt`, `wall_s.txt` (includes hook time: the hook runs inside the session),
`gate.jsonl`, `snap-<n>.diff`, `hidden-<n>.txt`, `gate1-<n>.txt`, `hook-input-1.json`, `post.diff`, `numstat.txt`,
`suite.txt` (full suite, no -x), `hidden.txt`, `gate_tests/`, `testwriter.json`, `gatetests-<n>.txt`, `gatetests-final.txt`, `result.json`, `score.json`, `meta.json`, `task.md`, `prompt.txt`,
`settings.json` + `stop.sh` (gated configs).
`score.json`: task, cfg, rep, hidden_pass/hidden_fail/failed_reqs (**requirement level**: 5 reqs per task), suite_pass/fail,
hook_calls, blocks_total, blocks_by_gate {g1,g3}, right_blocks / false_blocks (+ `_by_gate`), give_up, ended_blocked,
cost_worker_usd, cost_reviewers_usd, cost_testwriter_usd, cost_total_usd (all three), testwriter_ok, gate_tests_total,
gate_tests_fail_final, turns, wall_s, reviewer_findings_total, reviewer_errors, gate_crashes,
model_ok (worker init model + every modelUsage key + every reviewer model == claude-sonnet-5-5), hook_events_in_stream,
hidden_first_stop.
## Matrix
5 tasks x 5 configs x 2 reps = 50 runs. Rep 1 config blocks bare, g1, g3, g13, g13adv (tasks rotated per block), rep 2 the
reverse block order. Up to 3 runs at once, each in its own work copy (`matrix-one.sh` via `xargs -P 3`, bash 3.2 has no
`wait -n`). `matrix.log`: one `start` and one result line per run.
## Traps found
- **Gate 1 as first built passed by construction** (it only re-ran the implementer's own tests: `T2-g1-r1` one allow, hidden
R4 missed). Redesigned around an independent test writer (above). The writer hedges where the terse prompt is silent
(accepts "raises or no-op"), so it cannot recover unstated requirements; per-task overlap is in VALIDATION.md.
- **A blocked agent hunts for the tests.** After a gate-1 block the smoke agent ran `cat .claude/hooks/stop.sh; find / -name
test_T1.py` (denied only because `find` is not on the allow-list; `cat` is, and `stop.sh` names the bench dir, from where
`hidden/` is one `cat` away). Gated configs' `.claude/settings.json` now carries `permissions.deny`: Read of `./.claude/**`,
the bench's `hidden/ runs/ validation/ reference/`, the two sibling benches; Bash patterns containing `.claude/hooks`,
`.claude/settings`, `../hidden`, `../runs`, `gate_tests`, the bench sub-paths, `find *`. Probed with a Haiku session: all
five escape attempts (cat, Read, ls, python open) denied, in-tree reads fine. Bash rules match command text, not a security
boundary. Bare runs have no settings file (bare r1 ran before this change; bare has no motive to hunt).
Reviewers and the writer run with `--setting-sources ""`, so these rules do not apply to them.
- **Transient `FileNotFoundError` spawning a reviewer** inside the hook (once, both reviewers of one call, wall 0.0 s;
never reproduced by hand). Reviewer calls now retry once after 3 s; the error message records `filename`.
- **`--bare` cannot authenticate here.** It reads `ANTHROPIC_API_KEY` / `apiKeyHelper` only, never OAuth or the keychain;
this account is OAuth (`~/.claude`), so a `--bare` reviewer returns `terminal_reason: api_error`, cost 0. Used the
documented equivalents instead: `--setting-sources ""` + `--settings '{"disableAllHooks":true}'` + `--strict-mcp-config` +
`CLAUDE_CODE_DISABLE_CLAUDE_MDS=1` + `CLAUDE_CODE_DISABLE_AUTO_MEMORY=1` + `--tools Read,Grep,Glob`. The reviewers' cwd is the
work copy, whose `.claude/settings.json` carries the Stop hook: without `--setting-sources ""`/`disableAllHooks` a reviewer
would run the gate recursively. Verified: a dummy Stop hook in the probe copy never fired.
- **Inherited session env.** A matrix launched from inside Claude Code inherits `CLAUDE_EFFORT=medium`,
`CLAUDE_CODE_SUBAGENT_MODEL=opus`, `CLAUDE_CODE_SESSION_ID`, `CLAUDECODE`, `CLAUDE_CODE_CHILD_SESSION`, messaging tokens.
`env.sh` unsets every `CLAUDE*` var before setting its own; reviewers get a scrubbed env too (the hook process itself
inherits the worker's `CLAUDE_*` vars, e.g. the worker's session id).
- **The feedback line leaks the hook command.** Claude Code prefixes each block with the command line:
`Stop hook feedback:\n[<command>]: <reason>`. With the SPEC command inline the agent would read `GATE_CONFIG=g13adv` and
`gate-bench/gate.sh`. The settings command is therefore `bash "$CLAUDE_PROJECT_DIR/.claude/hooks/stop.sh"`, a one-line
wrapper holding the SPEC command. The agent can still `cat` it (it lives in the work copy) but it is not pushed at it.
- **Stop hook input keys (2.1.293):** `background_tasks, cwd, effort ({"level":"medium"}), hook_event_name,
last_assistant_message, permission_mode, prompt_id, session_crons, session_id, stop_hook_active, transcript_path`.
- **8-cap resets on tool calls.** Per the docs the consecutive-continuation count resets each time Claude calls a tool, so the
built-in cap only bites an agent that answers blocks with text alone. Our give-up (6 blocks per run, counted from
`gate.jsonl`, tool calls or not) fires first in every case.
- **Red start state: the agent refuses to fix code it did not write.** Validation step 2 (T1 + an uncommitted `>=` -> `>`
edit) shows Sonnet 5.5 implementing T1, then answering 6 consecutive gate-1 blocks with "the red test is your pre-existing
uncommitted edit, not my change" without touching it, until give_up. The hook mechanics are validated; the behaviour is a
finding in itself (gate 1 cannot force a fix the agent considers out of scope).
- **pytest-cov.** The repo's `pytest.ini` adds `--cov tinydb`; gate 1 adds `--no-cov` so the 40-line tail holds the failure,
not the coverage table (and no `.coverage` file is written). `.coverage` is gitignored anyway.
- **T5 R4 is parametrized** (several test ids, one requirement). hidden counts are per requirement (5 per task), a req
passes only when all its ids pass. `validate.sh` step 1: T5 upstream passes 1 req (R3, the regression guard) as designed.
- **Diff without touching the index.** `git add -N` would change what the agent sees in `git status`; untracked files are
diffed with `git diff --no-index /dev/null <f>` instead. `.claude/` is in `.git/info/exclude`, so the hook config never
shows in any diff or in `git status`.
- **Worker Bash denials.** The 2.1.293 permission layer denies some agent commands with "Contains brace with quote character
(expansion obfuscation)" even under acceptEdits + allow-list; agents route around it (visible as `permission_denied` events).
- **Reviewers on a correct solution still flag "blocking".** On the T1 reference patch, reviewer A marked "pull does not check
the field is a list" (not required by the task) as blocking; indep therefore blocks, the adv challenger rejected it. This is
exactly what false_blocks measures (hidden all green at that stop).
- **Shared plan meter.** Workers and reviewers bill the user's plan quota; `total_cost_usd` is the list-price equivalent the
CLI reports, not a bill. Nothing else heavy should run on the account during the matrix.
- **rtk rewrites interactive `grep`/`diff`** in the launching Claude session only; scripts here call binaries directly
(`/usr/bin/diff`/`cmp` for verification diffs).
- **CLI version:** `claude --version` = 2.1.293 (Claude Code); flags checked in `claude --help`: `--json-schema`, `--bare`, `--tools`,
`--permission-mode` (choices acceptEdits, auto, bypassPermissions, manual, dontAsk, plan), `--setting-sources`,
`--include-hook-events`. `--max-turns` is accepted though not listed in `--help`.
## Matrix launch (2026-10-08 16:33:54, after the gate-1 redesign)
`cd /Users/thomas/dev/gate-bench && SKIP_DONE=1 nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 &` -> PID 71317, 3 lanes.
The 5 bare r1 runs (terse prompts) are kept and skipped; 45 runs to go. Expected wall ~40-70 min: gated runs so far take
~30-70 s of worker wall + ~35-40 s of test writer (before the worker) + 1-35 s per hook call, more with blocks.
Then `.venv/bin/python analyze.py`.
HARNESS.md
# gate-bench
What does each of Shopify's Helix gates catch when rebuilt as a Claude Code **Stop hook**, and what does it
cost? Headless Sonnet 5.5 workers implement five tinydb features (T1..T5); a Stop hook runs gate 1
(behaviour tests) and/or gate 3 (adversarial review by headless Sonnet 5.5 reviewers) every time the worker
tries to stop; hidden tests (the oracle, never visible to the agent) record the ground truth at every stop
and at the end. Built 2026-10-08 from `SPEC.md`; upstream, hidden tests, tasks and reference patches copied
from `../verify-loops-bench` (nothing there or in `../haiku-routing-bench` is modified).
## Commands (from this dir)
```bash
./validate.sh # steps 1,3,4 (+ step 2 once its run exists) -> validation.log
INJECT=$PWD/inject/break.patch RUN_TAG=val2 ./run.sh g1 T1 1 # validation step 2 (red start state)
RUN_TAG=smoke ./run.sh g13adv T1 1 # smoke (tagged runs are ignored by analyze.py)
./validate_writer.sh # gate-1 test writer on T1..T5: tests on upstream vs reference (REWRITE=1 to redo)
./gen_matrix.py # matrix.txt (50 runs, already generated)
nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 & # 3 runs in parallel (MATRIX_JOBS=3)
SKIP_DONE=1 nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 & # resume
.venv/bin/python analyze.py # results.json + results.md
.venv/bin/python lib/finalize.py runs/<run> # rescore one run from its work copy
.venv/bin/python lib/review.py work/<run> T1 adv # one gate-3 round by hand
```
Single run: `./run.sh <cfg> <task> <rep>` -> `runs/<task>-<cfg>-r<rep>/`, work copy `work/<same name>/`.
## Layout
| path | what |
|---|---|
| `SPEC.md` | the spec this harness implements |
| `upstream/` | tinydb @ 18d73a1 (git repo) |
| `hidden/` | `test_T1..T6.py` (T6 unused), `conftest.py` (`$TINYDB_TREE` or cwd first on sys.path), `pytest.ini` |
| `tasks/T1..T5.md` + `tasks/README.md` | the task prompts (verbatim) and what each hidden req checks |
| `reference/T1..T5.patch` | reference solutions (validation step 1) |
| `inject/break.patch` | `>=` -> `>` in `Query.__ge__` (one repo test red; validation steps 2-3) |
| `inject/test_pushpull.py` | one thin test, makes the T1 reference copy "green with tests touched" (step 3) |
| `configs.json` | `bare` (no hook), `g1`, `g3` (indep), `g13` (g1 then g3 indep), `g13adv` (g1 then g3 adv) |
| `env.sh` | scrubs inherited `CLAUDE*` vars, default `~/.claude` auth, `CLAUDE_CODE_DISABLE_AUTO_MEMORY=1`, venv on PATH |
| `prep.sh <cfg> <work> <run>` | fresh copy of upstream, `.claude/` in `.git/info/exclude`, hook settings for gated configs |
| `run.sh <cfg> <task> <rep>` | one run (worker + hook + `lib/finalize.py`) |
| `gate.sh` | the Stop hook (bash 3.2 wrapper: `exec .venv/bin/python -I lib/hook.py`) |
| `lib/hook.py` | gate dispatch, snapshot, hidden run, gate 1, gate 3, decision, `gate.jsonl` line |
| `lib/review.py` | gate-3 reviewers (indep / adv), schemas, prompts, reason text; CLI for a manual round |
| `lib/testwriter.py` | gate-1 test writer (before the worker) + `run_gate_tests` (path rewritten to `acceptance/`) |
| `validate_writer.sh`, `validation/writer/<T>/` | writer validation files and their runs on upstream / reference |
| `tasks/` (terse prompts, used), `tasks-explicit/` + `runs-explicit/` | the matrix uses terse prompts; see `tasks/README.md` |
| `lib/common.py` | diff incl. untracked files, suite run, hidden run (requirement level) |
| `lib/finalize.py` | post-run diff/suite/hidden, `result.json`, `score.json` |
| `gen_matrix.py`, `matrix.txt`, `matrix.sh`, `matrix-one.sh`, `matrix.log` | 50-run matrix, 3 in parallel (xargs -P), resumable |
| `analyze.py` -> `results.json`, `results.md` | per config, per task, the five questions |
| `validate.sh`, `VALIDATION.md`, `validation.log` | pre-matrix checks |
| `.venv/` | python 3.14.7, pytest 9.1.1, pytest-cov 7.1.0, pyyaml, ruff 0.16.10 (same pins as haiku-routing-bench) |
### Worker (run.sh)
`claude -p "<T-file verbatim>\n\nWhen you are done, stop and summarise what you changed." --model claude-sonnet-5-5
--output-format stream-json --verbose --include-hook-events --max-turns 60 --permission-mode acceptEdits
--allowedTools <haiku-routing-bench list> --setting-sources project --strict-mcp-config`, cwd = `work/<run>/`, stdin /dev/null.
Gated configs get `work/<run>/.claude/settings.json`:
`{"permissions":{"deny":[...see traps...]},"hooks":{"Stop":[{"hooks":[{"type":"command","command":"bash \"$CLAUDE_PROJECT_DIR/.claude/hooks/stop.sh\"","timeout":1500}]}]}}`
and `.claude/hooks/stop.sh` = `GATE_RUN=<abs run dir> GATE_CONFIG=<cfg> bash <bench>/gate.sh` (the SPEC command, one level down: see traps).
Copies of both land in the run dir (`settings.json`, `stop.sh`).
### Hook (lib/hook.py), one invocation
1. read stdin JSON; n = lines already in `gate.jsonl` + 1; blocks so far from the same file
2. `snap-<n>.diff` = `git diff HEAD` + untracked files (index never touched)
3. hidden tests on the work copy (`TINYDB_TREE` = work copy, cwd = work copy) -> `hidden-<n>.txt`; never shown to the agent
4. gate 1 (if configured) = the repo suite `python -m pytest -q -x --no-header -p no:cacheprovider --no-cov` (300 s)
-> `gate1-<n>.txt` AND the writer's acceptance tests `runs/<run>/gate_tests/test_<T>.py` run with `TINYDB_TREE` = work
copy -> `gatetests-<n>.txt`. Block when either is red. Reason: "The acceptance tests for this change fail:" + last 40
lines (failing test names + assertion output, the gate_tests path rewritten to `acceptance/`), then the suite tail if the
suite is red too; <= 1500 chars. `tests_touched` is still logged but no longer blocks. No writer file (writer failed
twice) -> suite only, logged as `gate_tests_present: false`.
5. gate 3 (if configured and gate 1 passed or absent): see below
6. decision: allow / block / give_up (a would-be block when 6 blocks already happened -> allow, logged `give_up`).
`stop_hook_active` is logged, never used. Block = exit 2 + reason on stderr. A crash of the hook itself is
logged to `gate-errors.txt` and allows (counted as `gate_crashes` in score.json).
### Gate 1 test writer (lib/testwriter.py, run.sh before the worker, configs g1/g13/g13adv)
`claude -p` Sonnet 5.5 with the reviewer isolation flags, `--tools Read,Grep,Glob --max-turns 12 --json-schema
{"file": string}`, cwd = a pristine temp copy of upstream (deleted after). Prompt = framing + `tasks/<T>.md` + the
coordinator's instruction ("Write integration-style pytest cases for this request against the tinydb API, one test per
behaviour a careful senior would expect, pushing for edge cases beyond the happy path (missing fields, wrong types,
boundaries, persistence, cache, existing behaviour unchanged). Tests only, no implementation. Return the file content.")
+ mechanics (self-contained module, imports of the new API inside each test, MemoryStorage/tmp_path). The file must
compile and contain `def test_`, else one retry. Output `runs/<run>/gate_tests/` (+ hidden's conftest.py with
`TINYDB_TREE`, pytest.ini) and `testwriter.json` (cost, turns, attempts). Fixed per-run cost, outside the worker's wall.
### Gate 3 (lib/review.py)
Each reviewer: `claude -p "<prompt>" --model claude-sonnet-5-5 --tools Read,Grep,Glob --max-turns 15 --output-format json
--json-schema <schema> --permission-mode dontAsk --setting-sources "" --settings '{"disableAllHooks":true}' --strict-mcp-config`
with `CLAUDE_CODE_DISABLE_CLAUDE_MDS=1 CLAUDE_CODE_DISABLE_AUTO_MEMORY=1`, all inherited `CLAUDE*` vars dropped, cwd = work copy,
600 s timeout. Prompt = framing + task prompt (T-file) + diff (tracked + untracked, cap 60k chars) + the SPEC's "Find defects..."
sentence. `structured_output` is read from the JSON result; `total_cost_usd`, `num_turns`, wall and models are recorded.
- `indep`: reviewers A and B, same prompt, run in parallel, no shared context. Blocking findings = union. Pass when both approve.
- `adv`: A as above; challenger B gets A's numbered findings + the diff, answers `upheld|rejected` + reason per id, may add
findings (`added_findings`). Blocking = A's blocking findings B upheld + B's added blocking findings. B always runs (same
2-call budget as indep).
- Effective `approve` = "no blocking finding" (the raw `approve` is logged as `approve_raw`; a reviewer saying approve=false with
only minor findings does not block). A reviewer call that errors approves (fail open) with `error` recorded; an errored
challenger upholds nothing.
- Reason = "Code review found N blocking defect(s):" + numbered `file:line text` + "Fix every finding, re-run the tests, then stop.", <= 1500 chars.
### Per run (`runs/<task>-<cfg>-r<rep>/`)
`stream.jsonl`, `stderr.txt`, `exit_code.txt`, `wall_s.txt` (includes hook time: the hook runs inside the session),
`gate.jsonl`, `snap-<n>.diff`, `hidden-<n>.txt`, `gate1-<n>.txt`, `hook-input-1.json`, `post.diff`, `numstat.txt`,
`suite.txt` (full suite, no -x), `hidden.txt`, `gate_tests/`, `testwriter.json`, `gatetests-<n>.txt`, `gatetests-final.txt`, `result.json`, `score.json`, `meta.json`, `task.md`, `prompt.txt`,
`settings.json` + `stop.sh` (gated configs).
`score.json`: task, cfg, rep, hidden_pass/hidden_fail/failed_reqs (**requirement level**: 5 reqs per task), suite_pass/fail,
hook_calls, blocks_total, blocks_by_gate {g1,g3}, right_blocks / false_blocks (+ `_by_gate`), give_up, ended_blocked,
cost_worker_usd, cost_reviewers_usd, cost_testwriter_usd, cost_total_usd (all three), testwriter_ok, gate_tests_total,
gate_tests_fail_final, turns, wall_s, reviewer_findings_total, reviewer_errors, gate_crashes,
model_ok (worker init model + every modelUsage key + every reviewer model == claude-sonnet-5-5), hook_events_in_stream,
hidden_first_stop.
## Matrix
5 tasks x 5 configs x 2 reps = 50 runs. Rep 1 config blocks bare, g1, g3, g13, g13adv (tasks rotated per block), rep 2 the
reverse block order. Up to 3 runs at once, each in its own work copy (`matrix-one.sh` via `xargs -P 3`, bash 3.2 has no
`wait -n`). `matrix.log`: one `start` and one result line per run.
## Traps found
- **Gate 1 as first built passed by construction** (it only re-ran the implementer's own tests: `T2-g1-r1` one allow, hidden
R4 missed). Redesigned around an independent test writer (above). The writer hedges where the terse prompt is silent
(accepts "raises or no-op"), so it cannot recover unstated requirements; per-task overlap is in VALIDATION.md.
- **A blocked agent hunts for the tests.** After a gate-1 block the smoke agent ran `cat .claude/hooks/stop.sh; find / -name
test_T1.py` (denied only because `find` is not on the allow-list; `cat` is, and `stop.sh` names the bench dir, from where
`hidden/` is one `cat` away). Gated configs' `.claude/settings.json` now carries `permissions.deny`: Read of `./.claude/**`,
the bench's `hidden/ runs/ validation/ reference/`, the two sibling benches; Bash patterns containing `.claude/hooks`,
`.claude/settings`, `../hidden`, `../runs`, `gate_tests`, the bench sub-paths, `find *`. Probed with a Haiku session: all
five escape attempts (cat, Read, ls, python open) denied, in-tree reads fine. Bash rules match command text, not a security
boundary. Bare runs have no settings file (bare r1 ran before this change; bare has no motive to hunt).
Reviewers and the writer run with `--setting-sources ""`, so these rules do not apply to them.
- **Transient `FileNotFoundError` spawning a reviewer** inside the hook (once, both reviewers of one call, wall 0.0 s;
never reproduced by hand). Reviewer calls now retry once after 3 s; the error message records `filename`.
- **`--bare` cannot authenticate here.** It reads `ANTHROPIC_API_KEY` / `apiKeyHelper` only, never OAuth or the keychain;
this account is OAuth (`~/.claude`), so a `--bare` reviewer returns `terminal_reason: api_error`, cost 0. Used the
documented equivalents instead: `--setting-sources ""` + `--settings '{"disableAllHooks":true}'` + `--strict-mcp-config` +
`CLAUDE_CODE_DISABLE_CLAUDE_MDS=1` + `CLAUDE_CODE_DISABLE_AUTO_MEMORY=1` + `--tools Read,Grep,Glob`. The reviewers' cwd is the
work copy, whose `.claude/settings.json` carries the Stop hook: without `--setting-sources ""`/`disableAllHooks` a reviewer
would run the gate recursively. Verified: a dummy Stop hook in the probe copy never fired.
- **Inherited session env.** A matrix launched from inside Claude Code inherits `CLAUDE_EFFORT=medium`,
`CLAUDE_CODE_SUBAGENT_MODEL=opus`, `CLAUDE_CODE_SESSION_ID`, `CLAUDECODE`, `CLAUDE_CODE_CHILD_SESSION`, messaging tokens.
`env.sh` unsets every `CLAUDE*` var before setting its own; reviewers get a scrubbed env too (the hook process itself
inherits the worker's `CLAUDE_*` vars, e.g. the worker's session id).
- **The feedback line leaks the hook command.** Claude Code prefixes each block with the command line:
`Stop hook feedback:\n[<command>]: <reason>`. With the SPEC command inline the agent would read `GATE_CONFIG=g13adv` and
`gate-bench/gate.sh`. The settings command is therefore `bash "$CLAUDE_PROJECT_DIR/.claude/hooks/stop.sh"`, a one-line
wrapper holding the SPEC command. The agent can still `cat` it (it lives in the work copy) but it is not pushed at it.
- **Stop hook input keys (2.1.293):** `background_tasks, cwd, effort ({"level":"medium"}), hook_event_name,
last_assistant_message, permission_mode, prompt_id, session_crons, session_id, stop_hook_active, transcript_path`.
- **8-cap resets on tool calls.** Per the docs the consecutive-continuation count resets each time Claude calls a tool, so the
built-in cap only bites an agent that answers blocks with text alone. Our give-up (6 blocks per run, counted from
`gate.jsonl`, tool calls or not) fires first in every case.
- **Red start state: the agent refuses to fix code it did not write.** Validation step 2 (T1 + an uncommitted `>=` -> `>`
edit) shows Sonnet 5.5 implementing T1, then answering 6 consecutive gate-1 blocks with "the red test is your pre-existing
uncommitted edit, not my change" without touching it, until give_up. The hook mechanics are validated; the behaviour is a
finding in itself (gate 1 cannot force a fix the agent considers out of scope).
- **pytest-cov.** The repo's `pytest.ini` adds `--cov tinydb`; gate 1 adds `--no-cov` so the 40-line tail holds the failure,
not the coverage table (and no `.coverage` file is written). `.coverage` is gitignored anyway.
- **T5 R4 is parametrized** (several test ids, one requirement). hidden counts are per requirement (5 per task), a req
passes only when all its ids pass. `validate.sh` step 1: T5 upstream passes 1 req (R3, the regression guard) as designed.
- **Diff without touching the index.** `git add -N` would change what the agent sees in `git status`; untracked files are
diffed with `git diff --no-index /dev/null <f>` instead. `.claude/` is in `.git/info/exclude`, so the hook config never
shows in any diff or in `git status`.
- **Worker Bash denials.** The 2.1.293 permission layer denies some agent commands with "Contains brace with quote character
(expansion obfuscation)" even under acceptEdits + allow-list; agents route around it (visible as `permission_denied` events).
- **Reviewers on a correct solution still flag "blocking".** On the T1 reference patch, reviewer A marked "pull does not check
the field is a list" (not required by the task) as blocking; indep therefore blocks, the adv challenger rejected it. This is
exactly what false_blocks measures (hidden all green at that stop).
- **Shared plan meter.** Workers and reviewers bill the user's plan quota; `total_cost_usd` is the list-price equivalent the
CLI reports, not a bill. Nothing else heavy should run on the account during the matrix.
- **rtk rewrites interactive `grep`/`diff`** in the launching Claude session only; scripts here call binaries directly
(`/usr/bin/diff`/`cmp` for verification diffs).
- **CLI version:** `claude --version` = 2.1.293 (Claude Code); flags checked in `claude --help`: `--json-schema`, `--bare`, `--tools`,
`--permission-mode` (choices acceptEdits, auto, bypassPermissions, manual, dontAsk, plan), `--setting-sources`,
`--include-hook-events`. `--max-turns` is accepted though not listed in `--help`.
## Matrix launch (2026-10-08 16:33:54, after the gate-1 redesign)
`cd /Users/thomas/dev/gate-bench && SKIP_DONE=1 nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 &` -> PID 71317, 3 lanes.
The 5 bare r1 runs (terse prompts) are kept and skipped; 45 runs to go. Expected wall ~40-70 min: gated runs so far take
~30-70 s of worker wall + ~35-40 s of test writer (before the worker) + 1-35 s per hook call, more with blocks.
Then `.venv/bin/python analyze.py`.
SPEC.md
# gate-bench — SPEC (written 2026-10-08 by the /plan stage of the AIDive pipeline)
Goal: measure, on a real repo, what each of Shopify's Helix gates catches when rebuilt as a Claude Code
Stop hook, and what it costs. Output feeds a YouTube video; every number must be reproducible from this dir.
## Reuse (read-only, never modify)
- `/Users/thomas/dev/verify-loops-bench/upstream/` : tinydb @ 18d73a1 (git repo). Copy it as `upstream/` here.
- `/Users/thomas/dev/verify-loops-bench/hidden/` : `test_T1.py..test_T6.py`, `conftest.py`, `pytest.ini`. Copy as `hidden/`.
Hidden tests are the ORACLE: one `test_reqN` per requirement of the task prompt. They live OUTSIDE every work copy
and the agent never sees them. Run: `cd <tree> && python -m pytest -c <bench>/hidden/pytest.ini --rootdir <bench>/hidden <bench>/hidden/test_Tn.py -q`
(conftest puts cwd or `$TINYDB_TREE` first on sys.path). Check `tasks/README.md` there for the exact invocation and validate with `validate.sh`.
- `/Users/thomas/dev/verify-loops-bench/tasks/T1.md..T5.md` : the five task prompts, typed as a developer would. Copy as `tasks/`. T6 is NOT used.
- `/Users/thomas/dev/haiku-routing-bench/{env.sh,run.sh,extract.py,prep.sh,matrix.sh}` : patterns to copy (auth via the
user's default config dir, `CLAUDE_CODE_DISABLE_AUTO_MEMORY=1`, `.venv` with pytest, stream-json extraction of
`total_cost_usd`, `num_turns`, `usage`, `modelUsage`, fresh work copy per run, `--setting-sources project --strict-mcp-config`).
- Claude Code CLI 2.1.293 is installed. Docs that define the hook contract: https://code.claude.com/docs/en/hooks
(fetch with `curl -s https://r.jina.ai/https://code.claude.com/docs/en/hooks`): a Stop hook that exits 2 blocks the
stop and its stderr is fed back; `{"decision":"block","reason":"..."}` on stdout does the same; input JSON on stdin
carries `stop_hook_active`, `session_id`, `transcript_path`; after 8 consecutive continuations Claude Code overrides
the next block (`CLAUDE_CODE_STOP_HOOK_BLOCK_CAP` raises it); a hook past `timeout` (seconds, default 600) renders no decision.
## Worker
`claude -p "<task prompt>" --model claude-sonnet-5-5 --output-format stream-json --verbose --max-turns 60
--permission-mode acceptEdits --allowedTools "<same list as haiku-routing-bench/run.sh>" --setting-sources project
--strict-mcp-config`, cwd = fresh work copy `work/<run>/` (copy of upstream, `git` clean at HEAD). The task prompt is the
T-file verbatim plus one line appended for every config (bare included): "When you are done, stop and summarise what
you changed." No mention of gates in the prompt: the gate is the hook, not the instruction.
Work copy carries `.claude/settings.json` with the Stop hook for gated configs (none for `bare`):
`{"hooks":{"Stop":[{"hooks":[{"type":"command","command":"GATE_RUN=<abs run dir> GATE_CONFIG=<cfg> bash <bench>/gate.sh","timeout":1500}]}]}}`
(check the exact settings shape in the hooks doc; `timeout` is in seconds; verify the hook fires with a smoke run before the matrix).
## Gates (gate.sh, one script, dispatches on GATE_CONFIG)
Reads stdin JSON. Logs ONE line per invocation to `$GATE_RUN/gate.jsonl`: {ts, n (invocation index), stop_hook_active,
gate1:{ran, suite_pass, suite_fail, tests_touched (bool: any file under tests/ added or modified vs HEAD), pass},
gate3:{ran, variant, reviewers:[{id, approve, findings:[{severity, file, line, text}], cost_usd, turns, wall_s}], blocking_findings, pass},
hidden:{pass, fail, failed_reqs} (ground truth at this moment, run SILENTLY on the work copy; never shown to the agent),
decision ("block"|"allow"|"give_up"), reason (the text sent back, truncated 500), wall_s}.
Give-up: after 6 blocks in one run the hook allows and logs decision "give_up" (the video needs to count it; Claude
Code's own 8-cap would otherwise hide it). Never use stop_hook_active to allow: log it only.
Order inside one invocation: gate 1 (when the config has it) first; gate 3 only when gate 1 passed or is absent.
- Gate 1 (behaviour tests): `python -m pytest -q -x --no-header -p no:cacheprovider` on the work copy (the repo's own 226 tests plus
whatever the agent wrote). Block when red (reason = last 40 lines of pytest), OR when `tests_touched` is false
(reason: "No test covers the new behaviour. Add tests for every requirement in the task, run them, then stop.").
- Gate 3 (adversarial reviews): headless reviewers, each `claude -p --bare --model claude-sonnet-5-5 --tools "Read,Grep,Glob"
--max-turns 15 --output-format json --json-schema '<schema>' --permission-mode dontAsk` (confirm each flag exists in
`claude --help`; drop `--permission-mode dontAsk` if it errors), cwd = work copy, prompt = system framing + the task
prompt + `git diff HEAD` (tracked + untracked files, cap 60k chars) + "Find defects: requirements not met, edge cases
wrong, behaviour broken, tests that do not test the requirement. Report only real defects with file and line. approve=true
only when nothing blocks." Schema: {approve: boolean, findings: [{severity: "blocking"|"minor", file, line, text}]}.
Read `structured_output` from the JSON result; record `total_cost_usd`. Set `CLAUDE_CODE_DISABLE_AUTO_MEMORY=1`.
- variant `indep` (Shopify as written): two reviewers, same prompt, independent, no shared context; union of findings;
block if any blocking finding from either; reason = the blocking findings, numbered, "Fix every finding, re-run the
tests, then stop." Pass when BOTH approve.
- variant `adv` (reviewer + challenger, arXiv 2608.18167 shape): reviewer A as above; then challenger B gets A's
findings plus the diff and must, per finding, answer upheld|rejected with a reason, and may add findings A missed;
only upheld (or B-added) blocking findings block. Pass when nothing upheld blocks.
- Reason text is what the worker sees next turn: keep it factual, under 1500 chars.
## Configs (configs.json)
- `bare`: no hook.
- `g1`: gate 1 only.
- `g3`: gate 3 indep only.
- `g13`: gate 1 then gate 3 indep (Shopify's order).
- `g13adv`: gate 1 then gate 3 adv.
Gate 2 (UI) and gate 4 (engineer) are out of scope by design: tinydb is a library, Shopify allows skipping UI review for
logic-only changes; gate 4 is the human, measured as "what reached the human": the hidden failures left at the end.
## Per run outputs (runs/<task>-<cfg>-r<rep>/)
stream.jsonl, stderr.txt, exit_code.txt, wall_s.txt, gate.jsonl, post.diff, numstat.txt, suite.txt (final repo suite),
hidden.txt (final hidden run), result.json (worker: total_cost_usd, num_turns, usage, modelUsage, is_error, subtype,
final text), score.json: {task, cfg, rep, hidden_pass, hidden_fail, failed_reqs, suite_pass, suite_fail, blocks_total,
blocks_by_gate {g1, g3}, right_blocks (hidden failing at that moment), false_blocks (hidden all passing at that moment),
give_up, ended_blocked (the LAST gate.jsonl line is a block, i.e. Claude Code's cap or the agent ended anyway),
cost_worker_usd, cost_reviewers_usd, cost_total_usd, turns, wall_s, reviewer_findings_total, model_ok}.
Snapshot at every hook call: before gate 1 runs, `git diff HEAD` of the work copy to `$GATE_RUN/snap-<n>.diff`.
## Matrix
5 tasks (T1..T5) × 5 configs × 2 reps = 50 runs. `matrix.txt` ordered: rep 1 of everything first (config-contiguous:
bare, g1, g3, g13, g13adv, tasks rotated), then rep 2 in reverse config order. `matrix.sh` runs up to 3 runs in parallel
(each in its own work copy), appends one line per run to `matrix.log`, resumable (`SKIP_DONE=1` skips runs with score.json).
`analyze.py` → `results.json` + `results.md`: per config: hidden requirements passed / total, runs fully passing, mean
cost, mean wall, blocks per run, right vs false blocks per gate, give-ups, ended_blocked; per task the same; and the
answer to the video's questions: (1) does gate 1 alone move hidden pass rate vs bare, (2) does gate 3 indep add anything
over g1, (3) does adv beat indep, (4) false-block rate per gate, (5) cost per gate.
## Validation before the matrix (VALIDATION.md + validate.sh)
1. upstream fails every hidden test file; the reference patches in `/Users/thomas/dev/verify-loops-bench/reference/`
pass hidden + suite (reuse their validate.sh logic).
2. The hook fires: one smoke run of T1 on `g1` with a deliberately failing state (apply a patch that breaks one test)
must show a block in gate.jsonl, and the stream must show the agent continuing after "Stop hook feedback".
3. gate.sh called by hand with a fake stdin on a clean work copy returns allow, on a red copy returns exit 2 with the
pytest tail on stderr.
4. One reviewer call by hand returns valid structured_output against the schema.
5. Record the exact `claude --version`, the settings.json used, and the hook input keys seen.
## Deliverables
Everything under /Users/thomas/dev/gate-bench with README.md (layout, commands, traps found). When the matrix is launched,
run it with `nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 &` and report the PID; do not wait for it to finish.
Copy the scripts (not runs/, not work/) to /Users/thomas/dev/faceless-video-pipeline/jobs/shopify-four-gates-claude-code-20261008-1529/research/code/
with a one-line README. Report: what was validated, the smoke run's gate.jsonl, the matrix PID, expected duration.
VALIDATION.md
# Validation (2026-10-08, claude 2.1.293) — raw output in validation.log
1. **Oracle.** `./validate.sh step1`: on upstream every hidden req fails (T5: 1 passing, R3 = regression guard, as designed);
each reference patch passes its hidden file (T1-T4 5 tests, T5 6 tests = 5 reqs) and the repo's 226 tests. OK.
2. **Hook fires.** `INJECT=$PWD/inject/break.patch RUN_TAG=val2 ./run.sh g1 T1 1` (T1 + an uncommitted `>=` -> `>` break):
gate.jsonl = 7 calls, 6 `block` (g1, suite red 1 failed / 33 passed under -x) then `give_up`; the stream shows
`Stop hook feedback:` user messages followed by new assistant turns (the agent continued each time; it kept
explaining the red test is the user's pre-existing edit and never touched it). `stop_hook_active` false on call 1, true after. OK.
3. **gate.sh by hand** (`./validate.sh step3`, fake stdin): green copy (T1 reference + one new test) -> exit 0, `allow`,
empty stderr; red copy (break.patch) -> exit 2, `block`, stderr = pytest tail incl. `short test summary info`. OK.
4. **Reviewer by hand** (`./validate.sh step4`, `lib/review.py work/_val3-green T1 indep|adv`): every call returned a
schema-valid `structured_output`, cost $0.03-0.05 per pair, 10-45 s per call. OK.
5. **Recorded.** `claude --version` 2.1.293. settings.json (gated configs):
`{"hooks":{"Stop":[{"hooks":[{"type":"command","command":"bash \"$CLAUDE_PROJECT_DIR/.claude/hooks/stop.sh\"","timeout":1500}]}]}}`,
`.claude/hooks/stop.sh` = `GATE_RUN=<run dir> GATE_CONFIG=<cfg> bash /Users/thomas/dev/gate-bench/gate.sh`.
Hook input keys seen: background_tasks, cwd, effort, hook_event_name, last_assistant_message, permission_mode, prompt_id,
session_crons, session_id, stop_hook_active, transcript_path. Stream hook events in the smoke run: exactly one
`hook_started`/`hook_response` pair, `hook_name: Stop` (no user/global hooks: `--setting-sources project`).
Smoke (`runs/T1-g13adv-r1-smoke`, wrapper hook): hidden 5/5, suite 238/238, 1 hook call, gate 1 green with tests touched,
adv reviewer A approve (0 findings), challenger approve, allow. Worker $0.084 + reviewers $0.034 = $0.117, 12 turns, 66 s.
`runs/T1-g13adv-r1-smoke0` = the first smoke with the inline SPEC command (same outcome, $0.195, 69 s), superseded because the
feedback prefix would leak the command line.
## Gate 1 redesign (2026-10-08, after the coordinator's review): independent test writer
Gate 1 no longer re-runs only the implementer's own tests (it passed by construction: `T2-g1-r1` one allow, hidden R4
missed, suite green). Before the worker starts, `lib/testwriter.py` (headless Sonnet 5.5, reviewer isolation flags,
`--tools Read,Grep,Glob --max-turns 12 --json-schema {file}`, cwd = a pristine temp copy of upstream) gets ONLY
`tasks/<T>.md` and writes `runs/<run>/gate_tests/test_<T>.py` (+ hidden's conftest/pytest.ini) outside the work copy.
Gate 1 = repo suite AND those tests (`TINYDB_TREE` = work copy). `tests_touched` is logged, no longer a rule.
`./validate_writer.sh` (one writer call per task, files in `validation/writer/<T>/`; reference = verify-loops-bench patches):
| task | writer tests | pass on reference | pass on upstream | fail on reference (possible false blocks, kept) | writer $ / wall |
|---|---|---|---|---|---|
| T1 | 52 | 51 | 1 | `test_pull_on_non_list_field_raises_and_leaves_data_unchanged` | 0.099 / 39 s |
| T2 | 42 | 42 | 5 | none | 0.094 / 35 s |
| T3 | 35 | 34 | 11 | `test_rename_table_non_string_target_does_not_corrupt_database` | 0.077 / 34 s |
| T4 | 48 | 47 | 7 | `test_reverse_without_sort_by_reverses_default_order` | 0.092 / 42 s |
| T5 | 26 | 25 | 7 | `test_non_integer_size_is_rejected` | 0.086 / 30 s |
Tests passing on upstream are the "existing behaviour unchanged" guards the instruction asks for (plus a few that
accept "raises" as an outcome); every task's file fails on upstream overall. Each matrix run writes its own file, so
these counts are one sample; per-run counts land in `score.json` (`gate_tests_total`, `gate_tests_fail_final`).
Overlap with the 5 hidden requirements (judgement from test names + spot reads of the assertions):
- T1: 2/5 strict (R1 order, R5 push non-list raises). R2/R4 hedged (`try: ... except (KeyError, AttributeError, TypeError)`: a raise is accepted), R3 hedged (`count('x') < 2`, first-occurrence removal passes).
- T2: 4/5 (R1, R2, R3, R5 cache inclusive vs exclusive and other bounds); R4 hedged (accepts a TypeError).
- T3: 3/5 strict (R1 ids, R2 persistence + `tables()`, R5 stale cache / old name empty / id sequence); R3/R4 use `pytest.raises(Exception)`, not KeyError/ValueError (R4 does check both tables intact).
- T4: 4/5 (R1, R3, R4 limit 0 / negative ValueError, R5 cache not polluted + Document/doc_id); R2 hedged (checks order of docs that have the field and that none are dropped, not "missing last in both directions").
- T5: 5/5 (R1, R2 restart, R3 class/subclass default, R4 non-positive ValueError at construction, R5 close flushes).
The writer hedges exactly where the terse prompt is silent: it encodes "a raise or a no-op are both fine", so gate 1
cannot recover a requirement the prompt never states (T1 R2/R4, T4 R2).
Smoke after the redesign (`runs/T1-g13adv-r1-smoke`, with the deny rules): writer 45 tests ($0.071), first stop gate 1
green (232 suite + 45/45 gate tests), adv A+B approve, allow; hidden 2/5 (R2, R4, R5 missed). Total $0.189, 68 s.
`runs/T1-g13adv-r1-smoke-a` (first redesign smoke): stop 1 blocked by gate 1 (5 gate tests red, reason starts
"The acceptance tests for this change fail:"), the agent fixed push/pull type checks, stop 2 gate 1 green, both reviewers
errored (transient FileNotFoundError on spawn, logged, fail-open allow; one retry added since). The agent also tried
`cat .claude/hooks/stop.sh; find / -name test_T1.py` after the block (denied by the allow-list) -> deny rules added.
`runs/T1-g13adv-r1-smoke-b`: second redesign smoke, clean (reviewers ran, $0.167).
Zrób to w poniedziałek
- Napisz kontrolę jako skrypt, który kończy się kodem 2 z porażką na stderr, i podepnij go jako hook Stop typu command z timeoutem dłuższym niż twój zestaw testów.
- Niech osobny agent napisze testy akceptacyjne wyłącznie na podstawie zlecenia, zapisz je poza repo i zabroń implementującemu odczytu hooka, ustawień i tych testów.
- Loguj każde zatrzymanie do pliku z decyzją i informacją, czy bramka była czerwona, żeby bramka, która coś przepuściła, była widoczna.
- Dodaj własny licznik, który wypuszcza pętlę po sześciu blokadach. Limit Claude Code to osiem kolejnych blokad zerowanych przy każdym wywołaniu narzędzia.
- Uruchamiaj recenzentów w trybie headless z
--tools Read,Grep,Glob, pustym źródłem ustawień i wyłączonymi hookami, żeby nie mogli rekurencyjnie wywołać bramki. - Jeśli zachowujesz bramkę przeglądu, odsyłaj jej ustalenia jako radę w informacji zwrotnej zamiast robić z nich poprzeczkę.
- Przez dwa tygodnie licz słuszne i fałszywe blokady, zanim zaufasz bramce.
- Pamiętaj, że
--bareczyta tylko klucz API: przy logowaniu subskrypcją zwraca błąd, więc izoluj recenzentów flagami ustawień.
Pomiary
Z results.md: 50 runów, po 10 na konfigurację, 5 zadań razy 2 powtórzenia. Ukryte wymagania liczone są z 50 na konfigurację. Blokady zapisano jako słuszne/fałszywe: słuszna blokada oznacza, że ukryta wyrocznia miała niespełnione wymaganie przy tym zatrzymaniu, fałszywa, że była w pełni zielona.
| Konfiguracja | Zaliczone ukryte wymagania | W pełni czyste runy | Średni koszt na run | Średni czas w sekundach | Blokady bramki 1 słuszne/fałszywe | Blokady bramki 3 słuszne/fałszywe | Poddania się |
|---|---|---|---|---|---|---|---|
| bare (no hook) | 35/50 (70%) | 3 | $0.085 | 33 | 0/0 | 0/0 | 0 |
| g1 (acceptance tests) | 42/50 (84%) | 6 | $0.201 | 60 | 3/3 | 0/0 | 0 |
| g3 (two reviewers) | 34/50 (68%) | 3 | $0.151 | 54 | 0/0 | 1/0 | 0 |
| g13 (tests, then two reviewers) | 41/50 (82%) | 5 | $0.282 | 86 | 14/1 | 2/0 | 2 |
| g13adv (tests, then reviewer plus critic) | 39/50 (78%) | 6 | $0.276 | 86 | 11/1 | 0/0 | 1 |
Koszt to suma workera, piszącego bramki 1 i recenzentów bramki 3, tak jak raportuje go total_cost_usd, odpowiednik po cenach katalogowych, a nie rachunek. Każdy run miał skonfigurowany model. W runach z bramką bramka 1 odesłała agenta 28 razy z faktycznie niespełnionym wymaganiem i 5 razy, gdy wszystkie były spełnione. Bramka 3 zablokowała 3 razy i nigdy niesłusznie. Trzy pętle poddały się przy sześciu blokadach, a własny limit Claude Code nigdy nie zadziałał.
Zostaje dla człowieka z 50 wymagań: 15 bez bramki, 8 z testami, 16 z samymi recenzentami, 9 z testami i recenzentami, 11 z testami i recenzentem plus krytykiem. Tam, gdzie testy ruszyły zadanie: wyszukiwanie z sortowaniem poszło z 5 z 10 do 10 z 10, a zadanie z rozmiarem cache z 8 do 10. Zadanie push i pull zostało na 4 z 10, bo zlecenie nie mówiło, co zrobić z dokumentem bez tego pola.
Co dalej
- Bramka UI. Bramka 2 Shopify każe drugiemu modelowi działać jako recenzent designu na repo frontendowym s1. Pominęliśmy ją, bo repo jest biblioteką. Miejscem na jej zmierzenie jest późniejszy test na repo frontendowym.
- Pętla Ralph. Również utrzymuje agenta w pracy, dopóki warunek nie zostanie spełniony, ale wyjście siedzi w frazie, którą agent może napisać. Nasze wyjście jest zewnętrzne: skrypt i licznik poza zasięgiem agenta.
- Publikacja o przeglądzie adwersaryjnym. Czytaj tabele, nie streszczenie: układ recenzent plus krytyk s5 to ten, który na papierze zasługuje na swoje tokeny, a nasz run nie odtworzył zysku na tym repo, więc przetestuj go na swoim, zanim go przyjmiesz. Strona abstraktu: s23.
- Limit blokad i timeout. Przeczytaj sekcję Stop w referencji hooków, zanim zaczniesz polegać na hooku jako twardym zatrzymaniu s2, oraz sekcję rozwiązywania problemów w poradniku o hookach s19.
- Zmierzona bramka, z którą możesz porównać swoje raportowanie: s8.
FAQ
Czy hook Stop naprawdę może powstrzymać Claude Code przed zakończeniem?
W większości. Exit 2 uniemożliwia zakończenie i odsyła stderr s2, ale po ośmiu kolejnych kontynuacjach Claude Code nadpisuje następną blokadę, a hook, który przekroczy timeout, nie wydaje decyzji s2. Loguj każde zatrzymanie, żeby widzieć, kiedy coś przepuścił.
Dlaczego piszący testy jest osobnym agentem?
Nasza pierwsza wersja bramki 1 uruchamiała ponownie własne testy implementującego. Przepuszczała każdy run przy pierwszym zatrzymaniu, choć ukryte wymaganie padało, bo własne testy agenta testują to, co sam zbudował. Przeprojektowany piszący widzi tylko zlecenie i repo sprzed zmiany.
Dlaczego nie użyć --bare dla recenzentów?
Pomija hooki i pamięć s20, ale przy logowaniu subskrypcją czyta tylko klucz API i zwrócił tu api_error. Izolowaliśmy recenzentów pustym źródłem ustawień, wyłączonymi hookami, ścisłą konfiguracją MCP i narzędziami tylko do odczytu.
Czy recenzenci kiedykolwiek zablokowali dobrą pracę?
Nie w naszych 50 runach: 3 blokady, żadna niesłuszna. Kosztem było to, co przepuścili. W dwóch runach obaj recenzenci zobaczyli ten sam crash przy braku pola sortowania i ocenili go jako minor, a oba runy skończyły się na zero z pięciu.
Czy to dowodzi, że recenzenci nigdy nie pomagają?
Nie. To jedno repo, jeden model workera, pięć zadań i dwa powtórzenia na komórkę. Pokazuje, że recenzenci nie ruszyli tu odsetka zaliczeń, a bramka testów tak.