AIDiveTL;DR ve hüküm
- Shopify'ın Helix'i her checkpoint'i sırayla dört kapıdan geçiriyor: davranış testleri, UI incelemesi, iki bağımsız bağlamı izole reviewer ve bir mühendisin onayı s1. Yazı hiçbir pass rate vermiyor. Biz 1, 3 ve 4. kapıları gerçek bir repoda Claude Code Stop hook'u olarak yeniden kurduk ve saydık.
- Matrisimiz: 50 run, beş tinydb görevi, beş konfigürasyon, headless Sonnet 5.5. Hiç kapı yokken 50 gizli gereksinimden 35'i geçti (70%). Tek başına kabul testi kapısıyla 50'de 42 (84%).
- Yazıldığı haliyle iki reviewer'lı kapı pass rate'i oynatmadı: tek başına 50'de 34, testlerden sonra 50'de 41. Bir reviewer artı bir critic 50'de 39'a ulaştı.
- Hüküm: test kapısını tutun, uygulayan olmayan bir writer ile ve agent'ın düzenleyemeyeceği testlerle. Altı engelden birinin yanlış olmasını bekleyin ve döngüye bir sayaç verin.
- İki reviewer'lı kapıyı, tutacaksanız, bir bar olarak değil, geri bildirimde tavsiye olarak tutun.
- Sınırlar: bir repo, bir worker modeli, beş görev, hücre başına iki tekrar. Satırlar arasındaki iki puanlık fark gürültü. Kapısız ile test kapısı arasındaki 14 puan değil.
Kaynaklar ne diyor
Helix kuralı kendisi adlandırıyor: bir kapı başarısız olursa agent geri bildirimi uygulamayı düzeltmek için kullanır ve kontrolü yeniden çalıştırır; sonucun yeterince iyi olduğunu düşündüğü için başarısız bir kontrolü geçersiz kılamaz s1. 3. kapı, iki bağımsız, bağlamı izole reviewer agent'tır; her bulgunun düzeltilmesi gerekir ve ikisi de onaylayana kadar döngü tekrarlanır s1. Yalnızca mantık içeren bir değişiklik UI inceleme kapısını atlar s1; CLI tarzı kütüphane repomuzda 2. kapının olmamasının sebebi bu. Bağımsız bir yazı, Helix kanıtını vaka çalışması, birinci taraf ve ölçülmemiş olarak sınıflandırıyor ve hakemi hiçbir şeyin doğrulamadığını not ediyor: yanlış engelleme oranı raporlanmıyor s6.
Anthropic'in kendi rehberi sorunu veriyor: Claude iş bitmiş göründüğünde durur ve çalıştırabileceği bir kontrol olmadan "bitmiş görünüyor" tek sinyaldir s3. Deterministik sürüm, kontrolünüzü betik olarak çalıştıran ve geçene kadar turun bitmesini engelleyen bir Stop hook'tur s3. Rehber ayrıca, taze bir subagent bağlamındaki reviewer'ın yalnızca diff'i ve kriterleri gördüğünü, değişikliğin arkasındaki akıl yürütmeyi görmediğini ekliyor s3 ve subagent'lar taze, izole bir bağlam penceresiyle başlıyor s22.
Hook sözleşmesi: Stop'ta 2 ile çıkmak Claude'un durmasını engeller s2 ve stderr mesajı Claude'a sebep olarak ulaşır s2. Başka herhangi bir çıkış kodu, 1 dahil, kendi başına engellemez s2. JSON biçimi, zorunlu bir reason ile decision: "block" şeklindedir s2.
Kapının kendi belgelerinde üç çıkışı var:
- Üst sınır. Stop hook'ları turu art arda sekiz kez devam ettirdikten sonra Claude Code bir sonraki engeli geçersiz kılar ve turu bitirir. Sayaç Claude her tool çağırdığında sıfırlanır ve
CLAUDE_CODE_STOP_HOOK_BLOCK_CAPonu yükseltir s2 s19. - Timeout. Command hook'lar varsayılan olarak 600 saniyedir ve timeout'a ulaşan bir hook iptal edilir, çıktısı atılır; bu yüzden çoğu olayda karar üretmez s2.
- Bayrak.
stop_hook_active, Claude Code zaten bir stop hook yüzünden devam ediyorsa true olur; böylece bir betik asla çözülmeyecek bir koşulda engellemekten kaçınabilir s2.
Bir uygulayıcı, ucuz ve düşük bağlamlı bir denetçinin "gets things wrong a lot" olduğu ve iyi oturumları engellediği, bu yüzden son sözü almaması gerektiği uyarısında bulunuyor s7. Etiketli 100 durak üzerinde ölçülmüş bir Stop kapısı 100'de 8 turu engelledi, bunların 7'si haklıydı; etiketlerin elle değil bir modelden geldiği uyarısıyla s8. Biz de engelleri aynı şekilde raporluyoruz: haklı ve haksız.
Dayandığımız betik bayrakları: --bare hook'ları, skill'leri, plugin'leri, MCP sunucularını, otomatik belleği ve CLAUDE.md'yi atlar s20, --max-turns print modunda agentic turları sınırlar s20, --json-schema doğrulanmış çıktıyı structured_output alanında döndürür s21 ve --output-format json, faturanızdan farklı olabilen istemci tarafı bir tahmin olan total_cost_usd'yi taşır s21.
- kapıda, 2026'dan bir workshop makalesi deseni ölçtü. LiveCodeBench üzerinde: zero-shot 77%, tek reviewer 77%, 105 problemde iki reviewer 75% ve zor alt kümede 57'de 34 s5. Makale iki reviewer sonucunu hâlâ tek reviewer, self-refine ve zero-shot'ın gürültüsü içinde olarak niteliyor s5. Adversarial biçimi, yani ana agent düzenleme yapmadan önce incelemeyi denetleyen bir critic ile eşleştirilmiş bir reviewer, 87%'e ulaştı s5. SWE-bench Verified üzerinde zero-shot 71.6%'ye karşı 75.2%'ye ulaştı, yaklaşık 4.5x token ile s5. Makale: s23.
Kit
Bench, 18d73a1 commit'indeki tinydb ve 226 geçen test. Bir geliştiricinin yazdığı gibi yazılmış beş istek (push ve pull işlemleri, bir between sorgusu, bir tablo yeniden adlandırma, sıralı arama, bir yazma cache boyutu), her birinin agent'ın asla görmediği beş gizli gereksinimi var: toplam 25. Worker headless Sonnet 5.5. Aşağıdaki her şey harness ve kapılar. tinydb checkout'u, gizli testler ve görev prompt'ları dahil değil: kendi reponuza yöneltin ve kendi oracle'ınızı yazın.
Kapı:
gate.sh: Stop hook giriş noktası,lib/hook.py'yi çalıştıran tek satırlık bir sarmalayıcı.gate.sh
#!/usr/bin/env bash
# Stop hook entry point (bash 3.2 OK). All logic lives in lib/hook.py.
# Called as: GATE_RUN=<abs run dir> GATE_CONFIG=<cfg> bash <bench>/gate.sh (stdin = hook input JSON)
HERE="$(cd "$(dirname "$0")" && pwd)"
exec "$HERE/.venv/bin/python" -I "$HERE/lib/hook.py"
lib/hook.py
"""Stop hook: gate 1 (behaviour tests) and/or gate 3 (adversarial review), per GATE_CONFIG.
Env: GATE_RUN (abs run dir, holds meta.json + prompt.txt), GATE_CONFIG (key of configs.json).
stdin: the Stop hook input JSON. Block = exit 2 with the reason on stderr. Allow = exit 0, no output.
Every invocation appends ONE line to $GATE_RUN/gate.jsonl. Nothing but the reason ever reaches the agent.
"""
import json
import os
import sys
import time
import traceback
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from common import BENCH, run_hidden, run_suite, tests_touched, work_diff # noqa: E402
from testwriter import run_gate_tests # noqa: E402
GIVE_UP_AFTER = 6
def tail40(out, room=1400):
t = "\n".join(out.rstrip().splitlines()[-40:])
if len(t) > room:
t = "...\n" + t[-room:]
return t
def main():
t0 = time.time()
run = os.environ["GATE_RUN"]
cfg_name = os.environ["GATE_CONFIG"]
cfg = json.load(open(os.path.join(BENCH, "configs.json")))[cfg_name]
meta = json.load(open(os.path.join(run, "meta.json")))
work, task = meta["work"], meta["task"]
raw = sys.stdin.read()
try:
hin = json.loads(raw) if raw.strip() else {}
except Exception:
hin = {"_unparsed": raw[:500]}
log_path = os.path.join(run, "gate.jsonl")
prev = []
if os.path.exists(log_path):
prev = [json.loads(l) for l in open(log_path) if l.strip()]
n = len(prev) + 1
blocks_before = sum(1 for p in prev if p.get("decision") == "block")
# Snapshot of the work at this stop, before gate 1 runs.
open(os.path.join(run, "snap-%d.diff" % n), "w").write(work_diff(work))
# Ground truth at this moment (silent).
_, hout, hp, hf, hfailed = run_hidden(work, task)
open(os.path.join(run, "hidden-%d.txt" % n), "w").write(hout)
rec = {"ts": time.strftime("%Y-%m-%dT%H:%M:%S"), "n": n, "stop_hook_active": hin.get("stop_hook_active"),
"hook_input_keys": sorted(hin.keys()),
"gate1": {"ran": False}, "gate3": {"ran": False},
"hidden": {"pass": hp, "fail": hf, "failed_reqs": hfailed}}
reason, blocked_by = None, None
if cfg["gate1"]:
# Gate 1 = the repo suite AND the acceptance tests written before the run by the independent test writer
# (run dir gate_tests/, outside the work copy). A writer failure leaves the suite alone (logged).
rc, out, sp, sf = run_suite(work)
open(os.path.join(run, "gate1-%d.txt" % n), "w").write(out)
suite_ok = rc == 0 and sf == 0
gdir = os.path.join(run, "gate_tests")
has_gt = os.path.exists(os.path.join(gdir, "test_%s.py" % task))
g = {"ran": True, "suite_pass": sp, "suite_fail": sf, "tests_touched": tests_touched(work),
"gate_tests_present": has_gt}
gt_ok, gout = True, ""
if has_gt:
grc, gout, gp, gf, gnames = run_gate_tests(gdir, task, work)
open(os.path.join(run, "gatetests-%d.txt" % n), "w").write(gout)
gt_ok = grc == 0 and gf == 0
g.update(gate_tests_pass=gp, gate_tests_fail=gf, gate_tests_failed=gnames)
g["pass"] = suite_ok and gt_ok
rec["gate1"] = g
if not g["pass"]:
blocked_by = "g1"
parts = []
if not gt_ok:
parts.append("The acceptance tests for this change fail:\n" + tail40(gout, 1400 if suite_ok else 800))
if not suite_ok:
parts.append("The repo's test suite is red (python -m pytest -q -x):\n" + tail40(out, 1400 if gt_ok else 550))
reason = "\n\n".join(parts)
if cfg["gate3"] and blocked_by is None:
from review import gate3, reason_text
g3 = gate3(work, open(os.path.join(run, "task.md")).read(), cfg["gate3"])
rec["gate3"] = g3
if not g3["pass"]:
blocked_by = "g3"
reason = reason_text(g3)
if blocked_by is None:
decision = "allow"
elif blocks_before >= GIVE_UP_AFTER:
decision = "give_up"
else:
decision = "block"
rec["blocked_by"] = blocked_by
rec["decision"] = decision
rec["reason"] = (reason or "")[:500]
rec["reason_chars"] = len(reason or "")
rec["wall_s"] = round(time.time() - t0, 1)
with open(log_path, "a") as fh:
fh.write(json.dumps(rec) + "\n")
if n == 1:
json.dump(hin, open(os.path.join(run, "hook-input-1.json"), "w"), indent=2)
if decision == "block":
sys.stderr.write(reason)
sys.exit(2)
sys.exit(0)
if __name__ == "__main__":
try:
main()
except SystemExit:
raise
except Exception:
# A crashed gate must never deadlock a run: log and allow.
run = os.environ.get("GATE_RUN", "/tmp")
with open(os.path.join(run, "gate-errors.txt"), "a") as fh:
fh.write(traceback.format_exc() + "\n")
sys.exit(0)
lib/testwriter.py
"""Gate 1 test writer: an independent headless Sonnet 5.5 that sees ONLY the task prompt (plus read-only access to a
pristine upstream copy) and writes integration-style pytest cases. The file lives OUTSIDE the work copy, in
<out dir>/gate_tests/ next to a copy of hidden/conftest.py (TINYDB_TREE) and hidden/pytest.ini.
CLI: testwriter.py <task id> <out dir> -> writes <out dir>/gate_tests/test_<T>.py and <out dir>/testwriter.json
"""
import json
import os
import shutil
import subprocess
import sys
import tempfile
import time
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from common import BENCH, HIDDEN, PY, counts, env_clean, sh # noqa: E402
MODEL = "claude-sonnet-5-5"
CALL_TIMEOUT = 600
SCHEMA = {"type": "object", "properties": {"file": {"type": "string"}}, "required": ["file"]}
FRAMING = ("You are a senior engineer writing acceptance tests for a change request to tinydb, a small Python document "
"database. You can read the CURRENT repository (before the change) with Read, Grep and Glob to learn the API; "
"you cannot run code. The change is not implemented yet: your tests must describe the requested behaviour.")
ASK = ("Write integration-style pytest cases for this request against the tinydb API, one test per behaviour a careful "
"senior would expect, pushing for edge cases beyond the happy path (missing fields, wrong types, boundaries, "
"persistence, cache, existing behaviour unchanged). Tests only, no implementation. Return the file content.\n"
"Constraints: one self-contained pytest module (field `file`); `tinydb` is importable (the tree under test is put "
"first on sys.path); do not import anything from the repo's tests/ directory or its fixtures; use "
"tinydb.storages.MemoryStorage or pytest's tmp_path for files; import the new API inside each test, not at module "
"level, so one missing name fails its tests instead of the whole module; give every test a descriptive name.")
def write(task, out_dir, attempts=2):
gt = os.path.join(out_dir, "gate_tests")
os.makedirs(gt, exist_ok=True)
shutil.copy(os.path.join(HIDDEN, "conftest.py"), gt)
shutil.copy(os.path.join(HIDDEN, "pytest.ini"), gt)
task_prompt = open(os.path.join(BENCH, "tasks", task + ".md")).read().strip()
prompt = "%s\n\n## The request (as the developer typed it)\n%s\n\n%s" % (FRAMING, task_prompt, ASK)
rec = {"task": task, "attempts": [], "cost_usd": 0.0, "ok": False, "file": None}
pristine = tempfile.mkdtemp(prefix="gb-writer-")
try:
shutil.copytree(os.path.join(BENCH, "upstream"), os.path.join(pristine, "tinydb-repo"), symlinks=True)
cwd = os.path.join(pristine, "tinydb-repo")
for i in range(attempts):
t0 = time.time()
a = {"error": None}
args = ["claude", "-p", prompt, "--model", MODEL, "--tools", "Read,Grep,Glob", "--max-turns", "12",
"--output-format", "json", "--json-schema", json.dumps(SCHEMA), "--permission-mode", "dontAsk",
"--setting-sources", "", "--settings", '{"disableAllHooks":true}', "--strict-mcp-config"]
env = env_clean({"CLAUDE_CODE_DISABLE_AUTO_MEMORY": "1", "CLAUDE_CODE_DISABLE_CLAUDE_MDS": "1"})
try:
p = subprocess.run(args, cwd=cwd, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
stdin=subprocess.DEVNULL, timeout=CALL_TIMEOUT, env=env)
r = json.loads(p.stdout.decode("utf-8", "replace"))
a.update(cost_usd=r.get("total_cost_usd") or 0.0, turns=r.get("num_turns"),
models=sorted((r.get("modelUsage") or {}).keys()), subtype=r.get("subtype"))
so = r.get("structured_output")
src = so.get("file") if isinstance(so, dict) else None
if r.get("is_error") or not src:
a["error"] = "is_error=%s subtype=%s terminal=%s" % (r.get("is_error"), r.get("subtype"),
r.get("terminal_reason"))
else:
compile(src, "test_%s.py" % task, "exec")
if "def test_" not in src:
raise ValueError("no test functions")
open(os.path.join(gt, "test_%s.py" % task), "w").write(src)
rec["ok"], rec["file"] = True, os.path.join(gt, "test_%s.py" % task)
except subprocess.TimeoutExpired:
a["error"] = "timeout after %ss" % CALL_TIMEOUT
except Exception as e:
a["error"] = "%s: %s" % (type(e).__name__, str(e)[:300])
a["wall_s"] = round(time.time() - t0, 1)
rec["attempts"].append(a)
rec["cost_usd"] += a.get("cost_usd") or 0.0
if rec["ok"]:
break
finally:
shutil.rmtree(pristine, ignore_errors=True)
rec["cost_usd"] = round(rec["cost_usd"], 6)
rec["wall_s"] = round(sum(a["wall_s"] for a in rec["attempts"]), 1)
rec["models"] = sorted({m for a in rec["attempts"] for m in a.get("models") or []})
json.dump(rec, open(os.path.join(out_dir, "testwriter.json"), "w"), indent=2)
return rec
def run_gate_tests(gate_dir, task, tree, timeout=300):
"""Generated tests on `tree` (TINYDB_TREE = tree, cwd = tree). Returns (rc, out, passed, failed, failed_names).
Paths to the gate_tests dir are rewritten to `acceptance/` so the agent never learns where the file is."""
f = os.path.join(gate_dir, "test_%s.py" % task)
rc, out = sh([PY, "-m", "pytest", "-q", "--tb=short", "-rfE", "--no-header", "-c", os.path.join(gate_dir, "pytest.ini"),
"--rootdir", gate_dir, f], tree, timeout=timeout, env=env_clean({"TINYDB_TREE": tree}))
out = out.replace(os.path.realpath(gate_dir), "acceptance").replace(gate_dir, "acceptance")
rel = os.path.relpath(gate_dir, tree)
out = out.replace(rel, "acceptance")
import re
names = sorted(set(re.findall(r"(?:FAILED|ERROR) \S*::(test_\w+)", out)))
p, fl = counts(out)
if rc != 0 and fl == 0:
fl = max(1, len(names))
return rc, out, p, fl, names
if __name__ == "__main__":
r = write(sys.argv[1], os.path.abspath(sys.argv[2]))
print(json.dumps({k: r[k] for k in ("task", "ok", "cost_usd", "wall_s", "file")}))
lib/review.py
"""Gate 3 reviewers: headless `claude -p` calls returning schema-validated JSON.
CLI (validation step 4): review.py <work dir> <task id> [indep|adv] -> prints the gate3 record as JSON
"""
import json
import subprocess
import os
import sys
import time
from concurrent.futures import ThreadPoolExecutor
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from common import BENCH, env_clean, sh, work_diff # noqa: E402
MODEL = "claude-sonnet-5-5"
DIFF_CAP = 60000
CALL_TIMEOUT = 600 # seconds per reviewer call; the hook timeout is 1500
FINDING = {"type": "object",
"properties": {"severity": {"type": "string", "enum": ["blocking", "minor"]},
"file": {"type": "string"}, "line": {"type": "integer"}, "text": {"type": "string"}},
"required": ["severity", "file", "line", "text"]}
SCHEMA_REVIEW = {"type": "object",
"properties": {"approve": {"type": "boolean"}, "findings": {"type": "array", "items": FINDING}},
"required": ["approve", "findings"]}
SCHEMA_CHALLENGE = {"type": "object",
"properties": {
"verdicts": {"type": "array", "items": {
"type": "object",
"properties": {"id": {"type": "integer"},
"verdict": {"type": "string", "enum": ["upheld", "rejected"]},
"reason": {"type": "string"}},
"required": ["id", "verdict", "reason"]}},
"added_findings": {"type": "array", "items": FINDING},
"approve": {"type": "boolean"}},
"required": ["verdicts", "added_findings", "approve"]}
FRAMING = ("You are a senior engineer doing a pre-merge code review of a change to tinydb, a small Python "
"document database. You can read the repository with Read, Grep and Glob (cwd = the repo with the "
"change applied). You cannot run code. Below: the task the developer was given, then the full diff "
"(tracked and new files).")
ASK = ("Find defects: requirements not met, edge cases wrong, behaviour broken, tests that do not test the "
"requirement. Report only real defects with file and line. approve=true only when nothing blocks.")
def capped_diff(work):
d = work_diff(work)
if len(d) > DIFF_CAP:
d = d[:DIFF_CAP] + "\n[diff truncated at %d chars]\n" % DIFF_CAP
return d
def call(work, prompt, schema, rid):
"""One headless reviewer. Never raises: an error is recorded and counts as approve (fail open)."""
# Equivalent of --bare that still authenticates with the user's OAuth login (--bare reads
# ANTHROPIC_API_KEY only): no settings sources, hooks disabled, no MCP, no CLAUDE.md, no auto-memory.
args = ["claude", "-p", prompt, "--model", MODEL, "--tools", "Read,Grep,Glob", "--max-turns", "15",
"--output-format", "json", "--json-schema", json.dumps(schema), "--permission-mode", "dontAsk",
"--setting-sources", "", "--settings", '{"disableAllHooks":true}', "--strict-mcp-config"]
env = env_clean({"CLAUDE_CODE_DISABLE_AUTO_MEMORY": "1", "CLAUDE_CODE_DISABLE_CLAUDE_MDS": "1"})
t0 = time.time()
rec = {"id": rid, "approve": True, "findings": [], "cost_usd": 0.0, "turns": None, "wall_s": None, "error": None}
try:
try:
p = subprocess.run(args, cwd=work, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.DEVNULL,
timeout=CALL_TIMEOUT, env=env)
except OSError as e1: # seen once in a smoke run: transient FileNotFoundError on spawn -> one retry
rec["retry_after"] = "%r filename=%s" % (e1, getattr(e1, "filename", None))
time.sleep(3)
p = subprocess.run(args, cwd=work, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.DEVNULL,
timeout=CALL_TIMEOUT, env=env)
out = p.stdout.decode("utf-8", "replace")
try:
r = json.loads(out)
except Exception:
r = None
if not isinstance(r, dict):
rec["error"] = "no json (rc=%s): %s %s" % (p.returncode, out[-300:], p.stderr.decode("utf-8", "replace")[-300:])
else:
rec["cost_usd"] = r.get("total_cost_usd") or 0.0
rec["turns"] = r.get("num_turns")
rec["models"] = sorted((r.get("modelUsage") or {}).keys())
so = r.get("structured_output")
if r.get("is_error") or not isinstance(so, dict):
rec["error"] = "is_error=%s subtype=%s terminal=%s structured_output=%s" % (
r.get("is_error"), r.get("subtype"), r.get("terminal_reason"), type(so).__name__)
else:
rec["raw"] = so
except subprocess.TimeoutExpired:
rec["error"] = "timeout after %ss" % CALL_TIMEOUT
except Exception as e: # pragma: no cover
rec["error"] = "exception: %r filename=%s path_head=%s" % (e, getattr(e, "filename", None), os.environ.get("PATH", "")[:200])
rec["wall_s"] = round(time.time() - t0, 1)
return rec
def norm_findings(lst):
out = []
for f in lst or []:
if isinstance(f, dict):
out.append({"severity": f.get("severity") if f.get("severity") in ("blocking", "minor") else "minor",
"file": str(f.get("file", "")), "line": f.get("line"), "text": str(f.get("text", ""))})
return out
def review_prompt(task_prompt, diff):
return "%s\n\n## Task given to the developer\n%s\n\n## Diff (git diff HEAD, new files included)\n```diff\n%s\n```\n\n%s" % (
FRAMING, task_prompt.strip(), diff, ASK)
def challenge_prompt(task_prompt, diff, a_findings):
listing = "\n".join("[%d] (%s) %s:%s %s" % (i + 1, f["severity"], f["file"], f["line"], f["text"])
for i, f in enumerate(a_findings)) or "(reviewer A reported no findings)"
return ("%s\n\nA first reviewer (A) already reviewed this diff. You are the challenger: for EACH of A's findings "
"decide whether it is a real defect (upheld) or not (rejected: wrong, not required by the task, already "
"handled, or speculative), with a one-sentence reason grounded in the code; check the code before you "
"decide. Then add any real blocking defect A missed in added_findings (file and line). approve=true only "
"when no upheld or added finding blocks.\n\n## Task given to the developer\n%s\n\n## Diff (git diff HEAD, "
"new files included)\n```diff\n%s\n```\n\n## Reviewer A's findings (id in brackets)\n%s\n\n%s" % (
FRAMING, task_prompt.strip(), diff, listing,
"Answer every id in verdicts. Report only real defects."))
def gate3(work, task_prompt, variant):
diff = capped_diff(work)
reviewers, blocking = [], []
if variant == "indep":
prompt = review_prompt(task_prompt, diff)
with ThreadPoolExecutor(max_workers=2) as ex:
recs = list(ex.map(lambda rid: call(work, prompt, SCHEMA_REVIEW, rid), ["A", "B"]))
for rec in recs:
raw = rec.pop("raw", None) or {}
rec["findings"] = norm_findings(raw.get("findings"))
rec["approve_raw"] = raw.get("approve", True) if not rec["error"] else None
rb = [f for f in rec["findings"] if f["severity"] == "blocking"]
rec["approve"] = not rb # effective verdict = no blocking finding (an errored call approves)
blocking += [dict(f, by=rec["id"]) for f in rb]
reviewers.append(rec)
elif variant == "adv":
a = call(work, review_prompt(task_prompt, diff), SCHEMA_REVIEW, "A")
raw = a.pop("raw", None) or {}
a["findings"] = norm_findings(raw.get("findings"))
a["approve_raw"] = raw.get("approve", True) if not a["error"] else None
a["approve"] = not [f for f in a["findings"] if f["severity"] == "blocking"]
b = call(work, challenge_prompt(task_prompt, diff, a["findings"]), SCHEMA_CHALLENGE, "B")
rawb = b.pop("raw", None) or {}
verdicts = {}
for v in rawb.get("verdicts") or []:
if isinstance(v, dict) and isinstance(v.get("id"), int):
verdicts[v["id"]] = {"verdict": v.get("verdict"), "reason": str(v.get("reason", ""))}
b["verdicts"] = [dict(id=k, **verdicts[k]) for k in sorted(verdicts)]
b["findings"] = norm_findings(rawb.get("added_findings"))
b["approve_raw"] = rawb.get("approve", True) if not b["error"] else None
# Only upheld findings of A block. An errored challenger upholds nothing (fail open, logged).
for i, f in enumerate(a["findings"]):
v = verdicts.get(i + 1)
f["challenge"] = v["verdict"] if v else ("error" if b["error"] else "unanswered")
if f["severity"] == "blocking" and v and v["verdict"] == "upheld":
blocking.append(dict(f, by="A", upheld_by="B"))
blocking += [dict(f, by="B") for f in b["findings"] if f["severity"] == "blocking"]
b["approve"] = not blocking
reviewers = [a, b]
else:
raise ValueError(variant)
return {"ran": True, "variant": variant, "reviewers": reviewers, "blocking_findings": len(blocking),
"blocking": blocking, "pass": not blocking,
"cost_usd": round(sum(r.get("cost_usd") or 0 for r in reviewers), 6), "diff_chars": len(diff)}
def reason_text(g3):
lines = ["Code review found %d blocking defect(s):" % g3["blocking_findings"]]
for i, f in enumerate(g3["blocking"]):
lines.append("%d. %s:%s %s" % (i + 1, f["file"], f["line"], " ".join(f["text"].split())))
tail = "Fix every finding, re-run the tests, then stop."
body = "\n".join(lines)
room = 1500 - len(tail) - 2
if len(body) > room:
body = body[:room - 4] + " ..."
return body + "\n" + tail
if __name__ == "__main__":
work, task = sys.argv[1], sys.argv[2]
variant = sys.argv[3] if len(sys.argv) > 3 else "indep"
tp = open(os.path.join(BENCH, "tasks", task + ".md")).read()
g3 = gate3(os.path.abspath(work), tp, variant)
print(json.dumps(g3, indent=2))
if not g3["pass"]:
print("--- reason ---\n" + reason_text(g3))
lib/common.py
"""Shared helpers: work-copy diff, pytest runs and parsing. Used by hook.py, review.py, finalize.py."""
import os
import re
import subprocess
BENCH = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
PY = os.path.join(BENCH, ".venv", "bin", "python")
HIDDEN = os.path.join(BENCH, "hidden")
def sh(args, cwd, timeout=None, env=None):
"""Run a command, capture stdout+stderr together. Returns (rc, text). rc=124 on timeout."""
try:
p = subprocess.run(args, cwd=cwd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
stdin=subprocess.DEVNULL, timeout=timeout, env=env)
return p.returncode, p.stdout.decode("utf-8", "replace")
except subprocess.TimeoutExpired as e:
out = (e.stdout or b"").decode("utf-8", "replace")
return 124, out + "\n[timed out after %ss]" % timeout
def work_diff(work):
"""`git diff HEAD` of tracked files plus every untracked (non-ignored) file as a new-file diff.
Never touches the index (no `git add -N`), so the agent's `git status` is unaffected."""
_, d = sh(["git", "diff", "HEAD", "--no-color", "--", ".", ":(exclude).coverage"], work)
_, others = sh(["git", "ls-files", "--others", "--exclude-standard"], work)
for f in [l for l in others.splitlines() if l.strip()]:
_, nd = sh(["git", "diff", "--no-color", "--no-index", "--", "/dev/null", f], work)
d += nd
return d
def tests_touched(work):
_, mod = sh(["git", "diff", "HEAD", "--name-only", "--", "tests"], work)
_, new = sh(["git", "ls-files", "--others", "--exclude-standard", "--", "tests"], work)
return bool(mod.strip() or new.strip())
def counts(txt):
"""(passed, failed+errors) from a pytest summary line."""
p = re.findall(r"(\d+) passed", txt)
f = re.findall(r"(\d+) failed", txt)
e = re.findall(r"(\d+) errors?\b", txt)
return (int(p[-1]) if p else 0), (int(f[-1]) if f else 0) + (int(e[-1]) if e else 0)
def env_clean(extra=None):
env = {k: v for k, v in os.environ.items() if not k.startswith("CLAUDE") and not k.startswith("GATE_")}
env["PATH"] = os.path.join(BENCH, ".venv", "bin") + os.pathsep + env.get("PATH", "/usr/bin:/bin")
env["PYTHONDONTWRITEBYTECODE"] = "1"
env.update(extra or {})
return env
def run_suite(work, timeout=300):
"""Gate 1 / final suite: the repo's own pytest run (226 tests + whatever the agent wrote).
--no-cov: the repo's pytest.ini adds --cov; the coverage table would fill the 40-line tail."""
rc, out = sh([PY, "-m", "pytest", "-q", "-x", "--no-header", "-p", "no:cacheprovider", "--no-cov"],
work, timeout=timeout, env=env_clean())
p, f = counts(out)
if rc != 0 and f == 0:
f = 1 # collection error, timeout, crash: count as red
return rc, out, p, f
def run_hidden(work, task, timeout=300):
"""Ground truth: hidden tests on the work copy (cwd = work copy, TINYDB_TREE set). Output never reaches the agent."""
rc, out = sh([PY, "-m", "pytest", "-q", "-rA", "-c", os.path.join(HIDDEN, "pytest.ini"), "--rootdir", HIDDEN,
os.path.join(HIDDEN, "test_%s.py" % task)],
work, timeout=timeout, env=env_clean({"TINYDB_TREE": work}))
# Requirement level (T5 R4 is parametrized: one req = several test ids). A req passes iff none of its
# test ids failed or errored; a run that collected nothing counts every req as failed.
reqs = sorted(set(re.findall(r"^def (test_req\d+)", open(os.path.join(HIDDEN, "test_%s.py" % task)).read(), re.M)))
failed = sorted(set(re.findall(r"(?:FAILED|ERROR) \S+::(test_req\d+)", out)))
p_tests, f_tests = counts(out)
if rc != 0 and not failed:
failed = list(reqs)
if p_tests == 0 and f_tests == 0:
failed = list(reqs)
return rc, out, len(reqs) - len(failed), len(failed), failed
lib/finalize.py
"""After the worker exits: final diff, suite, hidden run, result.json and score.json. Prints one line.
Usage: finalize.py <run dir> (re-runnable: rescoring an existing run only reads/rewrites files in it)"""
import json
import os
import re
import sys
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from common import PY, env_clean, run_hidden, sh, work_diff # noqa: E402
from testwriter import run_gate_tests # noqa: E402
MODEL = "claude-sonnet-5-5"
out = os.path.abspath(sys.argv[1])
meta = json.load(open(os.path.join(out, "meta.json")))
work, task, cfg, rep = meta["work"], meta["task"], meta["cfg"], meta["rep"]
def w(name, txt):
open(os.path.join(out, name), "w").write(txt)
def rd(name, default=""):
try:
return open(os.path.join(out, name)).read().strip()
except Exception:
return default
# Final state of the work copy
w("post.diff", work_diff(work))
_, ns = sh(["git", "diff", "HEAD", "--numstat", "--", ".", ":(exclude).coverage"], work)
_, others = sh(["git", "ls-files", "--others", "--exclude-standard"], work)
for f in [l for l in others.splitlines() if l.strip()]:
try:
ns += "%d\t0\t%s\n" % (sum(1 for _ in open(os.path.join(work, f), errors="replace")), f)
except Exception:
pass
w("numstat.txt", ns)
rc, suite = sh([PY, "-m", "pytest", "-q", "--no-header", "-p", "no:cacheprovider", "--no-cov"], work,
timeout=300, env=env_clean())
w("suite.txt", suite)
sp = int((re.findall(r"(\d+) passed", suite) or [0])[-1])
sf = int((re.findall(r"(\d+) failed", suite) or [0])[-1]) + int((re.findall(r"(\d+) errors?\b", suite) or [0])[-1])
if rc != 0 and sf == 0:
sf = 1
_, hout, hp, hf, hfailed = run_hidden(work, task)
w("hidden.txt", hout)
# Worker result line
lines = []
for raw in open(os.path.join(out, "stream.jsonl"), errors="replace") if os.path.exists(os.path.join(out, "stream.jsonl")) else []:
raw = raw.strip()
if raw.startswith("{"):
try:
lines.append(json.loads(raw))
except Exception:
pass
result = next((o for o in reversed(lines) if o.get("type") == "result"), None) or {}
init = next((o for o in lines if o.get("type") == "system" and o.get("subtype") == "init"), {})
res = {k: result.get(k) for k in ("total_cost_usd", "num_turns", "usage", "modelUsage", "is_error", "subtype",
"stop_reason", "terminal_reason", "duration_ms")}
res["final_text"] = (result.get("result") or "")[-3000:]
res["has_result_line"] = bool(result)
json.dump(res, open(os.path.join(out, "result.json"), "w"), indent=2)
# Hook events seen in the stream (which hooks ran)
hook_events = sorted({"%s:%s" % (o.get("hook_event") or o.get("hook_event_name") or "?", o.get("hook_name") or "")
for o in lines if o.get("type") == "system" and str(o.get("subtype", "")).startswith("hook")})
# Gate log
gl = [json.loads(l) for l in open(os.path.join(out, "gate.jsonl")) if l.strip()] if os.path.exists(os.path.join(out, "gate.jsonl")) else []
blocks = [g for g in gl if g.get("decision") == "block"]
by_gate = {"g1": sum(1 for g in blocks if g.get("blocked_by") == "g1"),
"g3": sum(1 for g in blocks if g.get("blocked_by") == "g3")}
right = {k: sum(1 for g in blocks if g.get("blocked_by") == k and g["hidden"]["fail"] > 0) for k in ("g1", "g3")}
false = {k: sum(1 for g in blocks if g.get("blocked_by") == k and g["hidden"]["fail"] == 0) for k in ("g1", "g3")}
rev_cost, rev_findings, rev_errors, rev_models = 0.0, 0, 0, set()
for g in gl:
for r in (g.get("gate3") or {}).get("reviewers") or []:
rev_cost += r.get("cost_usd") or 0
rev_findings += len(r.get("findings") or [])
rev_errors += 1 if r.get("error") else 0
rev_models |= set(r.get("models") or [])
# Gate-1 test writer (gated-1 configs only): cost and the final state of its tests on the work copy
tw = {}
try:
tw = json.load(open(os.path.join(out, "testwriter.json")))
except Exception:
pass
cost_tw = tw.get("cost_usd") or 0.0
rev_models |= set(tw.get("models") or [])
gt_total = gt_fail = None
gdir = os.path.join(out, "gate_tests")
if os.path.exists(os.path.join(gdir, "test_%s.py" % task)):
_, gout, gp, gf, _ = run_gate_tests(gdir, task, work)
w("gatetests-final.txt", gout)
gt_total, gt_fail = gp + gf, gf
mu = result.get("modelUsage") or {}
cw = result.get("total_cost_usd") or 0.0
s = {
"task": task, "cfg": cfg, "rep": rep, "run": meta["name"],
"hidden_pass": hp, "hidden_fail": hf, "failed_reqs": hfailed,
"suite_pass": sp, "suite_fail": sf,
"hook_calls": len(gl), "blocks_total": len(blocks), "blocks_by_gate": by_gate,
"right_blocks": sum(right.values()), "false_blocks": sum(false.values()),
"right_blocks_by_gate": right, "false_blocks_by_gate": false,
"give_up": any(g.get("decision") == "give_up" for g in gl),
"ended_blocked": bool(gl) and gl[-1].get("decision") == "block",
"cost_worker_usd": round(cw, 6), "cost_reviewers_usd": round(rev_cost, 6), "cost_testwriter_usd": round(cost_tw, 6),
"cost_total_usd": round(cw + rev_cost + cost_tw, 6),
"testwriter_ok": tw.get("ok") if tw else None, "gate_tests_total": gt_total, "gate_tests_fail_final": gt_fail,
"turns": result.get("num_turns"), "wall_s": int(rd("wall_s.txt", "0") or 0),
"reviewer_findings_total": rev_findings, "reviewer_errors": rev_errors,
"gate_crashes": rd("gate-errors.txt").count("Traceback"),
"model_ok": init.get("model") == MODEL and bool(mu) and set(mu) <= {MODEL} and rev_models <= {MODEL},
"worker_models": sorted(mu), "reviewer_models": sorted(rev_models),
"is_error": result.get("is_error"), "subtype": result.get("subtype"), "exit_code": rd("exit_code.txt"),
"has_result_line": bool(result), "claude_code_version": init.get("claude_code_version"),
"hook_events_in_stream": hook_events,
"hidden_first_stop": gl[0]["hidden"] if gl else None,
}
json.dump(s, open(os.path.join(out, "score.json"), "w"), indent=2)
print("%s: hidden %d/%d suite %d/%d blocks=%d (g1 %d, g3 %d; right %d, false %d) give_up=%s ended_blocked=%s "
"gate_tests_fail=%s/%s cost=$%.3f (worker %.3f + rev %.3f + writer %.3f) turns=%s wall=%ss model_ok=%s" % (
s["run"], hp, hp + hf, sp, sp + sf, len(blocks), by_gate["g1"], by_gate["g3"], s["right_blocks"],
s["false_blocks"], s["give_up"], s["ended_blocked"], gt_fail, gt_total, s["cost_total_usd"], cw, rev_cost, cost_tw, s["turns"],
s["wall_s"], s["model_ok"]))
.claude ayarları HARNESS.md içinde anlatılıyor: tek bir Stop command hook'u (timeout 1500) olan ve tek satırlık bir stop.sh'yi çağıran bir settings.json, artı hook'un, ayarların ve gizli testlerin okunmasını engelleyen bir permissions.deny listesi. Reviewer'lar ve writer boş bir settings kaynağı ve kapalı hook'larla çalışıyor, böylece kapı kendini asla özyinelemeli çağırmıyor.
HARNESS.md
# gate-bench
What does each of Shopify's Helix gates catch when rebuilt as a Claude Code **Stop hook**, and what does it
cost? Headless Sonnet 5.5 workers implement five tinydb features (T1..T5); a Stop hook runs gate 1
(behaviour tests) and/or gate 3 (adversarial review by headless Sonnet 5.5 reviewers) every time the worker
tries to stop; hidden tests (the oracle, never visible to the agent) record the ground truth at every stop
and at the end. Built 2026-10-08 from `SPEC.md`; upstream, hidden tests, tasks and reference patches copied
from `../verify-loops-bench` (nothing there or in `../haiku-routing-bench` is modified).
## Commands (from this dir)
```bash
./validate.sh # steps 1,3,4 (+ step 2 once its run exists) -> validation.log
INJECT=$PWD/inject/break.patch RUN_TAG=val2 ./run.sh g1 T1 1 # validation step 2 (red start state)
RUN_TAG=smoke ./run.sh g13adv T1 1 # smoke (tagged runs are ignored by analyze.py)
./validate_writer.sh # gate-1 test writer on T1..T5: tests on upstream vs reference (REWRITE=1 to redo)
./gen_matrix.py # matrix.txt (50 runs, already generated)
nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 & # 3 runs in parallel (MATRIX_JOBS=3)
SKIP_DONE=1 nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 & # resume
.venv/bin/python analyze.py # results.json + results.md
.venv/bin/python lib/finalize.py runs/<run> # rescore one run from its work copy
.venv/bin/python lib/review.py work/<run> T1 adv # one gate-3 round by hand
```
Single run: `./run.sh <cfg> <task> <rep>` -> `runs/<task>-<cfg>-r<rep>/`, work copy `work/<same name>/`.
## Layout
| path | what |
|---|---|
| `SPEC.md` | the spec this harness implements |
| `upstream/` | tinydb @ 18d73a1 (git repo) |
| `hidden/` | `test_T1..T6.py` (T6 unused), `conftest.py` (`$TINYDB_TREE` or cwd first on sys.path), `pytest.ini` |
| `tasks/T1..T5.md` + `tasks/README.md` | the task prompts (verbatim) and what each hidden req checks |
| `reference/T1..T5.patch` | reference solutions (validation step 1) |
| `inject/break.patch` | `>=` -> `>` in `Query.__ge__` (one repo test red; validation steps 2-3) |
| `inject/test_pushpull.py` | one thin test, makes the T1 reference copy "green with tests touched" (step 3) |
| `configs.json` | `bare` (no hook), `g1`, `g3` (indep), `g13` (g1 then g3 indep), `g13adv` (g1 then g3 adv) |
| `env.sh` | scrubs inherited `CLAUDE*` vars, default `~/.claude` auth, `CLAUDE_CODE_DISABLE_AUTO_MEMORY=1`, venv on PATH |
| `prep.sh <cfg> <work> <run>` | fresh copy of upstream, `.claude/` in `.git/info/exclude`, hook settings for gated configs |
| `run.sh <cfg> <task> <rep>` | one run (worker + hook + `lib/finalize.py`) |
| `gate.sh` | the Stop hook (bash 3.2 wrapper: `exec .venv/bin/python -I lib/hook.py`) |
| `lib/hook.py` | gate dispatch, snapshot, hidden run, gate 1, gate 3, decision, `gate.jsonl` line |
| `lib/review.py` | gate-3 reviewers (indep / adv), schemas, prompts, reason text; CLI for a manual round |
| `lib/testwriter.py` | gate-1 test writer (before the worker) + `run_gate_tests` (path rewritten to `acceptance/`) |
| `validate_writer.sh`, `validation/writer/<T>/` | writer validation files and their runs on upstream / reference |
| `tasks/` (terse prompts, used), `tasks-explicit/` + `runs-explicit/` | the matrix uses terse prompts; see `tasks/README.md` |
| `lib/common.py` | diff incl. untracked files, suite run, hidden run (requirement level) |
| `lib/finalize.py` | post-run diff/suite/hidden, `result.json`, `score.json` |
| `gen_matrix.py`, `matrix.txt`, `matrix.sh`, `matrix-one.sh`, `matrix.log` | 50-run matrix, 3 in parallel (xargs -P), resumable |
| `analyze.py` -> `results.json`, `results.md` | per config, per task, the five questions |
| `validate.sh`, `VALIDATION.md`, `validation.log` | pre-matrix checks |
| `.venv/` | python 3.14.7, pytest 9.1.1, pytest-cov 7.1.0, pyyaml, ruff 0.16.10 (same pins as haiku-routing-bench) |
### Worker (run.sh)
`claude -p "<T-file verbatim>\n\nWhen you are done, stop and summarise what you changed." --model claude-sonnet-5-5
--output-format stream-json --verbose --include-hook-events --max-turns 60 --permission-mode acceptEdits
--allowedTools <haiku-routing-bench list> --setting-sources project --strict-mcp-config`, cwd = `work/<run>/`, stdin /dev/null.
Gated configs get `work/<run>/.claude/settings.json`:
`{"permissions":{"deny":[...see traps...]},"hooks":{"Stop":[{"hooks":[{"type":"command","command":"bash \"$CLAUDE_PROJECT_DIR/.claude/hooks/stop.sh\"","timeout":1500}]}]}}`
and `.claude/hooks/stop.sh` = `GATE_RUN=<abs run dir> GATE_CONFIG=<cfg> bash <bench>/gate.sh` (the SPEC command, one level down: see traps).
Copies of both land in the run dir (`settings.json`, `stop.sh`).
### Hook (lib/hook.py), one invocation
1. read stdin JSON; n = lines already in `gate.jsonl` + 1; blocks so far from the same file
2. `snap-<n>.diff` = `git diff HEAD` + untracked files (index never touched)
3. hidden tests on the work copy (`TINYDB_TREE` = work copy, cwd = work copy) -> `hidden-<n>.txt`; never shown to the agent
4. gate 1 (if configured) = the repo suite `python -m pytest -q -x --no-header -p no:cacheprovider --no-cov` (300 s)
-> `gate1-<n>.txt` AND the writer's acceptance tests `runs/<run>/gate_tests/test_<T>.py` run with `TINYDB_TREE` = work
copy -> `gatetests-<n>.txt`. Block when either is red. Reason: "The acceptance tests for this change fail:" + last 40
lines (failing test names + assertion output, the gate_tests path rewritten to `acceptance/`), then the suite tail if the
suite is red too; <= 1500 chars. `tests_touched` is still logged but no longer blocks. No writer file (writer failed
twice) -> suite only, logged as `gate_tests_present: false`.
5. gate 3 (if configured and gate 1 passed or absent): see below
6. decision: allow / block / give_up (a would-be block when 6 blocks already happened -> allow, logged `give_up`).
`stop_hook_active` is logged, never used. Block = exit 2 + reason on stderr. A crash of the hook itself is
logged to `gate-errors.txt` and allows (counted as `gate_crashes` in score.json).
### Gate 1 test writer (lib/testwriter.py, run.sh before the worker, configs g1/g13/g13adv)
`claude -p` Sonnet 5.5 with the reviewer isolation flags, `--tools Read,Grep,Glob --max-turns 12 --json-schema
{"file": string}`, cwd = a pristine temp copy of upstream (deleted after). Prompt = framing + `tasks/<T>.md` + the
coordinator's instruction ("Write integration-style pytest cases for this request against the tinydb API, one test per
behaviour a careful senior would expect, pushing for edge cases beyond the happy path (missing fields, wrong types,
boundaries, persistence, cache, existing behaviour unchanged). Tests only, no implementation. Return the file content.")
+ mechanics (self-contained module, imports of the new API inside each test, MemoryStorage/tmp_path). The file must
compile and contain `def test_`, else one retry. Output `runs/<run>/gate_tests/` (+ hidden's conftest.py with
`TINYDB_TREE`, pytest.ini) and `testwriter.json` (cost, turns, attempts). Fixed per-run cost, outside the worker's wall.
### Gate 3 (lib/review.py)
Each reviewer: `claude -p "<prompt>" --model claude-sonnet-5-5 --tools Read,Grep,Glob --max-turns 15 --output-format json
--json-schema <schema> --permission-mode dontAsk --setting-sources "" --settings '{"disableAllHooks":true}' --strict-mcp-config`
with `CLAUDE_CODE_DISABLE_CLAUDE_MDS=1 CLAUDE_CODE_DISABLE_AUTO_MEMORY=1`, all inherited `CLAUDE*` vars dropped, cwd = work copy,
600 s timeout. Prompt = framing + task prompt (T-file) + diff (tracked + untracked, cap 60k chars) + the SPEC's "Find defects..."
sentence. `structured_output` is read from the JSON result; `total_cost_usd`, `num_turns`, wall and models are recorded.
- `indep`: reviewers A and B, same prompt, run in parallel, no shared context. Blocking findings = union. Pass when both approve.
- `adv`: A as above; challenger B gets A's numbered findings + the diff, answers `upheld|rejected` + reason per id, may add
findings (`added_findings`). Blocking = A's blocking findings B upheld + B's added blocking findings. B always runs (same
2-call budget as indep).
- Effective `approve` = "no blocking finding" (the raw `approve` is logged as `approve_raw`; a reviewer saying approve=false with
only minor findings does not block). A reviewer call that errors approves (fail open) with `error` recorded; an errored
challenger upholds nothing.
- Reason = "Code review found N blocking defect(s):" + numbered `file:line text` + "Fix every finding, re-run the tests, then stop.", <= 1500 chars.
### Per run (`runs/<task>-<cfg>-r<rep>/`)
`stream.jsonl`, `stderr.txt`, `exit_code.txt`, `wall_s.txt` (includes hook time: the hook runs inside the session),
`gate.jsonl`, `snap-<n>.diff`, `hidden-<n>.txt`, `gate1-<n>.txt`, `hook-input-1.json`, `post.diff`, `numstat.txt`,
`suite.txt` (full suite, no -x), `hidden.txt`, `gate_tests/`, `testwriter.json`, `gatetests-<n>.txt`, `gatetests-final.txt`, `result.json`, `score.json`, `meta.json`, `task.md`, `prompt.txt`,
`settings.json` + `stop.sh` (gated configs).
`score.json`: task, cfg, rep, hidden_pass/hidden_fail/failed_reqs (**requirement level**: 5 reqs per task), suite_pass/fail,
hook_calls, blocks_total, blocks_by_gate {g1,g3}, right_blocks / false_blocks (+ `_by_gate`), give_up, ended_blocked,
cost_worker_usd, cost_reviewers_usd, cost_testwriter_usd, cost_total_usd (all three), testwriter_ok, gate_tests_total,
gate_tests_fail_final, turns, wall_s, reviewer_findings_total, reviewer_errors, gate_crashes,
model_ok (worker init model + every modelUsage key + every reviewer model == claude-sonnet-5-5), hook_events_in_stream,
hidden_first_stop.
## Matrix
5 tasks x 5 configs x 2 reps = 50 runs. Rep 1 config blocks bare, g1, g3, g13, g13adv (tasks rotated per block), rep 2 the
reverse block order. Up to 3 runs at once, each in its own work copy (`matrix-one.sh` via `xargs -P 3`, bash 3.2 has no
`wait -n`). `matrix.log`: one `start` and one result line per run.
## Traps found
- **Gate 1 as first built passed by construction** (it only re-ran the implementer's own tests: `T2-g1-r1` one allow, hidden
R4 missed). Redesigned around an independent test writer (above). The writer hedges where the terse prompt is silent
(accepts "raises or no-op"), so it cannot recover unstated requirements; per-task overlap is in VALIDATION.md.
- **A blocked agent hunts for the tests.** After a gate-1 block the smoke agent ran `cat .claude/hooks/stop.sh; find / -name
test_T1.py` (denied only because `find` is not on the allow-list; `cat` is, and `stop.sh` names the bench dir, from where
`hidden/` is one `cat` away). Gated configs' `.claude/settings.json` now carries `permissions.deny`: Read of `./.claude/**`,
the bench's `hidden/ runs/ validation/ reference/`, the two sibling benches; Bash patterns containing `.claude/hooks`,
`.claude/settings`, `../hidden`, `../runs`, `gate_tests`, the bench sub-paths, `find *`. Probed with a Haiku session: all
five escape attempts (cat, Read, ls, python open) denied, in-tree reads fine. Bash rules match command text, not a security
boundary. Bare runs have no settings file (bare r1 ran before this change; bare has no motive to hunt).
Reviewers and the writer run with `--setting-sources ""`, so these rules do not apply to them.
- **Transient `FileNotFoundError` spawning a reviewer** inside the hook (once, both reviewers of one call, wall 0.0 s;
never reproduced by hand). Reviewer calls now retry once after 3 s; the error message records `filename`.
- **`--bare` cannot authenticate here.** It reads `ANTHROPIC_API_KEY` / `apiKeyHelper` only, never OAuth or the keychain;
this account is OAuth (`~/.claude`), so a `--bare` reviewer returns `terminal_reason: api_error`, cost 0. Used the
documented equivalents instead: `--setting-sources ""` + `--settings '{"disableAllHooks":true}'` + `--strict-mcp-config` +
`CLAUDE_CODE_DISABLE_CLAUDE_MDS=1` + `CLAUDE_CODE_DISABLE_AUTO_MEMORY=1` + `--tools Read,Grep,Glob`. The reviewers' cwd is the
work copy, whose `.claude/settings.json` carries the Stop hook: without `--setting-sources ""`/`disableAllHooks` a reviewer
would run the gate recursively. Verified: a dummy Stop hook in the probe copy never fired.
- **Inherited session env.** A matrix launched from inside Claude Code inherits `CLAUDE_EFFORT=medium`,
`CLAUDE_CODE_SUBAGENT_MODEL=opus`, `CLAUDE_CODE_SESSION_ID`, `CLAUDECODE`, `CLAUDE_CODE_CHILD_SESSION`, messaging tokens.
`env.sh` unsets every `CLAUDE*` var before setting its own; reviewers get a scrubbed env too (the hook process itself
inherits the worker's `CLAUDE_*` vars, e.g. the worker's session id).
- **The feedback line leaks the hook command.** Claude Code prefixes each block with the command line:
`Stop hook feedback:\n[<command>]: <reason>`. With the SPEC command inline the agent would read `GATE_CONFIG=g13adv` and
`gate-bench/gate.sh`. The settings command is therefore `bash "$CLAUDE_PROJECT_DIR/.claude/hooks/stop.sh"`, a one-line
wrapper holding the SPEC command. The agent can still `cat` it (it lives in the work copy) but it is not pushed at it.
- **Stop hook input keys (2.1.293):** `background_tasks, cwd, effort ({"level":"medium"}), hook_event_name,
last_assistant_message, permission_mode, prompt_id, session_crons, session_id, stop_hook_active, transcript_path`.
- **8-cap resets on tool calls.** Per the docs the consecutive-continuation count resets each time Claude calls a tool, so the
built-in cap only bites an agent that answers blocks with text alone. Our give-up (6 blocks per run, counted from
`gate.jsonl`, tool calls or not) fires first in every case.
- **Red start state: the agent refuses to fix code it did not write.** Validation step 2 (T1 + an uncommitted `>=` -> `>`
edit) shows Sonnet 5.5 implementing T1, then answering 6 consecutive gate-1 blocks with "the red test is your pre-existing
uncommitted edit, not my change" without touching it, until give_up. The hook mechanics are validated; the behaviour is a
finding in itself (gate 1 cannot force a fix the agent considers out of scope).
- **pytest-cov.** The repo's `pytest.ini` adds `--cov tinydb`; gate 1 adds `--no-cov` so the 40-line tail holds the failure,
not the coverage table (and no `.coverage` file is written). `.coverage` is gitignored anyway.
- **T5 R4 is parametrized** (several test ids, one requirement). hidden counts are per requirement (5 per task), a req
passes only when all its ids pass. `validate.sh` step 1: T5 upstream passes 1 req (R3, the regression guard) as designed.
- **Diff without touching the index.** `git add -N` would change what the agent sees in `git status`; untracked files are
diffed with `git diff --no-index /dev/null <f>` instead. `.claude/` is in `.git/info/exclude`, so the hook config never
shows in any diff or in `git status`.
- **Worker Bash denials.** The 2.1.293 permission layer denies some agent commands with "Contains brace with quote character
(expansion obfuscation)" even under acceptEdits + allow-list; agents route around it (visible as `permission_denied` events).
- **Reviewers on a correct solution still flag "blocking".** On the T1 reference patch, reviewer A marked "pull does not check
the field is a list" (not required by the task) as blocking; indep therefore blocks, the adv challenger rejected it. This is
exactly what false_blocks measures (hidden all green at that stop).
- **Shared plan meter.** Workers and reviewers bill the user's plan quota; `total_cost_usd` is the list-price equivalent the
CLI reports, not a bill. Nothing else heavy should run on the account during the matrix.
- **rtk rewrites interactive `grep`/`diff`** in the launching Claude session only; scripts here call binaries directly
(`/usr/bin/diff`/`cmp` for verification diffs).
- **CLI version:** `claude --version` = 2.1.293 (Claude Code); flags checked in `claude --help`: `--json-schema`, `--bare`, `--tools`,
`--permission-mode` (choices acceptEdits, auto, bypassPermissions, manual, dontAsk, plan), `--setting-sources`,
`--include-hook-events`. `--max-turns` is accepted though not listed in `--help`.
## Matrix launch (2026-10-08 16:33:54, after the gate-1 redesign)
`cd /Users/thomas/dev/gate-bench && SKIP_DONE=1 nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 &` -> PID 71317, 3 lanes.
The 5 bare r1 runs (terse prompts) are kept and skipped; 45 runs to go. Expected wall ~40-70 min: gated runs so far take
~30-70 s of worker wall + ~35-40 s of test writer (before the worker) + 1-35 s per hook call, more with blocks.
Then `.venv/bin/python analyze.py`.
Çalıştırıcı:
run.sh: tek bir run, worker artı hook artı skorlama.run.sh
#!/usr/bin/env bash
# One run: fresh work copy, headless Sonnet 5.5 worker, Stop hook per config, final scoring.
# Usage: run.sh <cfg> <task> <rep> cfg: bare|g1|g3|g13|g13adv task: T1..T5
# RUN_TAG=x -> runs/<task>-<cfg>-r<rep>-x (smoke runs stay out of the matrix)
# INJECT=<patch> -> git apply it to the work copy before the worker starts (validation: red start state)
set -uo pipefail
CFG="$1"; TASK="$2"; REP="${3:-1}"
HERE="$(cd "$(dirname "$0")" && pwd)"
. "$HERE/env.sh"
PY="$HERE/.venv/bin/python"
MAX_TURNS="${BENCH_MAX_TURNS:-60}"
NAME="${TASK}-${CFG}-r${REP}${RUN_TAG:+-$RUN_TAG}"
OUT="$HERE/runs/$NAME"; W="$HERE/work/$NAME"
rm -rf "$OUT"; mkdir -p "$OUT"
"$HERE/prep.sh" "$CFG" "$W" "$OUT" > "$OUT/prep.txt" 2>&1 || { echo "$NAME: prep failed"; cat "$OUT/prep.txt"; exit 2; }
if [ -n "${INJECT:-}" ]; then ( cd "$W" && git apply "$INJECT" ) || { echo "$NAME: inject failed"; exit 2; }; fi
cp "$HERE/tasks/$TASK.md" "$OUT/task.md"
printf '%s\n\nWhen you are done, stop and summarise what you changed.\n' "$(cat "$HERE/tasks/$TASK.md")" > "$OUT/prompt.txt"
printf '{"task":"%s","cfg":"%s","rep":%s,"work":"%s","name":"%s"}\n' "$TASK" "$CFG" "$REP" "$W" "$NAME" > "$OUT/meta.json"
# Gate 1 configs: the independent test writer runs BEFORE the worker (fixed per-run cost), sees only tasks/<T>.md,
# writes $OUT/gate_tests/test_<T>.py outside the work copy.
if "$PY" -c "import json,sys; sys.exit(0 if json.load(open('$HERE/configs.json'))['$CFG']['gate1'] else 1)"; then
"$PY" "$HERE/lib/testwriter.py" "$TASK" "$OUT" > "$OUT/testwriter.out" 2>&1
fi
[ -f "$W/.claude/settings.json" ] && cp "$W/.claude/settings.json" "$OUT/settings.json" && cp "$W/.claude/hooks/stop.sh" "$OUT/stop.sh"
# Same allow-list as haiku-routing-bench/run.sh (verify-loops-bench's + Bash(ruff:*)).
TOOLS="Read,Edit,Write,MultiEdit,Glob,Grep,Skill,TodoWrite,Bash(uv:*),Bash(python:*),Bash(python3:*),Bash(pytest:*),Bash(ruff:*),Bash(git status:*),Bash(git diff:*),Bash(git log:*),Bash(ls:*),Bash(cat:*),Bash(mkdir:*),Bash(mktemp:*),Bash(head:*),Bash(tail:*),Bash(grep:*),Bash(wc:*)"
cd "$W"
START=$(date +%s)
claude -p "$(cat "$OUT/prompt.txt")" --model claude-sonnet-5-5 \
--output-format stream-json --verbose --include-hook-events --max-turns "$MAX_TURNS" \
--permission-mode acceptEdits --allowedTools "$TOOLS" \
--setting-sources project --strict-mcp-config \
< /dev/null > "$OUT/stream.jsonl" 2> "$OUT/stderr.txt"
echo "$?" > "$OUT/exit_code.txt"
END=$(date +%s)
echo "$((END-START))" > "$OUT/wall_s.txt"
"$PY" "$HERE/lib/finalize.py" "$OUT"
prep.sh
#!/usr/bin/env bash
# Fresh work copy of upstream tinydb (git clean at HEAD) + the Stop hook settings for gated configs.
# Usage: prep.sh <cfg> <work dir> <run dir>
set -euo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"
CFG="$1"; W="$2"; RUN="$3"
rm -rf "$W"; mkdir -p "$(dirname "$W")"
cp -R "$HERE/upstream" "$W"
cd "$W"
/usr/bin/find . -name __pycache__ -type d -prune -exec rm -rf {} +
rm -f .coverage
# .claude/ is excluded locally (not committed, not in .gitignore): `git status` stays clean,
# the hook config never shows up in the diffs the reviewers and the scorer read.
echo ".claude/" >> .git/info/exclude
git update-index -q --refresh || true
if [ "$CFG" != "bare" ]; then
# The settings command is a neutral wrapper: Claude Code prefixes every "Stop hook feedback" with the
# hook's command line, so the SPEC command (GATE_RUN=... GATE_CONFIG=<cfg> bash <bench>/gate.sh) would
# show the agent the config name and the bench path. The wrapper holds that exact command instead.
mkdir -p .claude/hooks
printf '#!/bin/bash\nGATE_RUN=%s GATE_CONFIG=%s bash %s/gate.sh\n' "$RUN" "$CFG" "$HERE" > .claude/hooks/stop.sh
"$HERE/.venv/bin/python" - "$HERE" > .claude/settings.json <<'PY'
import json
import sys
bench = sys.argv[1]
cmd = 'bash "$CLAUDE_PROJECT_DIR/.claude/hooks/stop.sh"'
# Once a gate blocks, the agent hunts for the tests behind it (seen in a smoke: `cat .claude/hooks/stop.sh;
# find / -name test_T1.py`). The oracle and the gate tests live under the bench: deny the usual forms.
# Read rules cover Read/Grep/Glob; Bash rules match the command text (not a security boundary).
deny = ["Read(./.claude/**)", "Read(/%s/hidden/**)" % bench, "Read(/%s/runs/**)" % bench,
"Read(/%s/validation/**)" % bench, "Read(/%s/reference/**)" % bench,
"Read(//Users/thomas/dev/verify-loops-bench/**)", "Read(//Users/thomas/dev/haiku-routing-bench/**)",
"Bash(*.claude/hooks*)", "Bash(*.claude/settings*)", "Bash(*gate-bench/hidden*)", "Bash(*../hidden*)",
"Bash(*gate-bench/runs*)", "Bash(*../runs*)", "Bash(*gate_tests*)", "Bash(*gate-bench/reference*)",
"Bash(*gate-bench/validation*)", "Bash(*verify-loops-bench*)", "Bash(*haiku-routing-bench*)", "Bash(find *)"]
print(json.dumps({"permissions": {"deny": deny},
"hooks": {"Stop": [{"hooks": [{"type": "command", "command": cmd, "timeout": 1500}]}]}}, indent=2))
PY
fi
test -z "$(git status --porcelain)" || { echo "work copy not clean"; git status --porcelain; exit 1; }
env.sh
# Sourced by run.sh, gate.sh and validate.sh.
# 1. Scrub every CLAUDE* variable inherited from a launching Claude Code session
# (CLAUDE_EFFORT, CLAUDE_CODE_SUBAGENT_MODEL, CLAUDE_CODE_SESSION_ID, CLAUDECODE,
# CLAUDE_CODE_CHILD_SESSION, ...): they would silently change the worker's effort,
# subagent model and session bookkeeping.
for _v in $(env | sed -n 's/^\(CLAUDE[A-Za-z0-9_]*\)=.*/\1/p'); do unset "$_v"; done
unset _v
# 2. Auth: the user's default config dir (~/.claude, OAuth). See README traps (--bare cannot auth).
unset CLAUDE_CONFIG_DIR
# 3. Fresh work copies at the same path must not share auto-memory across reps.
export CLAUDE_CODE_DISABLE_AUTO_MEMORY=1
# 4. No bytecode in the shared hidden/ dir when 3 runs execute hidden tests in parallel.
export PYTHONDONTWRITEBYTECODE=1
# 5. The bench venv first: `python -m pytest` inside the work copy uses it.
GATE_BENCH="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
export GATE_BENCH
export PATH="$GATE_BENCH/.venv/bin:$PATH"
configs.json
{
"bare": {"gate1": false, "gate3": null, "desc": "no hook"},
"g1": {"gate1": true, "gate3": null, "desc": "gate 1 (behaviour tests) only"},
"g3": {"gate1": false, "gate3": "indep", "desc": "gate 3, two independent reviewers"},
"g13": {"gate1": true, "gate3": "indep", "desc": "gate 1 then gate 3 indep (Shopify order)"},
"g13adv": {"gate1": true, "gate3": "adv", "desc": "gate 1 then gate 3 reviewer + challenger"}
}
gen_matrix.py
#!/usr/bin/env python3
"""Write matrix.txt: "<cfg> <task> <rep>" lines. Rep 1: configs bare,g1,g3,g13,g13adv as contiguous blocks,
tasks rotated per block; rep 2: same blocks in reverse config order (ABBA over time)."""
CFGS = ["bare", "g1", "g3", "g13", "g13adv"]
TASKS = ["T1", "T2", "T3", "T4", "T5"]
lines = []
for rep, order in ((1, CFGS), (2, CFGS[::-1])):
lines.append("# rep %d" % rep)
for i, c in enumerate(order):
k = (i + rep - 1) % len(TASKS)
for t in TASKS[k:] + TASKS[:k]:
lines.append("%s %s %d" % (c, t, rep))
open("matrix.txt", "w").write("\n".join(lines) + "\n")
print(sum(1 for l in lines if not l.startswith("#")), "runs")
matrix.txt
# rep 1
bare T1 1
bare T2 1
bare T3 1
bare T4 1
bare T5 1
g1 T2 1
g1 T3 1
g1 T4 1
g1 T5 1
g1 T1 1
g3 T3 1
g3 T4 1
g3 T5 1
g3 T1 1
g3 T2 1
g13 T4 1
g13 T5 1
g13 T1 1
g13 T2 1
g13 T3 1
g13adv T5 1
g13adv T1 1
g13adv T2 1
g13adv T3 1
g13adv T4 1
# rep 2
g13adv T2 2
g13adv T3 2
g13adv T4 2
g13adv T5 2
g13adv T1 2
g13 T3 2
g13 T4 2
g13 T5 2
g13 T1 2
g13 T2 2
g3 T4 2
g3 T5 2
g3 T1 2
g3 T2 2
g3 T3 2
g1 T5 2
g1 T1 2
g1 T2 2
g1 T3 2
g1 T4 2
bare T1 2
bare T2 2
bare T3 2
bare T4 2
bare T5 2
matrix.sh
#!/usr/bin/env bash
# Run an ordered list of "<cfg> <task> <rep>" lines, up to MATRIX_JOBS (default 3) at a time, each run in
# its own work copy. Blank lines and #comments skipped. Appends to matrix.log. Resumable: SKIP_DONE=1.
# Usage: nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 &
set -uo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"; cd "$HERE"
LIST="${1:-matrix.txt}"; JOBS="${MATRIX_JOBS:-3}"
mkdir "$HERE/.matrix.lock" 2>/dev/null || { echo "another matrix.sh is running (remove .matrix.lock if stale)" >&2; exit 1; }
trap 'rmdir "$HERE/.matrix.lock"' EXIT
echo "=== $(date '+%F %T') start $LIST jobs=$JOBS pid=$$" >> matrix.log
grep -vE '^[[:space:]]*(#|$)' "$LIST" | xargs -P "$JOBS" -L 1 ./matrix-one.sh
echo "=== $(date '+%F %T') done $LIST" >> matrix.log
matrix-one.sh
#!/usr/bin/env bash
# One matrix cell (called by matrix.sh through xargs). Args: <cfg> <task> <rep>
HERE="$(cd "$(dirname "$0")" && pwd)"; cd "$HERE"
CFG="$1"; TASK="$2"; REP="$3"
if [ "${SKIP_DONE:-0}" = 1 ] && [ -f "runs/${TASK}-${CFG}-r${REP}/score.json" ]; then
echo "$(date '+%F %T') skip ${TASK}-${CFG}-r${REP}" >> matrix.log; exit 0
fi
echo "$(date '+%F %T') start ${TASK}-${CFG}-r${REP}" >> matrix.log
line=$(./run.sh "$CFG" "$TASK" "$REP" < /dev/null 2>&1 | tail -1)
echo "$(date '+%F %T') $line" >> matrix.log
echo "$line"
analyze.py
#!/usr/bin/env python3
"""Aggregate runs/*/score.json (matrix runs only: <task>-<cfg>-r<rep>, no tag) -> results.json + results.md."""
import glob
import json
import os
import re
HERE = os.path.dirname(os.path.abspath(__file__))
CFGS = ["bare", "g1", "g3", "g13", "g13adv"]
TASKS = ["T1", "T2", "T3", "T4", "T5"]
rows = []
for p in sorted(glob.glob(os.path.join(HERE, "runs", "*", "score.json"))):
if re.fullmatch(r"T\d-[a-z0-9]+-r\d+", os.path.basename(os.path.dirname(p))):
rows.append(json.load(open(p)))
def mean(xs):
xs = [x for x in xs if x is not None]
return round(sum(xs) / len(xs), 4) if xs else None
def agg(rs):
if not rs:
return None
n = len(rs)
hp = sum(r["hidden_pass"] for r in rs)
ht = sum(r["hidden_pass"] + r["hidden_fail"] for r in rs)
rb = {g: sum(r["right_blocks_by_gate"][g] for r in rs) for g in ("g1", "g3")}
fb = {g: sum(r["false_blocks_by_gate"][g] for r in rs) for g in ("g1", "g3")}
return {
"runs": n, "hidden_reqs_passed": hp, "hidden_reqs_total": ht, "hidden_rate": round(hp / ht, 4) if ht else None,
"runs_fully_passing": sum(1 for r in rs if r["hidden_fail"] == 0),
"suite_green_runs": sum(1 for r in rs if r["suite_fail"] == 0),
"mean_cost_total_usd": mean([r["cost_total_usd"] for r in rs]),
"mean_cost_worker_usd": mean([r["cost_worker_usd"] for r in rs]),
"mean_cost_reviewers_usd": mean([r["cost_reviewers_usd"] for r in rs]),
"mean_cost_testwriter_usd": mean([r.get("cost_testwriter_usd") or 0 for r in rs]),
# per-gate cost: gate 1 = the test writer (its tests then run locally for free), gate 3 = the reviewers
"mean_cost_gate1_usd": mean([r.get("cost_testwriter_usd") or 0 for r in rs]),
"mean_cost_gate3_usd": mean([r["cost_reviewers_usd"] for r in rs]),
"gate_tests_total": sum(r.get("gate_tests_total") or 0 for r in rs),
"gate_tests_fail_final": sum(r.get("gate_tests_fail_final") or 0 for r in rs),
"mean_wall_s": mean([r["wall_s"] for r in rs]), "mean_turns": mean([r["turns"] for r in rs]),
"blocks_per_run": round(sum(r["blocks_total"] for r in rs) / n, 3),
"blocks_by_gate": {g: sum(r["blocks_by_gate"][g] for r in rs) for g in ("g1", "g3")},
"right_blocks": rb, "false_blocks": fb,
"false_block_rate": {g: (round(fb[g] / (rb[g] + fb[g]), 3) if rb[g] + fb[g] else None) for g in ("g1", "g3")},
"give_ups": sum(1 for r in rs if r["give_up"]), "ended_blocked": sum(1 for r in rs if r["ended_blocked"]),
"reviewer_findings": sum(r["reviewer_findings_total"] for r in rs),
"reviewer_errors": sum(r.get("reviewer_errors", 0) for r in rs),
"model_ok": all(r["model_ok"] for r in rs),
"missed_reqs": sorted(set("%s:%s" % (r["task"], q) for r in rs for q in r["failed_reqs"])),
}
by_cfg = {c: agg([r for r in rows if r["cfg"] == c]) for c in CFGS}
by_task = {t: {c: agg([r for r in rows if r["task"] == t and r["cfg"] == c]) for c in CFGS} for t in TASKS}
by_task_all = {t: agg([r for r in rows if r["task"] == t]) for t in TASKS}
def rate(c):
return (by_cfg.get(c) or {}).get("hidden_rate")
def d(a, b):
return round(a - b, 4) if a is not None and b is not None else None
answers = {
"q1_gate1_vs_bare": {"bare": rate("bare"), "g1": rate("g1"), "delta": d(rate("g1"), rate("bare"))},
"q2_gate3_over_g1": {"g1": rate("g1"), "g13": rate("g13"), "delta": d(rate("g13"), rate("g1")),
"g3_alone": rate("g3"), "g3_vs_bare": d(rate("g3"), rate("bare"))},
"q3_adv_vs_indep": {"g13": rate("g13"), "g13adv": rate("g13adv"), "delta": d(rate("g13adv"), rate("g13")),
"false_blocks_g3": {c: (by_cfg.get(c) or {}).get("false_blocks", {}).get("g3") for c in ("g13", "g13adv")}},
"q4_false_block_rate": {c: (by_cfg.get(c) or {}).get("false_block_rate") for c in CFGS},
"q4_gate1_false_blocks": {c: {"false": (by_cfg.get(c) or {}).get("false_blocks", {}).get("g1"),
"right": (by_cfg.get(c) or {}).get("right_blocks", {}).get("g1")}
for c in ("g1", "g13", "g13adv")},
"q5_cost": {c: {k: (by_cfg.get(c) or {}).get(k) for k in ("mean_cost_total_usd", "mean_cost_worker_usd",
"mean_cost_gate1_usd", "mean_cost_gate3_usd", "mean_wall_s")} for c in CFGS},
}
json.dump({"n_runs": len(rows), "by_cfg": by_cfg, "by_task": by_task, "by_task_all": by_task_all,
"answers": answers, "runs": rows}, open(os.path.join(HERE, "results.json"), "w"), indent=2)
L = ["# gate-bench results", "", "%d runs scored (matrix runs only)." % len(rows), "",
"## Per config", "",
"| cfg | runs | hidden reqs | fully passing | suite green | mean $ total (worker + g1 writer + g3 reviewers) | mean wall s | blocks/run | g1 right/false | g3 right/false | give-ups | ended blocked | model ok |",
"|---|---|---|---|---|---|---|---|---|---|---|---|---|"]
for c in CFGS:
a = by_cfg[c]
if not a:
L.append("| %s | 0 |" % c); continue
L.append("| %s | %d | %d/%d (%.0f%%) | %d | %d | %.3f (%.3f + %.3f + %.3f) | %.0f | %.2f | %d/%d | %d/%d | %d | %d | %s |" % (
c, a["runs"], a["hidden_reqs_passed"], a["hidden_reqs_total"], 100 * (a["hidden_rate"] or 0),
a["runs_fully_passing"], a["suite_green_runs"], a["mean_cost_total_usd"] or 0, a["mean_cost_worker_usd"] or 0,
a["mean_cost_testwriter_usd"] or 0, a["mean_cost_reviewers_usd"] or 0, a["mean_wall_s"] or 0, a["blocks_per_run"], a["right_blocks"]["g1"],
a["false_blocks"]["g1"], a["right_blocks"]["g3"], a["false_blocks"]["g3"], a["give_ups"], a["ended_blocked"],
a["model_ok"]))
L += ["", "## Per task (hidden reqs passed / total, per config)", "", "| task | " + " | ".join(CFGS) + " | missed reqs (all configs) |",
"|---|" + "---|" * (len(CFGS) + 1)]
for t in TASKS:
cells = []
for c in CFGS:
a = by_task[t][c]
cells.append("%d/%d, $%.2f, %.1f blk" % (a["hidden_reqs_passed"], a["hidden_reqs_total"], a["mean_cost_total_usd"] or 0,
a["blocks_per_run"]) if a else "-")
L.append("| %s | %s | %s |" % (t, " | ".join(cells), ", ".join((by_task_all[t] or {}).get("missed_reqs", []))))
L += ["", "## The video's questions", ""]
for k, v in answers.items():
L.append("- **%s**: `%s`" % (k, json.dumps(v)))
L += ["", "Gate 1 false blocks (gate 1 blocked while the hidden oracle was fully green): " + ", ".join(
"%s %s/%s" % (c, (by_cfg.get(c) or {}).get("false_blocks", {}).get("g1"),
((by_cfg.get(c) or {}).get("false_blocks", {}).get("g1") or 0) + ((by_cfg.get(c) or {}).get("right_blocks", {}).get("g1") or 0))
for c in ("g1", "g13", "g13adv")) + " (false / all g1 blocks)."]
L += ["", "Right block = the hidden oracle had at least one failing requirement at that stop; false block = the oracle "
"was fully green at that stop (the gate sent the agent back for nothing the user asked for)."]
open(os.path.join(HERE, "results.md"), "w").write("\n".join(L) + "\n")
print("\n".join(L))
results.md
# gate-bench results
50 runs scored (matrix runs only).
## Per config
| cfg | runs | hidden reqs | fully passing | suite green | mean $ total (worker + g1 writer + g3 reviewers) | mean wall s | blocks/run | g1 right/false | g3 right/false | give-ups | ended blocked | model ok |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bare | 10 | 35/50 (70%) | 3 | 10 | 0.085 (0.085 + 0.000 + 0.000) | 33 | 0.00 | 0/0 | 0/0 | 0 | 0 | True |
| g1 | 10 | 42/50 (84%) | 6 | 10 | 0.201 (0.130 + 0.071 + 0.000) | 60 | 0.60 | 3/3 | 0/0 | 0 | 0 | True |
| g3 | 10 | 34/50 (68%) | 3 | 10 | 0.151 (0.094 + 0.000 + 0.058) | 54 | 0.10 | 0/0 | 1/0 | 0 | 0 | True |
| g13 | 10 | 41/50 (82%) | 5 | 10 | 0.282 (0.153 + 0.071 + 0.057) | 86 | 1.70 | 14/1 | 2/0 | 2 | 0 | True |
| g13adv | 10 | 39/50 (78%) | 6 | 10 | 0.276 (0.157 + 0.072 + 0.048) | 86 | 1.20 | 11/1 | 0/0 | 1 | 0 | True |
## Per task (hidden reqs passed / total, per config)
| task | bare | g1 | g3 | g13 | g13adv | missed reqs (all configs) |
|---|---|---|---|---|---|---|
| T1 | 4/10, $0.08, 0.0 blk | 4/10, $0.22, 0.5 blk | 3/10, $0.16, 0.5 blk | 4/10, $0.36, 3.0 blk | 5/10, $0.31, 1.0 blk | T1:test_req2, T1:test_req3, T1:test_req4, T1:test_req5 |
| T2 | 8/10, $0.10, 0.0 blk | 8/10, $0.17, 0.0 blk | 8/10, $0.17, 0.0 blk | 8/10, $0.27, 3.0 blk | 9/10, $0.30, 3.5 blk | T2:test_req4 |
| T3 | 10/10, $0.09, 0.0 blk | 10/10, $0.21, 1.0 blk | 10/10, $0.14, 0.0 blk | 10/10, $0.26, 0.5 blk | 10/10, $0.31, 0.5 blk | |
| T4 | 5/10, $0.07, 0.0 blk | 10/10, $0.23, 0.5 blk | 5/10, $0.15, 0.0 blk | 10/10, $0.30, 1.0 blk | 5/10, $0.21, 0.0 blk | T4:test_req1, T4:test_req2, T4:test_req3, T4:test_req4, T4:test_req5 |
| T5 | 8/10, $0.08, 0.0 blk | 10/10, $0.18, 1.0 blk | 8/10, $0.12, 0.0 blk | 9/10, $0.23, 1.0 blk | 10/10, $0.25, 1.0 blk | T5:test_req4 |
## The video's questions
- **q1_gate1_vs_bare**: `{"bare": 0.7, "g1": 0.84, "delta": 0.14}`
- **q2_gate3_over_g1**: `{"g1": 0.84, "g13": 0.82, "delta": -0.02, "g3_alone": 0.68, "g3_vs_bare": -0.02}`
- **q3_adv_vs_indep**: `{"g13": 0.82, "g13adv": 0.78, "delta": -0.04, "false_blocks_g3": {"g13": 0, "g13adv": 0}}`
- **q4_false_block_rate**: `{"bare": {"g1": null, "g3": null}, "g1": {"g1": 0.5, "g3": null}, "g3": {"g1": null, "g3": 0.0}, "g13": {"g1": 0.067, "g3": 0.0}, "g13adv": {"g1": 0.083, "g3": null}}`
- **q4_gate1_false_blocks**: `{"g1": {"false": 3, "right": 3}, "g13": {"false": 1, "right": 14}, "g13adv": {"false": 1, "right": 11}}`
- **q5_cost**: `{"bare": {"mean_cost_total_usd": 0.0849, "mean_cost_worker_usd": 0.0849, "mean_cost_gate1_usd": 0.0, "mean_cost_gate3_usd": 0.0, "mean_wall_s": 32.7}, "g1": {"mean_cost_total_usd": 0.201, "mean_cost_worker_usd": 0.1302, "mean_cost_gate1_usd": 0.0709, "mean_cost_gate3_usd": 0.0, "mean_wall_s": 59.5}, "g3": {"mean_cost_total_usd": 0.1513, "mean_cost_worker_usd": 0.0938, "mean_cost_gate1_usd": 0.0, "mean_cost_gate3_usd": 0.0575, "mean_wall_s": 54.1}, "g13": {"mean_cost_total_usd": 0.2818, "mean_cost_worker_usd": 0.1534, "mean_cost_gate1_usd": 0.0712, "mean_cost_gate3_usd": 0.0573, "mean_wall_s": 86.4}, "g13adv": {"mean_cost_total_usd": 0.2763, "mean_cost_worker_usd": 0.1568, "mean_cost_gate1_usd": 0.0718, "mean_cost_gate3_usd": 0.0478, "mean_wall_s": 85.9}}`
Gate 1 false blocks (gate 1 blocked while the hidden oracle was fully green): g1 3/6, g13 1/15, g13adv 1/12 (false / all g1 blocks).
Right block = the hidden oracle had at least one failing requirement at that stop; false block = the oracle was fully green at that stop (the gate sent the agent back for nothing the user asked for).
Doğrulama ve demo:
validate.sh: oracle'ın upstream'de başarısız olduğunu, hook'un tetiklendiğini,gate.sh'nin kırmızı bir kopyayı engellediğini ve bir reviewer'ın geçerli JSON döndürdüğünü kontrol eder.validate.sh
#!/usr/bin/env bash
# Validation before the matrix (SPEC § Validation). Steps 1, 3, 4 here; step 2 = the g1 red-start smoke run:
# INJECT=$PWD/inject/break.patch RUN_TAG=val2 ./run.sh g1 T1 1 (then: ./validate.sh step2)
# Usage: validate.sh [all|step1|step2|step3|step4] -> appends to validation.log
set -u
HERE="$(cd "$(dirname "$0")" && pwd)"; cd "$HERE"
. "$HERE/env.sh"
PY="$HERE/.venv/bin/python"
STEP="${1:-all}"
status=0
count() { grep -Eo "[0-9]+ $2" <<<"$1" | tail -1 | awk '{print $1}'; }
log() { echo "$*" | tee -a validation.log; }
log "=== $(date '+%F %T') validate.sh $STEP ($(claude --version))"
if [ "$STEP" = all ] || [ "$STEP" = step1 ]; then
# 1. upstream fails the hidden tests (<= 1 req passing: T5 R3 is a regression guard); reference passes hidden + suite
TMP="$(mktemp -d "${TMPDIR:-/tmp}/gb-validate.XXXXXX")"
for t in T1 T2 T3 T4 T5; do
tree="$TMP/$t"; git clone -q "$HERE/upstream" "$tree"
v=$(cd "$tree" && TINYDB_TREE="$tree" "$PY" -m pytest -v -c "$HERE/hidden/pytest.ini" --rootdir "$HERE/hidden" "$HERE/hidden/test_$t.py" 2>&1)
up_pass=$(grep -Eo "test_req[0-9]+[^ ]* PASSED" <<<"$v" | wc -l | tr -d ' ')
(cd "$tree" && git apply "$HERE/reference/$t.patch") || { log "$t: reference patch does not apply"; status=1; continue; }
h=$(cd "$tree" && TINYDB_TREE="$tree" "$PY" -m pytest -q -c "$HERE/hidden/pytest.ini" --rootdir "$HERE/hidden" "$HERE/hidden/test_$t.py" 2>&1); hrc=$?
s=$(cd "$tree" && "$PY" -m pytest -q -p no:cacheprovider --no-cov 2>&1); src=$?
ok=OK; { [ "$up_pass" -gt 1 ] || [ $hrc -ne 0 ] || [ $src -ne 0 ]; } && { ok=FAIL; status=1; }
log "step1 $t: upstream reqs passing=$up_pass reference hidden rc=$hrc ($(count "$h" passed) passed) suite rc=$src ($(count "$s" passed) passed) [$ok]"
done
rm -rf "$TMP"
fi
if [ "$STEP" = all ] || [ "$STEP" = step3 ]; then
# 3. gate.sh by hand, fake stdin. green = T1 reference + one new test (g1 passes) -> exit 0 allow;
# red = break.patch (one repo test fails) -> exit 2, pytest tail on stderr.
FAKE='{"session_id":"validate","transcript_path":"/dev/null","cwd":"x","hook_event_name":"Stop","stop_hook_active":false}'
for k in green red; do
R="$HERE/runs/_val3-$k"; W="$HERE/work/_val3-$k"; rm -rf "$R"; mkdir -p "$R"
./prep.sh g1 "$W" "$R" > /dev/null
if [ $k = green ]; then
(cd "$W" && git apply "$HERE/reference/T1.patch" && cp "$HERE/inject/test_pushpull.py" tests/)
else
(cd "$W" && git apply "$HERE/inject/break.patch")
fi
printf '{"task":"T1","cfg":"g1","rep":0,"work":"%s","name":"_val3-%s"}\n' "$W" "$k" > "$R/meta.json"; cp tasks/T1.md "$R/task.md"
echo "$FAKE" | GATE_RUN="$R" GATE_CONFIG=g1 bash gate.sh > "$R/stdout.txt" 2> "$R/stderr.txt"; rc=$?
dec=$("$PY" -c "import json;print(json.loads(open('$R/gate.jsonl').readline())['decision'])")
want=0; [ $k = red ] && want=2
ok=OK; [ $rc -ne $want ] && { ok=FAIL; status=1; }
[ $k = red ] && ! grep -q "short test summary" "$R/stderr.txt" && { ok=FAIL; status=1; }
[ $k = green ] && [ -s "$R/stderr.txt" ] && { ok=FAIL; status=1; }
log "step3 $k: exit=$rc decision=$dec stderr_bytes=$(wc -c < "$R/stderr.txt" | tr -d ' ') [$ok]"
done
fi
if [ "$STEP" = all ] || [ "$STEP" = step4 ]; then
# 4. one reviewer pair by hand on the green copy (needs step3's work/_val3-green): structured_output valid
W="$HERE/work/_val3-green"
[ -d "$W" ] || { log "step4: run step3 first"; exit 1; }
for v in indep adv; do
"$PY" lib/review.py "$W" T1 $v > "runs/_val3-green/review-$v.txt" 2>&1
s=$("$PY" - "runs/_val3-green/review-$v.txt" <<'PY'
import json, sys
t = open(sys.argv[1]).read(); i = t.find("--- reason"); d = json.loads(t[:i] if i > 0 else t)
ok = all(r["error"] is None and isinstance(r["approve_raw"], bool) for r in d["reviewers"])
ok = ok and all(f["severity"] in ("blocking", "minor") and isinstance(f["line"], int) for r in d["reviewers"] for f in r["findings"])
print("%s pass=%s blocking=%d findings=%s cost=$%.4f wall=%s" % ("OK" if ok else "FAIL", d["pass"], d["blocking_findings"],
[len(r["findings"]) for r in d["reviewers"]], d["cost_usd"], [r["wall_s"] for r in d["reviewers"]]))
PY
)
case "$s" in OK*) ;; *) status=1 ;; esac
log "step4 $v: $s"
done
fi
if [ "$STEP" = all ] || [ "$STEP" = step2 ]; then
# 2. hook fires on a red start: runs/T1-g1-r1-val2 must log a block and the stream must show the agent continuing
R="$HERE/runs/T1-g1-r1-val2"
if [ -f "$R/gate.jsonl" ]; then
s=$("$PY" - "$R" <<'PY'
import json, sys, os
r = sys.argv[1]
g = [json.loads(l) for l in open(os.path.join(r, "gate.jsonl")) if l.strip()]
st = [json.loads(l) for l in open(os.path.join(r, "stream.jsonl")) if l.startswith("{")]
blocks = [x for x in g if x["decision"] == "block"]
# index in the stream of the first hook response that blocked, then any assistant tool_use after it
txt = [json.dumps(o) for o in st]
fb = next((i for i, t in enumerate(txt) if "Stop hook feedback" in t or ('"hook_event": "Stop"' in t and '"exit_code": 2' in t)), None)
after = fb is not None and any(o.get("type") == "assistant" for o in st[fb + 1:])
print("%s blocks=%d hook_calls=%d feedback_idx=%s assistant_after=%s last=%s" % (
"OK" if blocks and after else "FAIL", len(blocks), len(g), fb, after, g[-1]["decision"] if g else None))
PY
)
case "$s" in OK*) ;; *) status=1 ;; esac
log "step2: $s"
else
log "step2: not run yet (INJECT=\$PWD/inject/break.patch RUN_TAG=val2 ./run.sh g1 T1 1)"
fi
fi
[ $status -eq 0 ] && log "VALIDATION OK" || log "VALIDATION FAILED"
exit $status
validate_writer.sh
#!/usr/bin/env bash
# Validation of the gate-1 test writer: one writer call per task (in parallel) into validation/writer/<T>/,
# then its tests on upstream and on the reference patch. Usage: validate_writer.sh [T1 ...] (reuses an existing
# file unless REWRITE=1). Appends to validation.log.
set -u
HERE="$(cd "$(dirname "$0")" && pwd)"; cd "$HERE"
. "$HERE/env.sh"
PY="$HERE/.venv/bin/python"
TASKS="${*:-T1 T2 T3 T4 T5}"
for t in $TASKS; do
D="$HERE/validation/writer/$t"; mkdir -p "$D"
if [ "${REWRITE:-0}" = 1 ] || [ ! -f "$D/gate_tests/test_$t.py" ]; then "$PY" lib/testwriter.py "$t" "$D" > "$D/writer.out" 2>&1 & fi
done
wait
TMP="$(mktemp -d "${TMPDIR:-/tmp}/gb-vw.XXXXXX")"; trap 'rm -rf "$TMP"' EXIT
echo "=== $(date '+%F %T') validate_writer.sh $TASKS" >> validation.log
for t in $TASKS; do
D="$HERE/validation/writer/$t"
git clone -q "$HERE/upstream" "$TMP/up-$t"; git clone -q "$HERE/upstream" "$TMP/ref-$t"
(cd "$TMP/ref-$t" && git apply "$HERE/reference/$t.patch")
line=$("$PY" - "$D" "$t" "$TMP/up-$t" "$TMP/ref-$t" <<'PY'
import json, os, sys
sys.path.insert(0, "lib")
from testwriter import run_gate_tests
d, t, up, ref = sys.argv[1:5]
w = json.load(open(os.path.join(d, "testwriter.json")))
if not w["ok"]:
print("%s: writer FAILED %s" % (t, [a["error"] for a in w["attempts"]])); sys.exit()
g = os.path.join(d, "gate_tests")
_, uo, up_p, up_f, _ = run_gate_tests(g, t, up)
_, ro, rp, rf, rnames = run_gate_tests(g, t, ref)
open(os.path.join(d, "on-upstream.txt"), "w").write(uo); open(os.path.join(d, "on-reference.txt"), "w").write(ro)
print("%s: writer tests: %d, pass on reference: %d, pass on upstream: %d | fail on reference: %s | writer $%.3f %ss turns=%s" % (
t, rp + rf, rp, up_p, rnames, w["cost_usd"], w["wall_s"], [a.get("turns") for a in w["attempts"]]))
PY
)
echo "$line" | tee -a validation.log
done
demo.sh
#!/usr/bin/env bash
# Interactive demo of the gate for a screen recording: a gated tinydb work copy (config g13: acceptance
# tests, then two reviewers) with T1's acceptance tests already written. Usage: ./demo.sh [T1..T5]
# Then: cd work/demo && claude --setting-sources project --permission-mode acceptEdits
# and paste the request printed below. When Claude stops, the Stop hook runs; a red acceptance test shows
# as "Stop hook feedback: ... The acceptance tests for this change fail:" and Claude keeps working.
set -euo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"; TASK="${1:-T1}"
. "$HERE/env.sh"
RUN="$HERE/runs/demo"; W="$HERE/work/demo"
rm -rf "$RUN"; mkdir -p "$RUN"
"$HERE/prep.sh" g13 "$W" "$RUN" > "$RUN/prep.txt" 2>&1
printf '{"task":"%s","cfg":"g13","rep":0,"work":"%s","name":"demo"}\n' "$TASK" "$W" > "$RUN/meta.json"
cp "$HERE/tasks/$TASK.md" "$RUN/task.md"
printf '%s\n\nWhen you are done, stop and summarise what you changed.\n' "$(cat "$HERE/tasks/$TASK.md")" > "$RUN/prompt.txt"
# Reuse the acceptance tests of a matrix run of the same task (no new writer call).
SRC=$(ls -d "$HERE"/runs/"$TASK"-g13-r1/gate_tests 2>/dev/null | head -1)
cp -R "$SRC" "$RUN/gate_tests"
echo "ready: cd $W && claude --setting-sources project --permission-mode acceptEdits"
echo "--- paste this request ---"; cat "$RUN/prompt.txt"
inject/break.patch
diff --git a/tinydb/queries.py b/tinydb/queries.py
index cdab9fb..db00a2a 100644
--- a/tinydb/queries.py
+++ b/tinydb/queries.py
@@ -322,7 +322,7 @@ class Query(QueryInstance):
:param rhs: The value to compare against
"""
return self._generate_test(
- lambda value: value >= rhs,
+ lambda value: value > rhs,
('>=', self._path, rhs)
)
inject/test_pushpull.py
from tinydb import TinyDB, Query
from tinydb.storages import MemoryStorage
from tinydb.operations import push, pull
def test_push_pull():
db = TinyDB(storage=MemoryStorage)
db.insert({'n': 1, 't': ['a']})
db.update(push('t', 'b'), Query().n == 1)
db.update(pull('t', 'a'), Query().n == 1)
assert db.all()[0]['t'] == ['b']
inject/test_pushpull.py on GitHub
Belgeler:
README.md: yapı, komutlar ve harness'in ne olduğu.README.md
# gate-bench
What does each of Shopify's Helix gates catch when rebuilt as a Claude Code **Stop hook**, and what does it
cost? Headless Sonnet 5.5 workers implement five tinydb features (T1..T5); a Stop hook runs gate 1
(behaviour tests) and/or gate 3 (adversarial review by headless Sonnet 5.5 reviewers) every time the worker
tries to stop; hidden tests (the oracle, never visible to the agent) record the ground truth at every stop
and at the end. Built 2026-10-08 from `SPEC.md`; upstream, hidden tests, tasks and reference patches copied
from `../verify-loops-bench` (nothing there or in `../haiku-routing-bench` is modified).
## Commands (from this dir)
```bash
./validate.sh # steps 1,3,4 (+ step 2 once its run exists) -> validation.log
INJECT=$PWD/inject/break.patch RUN_TAG=val2 ./run.sh g1 T1 1 # validation step 2 (red start state)
RUN_TAG=smoke ./run.sh g13adv T1 1 # smoke (tagged runs are ignored by analyze.py)
./validate_writer.sh # gate-1 test writer on T1..T5: tests on upstream vs reference (REWRITE=1 to redo)
./gen_matrix.py # matrix.txt (50 runs, already generated)
nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 & # 3 runs in parallel (MATRIX_JOBS=3)
SKIP_DONE=1 nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 & # resume
.venv/bin/python analyze.py # results.json + results.md
.venv/bin/python lib/finalize.py runs/<run> # rescore one run from its work copy
.venv/bin/python lib/review.py work/<run> T1 adv # one gate-3 round by hand
```
Single run: `./run.sh <cfg> <task> <rep>` -> `runs/<task>-<cfg>-r<rep>/`, work copy `work/<same name>/`.
## Layout
| path | what |
|---|---|
| `SPEC.md` | the spec this harness implements |
| `upstream/` | tinydb @ 18d73a1 (git repo) |
| `hidden/` | `test_T1..T6.py` (T6 unused), `conftest.py` (`$TINYDB_TREE` or cwd first on sys.path), `pytest.ini` |
| `tasks/T1..T5.md` + `tasks/README.md` | the task prompts (verbatim) and what each hidden req checks |
| `reference/T1..T5.patch` | reference solutions (validation step 1) |
| `inject/break.patch` | `>=` -> `>` in `Query.__ge__` (one repo test red; validation steps 2-3) |
| `inject/test_pushpull.py` | one thin test, makes the T1 reference copy "green with tests touched" (step 3) |
| `configs.json` | `bare` (no hook), `g1`, `g3` (indep), `g13` (g1 then g3 indep), `g13adv` (g1 then g3 adv) |
| `env.sh` | scrubs inherited `CLAUDE*` vars, default `~/.claude` auth, `CLAUDE_CODE_DISABLE_AUTO_MEMORY=1`, venv on PATH |
| `prep.sh <cfg> <work> <run>` | fresh copy of upstream, `.claude/` in `.git/info/exclude`, hook settings for gated configs |
| `run.sh <cfg> <task> <rep>` | one run (worker + hook + `lib/finalize.py`) |
| `gate.sh` | the Stop hook (bash 3.2 wrapper: `exec .venv/bin/python -I lib/hook.py`) |
| `lib/hook.py` | gate dispatch, snapshot, hidden run, gate 1, gate 3, decision, `gate.jsonl` line |
| `lib/review.py` | gate-3 reviewers (indep / adv), schemas, prompts, reason text; CLI for a manual round |
| `lib/testwriter.py` | gate-1 test writer (before the worker) + `run_gate_tests` (path rewritten to `acceptance/`) |
| `validate_writer.sh`, `validation/writer/<T>/` | writer validation files and their runs on upstream / reference |
| `tasks/` (terse prompts, used), `tasks-explicit/` + `runs-explicit/` | the matrix uses terse prompts; see `tasks/README.md` |
| `lib/common.py` | diff incl. untracked files, suite run, hidden run (requirement level) |
| `lib/finalize.py` | post-run diff/suite/hidden, `result.json`, `score.json` |
| `gen_matrix.py`, `matrix.txt`, `matrix.sh`, `matrix-one.sh`, `matrix.log` | 50-run matrix, 3 in parallel (xargs -P), resumable |
| `analyze.py` -> `results.json`, `results.md` | per config, per task, the five questions |
| `validate.sh`, `VALIDATION.md`, `validation.log` | pre-matrix checks |
| `.venv/` | python 3.14.7, pytest 9.1.1, pytest-cov 7.1.0, pyyaml, ruff 0.16.10 (same pins as haiku-routing-bench) |
### Worker (run.sh)
`claude -p "<T-file verbatim>\n\nWhen you are done, stop and summarise what you changed." --model claude-sonnet-5-5
--output-format stream-json --verbose --include-hook-events --max-turns 60 --permission-mode acceptEdits
--allowedTools <haiku-routing-bench list> --setting-sources project --strict-mcp-config`, cwd = `work/<run>/`, stdin /dev/null.
Gated configs get `work/<run>/.claude/settings.json`:
`{"permissions":{"deny":[...see traps...]},"hooks":{"Stop":[{"hooks":[{"type":"command","command":"bash \"$CLAUDE_PROJECT_DIR/.claude/hooks/stop.sh\"","timeout":1500}]}]}}`
and `.claude/hooks/stop.sh` = `GATE_RUN=<abs run dir> GATE_CONFIG=<cfg> bash <bench>/gate.sh` (the SPEC command, one level down: see traps).
Copies of both land in the run dir (`settings.json`, `stop.sh`).
### Hook (lib/hook.py), one invocation
1. read stdin JSON; n = lines already in `gate.jsonl` + 1; blocks so far from the same file
2. `snap-<n>.diff` = `git diff HEAD` + untracked files (index never touched)
3. hidden tests on the work copy (`TINYDB_TREE` = work copy, cwd = work copy) -> `hidden-<n>.txt`; never shown to the agent
4. gate 1 (if configured) = the repo suite `python -m pytest -q -x --no-header -p no:cacheprovider --no-cov` (300 s)
-> `gate1-<n>.txt` AND the writer's acceptance tests `runs/<run>/gate_tests/test_<T>.py` run with `TINYDB_TREE` = work
copy -> `gatetests-<n>.txt`. Block when either is red. Reason: "The acceptance tests for this change fail:" + last 40
lines (failing test names + assertion output, the gate_tests path rewritten to `acceptance/`), then the suite tail if the
suite is red too; <= 1500 chars. `tests_touched` is still logged but no longer blocks. No writer file (writer failed
twice) -> suite only, logged as `gate_tests_present: false`.
5. gate 3 (if configured and gate 1 passed or absent): see below
6. decision: allow / block / give_up (a would-be block when 6 blocks already happened -> allow, logged `give_up`).
`stop_hook_active` is logged, never used. Block = exit 2 + reason on stderr. A crash of the hook itself is
logged to `gate-errors.txt` and allows (counted as `gate_crashes` in score.json).
### Gate 1 test writer (lib/testwriter.py, run.sh before the worker, configs g1/g13/g13adv)
`claude -p` Sonnet 5.5 with the reviewer isolation flags, `--tools Read,Grep,Glob --max-turns 12 --json-schema
{"file": string}`, cwd = a pristine temp copy of upstream (deleted after). Prompt = framing + `tasks/<T>.md` + the
coordinator's instruction ("Write integration-style pytest cases for this request against the tinydb API, one test per
behaviour a careful senior would expect, pushing for edge cases beyond the happy path (missing fields, wrong types,
boundaries, persistence, cache, existing behaviour unchanged). Tests only, no implementation. Return the file content.")
+ mechanics (self-contained module, imports of the new API inside each test, MemoryStorage/tmp_path). The file must
compile and contain `def test_`, else one retry. Output `runs/<run>/gate_tests/` (+ hidden's conftest.py with
`TINYDB_TREE`, pytest.ini) and `testwriter.json` (cost, turns, attempts). Fixed per-run cost, outside the worker's wall.
### Gate 3 (lib/review.py)
Each reviewer: `claude -p "<prompt>" --model claude-sonnet-5-5 --tools Read,Grep,Glob --max-turns 15 --output-format json
--json-schema <schema> --permission-mode dontAsk --setting-sources "" --settings '{"disableAllHooks":true}' --strict-mcp-config`
with `CLAUDE_CODE_DISABLE_CLAUDE_MDS=1 CLAUDE_CODE_DISABLE_AUTO_MEMORY=1`, all inherited `CLAUDE*` vars dropped, cwd = work copy,
600 s timeout. Prompt = framing + task prompt (T-file) + diff (tracked + untracked, cap 60k chars) + the SPEC's "Find defects..."
sentence. `structured_output` is read from the JSON result; `total_cost_usd`, `num_turns`, wall and models are recorded.
- `indep`: reviewers A and B, same prompt, run in parallel, no shared context. Blocking findings = union. Pass when both approve.
- `adv`: A as above; challenger B gets A's numbered findings + the diff, answers `upheld|rejected` + reason per id, may add
findings (`added_findings`). Blocking = A's blocking findings B upheld + B's added blocking findings. B always runs (same
2-call budget as indep).
- Effective `approve` = "no blocking finding" (the raw `approve` is logged as `approve_raw`; a reviewer saying approve=false with
only minor findings does not block). A reviewer call that errors approves (fail open) with `error` recorded; an errored
challenger upholds nothing.
- Reason = "Code review found N blocking defect(s):" + numbered `file:line text` + "Fix every finding, re-run the tests, then stop.", <= 1500 chars.
### Per run (`runs/<task>-<cfg>-r<rep>/`)
`stream.jsonl`, `stderr.txt`, `exit_code.txt`, `wall_s.txt` (includes hook time: the hook runs inside the session),
`gate.jsonl`, `snap-<n>.diff`, `hidden-<n>.txt`, `gate1-<n>.txt`, `hook-input-1.json`, `post.diff`, `numstat.txt`,
`suite.txt` (full suite, no -x), `hidden.txt`, `gate_tests/`, `testwriter.json`, `gatetests-<n>.txt`, `gatetests-final.txt`, `result.json`, `score.json`, `meta.json`, `task.md`, `prompt.txt`,
`settings.json` + `stop.sh` (gated configs).
`score.json`: task, cfg, rep, hidden_pass/hidden_fail/failed_reqs (**requirement level**: 5 reqs per task), suite_pass/fail,
hook_calls, blocks_total, blocks_by_gate {g1,g3}, right_blocks / false_blocks (+ `_by_gate`), give_up, ended_blocked,
cost_worker_usd, cost_reviewers_usd, cost_testwriter_usd, cost_total_usd (all three), testwriter_ok, gate_tests_total,
gate_tests_fail_final, turns, wall_s, reviewer_findings_total, reviewer_errors, gate_crashes,
model_ok (worker init model + every modelUsage key + every reviewer model == claude-sonnet-5-5), hook_events_in_stream,
hidden_first_stop.
## Matrix
5 tasks x 5 configs x 2 reps = 50 runs. Rep 1 config blocks bare, g1, g3, g13, g13adv (tasks rotated per block), rep 2 the
reverse block order. Up to 3 runs at once, each in its own work copy (`matrix-one.sh` via `xargs -P 3`, bash 3.2 has no
`wait -n`). `matrix.log`: one `start` and one result line per run.
## Traps found
- **Gate 1 as first built passed by construction** (it only re-ran the implementer's own tests: `T2-g1-r1` one allow, hidden
R4 missed). Redesigned around an independent test writer (above). The writer hedges where the terse prompt is silent
(accepts "raises or no-op"), so it cannot recover unstated requirements; per-task overlap is in VALIDATION.md.
- **A blocked agent hunts for the tests.** After a gate-1 block the smoke agent ran `cat .claude/hooks/stop.sh; find / -name
test_T1.py` (denied only because `find` is not on the allow-list; `cat` is, and `stop.sh` names the bench dir, from where
`hidden/` is one `cat` away). Gated configs' `.claude/settings.json` now carries `permissions.deny`: Read of `./.claude/**`,
the bench's `hidden/ runs/ validation/ reference/`, the two sibling benches; Bash patterns containing `.claude/hooks`,
`.claude/settings`, `../hidden`, `../runs`, `gate_tests`, the bench sub-paths, `find *`. Probed with a Haiku session: all
five escape attempts (cat, Read, ls, python open) denied, in-tree reads fine. Bash rules match command text, not a security
boundary. Bare runs have no settings file (bare r1 ran before this change; bare has no motive to hunt).
Reviewers and the writer run with `--setting-sources ""`, so these rules do not apply to them.
- **Transient `FileNotFoundError` spawning a reviewer** inside the hook (once, both reviewers of one call, wall 0.0 s;
never reproduced by hand). Reviewer calls now retry once after 3 s; the error message records `filename`.
- **`--bare` cannot authenticate here.** It reads `ANTHROPIC_API_KEY` / `apiKeyHelper` only, never OAuth or the keychain;
this account is OAuth (`~/.claude`), so a `--bare` reviewer returns `terminal_reason: api_error`, cost 0. Used the
documented equivalents instead: `--setting-sources ""` + `--settings '{"disableAllHooks":true}'` + `--strict-mcp-config` +
`CLAUDE_CODE_DISABLE_CLAUDE_MDS=1` + `CLAUDE_CODE_DISABLE_AUTO_MEMORY=1` + `--tools Read,Grep,Glob`. The reviewers' cwd is the
work copy, whose `.claude/settings.json` carries the Stop hook: without `--setting-sources ""`/`disableAllHooks` a reviewer
would run the gate recursively. Verified: a dummy Stop hook in the probe copy never fired.
- **Inherited session env.** A matrix launched from inside Claude Code inherits `CLAUDE_EFFORT=medium`,
`CLAUDE_CODE_SUBAGENT_MODEL=opus`, `CLAUDE_CODE_SESSION_ID`, `CLAUDECODE`, `CLAUDE_CODE_CHILD_SESSION`, messaging tokens.
`env.sh` unsets every `CLAUDE*` var before setting its own; reviewers get a scrubbed env too (the hook process itself
inherits the worker's `CLAUDE_*` vars, e.g. the worker's session id).
- **The feedback line leaks the hook command.** Claude Code prefixes each block with the command line:
`Stop hook feedback:\n[<command>]: <reason>`. With the SPEC command inline the agent would read `GATE_CONFIG=g13adv` and
`gate-bench/gate.sh`. The settings command is therefore `bash "$CLAUDE_PROJECT_DIR/.claude/hooks/stop.sh"`, a one-line
wrapper holding the SPEC command. The agent can still `cat` it (it lives in the work copy) but it is not pushed at it.
- **Stop hook input keys (2.1.293):** `background_tasks, cwd, effort ({"level":"medium"}), hook_event_name,
last_assistant_message, permission_mode, prompt_id, session_crons, session_id, stop_hook_active, transcript_path`.
- **8-cap resets on tool calls.** Per the docs the consecutive-continuation count resets each time Claude calls a tool, so the
built-in cap only bites an agent that answers blocks with text alone. Our give-up (6 blocks per run, counted from
`gate.jsonl`, tool calls or not) fires first in every case.
- **Red start state: the agent refuses to fix code it did not write.** Validation step 2 (T1 + an uncommitted `>=` -> `>`
edit) shows Sonnet 5.5 implementing T1, then answering 6 consecutive gate-1 blocks with "the red test is your pre-existing
uncommitted edit, not my change" without touching it, until give_up. The hook mechanics are validated; the behaviour is a
finding in itself (gate 1 cannot force a fix the agent considers out of scope).
- **pytest-cov.** The repo's `pytest.ini` adds `--cov tinydb`; gate 1 adds `--no-cov` so the 40-line tail holds the failure,
not the coverage table (and no `.coverage` file is written). `.coverage` is gitignored anyway.
- **T5 R4 is parametrized** (several test ids, one requirement). hidden counts are per requirement (5 per task), a req
passes only when all its ids pass. `validate.sh` step 1: T5 upstream passes 1 req (R3, the regression guard) as designed.
- **Diff without touching the index.** `git add -N` would change what the agent sees in `git status`; untracked files are
diffed with `git diff --no-index /dev/null <f>` instead. `.claude/` is in `.git/info/exclude`, so the hook config never
shows in any diff or in `git status`.
- **Worker Bash denials.** The 2.1.293 permission layer denies some agent commands with "Contains brace with quote character
(expansion obfuscation)" even under acceptEdits + allow-list; agents route around it (visible as `permission_denied` events).
- **Reviewers on a correct solution still flag "blocking".** On the T1 reference patch, reviewer A marked "pull does not check
the field is a list" (not required by the task) as blocking; indep therefore blocks, the adv challenger rejected it. This is
exactly what false_blocks measures (hidden all green at that stop).
- **Shared plan meter.** Workers and reviewers bill the user's plan quota; `total_cost_usd` is the list-price equivalent the
CLI reports, not a bill. Nothing else heavy should run on the account during the matrix.
- **rtk rewrites interactive `grep`/`diff`** in the launching Claude session only; scripts here call binaries directly
(`/usr/bin/diff`/`cmp` for verification diffs).
- **CLI version:** `claude --version` = 2.1.293 (Claude Code); flags checked in `claude --help`: `--json-schema`, `--bare`, `--tools`,
`--permission-mode` (choices acceptEdits, auto, bypassPermissions, manual, dontAsk, plan), `--setting-sources`,
`--include-hook-events`. `--max-turns` is accepted though not listed in `--help`.
## Matrix launch (2026-10-08 16:33:54, after the gate-1 redesign)
`cd /Users/thomas/dev/gate-bench && SKIP_DONE=1 nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 &` -> PID 71317, 3 lanes.
The 5 bare r1 runs (terse prompts) are kept and skipped; 45 runs to go. Expected wall ~40-70 min: gated runs so far take
~30-70 s of worker wall + ~35-40 s of test writer (before the worker) + 1-35 s per hook call, more with blocks.
Then `.venv/bin/python analyze.py`.
HARNESS.md
# gate-bench
What does each of Shopify's Helix gates catch when rebuilt as a Claude Code **Stop hook**, and what does it
cost? Headless Sonnet 5.5 workers implement five tinydb features (T1..T5); a Stop hook runs gate 1
(behaviour tests) and/or gate 3 (adversarial review by headless Sonnet 5.5 reviewers) every time the worker
tries to stop; hidden tests (the oracle, never visible to the agent) record the ground truth at every stop
and at the end. Built 2026-10-08 from `SPEC.md`; upstream, hidden tests, tasks and reference patches copied
from `../verify-loops-bench` (nothing there or in `../haiku-routing-bench` is modified).
## Commands (from this dir)
```bash
./validate.sh # steps 1,3,4 (+ step 2 once its run exists) -> validation.log
INJECT=$PWD/inject/break.patch RUN_TAG=val2 ./run.sh g1 T1 1 # validation step 2 (red start state)
RUN_TAG=smoke ./run.sh g13adv T1 1 # smoke (tagged runs are ignored by analyze.py)
./validate_writer.sh # gate-1 test writer on T1..T5: tests on upstream vs reference (REWRITE=1 to redo)
./gen_matrix.py # matrix.txt (50 runs, already generated)
nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 & # 3 runs in parallel (MATRIX_JOBS=3)
SKIP_DONE=1 nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 & # resume
.venv/bin/python analyze.py # results.json + results.md
.venv/bin/python lib/finalize.py runs/<run> # rescore one run from its work copy
.venv/bin/python lib/review.py work/<run> T1 adv # one gate-3 round by hand
```
Single run: `./run.sh <cfg> <task> <rep>` -> `runs/<task>-<cfg>-r<rep>/`, work copy `work/<same name>/`.
## Layout
| path | what |
|---|---|
| `SPEC.md` | the spec this harness implements |
| `upstream/` | tinydb @ 18d73a1 (git repo) |
| `hidden/` | `test_T1..T6.py` (T6 unused), `conftest.py` (`$TINYDB_TREE` or cwd first on sys.path), `pytest.ini` |
| `tasks/T1..T5.md` + `tasks/README.md` | the task prompts (verbatim) and what each hidden req checks |
| `reference/T1..T5.patch` | reference solutions (validation step 1) |
| `inject/break.patch` | `>=` -> `>` in `Query.__ge__` (one repo test red; validation steps 2-3) |
| `inject/test_pushpull.py` | one thin test, makes the T1 reference copy "green with tests touched" (step 3) |
| `configs.json` | `bare` (no hook), `g1`, `g3` (indep), `g13` (g1 then g3 indep), `g13adv` (g1 then g3 adv) |
| `env.sh` | scrubs inherited `CLAUDE*` vars, default `~/.claude` auth, `CLAUDE_CODE_DISABLE_AUTO_MEMORY=1`, venv on PATH |
| `prep.sh <cfg> <work> <run>` | fresh copy of upstream, `.claude/` in `.git/info/exclude`, hook settings for gated configs |
| `run.sh <cfg> <task> <rep>` | one run (worker + hook + `lib/finalize.py`) |
| `gate.sh` | the Stop hook (bash 3.2 wrapper: `exec .venv/bin/python -I lib/hook.py`) |
| `lib/hook.py` | gate dispatch, snapshot, hidden run, gate 1, gate 3, decision, `gate.jsonl` line |
| `lib/review.py` | gate-3 reviewers (indep / adv), schemas, prompts, reason text; CLI for a manual round |
| `lib/testwriter.py` | gate-1 test writer (before the worker) + `run_gate_tests` (path rewritten to `acceptance/`) |
| `validate_writer.sh`, `validation/writer/<T>/` | writer validation files and their runs on upstream / reference |
| `tasks/` (terse prompts, used), `tasks-explicit/` + `runs-explicit/` | the matrix uses terse prompts; see `tasks/README.md` |
| `lib/common.py` | diff incl. untracked files, suite run, hidden run (requirement level) |
| `lib/finalize.py` | post-run diff/suite/hidden, `result.json`, `score.json` |
| `gen_matrix.py`, `matrix.txt`, `matrix.sh`, `matrix-one.sh`, `matrix.log` | 50-run matrix, 3 in parallel (xargs -P), resumable |
| `analyze.py` -> `results.json`, `results.md` | per config, per task, the five questions |
| `validate.sh`, `VALIDATION.md`, `validation.log` | pre-matrix checks |
| `.venv/` | python 3.14.7, pytest 9.1.1, pytest-cov 7.1.0, pyyaml, ruff 0.16.10 (same pins as haiku-routing-bench) |
### Worker (run.sh)
`claude -p "<T-file verbatim>\n\nWhen you are done, stop and summarise what you changed." --model claude-sonnet-5-5
--output-format stream-json --verbose --include-hook-events --max-turns 60 --permission-mode acceptEdits
--allowedTools <haiku-routing-bench list> --setting-sources project --strict-mcp-config`, cwd = `work/<run>/`, stdin /dev/null.
Gated configs get `work/<run>/.claude/settings.json`:
`{"permissions":{"deny":[...see traps...]},"hooks":{"Stop":[{"hooks":[{"type":"command","command":"bash \"$CLAUDE_PROJECT_DIR/.claude/hooks/stop.sh\"","timeout":1500}]}]}}`
and `.claude/hooks/stop.sh` = `GATE_RUN=<abs run dir> GATE_CONFIG=<cfg> bash <bench>/gate.sh` (the SPEC command, one level down: see traps).
Copies of both land in the run dir (`settings.json`, `stop.sh`).
### Hook (lib/hook.py), one invocation
1. read stdin JSON; n = lines already in `gate.jsonl` + 1; blocks so far from the same file
2. `snap-<n>.diff` = `git diff HEAD` + untracked files (index never touched)
3. hidden tests on the work copy (`TINYDB_TREE` = work copy, cwd = work copy) -> `hidden-<n>.txt`; never shown to the agent
4. gate 1 (if configured) = the repo suite `python -m pytest -q -x --no-header -p no:cacheprovider --no-cov` (300 s)
-> `gate1-<n>.txt` AND the writer's acceptance tests `runs/<run>/gate_tests/test_<T>.py` run with `TINYDB_TREE` = work
copy -> `gatetests-<n>.txt`. Block when either is red. Reason: "The acceptance tests for this change fail:" + last 40
lines (failing test names + assertion output, the gate_tests path rewritten to `acceptance/`), then the suite tail if the
suite is red too; <= 1500 chars. `tests_touched` is still logged but no longer blocks. No writer file (writer failed
twice) -> suite only, logged as `gate_tests_present: false`.
5. gate 3 (if configured and gate 1 passed or absent): see below
6. decision: allow / block / give_up (a would-be block when 6 blocks already happened -> allow, logged `give_up`).
`stop_hook_active` is logged, never used. Block = exit 2 + reason on stderr. A crash of the hook itself is
logged to `gate-errors.txt` and allows (counted as `gate_crashes` in score.json).
### Gate 1 test writer (lib/testwriter.py, run.sh before the worker, configs g1/g13/g13adv)
`claude -p` Sonnet 5.5 with the reviewer isolation flags, `--tools Read,Grep,Glob --max-turns 12 --json-schema
{"file": string}`, cwd = a pristine temp copy of upstream (deleted after). Prompt = framing + `tasks/<T>.md` + the
coordinator's instruction ("Write integration-style pytest cases for this request against the tinydb API, one test per
behaviour a careful senior would expect, pushing for edge cases beyond the happy path (missing fields, wrong types,
boundaries, persistence, cache, existing behaviour unchanged). Tests only, no implementation. Return the file content.")
+ mechanics (self-contained module, imports of the new API inside each test, MemoryStorage/tmp_path). The file must
compile and contain `def test_`, else one retry. Output `runs/<run>/gate_tests/` (+ hidden's conftest.py with
`TINYDB_TREE`, pytest.ini) and `testwriter.json` (cost, turns, attempts). Fixed per-run cost, outside the worker's wall.
### Gate 3 (lib/review.py)
Each reviewer: `claude -p "<prompt>" --model claude-sonnet-5-5 --tools Read,Grep,Glob --max-turns 15 --output-format json
--json-schema <schema> --permission-mode dontAsk --setting-sources "" --settings '{"disableAllHooks":true}' --strict-mcp-config`
with `CLAUDE_CODE_DISABLE_CLAUDE_MDS=1 CLAUDE_CODE_DISABLE_AUTO_MEMORY=1`, all inherited `CLAUDE*` vars dropped, cwd = work copy,
600 s timeout. Prompt = framing + task prompt (T-file) + diff (tracked + untracked, cap 60k chars) + the SPEC's "Find defects..."
sentence. `structured_output` is read from the JSON result; `total_cost_usd`, `num_turns`, wall and models are recorded.
- `indep`: reviewers A and B, same prompt, run in parallel, no shared context. Blocking findings = union. Pass when both approve.
- `adv`: A as above; challenger B gets A's numbered findings + the diff, answers `upheld|rejected` + reason per id, may add
findings (`added_findings`). Blocking = A's blocking findings B upheld + B's added blocking findings. B always runs (same
2-call budget as indep).
- Effective `approve` = "no blocking finding" (the raw `approve` is logged as `approve_raw`; a reviewer saying approve=false with
only minor findings does not block). A reviewer call that errors approves (fail open) with `error` recorded; an errored
challenger upholds nothing.
- Reason = "Code review found N blocking defect(s):" + numbered `file:line text` + "Fix every finding, re-run the tests, then stop.", <= 1500 chars.
### Per run (`runs/<task>-<cfg>-r<rep>/`)
`stream.jsonl`, `stderr.txt`, `exit_code.txt`, `wall_s.txt` (includes hook time: the hook runs inside the session),
`gate.jsonl`, `snap-<n>.diff`, `hidden-<n>.txt`, `gate1-<n>.txt`, `hook-input-1.json`, `post.diff`, `numstat.txt`,
`suite.txt` (full suite, no -x), `hidden.txt`, `gate_tests/`, `testwriter.json`, `gatetests-<n>.txt`, `gatetests-final.txt`, `result.json`, `score.json`, `meta.json`, `task.md`, `prompt.txt`,
`settings.json` + `stop.sh` (gated configs).
`score.json`: task, cfg, rep, hidden_pass/hidden_fail/failed_reqs (**requirement level**: 5 reqs per task), suite_pass/fail,
hook_calls, blocks_total, blocks_by_gate {g1,g3}, right_blocks / false_blocks (+ `_by_gate`), give_up, ended_blocked,
cost_worker_usd, cost_reviewers_usd, cost_testwriter_usd, cost_total_usd (all three), testwriter_ok, gate_tests_total,
gate_tests_fail_final, turns, wall_s, reviewer_findings_total, reviewer_errors, gate_crashes,
model_ok (worker init model + every modelUsage key + every reviewer model == claude-sonnet-5-5), hook_events_in_stream,
hidden_first_stop.
## Matrix
5 tasks x 5 configs x 2 reps = 50 runs. Rep 1 config blocks bare, g1, g3, g13, g13adv (tasks rotated per block), rep 2 the
reverse block order. Up to 3 runs at once, each in its own work copy (`matrix-one.sh` via `xargs -P 3`, bash 3.2 has no
`wait -n`). `matrix.log`: one `start` and one result line per run.
## Traps found
- **Gate 1 as first built passed by construction** (it only re-ran the implementer's own tests: `T2-g1-r1` one allow, hidden
R4 missed). Redesigned around an independent test writer (above). The writer hedges where the terse prompt is silent
(accepts "raises or no-op"), so it cannot recover unstated requirements; per-task overlap is in VALIDATION.md.
- **A blocked agent hunts for the tests.** After a gate-1 block the smoke agent ran `cat .claude/hooks/stop.sh; find / -name
test_T1.py` (denied only because `find` is not on the allow-list; `cat` is, and `stop.sh` names the bench dir, from where
`hidden/` is one `cat` away). Gated configs' `.claude/settings.json` now carries `permissions.deny`: Read of `./.claude/**`,
the bench's `hidden/ runs/ validation/ reference/`, the two sibling benches; Bash patterns containing `.claude/hooks`,
`.claude/settings`, `../hidden`, `../runs`, `gate_tests`, the bench sub-paths, `find *`. Probed with a Haiku session: all
five escape attempts (cat, Read, ls, python open) denied, in-tree reads fine. Bash rules match command text, not a security
boundary. Bare runs have no settings file (bare r1 ran before this change; bare has no motive to hunt).
Reviewers and the writer run with `--setting-sources ""`, so these rules do not apply to them.
- **Transient `FileNotFoundError` spawning a reviewer** inside the hook (once, both reviewers of one call, wall 0.0 s;
never reproduced by hand). Reviewer calls now retry once after 3 s; the error message records `filename`.
- **`--bare` cannot authenticate here.** It reads `ANTHROPIC_API_KEY` / `apiKeyHelper` only, never OAuth or the keychain;
this account is OAuth (`~/.claude`), so a `--bare` reviewer returns `terminal_reason: api_error`, cost 0. Used the
documented equivalents instead: `--setting-sources ""` + `--settings '{"disableAllHooks":true}'` + `--strict-mcp-config` +
`CLAUDE_CODE_DISABLE_CLAUDE_MDS=1` + `CLAUDE_CODE_DISABLE_AUTO_MEMORY=1` + `--tools Read,Grep,Glob`. The reviewers' cwd is the
work copy, whose `.claude/settings.json` carries the Stop hook: without `--setting-sources ""`/`disableAllHooks` a reviewer
would run the gate recursively. Verified: a dummy Stop hook in the probe copy never fired.
- **Inherited session env.** A matrix launched from inside Claude Code inherits `CLAUDE_EFFORT=medium`,
`CLAUDE_CODE_SUBAGENT_MODEL=opus`, `CLAUDE_CODE_SESSION_ID`, `CLAUDECODE`, `CLAUDE_CODE_CHILD_SESSION`, messaging tokens.
`env.sh` unsets every `CLAUDE*` var before setting its own; reviewers get a scrubbed env too (the hook process itself
inherits the worker's `CLAUDE_*` vars, e.g. the worker's session id).
- **The feedback line leaks the hook command.** Claude Code prefixes each block with the command line:
`Stop hook feedback:\n[<command>]: <reason>`. With the SPEC command inline the agent would read `GATE_CONFIG=g13adv` and
`gate-bench/gate.sh`. The settings command is therefore `bash "$CLAUDE_PROJECT_DIR/.claude/hooks/stop.sh"`, a one-line
wrapper holding the SPEC command. The agent can still `cat` it (it lives in the work copy) but it is not pushed at it.
- **Stop hook input keys (2.1.293):** `background_tasks, cwd, effort ({"level":"medium"}), hook_event_name,
last_assistant_message, permission_mode, prompt_id, session_crons, session_id, stop_hook_active, transcript_path`.
- **8-cap resets on tool calls.** Per the docs the consecutive-continuation count resets each time Claude calls a tool, so the
built-in cap only bites an agent that answers blocks with text alone. Our give-up (6 blocks per run, counted from
`gate.jsonl`, tool calls or not) fires first in every case.
- **Red start state: the agent refuses to fix code it did not write.** Validation step 2 (T1 + an uncommitted `>=` -> `>`
edit) shows Sonnet 5.5 implementing T1, then answering 6 consecutive gate-1 blocks with "the red test is your pre-existing
uncommitted edit, not my change" without touching it, until give_up. The hook mechanics are validated; the behaviour is a
finding in itself (gate 1 cannot force a fix the agent considers out of scope).
- **pytest-cov.** The repo's `pytest.ini` adds `--cov tinydb`; gate 1 adds `--no-cov` so the 40-line tail holds the failure,
not the coverage table (and no `.coverage` file is written). `.coverage` is gitignored anyway.
- **T5 R4 is parametrized** (several test ids, one requirement). hidden counts are per requirement (5 per task), a req
passes only when all its ids pass. `validate.sh` step 1: T5 upstream passes 1 req (R3, the regression guard) as designed.
- **Diff without touching the index.** `git add -N` would change what the agent sees in `git status`; untracked files are
diffed with `git diff --no-index /dev/null <f>` instead. `.claude/` is in `.git/info/exclude`, so the hook config never
shows in any diff or in `git status`.
- **Worker Bash denials.** The 2.1.293 permission layer denies some agent commands with "Contains brace with quote character
(expansion obfuscation)" even under acceptEdits + allow-list; agents route around it (visible as `permission_denied` events).
- **Reviewers on a correct solution still flag "blocking".** On the T1 reference patch, reviewer A marked "pull does not check
the field is a list" (not required by the task) as blocking; indep therefore blocks, the adv challenger rejected it. This is
exactly what false_blocks measures (hidden all green at that stop).
- **Shared plan meter.** Workers and reviewers bill the user's plan quota; `total_cost_usd` is the list-price equivalent the
CLI reports, not a bill. Nothing else heavy should run on the account during the matrix.
- **rtk rewrites interactive `grep`/`diff`** in the launching Claude session only; scripts here call binaries directly
(`/usr/bin/diff`/`cmp` for verification diffs).
- **CLI version:** `claude --version` = 2.1.293 (Claude Code); flags checked in `claude --help`: `--json-schema`, `--bare`, `--tools`,
`--permission-mode` (choices acceptEdits, auto, bypassPermissions, manual, dontAsk, plan), `--setting-sources`,
`--include-hook-events`. `--max-turns` is accepted though not listed in `--help`.
## Matrix launch (2026-10-08 16:33:54, after the gate-1 redesign)
`cd /Users/thomas/dev/gate-bench && SKIP_DONE=1 nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 &` -> PID 71317, 3 lanes.
The 5 bare r1 runs (terse prompts) are kept and skipped; 45 runs to go. Expected wall ~40-70 min: gated runs so far take
~30-70 s of worker wall + ~35-40 s of test writer (before the worker) + 1-35 s per hook call, more with blocks.
Then `.venv/bin/python analyze.py`.
SPEC.md
# gate-bench — SPEC (written 2026-10-08 by the /plan stage of the AIDive pipeline)
Goal: measure, on a real repo, what each of Shopify's Helix gates catches when rebuilt as a Claude Code
Stop hook, and what it costs. Output feeds a YouTube video; every number must be reproducible from this dir.
## Reuse (read-only, never modify)
- `/Users/thomas/dev/verify-loops-bench/upstream/` : tinydb @ 18d73a1 (git repo). Copy it as `upstream/` here.
- `/Users/thomas/dev/verify-loops-bench/hidden/` : `test_T1.py..test_T6.py`, `conftest.py`, `pytest.ini`. Copy as `hidden/`.
Hidden tests are the ORACLE: one `test_reqN` per requirement of the task prompt. They live OUTSIDE every work copy
and the agent never sees them. Run: `cd <tree> && python -m pytest -c <bench>/hidden/pytest.ini --rootdir <bench>/hidden <bench>/hidden/test_Tn.py -q`
(conftest puts cwd or `$TINYDB_TREE` first on sys.path). Check `tasks/README.md` there for the exact invocation and validate with `validate.sh`.
- `/Users/thomas/dev/verify-loops-bench/tasks/T1.md..T5.md` : the five task prompts, typed as a developer would. Copy as `tasks/`. T6 is NOT used.
- `/Users/thomas/dev/haiku-routing-bench/{env.sh,run.sh,extract.py,prep.sh,matrix.sh}` : patterns to copy (auth via the
user's default config dir, `CLAUDE_CODE_DISABLE_AUTO_MEMORY=1`, `.venv` with pytest, stream-json extraction of
`total_cost_usd`, `num_turns`, `usage`, `modelUsage`, fresh work copy per run, `--setting-sources project --strict-mcp-config`).
- Claude Code CLI 2.1.293 is installed. Docs that define the hook contract: https://code.claude.com/docs/en/hooks
(fetch with `curl -s https://r.jina.ai/https://code.claude.com/docs/en/hooks`): a Stop hook that exits 2 blocks the
stop and its stderr is fed back; `{"decision":"block","reason":"..."}` on stdout does the same; input JSON on stdin
carries `stop_hook_active`, `session_id`, `transcript_path`; after 8 consecutive continuations Claude Code overrides
the next block (`CLAUDE_CODE_STOP_HOOK_BLOCK_CAP` raises it); a hook past `timeout` (seconds, default 600) renders no decision.
## Worker
`claude -p "<task prompt>" --model claude-sonnet-5-5 --output-format stream-json --verbose --max-turns 60
--permission-mode acceptEdits --allowedTools "<same list as haiku-routing-bench/run.sh>" --setting-sources project
--strict-mcp-config`, cwd = fresh work copy `work/<run>/` (copy of upstream, `git` clean at HEAD). The task prompt is the
T-file verbatim plus one line appended for every config (bare included): "When you are done, stop and summarise what
you changed." No mention of gates in the prompt: the gate is the hook, not the instruction.
Work copy carries `.claude/settings.json` with the Stop hook for gated configs (none for `bare`):
`{"hooks":{"Stop":[{"hooks":[{"type":"command","command":"GATE_RUN=<abs run dir> GATE_CONFIG=<cfg> bash <bench>/gate.sh","timeout":1500}]}]}}`
(check the exact settings shape in the hooks doc; `timeout` is in seconds; verify the hook fires with a smoke run before the matrix).
## Gates (gate.sh, one script, dispatches on GATE_CONFIG)
Reads stdin JSON. Logs ONE line per invocation to `$GATE_RUN/gate.jsonl`: {ts, n (invocation index), stop_hook_active,
gate1:{ran, suite_pass, suite_fail, tests_touched (bool: any file under tests/ added or modified vs HEAD), pass},
gate3:{ran, variant, reviewers:[{id, approve, findings:[{severity, file, line, text}], cost_usd, turns, wall_s}], blocking_findings, pass},
hidden:{pass, fail, failed_reqs} (ground truth at this moment, run SILENTLY on the work copy; never shown to the agent),
decision ("block"|"allow"|"give_up"), reason (the text sent back, truncated 500), wall_s}.
Give-up: after 6 blocks in one run the hook allows and logs decision "give_up" (the video needs to count it; Claude
Code's own 8-cap would otherwise hide it). Never use stop_hook_active to allow: log it only.
Order inside one invocation: gate 1 (when the config has it) first; gate 3 only when gate 1 passed or is absent.
- Gate 1 (behaviour tests): `python -m pytest -q -x --no-header -p no:cacheprovider` on the work copy (the repo's own 226 tests plus
whatever the agent wrote). Block when red (reason = last 40 lines of pytest), OR when `tests_touched` is false
(reason: "No test covers the new behaviour. Add tests for every requirement in the task, run them, then stop.").
- Gate 3 (adversarial reviews): headless reviewers, each `claude -p --bare --model claude-sonnet-5-5 --tools "Read,Grep,Glob"
--max-turns 15 --output-format json --json-schema '<schema>' --permission-mode dontAsk` (confirm each flag exists in
`claude --help`; drop `--permission-mode dontAsk` if it errors), cwd = work copy, prompt = system framing + the task
prompt + `git diff HEAD` (tracked + untracked files, cap 60k chars) + "Find defects: requirements not met, edge cases
wrong, behaviour broken, tests that do not test the requirement. Report only real defects with file and line. approve=true
only when nothing blocks." Schema: {approve: boolean, findings: [{severity: "blocking"|"minor", file, line, text}]}.
Read `structured_output` from the JSON result; record `total_cost_usd`. Set `CLAUDE_CODE_DISABLE_AUTO_MEMORY=1`.
- variant `indep` (Shopify as written): two reviewers, same prompt, independent, no shared context; union of findings;
block if any blocking finding from either; reason = the blocking findings, numbered, "Fix every finding, re-run the
tests, then stop." Pass when BOTH approve.
- variant `adv` (reviewer + challenger, arXiv 2608.18167 shape): reviewer A as above; then challenger B gets A's
findings plus the diff and must, per finding, answer upheld|rejected with a reason, and may add findings A missed;
only upheld (or B-added) blocking findings block. Pass when nothing upheld blocks.
- Reason text is what the worker sees next turn: keep it factual, under 1500 chars.
## Configs (configs.json)
- `bare`: no hook.
- `g1`: gate 1 only.
- `g3`: gate 3 indep only.
- `g13`: gate 1 then gate 3 indep (Shopify's order).
- `g13adv`: gate 1 then gate 3 adv.
Gate 2 (UI) and gate 4 (engineer) are out of scope by design: tinydb is a library, Shopify allows skipping UI review for
logic-only changes; gate 4 is the human, measured as "what reached the human": the hidden failures left at the end.
## Per run outputs (runs/<task>-<cfg>-r<rep>/)
stream.jsonl, stderr.txt, exit_code.txt, wall_s.txt, gate.jsonl, post.diff, numstat.txt, suite.txt (final repo suite),
hidden.txt (final hidden run), result.json (worker: total_cost_usd, num_turns, usage, modelUsage, is_error, subtype,
final text), score.json: {task, cfg, rep, hidden_pass, hidden_fail, failed_reqs, suite_pass, suite_fail, blocks_total,
blocks_by_gate {g1, g3}, right_blocks (hidden failing at that moment), false_blocks (hidden all passing at that moment),
give_up, ended_blocked (the LAST gate.jsonl line is a block, i.e. Claude Code's cap or the agent ended anyway),
cost_worker_usd, cost_reviewers_usd, cost_total_usd, turns, wall_s, reviewer_findings_total, model_ok}.
Snapshot at every hook call: before gate 1 runs, `git diff HEAD` of the work copy to `$GATE_RUN/snap-<n>.diff`.
## Matrix
5 tasks (T1..T5) × 5 configs × 2 reps = 50 runs. `matrix.txt` ordered: rep 1 of everything first (config-contiguous:
bare, g1, g3, g13, g13adv, tasks rotated), then rep 2 in reverse config order. `matrix.sh` runs up to 3 runs in parallel
(each in its own work copy), appends one line per run to `matrix.log`, resumable (`SKIP_DONE=1` skips runs with score.json).
`analyze.py` → `results.json` + `results.md`: per config: hidden requirements passed / total, runs fully passing, mean
cost, mean wall, blocks per run, right vs false blocks per gate, give-ups, ended_blocked; per task the same; and the
answer to the video's questions: (1) does gate 1 alone move hidden pass rate vs bare, (2) does gate 3 indep add anything
over g1, (3) does adv beat indep, (4) false-block rate per gate, (5) cost per gate.
## Validation before the matrix (VALIDATION.md + validate.sh)
1. upstream fails every hidden test file; the reference patches in `/Users/thomas/dev/verify-loops-bench/reference/`
pass hidden + suite (reuse their validate.sh logic).
2. The hook fires: one smoke run of T1 on `g1` with a deliberately failing state (apply a patch that breaks one test)
must show a block in gate.jsonl, and the stream must show the agent continuing after "Stop hook feedback".
3. gate.sh called by hand with a fake stdin on a clean work copy returns allow, on a red copy returns exit 2 with the
pytest tail on stderr.
4. One reviewer call by hand returns valid structured_output against the schema.
5. Record the exact `claude --version`, the settings.json used, and the hook input keys seen.
## Deliverables
Everything under /Users/thomas/dev/gate-bench with README.md (layout, commands, traps found). When the matrix is launched,
run it with `nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 &` and report the PID; do not wait for it to finish.
Copy the scripts (not runs/, not work/) to /Users/thomas/dev/faceless-video-pipeline/jobs/shopify-four-gates-claude-code-20261008-1529/research/code/
with a one-line README. Report: what was validated, the smoke run's gate.jsonl, the matrix PID, expected duration.
VALIDATION.md
# Validation (2026-10-08, claude 2.1.293) — raw output in validation.log
1. **Oracle.** `./validate.sh step1`: on upstream every hidden req fails (T5: 1 passing, R3 = regression guard, as designed);
each reference patch passes its hidden file (T1-T4 5 tests, T5 6 tests = 5 reqs) and the repo's 226 tests. OK.
2. **Hook fires.** `INJECT=$PWD/inject/break.patch RUN_TAG=val2 ./run.sh g1 T1 1` (T1 + an uncommitted `>=` -> `>` break):
gate.jsonl = 7 calls, 6 `block` (g1, suite red 1 failed / 33 passed under -x) then `give_up`; the stream shows
`Stop hook feedback:` user messages followed by new assistant turns (the agent continued each time; it kept
explaining the red test is the user's pre-existing edit and never touched it). `stop_hook_active` false on call 1, true after. OK.
3. **gate.sh by hand** (`./validate.sh step3`, fake stdin): green copy (T1 reference + one new test) -> exit 0, `allow`,
empty stderr; red copy (break.patch) -> exit 2, `block`, stderr = pytest tail incl. `short test summary info`. OK.
4. **Reviewer by hand** (`./validate.sh step4`, `lib/review.py work/_val3-green T1 indep|adv`): every call returned a
schema-valid `structured_output`, cost $0.03-0.05 per pair, 10-45 s per call. OK.
5. **Recorded.** `claude --version` 2.1.293. settings.json (gated configs):
`{"hooks":{"Stop":[{"hooks":[{"type":"command","command":"bash \"$CLAUDE_PROJECT_DIR/.claude/hooks/stop.sh\"","timeout":1500}]}]}}`,
`.claude/hooks/stop.sh` = `GATE_RUN=<run dir> GATE_CONFIG=<cfg> bash /Users/thomas/dev/gate-bench/gate.sh`.
Hook input keys seen: background_tasks, cwd, effort, hook_event_name, last_assistant_message, permission_mode, prompt_id,
session_crons, session_id, stop_hook_active, transcript_path. Stream hook events in the smoke run: exactly one
`hook_started`/`hook_response` pair, `hook_name: Stop` (no user/global hooks: `--setting-sources project`).
Smoke (`runs/T1-g13adv-r1-smoke`, wrapper hook): hidden 5/5, suite 238/238, 1 hook call, gate 1 green with tests touched,
adv reviewer A approve (0 findings), challenger approve, allow. Worker $0.084 + reviewers $0.034 = $0.117, 12 turns, 66 s.
`runs/T1-g13adv-r1-smoke0` = the first smoke with the inline SPEC command (same outcome, $0.195, 69 s), superseded because the
feedback prefix would leak the command line.
## Gate 1 redesign (2026-10-08, after the coordinator's review): independent test writer
Gate 1 no longer re-runs only the implementer's own tests (it passed by construction: `T2-g1-r1` one allow, hidden R4
missed, suite green). Before the worker starts, `lib/testwriter.py` (headless Sonnet 5.5, reviewer isolation flags,
`--tools Read,Grep,Glob --max-turns 12 --json-schema {file}`, cwd = a pristine temp copy of upstream) gets ONLY
`tasks/<T>.md` and writes `runs/<run>/gate_tests/test_<T>.py` (+ hidden's conftest/pytest.ini) outside the work copy.
Gate 1 = repo suite AND those tests (`TINYDB_TREE` = work copy). `tests_touched` is logged, no longer a rule.
`./validate_writer.sh` (one writer call per task, files in `validation/writer/<T>/`; reference = verify-loops-bench patches):
| task | writer tests | pass on reference | pass on upstream | fail on reference (possible false blocks, kept) | writer $ / wall |
|---|---|---|---|---|---|
| T1 | 52 | 51 | 1 | `test_pull_on_non_list_field_raises_and_leaves_data_unchanged` | 0.099 / 39 s |
| T2 | 42 | 42 | 5 | none | 0.094 / 35 s |
| T3 | 35 | 34 | 11 | `test_rename_table_non_string_target_does_not_corrupt_database` | 0.077 / 34 s |
| T4 | 48 | 47 | 7 | `test_reverse_without_sort_by_reverses_default_order` | 0.092 / 42 s |
| T5 | 26 | 25 | 7 | `test_non_integer_size_is_rejected` | 0.086 / 30 s |
Tests passing on upstream are the "existing behaviour unchanged" guards the instruction asks for (plus a few that
accept "raises" as an outcome); every task's file fails on upstream overall. Each matrix run writes its own file, so
these counts are one sample; per-run counts land in `score.json` (`gate_tests_total`, `gate_tests_fail_final`).
Overlap with the 5 hidden requirements (judgement from test names + spot reads of the assertions):
- T1: 2/5 strict (R1 order, R5 push non-list raises). R2/R4 hedged (`try: ... except (KeyError, AttributeError, TypeError)`: a raise is accepted), R3 hedged (`count('x') < 2`, first-occurrence removal passes).
- T2: 4/5 (R1, R2, R3, R5 cache inclusive vs exclusive and other bounds); R4 hedged (accepts a TypeError).
- T3: 3/5 strict (R1 ids, R2 persistence + `tables()`, R5 stale cache / old name empty / id sequence); R3/R4 use `pytest.raises(Exception)`, not KeyError/ValueError (R4 does check both tables intact).
- T4: 4/5 (R1, R3, R4 limit 0 / negative ValueError, R5 cache not polluted + Document/doc_id); R2 hedged (checks order of docs that have the field and that none are dropped, not "missing last in both directions").
- T5: 5/5 (R1, R2 restart, R3 class/subclass default, R4 non-positive ValueError at construction, R5 close flushes).
The writer hedges exactly where the terse prompt is silent: it encodes "a raise or a no-op are both fine", so gate 1
cannot recover a requirement the prompt never states (T1 R2/R4, T4 R2).
Smoke after the redesign (`runs/T1-g13adv-r1-smoke`, with the deny rules): writer 45 tests ($0.071), first stop gate 1
green (232 suite + 45/45 gate tests), adv A+B approve, allow; hidden 2/5 (R2, R4, R5 missed). Total $0.189, 68 s.
`runs/T1-g13adv-r1-smoke-a` (first redesign smoke): stop 1 blocked by gate 1 (5 gate tests red, reason starts
"The acceptance tests for this change fail:"), the agent fixed push/pull type checks, stop 2 gate 1 green, both reviewers
errored (transient FileNotFoundError on spawn, logged, fail-open allow; one retry added since). The agent also tried
`cat .claude/hooks/stop.sh; find / -name test_T1.py` after the block (denied by the allow-list) -> deny rules added.
`runs/T1-g13adv-r1-smoke-b`: second redesign smoke, clean (reviewers ran, $0.167).
Bunu Pazartesi yap
- Kontrolü, başarısızlığı stderr'e yazarak 2 ile çıkan bir betik olarak yazın ve paketinizden daha uzun bir timeout ile Stop command hook'u olarak bağlayın.
- Kabul testlerini yalnızca istekten ayrı bir agent'a yazdırın, repo dışında saklayın ve uygulayıcının hook'u, ayarları ve bu testleri okumasını reddedin.
- Her durağı karar ve kapının kırmızı olup olmadığıyla bir dosyaya loglayın; böylece iş geçiren bir kapı görünür olur.
- Altı engelden sonra döngüyü bırakan kendi sayacınızı ekleyin. Claude Code'un üst sınırı art arda sekiz engeldir ve her tool çağrısında sıfırlanır.
- Reviewer'ları
--tools Read,Grep,Glob, boş bir settings kaynağı ve kapalı hook'larla headless çalıştırın; böylece kapıyı özyinelemeli tetikleyemezler. - Bir inceleme kapısı tutuyorsanız, bulgularını bir bar yapmak yerine geri bildirimde tavsiye olarak gönderin.
- Kapıya güvenmeden önce iki hafta boyunca haklı ve haksız engelleri sayın.
-
--bare'in yalnızca bir API anahtarı okuduğunu unutmayın: bir abonelik girişinde hata döndürür, bu yüzden reviewer'ları settings bayraklarıyla yalıtın.
Ölçümler
results.md'den: 50 run, konfigürasyon başına 10, 5 görev çarpı 2 tekrar. Gizli gereksinimler konfigürasyon başına 50 üzerinden sayılır. Engeller haklı/haksız olarak yazılır: haklı engel, gizli oracle'ın o durakta başarısız bir gereksinimi olduğu anlamına gelir; haksız engel, oracle'ın tamamen yeşil olduğu anlamına gelir.
| Konfigürasyon | Geçen gizli gereksinim | Tamamen temiz run | Run başına ortalama maliyet | Ortalama duvar saniyesi | 1. kapı engelleri haklı/haksız | 3. kapı engelleri haklı/haksız | Vazgeçmeler |
|---|---|---|---|---|---|---|---|
| bare (hook yok) | 35/50 (70%) | 3 | $0.085 | 33 | 0/0 | 0/0 | 0 |
| g1 (kabul testleri) | 42/50 (84%) | 6 | $0.201 | 60 | 3/3 | 0/0 | 0 |
| g3 (iki reviewer) | 34/50 (68%) | 3 | $0.151 | 54 | 0/0 | 1/0 | 0 |
| g13 (testler, sonra iki reviewer) | 41/50 (82%) | 5 | $0.282 | 86 | 14/1 | 2/0 | 2 |
| g13adv (testler, sonra reviewer artı critic) | 39/50 (78%) | 6 | $0.276 | 86 | 11/1 | 0/0 | 1 |
Maliyet, worker, 1. kapının writer'ı ve 3. kapının reviewer'larının toplamıdır; total_cost_usd'nin raporladığı gibi, bir fatura değil, liste fiyatı eşdeğeri. Her run yapılandırılmış modeli taşıdı. Kapılı run'lar boyunca 1. kapı agent'ı 28 kez gerçek bir gereksinim başarısızken ve 5 kez her gereksinim geçerken geri gönderdi. 3. kapı 3 kez engelledi ve hiç haksız değildi. Üç döngü altı engelde vazgeçti ve Claude Code'un kendi üst sınırı hiç devreye girmedi.
50 gereksinimden insana kalan: kapısız 15, testlerle 8, tek başına reviewer'larla 16, testler ve reviewer'larla 9, testler ve reviewer artı critic ile 11. Testlerin bir görevi oynattığı yerler: sıralı arama 10'da 5'ten 10'da 10'a, cache boyutu görevi 8'den 10'a. Push ve pull görevi 10'da 4'te kaldı, çünkü istek alandan yoksun bir dokümanla ne yapılacağını belirtmedi.
Daha ileri
- UI kapısı. Shopify'ın 2. kapısı, bir front-end reposunda ikinci bir modelin tasarım reviewer'ı olarak davranmasını sağlar s1. Repo bir kütüphane olduğu için atladık. Onu ölçmek için doğru yer, bir front-end reposunda ileride yapılacak bir test.
- Ralph döngüsü. O da bir koşul sağlanana kadar agent'ı çalışır tutar, ama çıkış agent'ın yazabileceği bir ifadede durur. Bizim çıkışımız harici: agent'ın erişemeyeceği bir betik ve bir sayaç.
- Adversarial review makalesi. Özeti değil tabloları okuyun: reviewer artı critic biçimi s5 kâğıt üzerinde token'larını hak eden biçim, ama bizim run'ımız bu repoda bir kazanç yeniden üretmedi; benimsemeden önce kendi reponuzda test edin. Özet sayfası: s23.
- Engel üst sınırı ve timeout. Bir hook'a sert bir durdurma olarak güvenmeden önce hooks referansının Stop bölümünü s2 ve hooks rehberinin sorun giderme bölümünü okuyun s19.
- Raporlamanızı karşılaştırmak için ölçülmüş bir kapı: s8.
FAQ
Bir Stop hook gerçekten Claude Code'un bitirmesini durdurabilir mi?
Çoğunlukla. 2 ile çıkmak durmayı engeller ve stderr'i geri besler s2, ama art arda sekiz devamdan sonra Claude Code bir sonraki engeli geçersiz kılar ve timeout'a uğrayan bir hook karar üretmez s2. Her durağı loglayın; böylece ne zaman iş geçirdiğini görebilirsiniz.
Test writer neden ayrı bir agent?
- kapının ilk sürümü, uygulayanın kendi testlerini yeniden çalıştırıyordu. Gizli bir gereksinim başarısızken her run'a ilk durağında izin verdi, çünkü bir agent'ın kendi testleri onun inşa ettiği şeyi test eder. Yeniden tasarlanan writer yalnızca isteği ve değişiklik öncesi repoyu görüyor.
Reviewer'lar için neden --bare kullanmıyoruz?
Hook'ları ve belleği atlar s20, ama bir abonelik girişinde yalnızca bir API anahtarı okur ve burada bir api_error döndürdü. Reviewer'ları boş bir settings kaynağı, kapalı hook'lar, katı bir MCP yapılandırması ve salt okunur tool'larla yalıttık.
Reviewer'lar iyi işi hiç engelledi mi?
50 run'ımızda hayır: 3 engel, hiçbiri yanlış değil. Bedel, geçirdikleri şeydi. İki run'da iki reviewer da eksik bir sıralama alanında aynı çökmeyi gördü ve minor olarak değerlendirdi, ve iki run da beşte sıfırla bitti.
Bu, reviewer'ların asla işe yaramadığını kanıtlıyor mu?
Hayır. Bir repo, bir worker modeli, beş görev ve hücre başına iki tekrar. Reviewer'ların burada pass rate'i oynatmadığını, test kapısının oynattığını söylüyor.