AIDiveTL;DR와 결론
- Shopify의 Helix는 모든 체크포인트를 동작 테스트, UI 리뷰, 독립적이고 컨텍스트가 격리된 리뷰어 둘, 엔지니어의 승인 순서로 네 개의 게이트에 통과시켜요 s1. 글에는 합격률이 없어요. 우리는 게이트 1, 3, 4를 실제 저장소 위의 Claude Code Stop hook으로 다시 만들고 세어 봤어요.
- 우리 매트릭스는 50번 실행, tinydb 작업 다섯 개, 구성 다섯 개, 헤드리스 Sonnet 5.5예요. 게이트가 없으면 숨은 요구사항 50개 중 35개가 통과했어요(70%). 인수 테스트 게이트만 두면 50개 중 42개(84%)였어요.
- 글에 적힌 대로의 리뷰어 둘 게이트는 합격률을 움직이지 못했어요. 단독으로 50개 중 34개, 테스트 뒤에는 50개 중 41개예요. 리뷰어와 크리틱은 50개 중 39개에 도달했어요.
- 결론은 이래요. 테스트 게이트는 남기되, 구현자가 아닌 작성자와 에이전트가 수정할 수 없는 테스트를 쓰세요. 여섯 번 중 한 번꼴로 잘못된 차단이 나온다고 예상하고, 루프에 카운터를 달아 두세요.
- 리뷰어 둘 게이트를 남긴다면 장벽이 아니라 피드백 속 조언으로 두세요.
- 한계: 저장소 하나, 워커 모델 하나, 작업 다섯 개, 칸당 반복 두 번이에요. 줄 사이의 2점 차이는 잡음이에요. 게이트 없음과 테스트 게이트 사이의 14점은 잡음이 아니에요.
출처가 말하는 것
Helix는 규칙을 스스로 밝혀요. 게이트가 실패하면 에이전트는 피드백으로 구현을 고치고 검사를 다시 돌리며, 결과가 충분히 괜찮다고 생각한다는 이유로 실패한 검사를 무시할 수 없어요 s1. 게이트 3은 독립적이고 컨텍스트가 격리된 리뷰어 에이전트 둘이고, 모든 발견 사항은 고쳐야 하며, 둘 다 승인할 때까지 루프를 반복해요 s1. 로직만 바뀌는 변경은 UI 리뷰 게이트를 그냥 건너뛰어요 s1. 그래서 CLI 형태의 라이브러리 저장소인 우리에게는 게이트 2가 없어요. 한 독립 기고문은 Helix의 근거를 사례 연구이자 1차 자료이며 측정되지 않은 것으로 분류하고, 판정자를 검증하는 것이 없고 잘못된 차단 비율도 보고되지 않았다고 지적해요 s6.
Anthropic의 가이드는 문제를 이렇게 짚어요. Claude는 작업이 끝난 것처럼 보이면 멈추고, 돌려 볼 검사가 없으면 "끝난 것처럼 보인다"가 유일한 신호예요 s3. 결정론적인 방법은 검사를 스크립트로 돌리고 통과할 때까지 턴이 끝나지 못하게 막는 Stop hook이에요 s3. 가이드는 새 서브에이전트 컨텍스트의 리뷰어가 변경 뒤의 추론이 아니라 diff와 기준만 본다고 덧붙이고 s3, 서브에이전트는 실제로 새롭고 격리된 컨텍스트 윈도우에서 시작해요 s22.
훅 계약은 이래요. Stop에서 종료 코드 2는 Claude가 멈추는 것을 막고 s2, stderr 메시지가 Claude에게 이유로 전달돼요 s2. 종료 코드 1을 포함해 다른 종료 코드는 그것만으로는 막지 않아요 s2. JSON 형식은 필수 reason이 붙은 decision: "block"이에요 s2.
게이트에는 문서에 적힌 출구가 세 개 있어요.
- 상한. stop hook이 연속으로 턴을 8번 이어 가게 하면, Claude Code는 다음 차단을 무시하고 턴을 끝내요. Claude가 도구를 부를 때마다 횟수가 초기화되고,
CLAUDE_CODE_STOP_HOOK_BLOCK_CAP으로 올릴 수 있어요 s2 s19. - 타임아웃. 명령 훅의 기본값은 600초이고, 타임아웃에 도달한 훅은 취소되어 출력이 버려지므로 대부분의 이벤트에서 판단을 내리지 않아요 s2.
- 플래그.
stop_hook_active는 Claude Code가 이미 stop hook 때문에 이어지는 중이면 true라서, 스크립트가 풀리지 않을 조건 때문에 계속 막는 일을 피할 수 있어요 s2.
한 실무자는 저렴하고 컨텍스트가 적은 감사자가 "자주 틀리고" 멀쩡한 세션을 막으니 최종 결정권을 주면 안 된다고 경고해요 s7. 라벨이 붙은 멈춤 100번에 대한 측정된 Stop 게이트는 100번 중 8번을 막았고 그중 7번이 정당했어요. 다만 라벨은 사람이 아니라 모델이 붙였다는 단서가 있어요 s8. 우리도 차단을 같은 방식으로, 정당한 것과 잘못된 것으로 나눠 보고해요.
우리가 쓴 스크립트용 플래그: --bare는 훅, 스킬, 플러그인, MCP 서버, 자동 메모리, CLAUDE.md를 건너뛰고 s20, --max-turns는 print 모드에서 에이전트 턴 수를 제한하며 s20, --json-schema는 검증된 출력을 structured_output 필드로 돌려주고 s21, --output-format json에는 실제 청구서와 다를 수 있는 클라이언트 측 추정치인 total_cost_usd가 들어 있어요 s21.
게이트 3에 대해서는 2026년 워크숍 논문이 이 모양을 측정했어요. LiveCodeBench에서 제로샷 77%, 리뷰어 하나 77%, 리뷰어 둘 75%(문제 105개), 어려운 부분집합에서는 57개 중 34개예요 s5. 논문은 리뷰어 둘의 결과가 여전히 단일 리뷰어, self-refine, 제로샷과 오차 범위 안이라고 말해요 s5. 메인 에이전트가 수정하기 전에 크리틱이 리뷰를 검증하는 적대적 형태인 리뷰어와 크리틱은 87%에 도달했어요 s5. SWE-bench Verified에서는 제로샷 71.6%에 비해 75.2%였고, 토큰은 약 4.5배가 들었어요 s5. 논문: s23.
키트
벤치는 커밋 18d73a1의 tinydb이고 통과하는 테스트는 226개예요. 개발자가 평소 치는 식으로 쓴 요청 다섯 개(push와 pull 연산, between 쿼리, 테이블 이름 변경, 정렬 검색, 쓰기 캐시 크기)에 각각 에이전트가 절대 보지 못하는 숨은 요구사항이 다섯 개씩 있어서 모두 25개예요. 워커는 헤드리스 Sonnet 5.5예요. 아래는 모두 하네스와 게이트예요. tinydb 체크아웃, 숨은 테스트, 작업 프롬프트는 포함되어 있지 않아요. 직접 쓰는 저장소에 연결하고 자신만의 오라클을 쓰세요.
게이트:
gate.sh: Stop hook 진입점으로,lib/hook.py를 실행하는 한 줄짜리 래퍼예요.gate.sh
#!/usr/bin/env bash
# Stop hook entry point (bash 3.2 OK). All logic lives in lib/hook.py.
# Called as: GATE_RUN=<abs run dir> GATE_CONFIG=<cfg> bash <bench>/gate.sh (stdin = hook input JSON)
HERE="$(cd "$(dirname "$0")" && pwd)"
exec "$HERE/.venv/bin/python" -I "$HERE/lib/hook.py"
lib/hook.py
"""Stop hook: gate 1 (behaviour tests) and/or gate 3 (adversarial review), per GATE_CONFIG.
Env: GATE_RUN (abs run dir, holds meta.json + prompt.txt), GATE_CONFIG (key of configs.json).
stdin: the Stop hook input JSON. Block = exit 2 with the reason on stderr. Allow = exit 0, no output.
Every invocation appends ONE line to $GATE_RUN/gate.jsonl. Nothing but the reason ever reaches the agent.
"""
import json
import os
import sys
import time
import traceback
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from common import BENCH, run_hidden, run_suite, tests_touched, work_diff # noqa: E402
from testwriter import run_gate_tests # noqa: E402
GIVE_UP_AFTER = 6
def tail40(out, room=1400):
t = "\n".join(out.rstrip().splitlines()[-40:])
if len(t) > room:
t = "...\n" + t[-room:]
return t
def main():
t0 = time.time()
run = os.environ["GATE_RUN"]
cfg_name = os.environ["GATE_CONFIG"]
cfg = json.load(open(os.path.join(BENCH, "configs.json")))[cfg_name]
meta = json.load(open(os.path.join(run, "meta.json")))
work, task = meta["work"], meta["task"]
raw = sys.stdin.read()
try:
hin = json.loads(raw) if raw.strip() else {}
except Exception:
hin = {"_unparsed": raw[:500]}
log_path = os.path.join(run, "gate.jsonl")
prev = []
if os.path.exists(log_path):
prev = [json.loads(l) for l in open(log_path) if l.strip()]
n = len(prev) + 1
blocks_before = sum(1 for p in prev if p.get("decision") == "block")
# Snapshot of the work at this stop, before gate 1 runs.
open(os.path.join(run, "snap-%d.diff" % n), "w").write(work_diff(work))
# Ground truth at this moment (silent).
_, hout, hp, hf, hfailed = run_hidden(work, task)
open(os.path.join(run, "hidden-%d.txt" % n), "w").write(hout)
rec = {"ts": time.strftime("%Y-%m-%dT%H:%M:%S"), "n": n, "stop_hook_active": hin.get("stop_hook_active"),
"hook_input_keys": sorted(hin.keys()),
"gate1": {"ran": False}, "gate3": {"ran": False},
"hidden": {"pass": hp, "fail": hf, "failed_reqs": hfailed}}
reason, blocked_by = None, None
if cfg["gate1"]:
# Gate 1 = the repo suite AND the acceptance tests written before the run by the independent test writer
# (run dir gate_tests/, outside the work copy). A writer failure leaves the suite alone (logged).
rc, out, sp, sf = run_suite(work)
open(os.path.join(run, "gate1-%d.txt" % n), "w").write(out)
suite_ok = rc == 0 and sf == 0
gdir = os.path.join(run, "gate_tests")
has_gt = os.path.exists(os.path.join(gdir, "test_%s.py" % task))
g = {"ran": True, "suite_pass": sp, "suite_fail": sf, "tests_touched": tests_touched(work),
"gate_tests_present": has_gt}
gt_ok, gout = True, ""
if has_gt:
grc, gout, gp, gf, gnames = run_gate_tests(gdir, task, work)
open(os.path.join(run, "gatetests-%d.txt" % n), "w").write(gout)
gt_ok = grc == 0 and gf == 0
g.update(gate_tests_pass=gp, gate_tests_fail=gf, gate_tests_failed=gnames)
g["pass"] = suite_ok and gt_ok
rec["gate1"] = g
if not g["pass"]:
blocked_by = "g1"
parts = []
if not gt_ok:
parts.append("The acceptance tests for this change fail:\n" + tail40(gout, 1400 if suite_ok else 800))
if not suite_ok:
parts.append("The repo's test suite is red (python -m pytest -q -x):\n" + tail40(out, 1400 if gt_ok else 550))
reason = "\n\n".join(parts)
if cfg["gate3"] and blocked_by is None:
from review import gate3, reason_text
g3 = gate3(work, open(os.path.join(run, "task.md")).read(), cfg["gate3"])
rec["gate3"] = g3
if not g3["pass"]:
blocked_by = "g3"
reason = reason_text(g3)
if blocked_by is None:
decision = "allow"
elif blocks_before >= GIVE_UP_AFTER:
decision = "give_up"
else:
decision = "block"
rec["blocked_by"] = blocked_by
rec["decision"] = decision
rec["reason"] = (reason or "")[:500]
rec["reason_chars"] = len(reason or "")
rec["wall_s"] = round(time.time() - t0, 1)
with open(log_path, "a") as fh:
fh.write(json.dumps(rec) + "\n")
if n == 1:
json.dump(hin, open(os.path.join(run, "hook-input-1.json"), "w"), indent=2)
if decision == "block":
sys.stderr.write(reason)
sys.exit(2)
sys.exit(0)
if __name__ == "__main__":
try:
main()
except SystemExit:
raise
except Exception:
# A crashed gate must never deadlock a run: log and allow.
run = os.environ.get("GATE_RUN", "/tmp")
with open(os.path.join(run, "gate-errors.txt"), "a") as fh:
fh.write(traceback.format_exc() + "\n")
sys.exit(0)
lib/testwriter.py
"""Gate 1 test writer: an independent headless Sonnet 5.5 that sees ONLY the task prompt (plus read-only access to a
pristine upstream copy) and writes integration-style pytest cases. The file lives OUTSIDE the work copy, in
<out dir>/gate_tests/ next to a copy of hidden/conftest.py (TINYDB_TREE) and hidden/pytest.ini.
CLI: testwriter.py <task id> <out dir> -> writes <out dir>/gate_tests/test_<T>.py and <out dir>/testwriter.json
"""
import json
import os
import shutil
import subprocess
import sys
import tempfile
import time
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from common import BENCH, HIDDEN, PY, counts, env_clean, sh # noqa: E402
MODEL = "claude-sonnet-5-5"
CALL_TIMEOUT = 600
SCHEMA = {"type": "object", "properties": {"file": {"type": "string"}}, "required": ["file"]}
FRAMING = ("You are a senior engineer writing acceptance tests for a change request to tinydb, a small Python document "
"database. You can read the CURRENT repository (before the change) with Read, Grep and Glob to learn the API; "
"you cannot run code. The change is not implemented yet: your tests must describe the requested behaviour.")
ASK = ("Write integration-style pytest cases for this request against the tinydb API, one test per behaviour a careful "
"senior would expect, pushing for edge cases beyond the happy path (missing fields, wrong types, boundaries, "
"persistence, cache, existing behaviour unchanged). Tests only, no implementation. Return the file content.\n"
"Constraints: one self-contained pytest module (field `file`); `tinydb` is importable (the tree under test is put "
"first on sys.path); do not import anything from the repo's tests/ directory or its fixtures; use "
"tinydb.storages.MemoryStorage or pytest's tmp_path for files; import the new API inside each test, not at module "
"level, so one missing name fails its tests instead of the whole module; give every test a descriptive name.")
def write(task, out_dir, attempts=2):
gt = os.path.join(out_dir, "gate_tests")
os.makedirs(gt, exist_ok=True)
shutil.copy(os.path.join(HIDDEN, "conftest.py"), gt)
shutil.copy(os.path.join(HIDDEN, "pytest.ini"), gt)
task_prompt = open(os.path.join(BENCH, "tasks", task + ".md")).read().strip()
prompt = "%s\n\n## The request (as the developer typed it)\n%s\n\n%s" % (FRAMING, task_prompt, ASK)
rec = {"task": task, "attempts": [], "cost_usd": 0.0, "ok": False, "file": None}
pristine = tempfile.mkdtemp(prefix="gb-writer-")
try:
shutil.copytree(os.path.join(BENCH, "upstream"), os.path.join(pristine, "tinydb-repo"), symlinks=True)
cwd = os.path.join(pristine, "tinydb-repo")
for i in range(attempts):
t0 = time.time()
a = {"error": None}
args = ["claude", "-p", prompt, "--model", MODEL, "--tools", "Read,Grep,Glob", "--max-turns", "12",
"--output-format", "json", "--json-schema", json.dumps(SCHEMA), "--permission-mode", "dontAsk",
"--setting-sources", "", "--settings", '{"disableAllHooks":true}', "--strict-mcp-config"]
env = env_clean({"CLAUDE_CODE_DISABLE_AUTO_MEMORY": "1", "CLAUDE_CODE_DISABLE_CLAUDE_MDS": "1"})
try:
p = subprocess.run(args, cwd=cwd, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
stdin=subprocess.DEVNULL, timeout=CALL_TIMEOUT, env=env)
r = json.loads(p.stdout.decode("utf-8", "replace"))
a.update(cost_usd=r.get("total_cost_usd") or 0.0, turns=r.get("num_turns"),
models=sorted((r.get("modelUsage") or {}).keys()), subtype=r.get("subtype"))
so = r.get("structured_output")
src = so.get("file") if isinstance(so, dict) else None
if r.get("is_error") or not src:
a["error"] = "is_error=%s subtype=%s terminal=%s" % (r.get("is_error"), r.get("subtype"),
r.get("terminal_reason"))
else:
compile(src, "test_%s.py" % task, "exec")
if "def test_" not in src:
raise ValueError("no test functions")
open(os.path.join(gt, "test_%s.py" % task), "w").write(src)
rec["ok"], rec["file"] = True, os.path.join(gt, "test_%s.py" % task)
except subprocess.TimeoutExpired:
a["error"] = "timeout after %ss" % CALL_TIMEOUT
except Exception as e:
a["error"] = "%s: %s" % (type(e).__name__, str(e)[:300])
a["wall_s"] = round(time.time() - t0, 1)
rec["attempts"].append(a)
rec["cost_usd"] += a.get("cost_usd") or 0.0
if rec["ok"]:
break
finally:
shutil.rmtree(pristine, ignore_errors=True)
rec["cost_usd"] = round(rec["cost_usd"], 6)
rec["wall_s"] = round(sum(a["wall_s"] for a in rec["attempts"]), 1)
rec["models"] = sorted({m for a in rec["attempts"] for m in a.get("models") or []})
json.dump(rec, open(os.path.join(out_dir, "testwriter.json"), "w"), indent=2)
return rec
def run_gate_tests(gate_dir, task, tree, timeout=300):
"""Generated tests on `tree` (TINYDB_TREE = tree, cwd = tree). Returns (rc, out, passed, failed, failed_names).
Paths to the gate_tests dir are rewritten to `acceptance/` so the agent never learns where the file is."""
f = os.path.join(gate_dir, "test_%s.py" % task)
rc, out = sh([PY, "-m", "pytest", "-q", "--tb=short", "-rfE", "--no-header", "-c", os.path.join(gate_dir, "pytest.ini"),
"--rootdir", gate_dir, f], tree, timeout=timeout, env=env_clean({"TINYDB_TREE": tree}))
out = out.replace(os.path.realpath(gate_dir), "acceptance").replace(gate_dir, "acceptance")
rel = os.path.relpath(gate_dir, tree)
out = out.replace(rel, "acceptance")
import re
names = sorted(set(re.findall(r"(?:FAILED|ERROR) \S*::(test_\w+)", out)))
p, fl = counts(out)
if rc != 0 and fl == 0:
fl = max(1, len(names))
return rc, out, p, fl, names
if __name__ == "__main__":
r = write(sys.argv[1], os.path.abspath(sys.argv[2]))
print(json.dumps({k: r[k] for k in ("task", "ok", "cost_usd", "wall_s", "file")}))
lib/review.py
"""Gate 3 reviewers: headless `claude -p` calls returning schema-validated JSON.
CLI (validation step 4): review.py <work dir> <task id> [indep|adv] -> prints the gate3 record as JSON
"""
import json
import subprocess
import os
import sys
import time
from concurrent.futures import ThreadPoolExecutor
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from common import BENCH, env_clean, sh, work_diff # noqa: E402
MODEL = "claude-sonnet-5-5"
DIFF_CAP = 60000
CALL_TIMEOUT = 600 # seconds per reviewer call; the hook timeout is 1500
FINDING = {"type": "object",
"properties": {"severity": {"type": "string", "enum": ["blocking", "minor"]},
"file": {"type": "string"}, "line": {"type": "integer"}, "text": {"type": "string"}},
"required": ["severity", "file", "line", "text"]}
SCHEMA_REVIEW = {"type": "object",
"properties": {"approve": {"type": "boolean"}, "findings": {"type": "array", "items": FINDING}},
"required": ["approve", "findings"]}
SCHEMA_CHALLENGE = {"type": "object",
"properties": {
"verdicts": {"type": "array", "items": {
"type": "object",
"properties": {"id": {"type": "integer"},
"verdict": {"type": "string", "enum": ["upheld", "rejected"]},
"reason": {"type": "string"}},
"required": ["id", "verdict", "reason"]}},
"added_findings": {"type": "array", "items": FINDING},
"approve": {"type": "boolean"}},
"required": ["verdicts", "added_findings", "approve"]}
FRAMING = ("You are a senior engineer doing a pre-merge code review of a change to tinydb, a small Python "
"document database. You can read the repository with Read, Grep and Glob (cwd = the repo with the "
"change applied). You cannot run code. Below: the task the developer was given, then the full diff "
"(tracked and new files).")
ASK = ("Find defects: requirements not met, edge cases wrong, behaviour broken, tests that do not test the "
"requirement. Report only real defects with file and line. approve=true only when nothing blocks.")
def capped_diff(work):
d = work_diff(work)
if len(d) > DIFF_CAP:
d = d[:DIFF_CAP] + "\n[diff truncated at %d chars]\n" % DIFF_CAP
return d
def call(work, prompt, schema, rid):
"""One headless reviewer. Never raises: an error is recorded and counts as approve (fail open)."""
# Equivalent of --bare that still authenticates with the user's OAuth login (--bare reads
# ANTHROPIC_API_KEY only): no settings sources, hooks disabled, no MCP, no CLAUDE.md, no auto-memory.
args = ["claude", "-p", prompt, "--model", MODEL, "--tools", "Read,Grep,Glob", "--max-turns", "15",
"--output-format", "json", "--json-schema", json.dumps(schema), "--permission-mode", "dontAsk",
"--setting-sources", "", "--settings", '{"disableAllHooks":true}', "--strict-mcp-config"]
env = env_clean({"CLAUDE_CODE_DISABLE_AUTO_MEMORY": "1", "CLAUDE_CODE_DISABLE_CLAUDE_MDS": "1"})
t0 = time.time()
rec = {"id": rid, "approve": True, "findings": [], "cost_usd": 0.0, "turns": None, "wall_s": None, "error": None}
try:
try:
p = subprocess.run(args, cwd=work, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.DEVNULL,
timeout=CALL_TIMEOUT, env=env)
except OSError as e1: # seen once in a smoke run: transient FileNotFoundError on spawn -> one retry
rec["retry_after"] = "%r filename=%s" % (e1, getattr(e1, "filename", None))
time.sleep(3)
p = subprocess.run(args, cwd=work, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.DEVNULL,
timeout=CALL_TIMEOUT, env=env)
out = p.stdout.decode("utf-8", "replace")
try:
r = json.loads(out)
except Exception:
r = None
if not isinstance(r, dict):
rec["error"] = "no json (rc=%s): %s %s" % (p.returncode, out[-300:], p.stderr.decode("utf-8", "replace")[-300:])
else:
rec["cost_usd"] = r.get("total_cost_usd") or 0.0
rec["turns"] = r.get("num_turns")
rec["models"] = sorted((r.get("modelUsage") or {}).keys())
so = r.get("structured_output")
if r.get("is_error") or not isinstance(so, dict):
rec["error"] = "is_error=%s subtype=%s terminal=%s structured_output=%s" % (
r.get("is_error"), r.get("subtype"), r.get("terminal_reason"), type(so).__name__)
else:
rec["raw"] = so
except subprocess.TimeoutExpired:
rec["error"] = "timeout after %ss" % CALL_TIMEOUT
except Exception as e: # pragma: no cover
rec["error"] = "exception: %r filename=%s path_head=%s" % (e, getattr(e, "filename", None), os.environ.get("PATH", "")[:200])
rec["wall_s"] = round(time.time() - t0, 1)
return rec
def norm_findings(lst):
out = []
for f in lst or []:
if isinstance(f, dict):
out.append({"severity": f.get("severity") if f.get("severity") in ("blocking", "minor") else "minor",
"file": str(f.get("file", "")), "line": f.get("line"), "text": str(f.get("text", ""))})
return out
def review_prompt(task_prompt, diff):
return "%s\n\n## Task given to the developer\n%s\n\n## Diff (git diff HEAD, new files included)\n```diff\n%s\n```\n\n%s" % (
FRAMING, task_prompt.strip(), diff, ASK)
def challenge_prompt(task_prompt, diff, a_findings):
listing = "\n".join("[%d] (%s) %s:%s %s" % (i + 1, f["severity"], f["file"], f["line"], f["text"])
for i, f in enumerate(a_findings)) or "(reviewer A reported no findings)"
return ("%s\n\nA first reviewer (A) already reviewed this diff. You are the challenger: for EACH of A's findings "
"decide whether it is a real defect (upheld) or not (rejected: wrong, not required by the task, already "
"handled, or speculative), with a one-sentence reason grounded in the code; check the code before you "
"decide. Then add any real blocking defect A missed in added_findings (file and line). approve=true only "
"when no upheld or added finding blocks.\n\n## Task given to the developer\n%s\n\n## Diff (git diff HEAD, "
"new files included)\n```diff\n%s\n```\n\n## Reviewer A's findings (id in brackets)\n%s\n\n%s" % (
FRAMING, task_prompt.strip(), diff, listing,
"Answer every id in verdicts. Report only real defects."))
def gate3(work, task_prompt, variant):
diff = capped_diff(work)
reviewers, blocking = [], []
if variant == "indep":
prompt = review_prompt(task_prompt, diff)
with ThreadPoolExecutor(max_workers=2) as ex:
recs = list(ex.map(lambda rid: call(work, prompt, SCHEMA_REVIEW, rid), ["A", "B"]))
for rec in recs:
raw = rec.pop("raw", None) or {}
rec["findings"] = norm_findings(raw.get("findings"))
rec["approve_raw"] = raw.get("approve", True) if not rec["error"] else None
rb = [f for f in rec["findings"] if f["severity"] == "blocking"]
rec["approve"] = not rb # effective verdict = no blocking finding (an errored call approves)
blocking += [dict(f, by=rec["id"]) for f in rb]
reviewers.append(rec)
elif variant == "adv":
a = call(work, review_prompt(task_prompt, diff), SCHEMA_REVIEW, "A")
raw = a.pop("raw", None) or {}
a["findings"] = norm_findings(raw.get("findings"))
a["approve_raw"] = raw.get("approve", True) if not a["error"] else None
a["approve"] = not [f for f in a["findings"] if f["severity"] == "blocking"]
b = call(work, challenge_prompt(task_prompt, diff, a["findings"]), SCHEMA_CHALLENGE, "B")
rawb = b.pop("raw", None) or {}
verdicts = {}
for v in rawb.get("verdicts") or []:
if isinstance(v, dict) and isinstance(v.get("id"), int):
verdicts[v["id"]] = {"verdict": v.get("verdict"), "reason": str(v.get("reason", ""))}
b["verdicts"] = [dict(id=k, **verdicts[k]) for k in sorted(verdicts)]
b["findings"] = norm_findings(rawb.get("added_findings"))
b["approve_raw"] = rawb.get("approve", True) if not b["error"] else None
# Only upheld findings of A block. An errored challenger upholds nothing (fail open, logged).
for i, f in enumerate(a["findings"]):
v = verdicts.get(i + 1)
f["challenge"] = v["verdict"] if v else ("error" if b["error"] else "unanswered")
if f["severity"] == "blocking" and v and v["verdict"] == "upheld":
blocking.append(dict(f, by="A", upheld_by="B"))
blocking += [dict(f, by="B") for f in b["findings"] if f["severity"] == "blocking"]
b["approve"] = not blocking
reviewers = [a, b]
else:
raise ValueError(variant)
return {"ran": True, "variant": variant, "reviewers": reviewers, "blocking_findings": len(blocking),
"blocking": blocking, "pass": not blocking,
"cost_usd": round(sum(r.get("cost_usd") or 0 for r in reviewers), 6), "diff_chars": len(diff)}
def reason_text(g3):
lines = ["Code review found %d blocking defect(s):" % g3["blocking_findings"]]
for i, f in enumerate(g3["blocking"]):
lines.append("%d. %s:%s %s" % (i + 1, f["file"], f["line"], " ".join(f["text"].split())))
tail = "Fix every finding, re-run the tests, then stop."
body = "\n".join(lines)
room = 1500 - len(tail) - 2
if len(body) > room:
body = body[:room - 4] + " ..."
return body + "\n" + tail
if __name__ == "__main__":
work, task = sys.argv[1], sys.argv[2]
variant = sys.argv[3] if len(sys.argv) > 3 else "indep"
tp = open(os.path.join(BENCH, "tasks", task + ".md")).read()
g3 = gate3(os.path.abspath(work), tp, variant)
print(json.dumps(g3, indent=2))
if not g3["pass"]:
print("--- reason ---\n" + reason_text(g3))
lib/common.py
"""Shared helpers: work-copy diff, pytest runs and parsing. Used by hook.py, review.py, finalize.py."""
import os
import re
import subprocess
BENCH = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
PY = os.path.join(BENCH, ".venv", "bin", "python")
HIDDEN = os.path.join(BENCH, "hidden")
def sh(args, cwd, timeout=None, env=None):
"""Run a command, capture stdout+stderr together. Returns (rc, text). rc=124 on timeout."""
try:
p = subprocess.run(args, cwd=cwd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
stdin=subprocess.DEVNULL, timeout=timeout, env=env)
return p.returncode, p.stdout.decode("utf-8", "replace")
except subprocess.TimeoutExpired as e:
out = (e.stdout or b"").decode("utf-8", "replace")
return 124, out + "\n[timed out after %ss]" % timeout
def work_diff(work):
"""`git diff HEAD` of tracked files plus every untracked (non-ignored) file as a new-file diff.
Never touches the index (no `git add -N`), so the agent's `git status` is unaffected."""
_, d = sh(["git", "diff", "HEAD", "--no-color", "--", ".", ":(exclude).coverage"], work)
_, others = sh(["git", "ls-files", "--others", "--exclude-standard"], work)
for f in [l for l in others.splitlines() if l.strip()]:
_, nd = sh(["git", "diff", "--no-color", "--no-index", "--", "/dev/null", f], work)
d += nd
return d
def tests_touched(work):
_, mod = sh(["git", "diff", "HEAD", "--name-only", "--", "tests"], work)
_, new = sh(["git", "ls-files", "--others", "--exclude-standard", "--", "tests"], work)
return bool(mod.strip() or new.strip())
def counts(txt):
"""(passed, failed+errors) from a pytest summary line."""
p = re.findall(r"(\d+) passed", txt)
f = re.findall(r"(\d+) failed", txt)
e = re.findall(r"(\d+) errors?\b", txt)
return (int(p[-1]) if p else 0), (int(f[-1]) if f else 0) + (int(e[-1]) if e else 0)
def env_clean(extra=None):
env = {k: v for k, v in os.environ.items() if not k.startswith("CLAUDE") and not k.startswith("GATE_")}
env["PATH"] = os.path.join(BENCH, ".venv", "bin") + os.pathsep + env.get("PATH", "/usr/bin:/bin")
env["PYTHONDONTWRITEBYTECODE"] = "1"
env.update(extra or {})
return env
def run_suite(work, timeout=300):
"""Gate 1 / final suite: the repo's own pytest run (226 tests + whatever the agent wrote).
--no-cov: the repo's pytest.ini adds --cov; the coverage table would fill the 40-line tail."""
rc, out = sh([PY, "-m", "pytest", "-q", "-x", "--no-header", "-p", "no:cacheprovider", "--no-cov"],
work, timeout=timeout, env=env_clean())
p, f = counts(out)
if rc != 0 and f == 0:
f = 1 # collection error, timeout, crash: count as red
return rc, out, p, f
def run_hidden(work, task, timeout=300):
"""Ground truth: hidden tests on the work copy (cwd = work copy, TINYDB_TREE set). Output never reaches the agent."""
rc, out = sh([PY, "-m", "pytest", "-q", "-rA", "-c", os.path.join(HIDDEN, "pytest.ini"), "--rootdir", HIDDEN,
os.path.join(HIDDEN, "test_%s.py" % task)],
work, timeout=timeout, env=env_clean({"TINYDB_TREE": work}))
# Requirement level (T5 R4 is parametrized: one req = several test ids). A req passes iff none of its
# test ids failed or errored; a run that collected nothing counts every req as failed.
reqs = sorted(set(re.findall(r"^def (test_req\d+)", open(os.path.join(HIDDEN, "test_%s.py" % task)).read(), re.M)))
failed = sorted(set(re.findall(r"(?:FAILED|ERROR) \S+::(test_req\d+)", out)))
p_tests, f_tests = counts(out)
if rc != 0 and not failed:
failed = list(reqs)
if p_tests == 0 and f_tests == 0:
failed = list(reqs)
return rc, out, len(reqs) - len(failed), len(failed), failed
lib/finalize.py
"""After the worker exits: final diff, suite, hidden run, result.json and score.json. Prints one line.
Usage: finalize.py <run dir> (re-runnable: rescoring an existing run only reads/rewrites files in it)"""
import json
import os
import re
import sys
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from common import PY, env_clean, run_hidden, sh, work_diff # noqa: E402
from testwriter import run_gate_tests # noqa: E402
MODEL = "claude-sonnet-5-5"
out = os.path.abspath(sys.argv[1])
meta = json.load(open(os.path.join(out, "meta.json")))
work, task, cfg, rep = meta["work"], meta["task"], meta["cfg"], meta["rep"]
def w(name, txt):
open(os.path.join(out, name), "w").write(txt)
def rd(name, default=""):
try:
return open(os.path.join(out, name)).read().strip()
except Exception:
return default
# Final state of the work copy
w("post.diff", work_diff(work))
_, ns = sh(["git", "diff", "HEAD", "--numstat", "--", ".", ":(exclude).coverage"], work)
_, others = sh(["git", "ls-files", "--others", "--exclude-standard"], work)
for f in [l for l in others.splitlines() if l.strip()]:
try:
ns += "%d\t0\t%s\n" % (sum(1 for _ in open(os.path.join(work, f), errors="replace")), f)
except Exception:
pass
w("numstat.txt", ns)
rc, suite = sh([PY, "-m", "pytest", "-q", "--no-header", "-p", "no:cacheprovider", "--no-cov"], work,
timeout=300, env=env_clean())
w("suite.txt", suite)
sp = int((re.findall(r"(\d+) passed", suite) or [0])[-1])
sf = int((re.findall(r"(\d+) failed", suite) or [0])[-1]) + int((re.findall(r"(\d+) errors?\b", suite) or [0])[-1])
if rc != 0 and sf == 0:
sf = 1
_, hout, hp, hf, hfailed = run_hidden(work, task)
w("hidden.txt", hout)
# Worker result line
lines = []
for raw in open(os.path.join(out, "stream.jsonl"), errors="replace") if os.path.exists(os.path.join(out, "stream.jsonl")) else []:
raw = raw.strip()
if raw.startswith("{"):
try:
lines.append(json.loads(raw))
except Exception:
pass
result = next((o for o in reversed(lines) if o.get("type") == "result"), None) or {}
init = next((o for o in lines if o.get("type") == "system" and o.get("subtype") == "init"), {})
res = {k: result.get(k) for k in ("total_cost_usd", "num_turns", "usage", "modelUsage", "is_error", "subtype",
"stop_reason", "terminal_reason", "duration_ms")}
res["final_text"] = (result.get("result") or "")[-3000:]
res["has_result_line"] = bool(result)
json.dump(res, open(os.path.join(out, "result.json"), "w"), indent=2)
# Hook events seen in the stream (which hooks ran)
hook_events = sorted({"%s:%s" % (o.get("hook_event") or o.get("hook_event_name") or "?", o.get("hook_name") or "")
for o in lines if o.get("type") == "system" and str(o.get("subtype", "")).startswith("hook")})
# Gate log
gl = [json.loads(l) for l in open(os.path.join(out, "gate.jsonl")) if l.strip()] if os.path.exists(os.path.join(out, "gate.jsonl")) else []
blocks = [g for g in gl if g.get("decision") == "block"]
by_gate = {"g1": sum(1 for g in blocks if g.get("blocked_by") == "g1"),
"g3": sum(1 for g in blocks if g.get("blocked_by") == "g3")}
right = {k: sum(1 for g in blocks if g.get("blocked_by") == k and g["hidden"]["fail"] > 0) for k in ("g1", "g3")}
false = {k: sum(1 for g in blocks if g.get("blocked_by") == k and g["hidden"]["fail"] == 0) for k in ("g1", "g3")}
rev_cost, rev_findings, rev_errors, rev_models = 0.0, 0, 0, set()
for g in gl:
for r in (g.get("gate3") or {}).get("reviewers") or []:
rev_cost += r.get("cost_usd") or 0
rev_findings += len(r.get("findings") or [])
rev_errors += 1 if r.get("error") else 0
rev_models |= set(r.get("models") or [])
# Gate-1 test writer (gated-1 configs only): cost and the final state of its tests on the work copy
tw = {}
try:
tw = json.load(open(os.path.join(out, "testwriter.json")))
except Exception:
pass
cost_tw = tw.get("cost_usd") or 0.0
rev_models |= set(tw.get("models") or [])
gt_total = gt_fail = None
gdir = os.path.join(out, "gate_tests")
if os.path.exists(os.path.join(gdir, "test_%s.py" % task)):
_, gout, gp, gf, _ = run_gate_tests(gdir, task, work)
w("gatetests-final.txt", gout)
gt_total, gt_fail = gp + gf, gf
mu = result.get("modelUsage") or {}
cw = result.get("total_cost_usd") or 0.0
s = {
"task": task, "cfg": cfg, "rep": rep, "run": meta["name"],
"hidden_pass": hp, "hidden_fail": hf, "failed_reqs": hfailed,
"suite_pass": sp, "suite_fail": sf,
"hook_calls": len(gl), "blocks_total": len(blocks), "blocks_by_gate": by_gate,
"right_blocks": sum(right.values()), "false_blocks": sum(false.values()),
"right_blocks_by_gate": right, "false_blocks_by_gate": false,
"give_up": any(g.get("decision") == "give_up" for g in gl),
"ended_blocked": bool(gl) and gl[-1].get("decision") == "block",
"cost_worker_usd": round(cw, 6), "cost_reviewers_usd": round(rev_cost, 6), "cost_testwriter_usd": round(cost_tw, 6),
"cost_total_usd": round(cw + rev_cost + cost_tw, 6),
"testwriter_ok": tw.get("ok") if tw else None, "gate_tests_total": gt_total, "gate_tests_fail_final": gt_fail,
"turns": result.get("num_turns"), "wall_s": int(rd("wall_s.txt", "0") or 0),
"reviewer_findings_total": rev_findings, "reviewer_errors": rev_errors,
"gate_crashes": rd("gate-errors.txt").count("Traceback"),
"model_ok": init.get("model") == MODEL and bool(mu) and set(mu) <= {MODEL} and rev_models <= {MODEL},
"worker_models": sorted(mu), "reviewer_models": sorted(rev_models),
"is_error": result.get("is_error"), "subtype": result.get("subtype"), "exit_code": rd("exit_code.txt"),
"has_result_line": bool(result), "claude_code_version": init.get("claude_code_version"),
"hook_events_in_stream": hook_events,
"hidden_first_stop": gl[0]["hidden"] if gl else None,
}
json.dump(s, open(os.path.join(out, "score.json"), "w"), indent=2)
print("%s: hidden %d/%d suite %d/%d blocks=%d (g1 %d, g3 %d; right %d, false %d) give_up=%s ended_blocked=%s "
"gate_tests_fail=%s/%s cost=$%.3f (worker %.3f + rev %.3f + writer %.3f) turns=%s wall=%ss model_ok=%s" % (
s["run"], hp, hp + hf, sp, sp + sf, len(blocks), by_gate["g1"], by_gate["g3"], s["right_blocks"],
s["false_blocks"], s["give_up"], s["ended_blocked"], gt_fail, gt_total, s["cost_total_usd"], cw, rev_cost, cost_tw, s["turns"],
s["wall_s"], s["model_ok"]))
.claude 설정은 HARNESS.md에 설명되어 있어요. 타임아웃 1500의 Stop 명령 훅 하나가 한 줄짜리 stop.sh를 부르는 settings.json, 그리고 훅, 설정, 숨은 테스트의 읽기를 막는 permissions.deny 목록이에요. 리뷰어와 작성자는 빈 설정 소스와 비활성화된 훅으로 돌아서 게이트가 재귀하지 않아요.
HARNESS.md
# gate-bench
What does each of Shopify's Helix gates catch when rebuilt as a Claude Code **Stop hook**, and what does it
cost? Headless Sonnet 5.5 workers implement five tinydb features (T1..T5); a Stop hook runs gate 1
(behaviour tests) and/or gate 3 (adversarial review by headless Sonnet 5.5 reviewers) every time the worker
tries to stop; hidden tests (the oracle, never visible to the agent) record the ground truth at every stop
and at the end. Built 2026-10-08 from `SPEC.md`; upstream, hidden tests, tasks and reference patches copied
from `../verify-loops-bench` (nothing there or in `../haiku-routing-bench` is modified).
## Commands (from this dir)
```bash
./validate.sh # steps 1,3,4 (+ step 2 once its run exists) -> validation.log
INJECT=$PWD/inject/break.patch RUN_TAG=val2 ./run.sh g1 T1 1 # validation step 2 (red start state)
RUN_TAG=smoke ./run.sh g13adv T1 1 # smoke (tagged runs are ignored by analyze.py)
./validate_writer.sh # gate-1 test writer on T1..T5: tests on upstream vs reference (REWRITE=1 to redo)
./gen_matrix.py # matrix.txt (50 runs, already generated)
nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 & # 3 runs in parallel (MATRIX_JOBS=3)
SKIP_DONE=1 nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 & # resume
.venv/bin/python analyze.py # results.json + results.md
.venv/bin/python lib/finalize.py runs/<run> # rescore one run from its work copy
.venv/bin/python lib/review.py work/<run> T1 adv # one gate-3 round by hand
```
Single run: `./run.sh <cfg> <task> <rep>` -> `runs/<task>-<cfg>-r<rep>/`, work copy `work/<same name>/`.
## Layout
| path | what |
|---|---|
| `SPEC.md` | the spec this harness implements |
| `upstream/` | tinydb @ 18d73a1 (git repo) |
| `hidden/` | `test_T1..T6.py` (T6 unused), `conftest.py` (`$TINYDB_TREE` or cwd first on sys.path), `pytest.ini` |
| `tasks/T1..T5.md` + `tasks/README.md` | the task prompts (verbatim) and what each hidden req checks |
| `reference/T1..T5.patch` | reference solutions (validation step 1) |
| `inject/break.patch` | `>=` -> `>` in `Query.__ge__` (one repo test red; validation steps 2-3) |
| `inject/test_pushpull.py` | one thin test, makes the T1 reference copy "green with tests touched" (step 3) |
| `configs.json` | `bare` (no hook), `g1`, `g3` (indep), `g13` (g1 then g3 indep), `g13adv` (g1 then g3 adv) |
| `env.sh` | scrubs inherited `CLAUDE*` vars, default `~/.claude` auth, `CLAUDE_CODE_DISABLE_AUTO_MEMORY=1`, venv on PATH |
| `prep.sh <cfg> <work> <run>` | fresh copy of upstream, `.claude/` in `.git/info/exclude`, hook settings for gated configs |
| `run.sh <cfg> <task> <rep>` | one run (worker + hook + `lib/finalize.py`) |
| `gate.sh` | the Stop hook (bash 3.2 wrapper: `exec .venv/bin/python -I lib/hook.py`) |
| `lib/hook.py` | gate dispatch, snapshot, hidden run, gate 1, gate 3, decision, `gate.jsonl` line |
| `lib/review.py` | gate-3 reviewers (indep / adv), schemas, prompts, reason text; CLI for a manual round |
| `lib/testwriter.py` | gate-1 test writer (before the worker) + `run_gate_tests` (path rewritten to `acceptance/`) |
| `validate_writer.sh`, `validation/writer/<T>/` | writer validation files and their runs on upstream / reference |
| `tasks/` (terse prompts, used), `tasks-explicit/` + `runs-explicit/` | the matrix uses terse prompts; see `tasks/README.md` |
| `lib/common.py` | diff incl. untracked files, suite run, hidden run (requirement level) |
| `lib/finalize.py` | post-run diff/suite/hidden, `result.json`, `score.json` |
| `gen_matrix.py`, `matrix.txt`, `matrix.sh`, `matrix-one.sh`, `matrix.log` | 50-run matrix, 3 in parallel (xargs -P), resumable |
| `analyze.py` -> `results.json`, `results.md` | per config, per task, the five questions |
| `validate.sh`, `VALIDATION.md`, `validation.log` | pre-matrix checks |
| `.venv/` | python 3.14.7, pytest 9.1.1, pytest-cov 7.1.0, pyyaml, ruff 0.16.10 (same pins as haiku-routing-bench) |
### Worker (run.sh)
`claude -p "<T-file verbatim>\n\nWhen you are done, stop and summarise what you changed." --model claude-sonnet-5-5
--output-format stream-json --verbose --include-hook-events --max-turns 60 --permission-mode acceptEdits
--allowedTools <haiku-routing-bench list> --setting-sources project --strict-mcp-config`, cwd = `work/<run>/`, stdin /dev/null.
Gated configs get `work/<run>/.claude/settings.json`:
`{"permissions":{"deny":[...see traps...]},"hooks":{"Stop":[{"hooks":[{"type":"command","command":"bash \"$CLAUDE_PROJECT_DIR/.claude/hooks/stop.sh\"","timeout":1500}]}]}}`
and `.claude/hooks/stop.sh` = `GATE_RUN=<abs run dir> GATE_CONFIG=<cfg> bash <bench>/gate.sh` (the SPEC command, one level down: see traps).
Copies of both land in the run dir (`settings.json`, `stop.sh`).
### Hook (lib/hook.py), one invocation
1. read stdin JSON; n = lines already in `gate.jsonl` + 1; blocks so far from the same file
2. `snap-<n>.diff` = `git diff HEAD` + untracked files (index never touched)
3. hidden tests on the work copy (`TINYDB_TREE` = work copy, cwd = work copy) -> `hidden-<n>.txt`; never shown to the agent
4. gate 1 (if configured) = the repo suite `python -m pytest -q -x --no-header -p no:cacheprovider --no-cov` (300 s)
-> `gate1-<n>.txt` AND the writer's acceptance tests `runs/<run>/gate_tests/test_<T>.py` run with `TINYDB_TREE` = work
copy -> `gatetests-<n>.txt`. Block when either is red. Reason: "The acceptance tests for this change fail:" + last 40
lines (failing test names + assertion output, the gate_tests path rewritten to `acceptance/`), then the suite tail if the
suite is red too; <= 1500 chars. `tests_touched` is still logged but no longer blocks. No writer file (writer failed
twice) -> suite only, logged as `gate_tests_present: false`.
5. gate 3 (if configured and gate 1 passed or absent): see below
6. decision: allow / block / give_up (a would-be block when 6 blocks already happened -> allow, logged `give_up`).
`stop_hook_active` is logged, never used. Block = exit 2 + reason on stderr. A crash of the hook itself is
logged to `gate-errors.txt` and allows (counted as `gate_crashes` in score.json).
### Gate 1 test writer (lib/testwriter.py, run.sh before the worker, configs g1/g13/g13adv)
`claude -p` Sonnet 5.5 with the reviewer isolation flags, `--tools Read,Grep,Glob --max-turns 12 --json-schema
{"file": string}`, cwd = a pristine temp copy of upstream (deleted after). Prompt = framing + `tasks/<T>.md` + the
coordinator's instruction ("Write integration-style pytest cases for this request against the tinydb API, one test per
behaviour a careful senior would expect, pushing for edge cases beyond the happy path (missing fields, wrong types,
boundaries, persistence, cache, existing behaviour unchanged). Tests only, no implementation. Return the file content.")
+ mechanics (self-contained module, imports of the new API inside each test, MemoryStorage/tmp_path). The file must
compile and contain `def test_`, else one retry. Output `runs/<run>/gate_tests/` (+ hidden's conftest.py with
`TINYDB_TREE`, pytest.ini) and `testwriter.json` (cost, turns, attempts). Fixed per-run cost, outside the worker's wall.
### Gate 3 (lib/review.py)
Each reviewer: `claude -p "<prompt>" --model claude-sonnet-5-5 --tools Read,Grep,Glob --max-turns 15 --output-format json
--json-schema <schema> --permission-mode dontAsk --setting-sources "" --settings '{"disableAllHooks":true}' --strict-mcp-config`
with `CLAUDE_CODE_DISABLE_CLAUDE_MDS=1 CLAUDE_CODE_DISABLE_AUTO_MEMORY=1`, all inherited `CLAUDE*` vars dropped, cwd = work copy,
600 s timeout. Prompt = framing + task prompt (T-file) + diff (tracked + untracked, cap 60k chars) + the SPEC's "Find defects..."
sentence. `structured_output` is read from the JSON result; `total_cost_usd`, `num_turns`, wall and models are recorded.
- `indep`: reviewers A and B, same prompt, run in parallel, no shared context. Blocking findings = union. Pass when both approve.
- `adv`: A as above; challenger B gets A's numbered findings + the diff, answers `upheld|rejected` + reason per id, may add
findings (`added_findings`). Blocking = A's blocking findings B upheld + B's added blocking findings. B always runs (same
2-call budget as indep).
- Effective `approve` = "no blocking finding" (the raw `approve` is logged as `approve_raw`; a reviewer saying approve=false with
only minor findings does not block). A reviewer call that errors approves (fail open) with `error` recorded; an errored
challenger upholds nothing.
- Reason = "Code review found N blocking defect(s):" + numbered `file:line text` + "Fix every finding, re-run the tests, then stop.", <= 1500 chars.
### Per run (`runs/<task>-<cfg>-r<rep>/`)
`stream.jsonl`, `stderr.txt`, `exit_code.txt`, `wall_s.txt` (includes hook time: the hook runs inside the session),
`gate.jsonl`, `snap-<n>.diff`, `hidden-<n>.txt`, `gate1-<n>.txt`, `hook-input-1.json`, `post.diff`, `numstat.txt`,
`suite.txt` (full suite, no -x), `hidden.txt`, `gate_tests/`, `testwriter.json`, `gatetests-<n>.txt`, `gatetests-final.txt`, `result.json`, `score.json`, `meta.json`, `task.md`, `prompt.txt`,
`settings.json` + `stop.sh` (gated configs).
`score.json`: task, cfg, rep, hidden_pass/hidden_fail/failed_reqs (**requirement level**: 5 reqs per task), suite_pass/fail,
hook_calls, blocks_total, blocks_by_gate {g1,g3}, right_blocks / false_blocks (+ `_by_gate`), give_up, ended_blocked,
cost_worker_usd, cost_reviewers_usd, cost_testwriter_usd, cost_total_usd (all three), testwriter_ok, gate_tests_total,
gate_tests_fail_final, turns, wall_s, reviewer_findings_total, reviewer_errors, gate_crashes,
model_ok (worker init model + every modelUsage key + every reviewer model == claude-sonnet-5-5), hook_events_in_stream,
hidden_first_stop.
## Matrix
5 tasks x 5 configs x 2 reps = 50 runs. Rep 1 config blocks bare, g1, g3, g13, g13adv (tasks rotated per block), rep 2 the
reverse block order. Up to 3 runs at once, each in its own work copy (`matrix-one.sh` via `xargs -P 3`, bash 3.2 has no
`wait -n`). `matrix.log`: one `start` and one result line per run.
## Traps found
- **Gate 1 as first built passed by construction** (it only re-ran the implementer's own tests: `T2-g1-r1` one allow, hidden
R4 missed). Redesigned around an independent test writer (above). The writer hedges where the terse prompt is silent
(accepts "raises or no-op"), so it cannot recover unstated requirements; per-task overlap is in VALIDATION.md.
- **A blocked agent hunts for the tests.** After a gate-1 block the smoke agent ran `cat .claude/hooks/stop.sh; find / -name
test_T1.py` (denied only because `find` is not on the allow-list; `cat` is, and `stop.sh` names the bench dir, from where
`hidden/` is one `cat` away). Gated configs' `.claude/settings.json` now carries `permissions.deny`: Read of `./.claude/**`,
the bench's `hidden/ runs/ validation/ reference/`, the two sibling benches; Bash patterns containing `.claude/hooks`,
`.claude/settings`, `../hidden`, `../runs`, `gate_tests`, the bench sub-paths, `find *`. Probed with a Haiku session: all
five escape attempts (cat, Read, ls, python open) denied, in-tree reads fine. Bash rules match command text, not a security
boundary. Bare runs have no settings file (bare r1 ran before this change; bare has no motive to hunt).
Reviewers and the writer run with `--setting-sources ""`, so these rules do not apply to them.
- **Transient `FileNotFoundError` spawning a reviewer** inside the hook (once, both reviewers of one call, wall 0.0 s;
never reproduced by hand). Reviewer calls now retry once after 3 s; the error message records `filename`.
- **`--bare` cannot authenticate here.** It reads `ANTHROPIC_API_KEY` / `apiKeyHelper` only, never OAuth or the keychain;
this account is OAuth (`~/.claude`), so a `--bare` reviewer returns `terminal_reason: api_error`, cost 0. Used the
documented equivalents instead: `--setting-sources ""` + `--settings '{"disableAllHooks":true}'` + `--strict-mcp-config` +
`CLAUDE_CODE_DISABLE_CLAUDE_MDS=1` + `CLAUDE_CODE_DISABLE_AUTO_MEMORY=1` + `--tools Read,Grep,Glob`. The reviewers' cwd is the
work copy, whose `.claude/settings.json` carries the Stop hook: without `--setting-sources ""`/`disableAllHooks` a reviewer
would run the gate recursively. Verified: a dummy Stop hook in the probe copy never fired.
- **Inherited session env.** A matrix launched from inside Claude Code inherits `CLAUDE_EFFORT=medium`,
`CLAUDE_CODE_SUBAGENT_MODEL=opus`, `CLAUDE_CODE_SESSION_ID`, `CLAUDECODE`, `CLAUDE_CODE_CHILD_SESSION`, messaging tokens.
`env.sh` unsets every `CLAUDE*` var before setting its own; reviewers get a scrubbed env too (the hook process itself
inherits the worker's `CLAUDE_*` vars, e.g. the worker's session id).
- **The feedback line leaks the hook command.** Claude Code prefixes each block with the command line:
`Stop hook feedback:\n[<command>]: <reason>`. With the SPEC command inline the agent would read `GATE_CONFIG=g13adv` and
`gate-bench/gate.sh`. The settings command is therefore `bash "$CLAUDE_PROJECT_DIR/.claude/hooks/stop.sh"`, a one-line
wrapper holding the SPEC command. The agent can still `cat` it (it lives in the work copy) but it is not pushed at it.
- **Stop hook input keys (2.1.293):** `background_tasks, cwd, effort ({"level":"medium"}), hook_event_name,
last_assistant_message, permission_mode, prompt_id, session_crons, session_id, stop_hook_active, transcript_path`.
- **8-cap resets on tool calls.** Per the docs the consecutive-continuation count resets each time Claude calls a tool, so the
built-in cap only bites an agent that answers blocks with text alone. Our give-up (6 blocks per run, counted from
`gate.jsonl`, tool calls or not) fires first in every case.
- **Red start state: the agent refuses to fix code it did not write.** Validation step 2 (T1 + an uncommitted `>=` -> `>`
edit) shows Sonnet 5.5 implementing T1, then answering 6 consecutive gate-1 blocks with "the red test is your pre-existing
uncommitted edit, not my change" without touching it, until give_up. The hook mechanics are validated; the behaviour is a
finding in itself (gate 1 cannot force a fix the agent considers out of scope).
- **pytest-cov.** The repo's `pytest.ini` adds `--cov tinydb`; gate 1 adds `--no-cov` so the 40-line tail holds the failure,
not the coverage table (and no `.coverage` file is written). `.coverage` is gitignored anyway.
- **T5 R4 is parametrized** (several test ids, one requirement). hidden counts are per requirement (5 per task), a req
passes only when all its ids pass. `validate.sh` step 1: T5 upstream passes 1 req (R3, the regression guard) as designed.
- **Diff without touching the index.** `git add -N` would change what the agent sees in `git status`; untracked files are
diffed with `git diff --no-index /dev/null <f>` instead. `.claude/` is in `.git/info/exclude`, so the hook config never
shows in any diff or in `git status`.
- **Worker Bash denials.** The 2.1.293 permission layer denies some agent commands with "Contains brace with quote character
(expansion obfuscation)" even under acceptEdits + allow-list; agents route around it (visible as `permission_denied` events).
- **Reviewers on a correct solution still flag "blocking".** On the T1 reference patch, reviewer A marked "pull does not check
the field is a list" (not required by the task) as blocking; indep therefore blocks, the adv challenger rejected it. This is
exactly what false_blocks measures (hidden all green at that stop).
- **Shared plan meter.** Workers and reviewers bill the user's plan quota; `total_cost_usd` is the list-price equivalent the
CLI reports, not a bill. Nothing else heavy should run on the account during the matrix.
- **rtk rewrites interactive `grep`/`diff`** in the launching Claude session only; scripts here call binaries directly
(`/usr/bin/diff`/`cmp` for verification diffs).
- **CLI version:** `claude --version` = 2.1.293 (Claude Code); flags checked in `claude --help`: `--json-schema`, `--bare`, `--tools`,
`--permission-mode` (choices acceptEdits, auto, bypassPermissions, manual, dontAsk, plan), `--setting-sources`,
`--include-hook-events`. `--max-turns` is accepted though not listed in `--help`.
## Matrix launch (2026-10-08 16:33:54, after the gate-1 redesign)
`cd /Users/thomas/dev/gate-bench && SKIP_DONE=1 nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 &` -> PID 71317, 3 lanes.
The 5 bare r1 runs (terse prompts) are kept and skipped; 45 runs to go. Expected wall ~40-70 min: gated runs so far take
~30-70 s of worker wall + ~35-40 s of test writer (before the worker) + 1-35 s per hook call, more with blocks.
Then `.venv/bin/python analyze.py`.
러너:
run.sh: 한 번의 실행으로, 워커와 훅과 채점이에요.run.sh
#!/usr/bin/env bash
# One run: fresh work copy, headless Sonnet 5.5 worker, Stop hook per config, final scoring.
# Usage: run.sh <cfg> <task> <rep> cfg: bare|g1|g3|g13|g13adv task: T1..T5
# RUN_TAG=x -> runs/<task>-<cfg>-r<rep>-x (smoke runs stay out of the matrix)
# INJECT=<patch> -> git apply it to the work copy before the worker starts (validation: red start state)
set -uo pipefail
CFG="$1"; TASK="$2"; REP="${3:-1}"
HERE="$(cd "$(dirname "$0")" && pwd)"
. "$HERE/env.sh"
PY="$HERE/.venv/bin/python"
MAX_TURNS="${BENCH_MAX_TURNS:-60}"
NAME="${TASK}-${CFG}-r${REP}${RUN_TAG:+-$RUN_TAG}"
OUT="$HERE/runs/$NAME"; W="$HERE/work/$NAME"
rm -rf "$OUT"; mkdir -p "$OUT"
"$HERE/prep.sh" "$CFG" "$W" "$OUT" > "$OUT/prep.txt" 2>&1 || { echo "$NAME: prep failed"; cat "$OUT/prep.txt"; exit 2; }
if [ -n "${INJECT:-}" ]; then ( cd "$W" && git apply "$INJECT" ) || { echo "$NAME: inject failed"; exit 2; }; fi
cp "$HERE/tasks/$TASK.md" "$OUT/task.md"
printf '%s\n\nWhen you are done, stop and summarise what you changed.\n' "$(cat "$HERE/tasks/$TASK.md")" > "$OUT/prompt.txt"
printf '{"task":"%s","cfg":"%s","rep":%s,"work":"%s","name":"%s"}\n' "$TASK" "$CFG" "$REP" "$W" "$NAME" > "$OUT/meta.json"
# Gate 1 configs: the independent test writer runs BEFORE the worker (fixed per-run cost), sees only tasks/<T>.md,
# writes $OUT/gate_tests/test_<T>.py outside the work copy.
if "$PY" -c "import json,sys; sys.exit(0 if json.load(open('$HERE/configs.json'))['$CFG']['gate1'] else 1)"; then
"$PY" "$HERE/lib/testwriter.py" "$TASK" "$OUT" > "$OUT/testwriter.out" 2>&1
fi
[ -f "$W/.claude/settings.json" ] && cp "$W/.claude/settings.json" "$OUT/settings.json" && cp "$W/.claude/hooks/stop.sh" "$OUT/stop.sh"
# Same allow-list as haiku-routing-bench/run.sh (verify-loops-bench's + Bash(ruff:*)).
TOOLS="Read,Edit,Write,MultiEdit,Glob,Grep,Skill,TodoWrite,Bash(uv:*),Bash(python:*),Bash(python3:*),Bash(pytest:*),Bash(ruff:*),Bash(git status:*),Bash(git diff:*),Bash(git log:*),Bash(ls:*),Bash(cat:*),Bash(mkdir:*),Bash(mktemp:*),Bash(head:*),Bash(tail:*),Bash(grep:*),Bash(wc:*)"
cd "$W"
START=$(date +%s)
claude -p "$(cat "$OUT/prompt.txt")" --model claude-sonnet-5-5 \
--output-format stream-json --verbose --include-hook-events --max-turns "$MAX_TURNS" \
--permission-mode acceptEdits --allowedTools "$TOOLS" \
--setting-sources project --strict-mcp-config \
< /dev/null > "$OUT/stream.jsonl" 2> "$OUT/stderr.txt"
echo "$?" > "$OUT/exit_code.txt"
END=$(date +%s)
echo "$((END-START))" > "$OUT/wall_s.txt"
"$PY" "$HERE/lib/finalize.py" "$OUT"
prep.sh
#!/usr/bin/env bash
# Fresh work copy of upstream tinydb (git clean at HEAD) + the Stop hook settings for gated configs.
# Usage: prep.sh <cfg> <work dir> <run dir>
set -euo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"
CFG="$1"; W="$2"; RUN="$3"
rm -rf "$W"; mkdir -p "$(dirname "$W")"
cp -R "$HERE/upstream" "$W"
cd "$W"
/usr/bin/find . -name __pycache__ -type d -prune -exec rm -rf {} +
rm -f .coverage
# .claude/ is excluded locally (not committed, not in .gitignore): `git status` stays clean,
# the hook config never shows up in the diffs the reviewers and the scorer read.
echo ".claude/" >> .git/info/exclude
git update-index -q --refresh || true
if [ "$CFG" != "bare" ]; then
# The settings command is a neutral wrapper: Claude Code prefixes every "Stop hook feedback" with the
# hook's command line, so the SPEC command (GATE_RUN=... GATE_CONFIG=<cfg> bash <bench>/gate.sh) would
# show the agent the config name and the bench path. The wrapper holds that exact command instead.
mkdir -p .claude/hooks
printf '#!/bin/bash\nGATE_RUN=%s GATE_CONFIG=%s bash %s/gate.sh\n' "$RUN" "$CFG" "$HERE" > .claude/hooks/stop.sh
"$HERE/.venv/bin/python" - "$HERE" > .claude/settings.json <<'PY'
import json
import sys
bench = sys.argv[1]
cmd = 'bash "$CLAUDE_PROJECT_DIR/.claude/hooks/stop.sh"'
# Once a gate blocks, the agent hunts for the tests behind it (seen in a smoke: `cat .claude/hooks/stop.sh;
# find / -name test_T1.py`). The oracle and the gate tests live under the bench: deny the usual forms.
# Read rules cover Read/Grep/Glob; Bash rules match the command text (not a security boundary).
deny = ["Read(./.claude/**)", "Read(/%s/hidden/**)" % bench, "Read(/%s/runs/**)" % bench,
"Read(/%s/validation/**)" % bench, "Read(/%s/reference/**)" % bench,
"Read(//Users/thomas/dev/verify-loops-bench/**)", "Read(//Users/thomas/dev/haiku-routing-bench/**)",
"Bash(*.claude/hooks*)", "Bash(*.claude/settings*)", "Bash(*gate-bench/hidden*)", "Bash(*../hidden*)",
"Bash(*gate-bench/runs*)", "Bash(*../runs*)", "Bash(*gate_tests*)", "Bash(*gate-bench/reference*)",
"Bash(*gate-bench/validation*)", "Bash(*verify-loops-bench*)", "Bash(*haiku-routing-bench*)", "Bash(find *)"]
print(json.dumps({"permissions": {"deny": deny},
"hooks": {"Stop": [{"hooks": [{"type": "command", "command": cmd, "timeout": 1500}]}]}}, indent=2))
PY
fi
test -z "$(git status --porcelain)" || { echo "work copy not clean"; git status --porcelain; exit 1; }
env.sh
# Sourced by run.sh, gate.sh and validate.sh.
# 1. Scrub every CLAUDE* variable inherited from a launching Claude Code session
# (CLAUDE_EFFORT, CLAUDE_CODE_SUBAGENT_MODEL, CLAUDE_CODE_SESSION_ID, CLAUDECODE,
# CLAUDE_CODE_CHILD_SESSION, ...): they would silently change the worker's effort,
# subagent model and session bookkeeping.
for _v in $(env | sed -n 's/^\(CLAUDE[A-Za-z0-9_]*\)=.*/\1/p'); do unset "$_v"; done
unset _v
# 2. Auth: the user's default config dir (~/.claude, OAuth). See README traps (--bare cannot auth).
unset CLAUDE_CONFIG_DIR
# 3. Fresh work copies at the same path must not share auto-memory across reps.
export CLAUDE_CODE_DISABLE_AUTO_MEMORY=1
# 4. No bytecode in the shared hidden/ dir when 3 runs execute hidden tests in parallel.
export PYTHONDONTWRITEBYTECODE=1
# 5. The bench venv first: `python -m pytest` inside the work copy uses it.
GATE_BENCH="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
export GATE_BENCH
export PATH="$GATE_BENCH/.venv/bin:$PATH"
configs.json
{
"bare": {"gate1": false, "gate3": null, "desc": "no hook"},
"g1": {"gate1": true, "gate3": null, "desc": "gate 1 (behaviour tests) only"},
"g3": {"gate1": false, "gate3": "indep", "desc": "gate 3, two independent reviewers"},
"g13": {"gate1": true, "gate3": "indep", "desc": "gate 1 then gate 3 indep (Shopify order)"},
"g13adv": {"gate1": true, "gate3": "adv", "desc": "gate 1 then gate 3 reviewer + challenger"}
}
gen_matrix.py
#!/usr/bin/env python3
"""Write matrix.txt: "<cfg> <task> <rep>" lines. Rep 1: configs bare,g1,g3,g13,g13adv as contiguous blocks,
tasks rotated per block; rep 2: same blocks in reverse config order (ABBA over time)."""
CFGS = ["bare", "g1", "g3", "g13", "g13adv"]
TASKS = ["T1", "T2", "T3", "T4", "T5"]
lines = []
for rep, order in ((1, CFGS), (2, CFGS[::-1])):
lines.append("# rep %d" % rep)
for i, c in enumerate(order):
k = (i + rep - 1) % len(TASKS)
for t in TASKS[k:] + TASKS[:k]:
lines.append("%s %s %d" % (c, t, rep))
open("matrix.txt", "w").write("\n".join(lines) + "\n")
print(sum(1 for l in lines if not l.startswith("#")), "runs")
matrix.txt
# rep 1
bare T1 1
bare T2 1
bare T3 1
bare T4 1
bare T5 1
g1 T2 1
g1 T3 1
g1 T4 1
g1 T5 1
g1 T1 1
g3 T3 1
g3 T4 1
g3 T5 1
g3 T1 1
g3 T2 1
g13 T4 1
g13 T5 1
g13 T1 1
g13 T2 1
g13 T3 1
g13adv T5 1
g13adv T1 1
g13adv T2 1
g13adv T3 1
g13adv T4 1
# rep 2
g13adv T2 2
g13adv T3 2
g13adv T4 2
g13adv T5 2
g13adv T1 2
g13 T3 2
g13 T4 2
g13 T5 2
g13 T1 2
g13 T2 2
g3 T4 2
g3 T5 2
g3 T1 2
g3 T2 2
g3 T3 2
g1 T5 2
g1 T1 2
g1 T2 2
g1 T3 2
g1 T4 2
bare T1 2
bare T2 2
bare T3 2
bare T4 2
bare T5 2
matrix.sh
#!/usr/bin/env bash
# Run an ordered list of "<cfg> <task> <rep>" lines, up to MATRIX_JOBS (default 3) at a time, each run in
# its own work copy. Blank lines and #comments skipped. Appends to matrix.log. Resumable: SKIP_DONE=1.
# Usage: nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 &
set -uo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"; cd "$HERE"
LIST="${1:-matrix.txt}"; JOBS="${MATRIX_JOBS:-3}"
mkdir "$HERE/.matrix.lock" 2>/dev/null || { echo "another matrix.sh is running (remove .matrix.lock if stale)" >&2; exit 1; }
trap 'rmdir "$HERE/.matrix.lock"' EXIT
echo "=== $(date '+%F %T') start $LIST jobs=$JOBS pid=$$" >> matrix.log
grep -vE '^[[:space:]]*(#|$)' "$LIST" | xargs -P "$JOBS" -L 1 ./matrix-one.sh
echo "=== $(date '+%F %T') done $LIST" >> matrix.log
matrix-one.sh
#!/usr/bin/env bash
# One matrix cell (called by matrix.sh through xargs). Args: <cfg> <task> <rep>
HERE="$(cd "$(dirname "$0")" && pwd)"; cd "$HERE"
CFG="$1"; TASK="$2"; REP="$3"
if [ "${SKIP_DONE:-0}" = 1 ] && [ -f "runs/${TASK}-${CFG}-r${REP}/score.json" ]; then
echo "$(date '+%F %T') skip ${TASK}-${CFG}-r${REP}" >> matrix.log; exit 0
fi
echo "$(date '+%F %T') start ${TASK}-${CFG}-r${REP}" >> matrix.log
line=$(./run.sh "$CFG" "$TASK" "$REP" < /dev/null 2>&1 | tail -1)
echo "$(date '+%F %T') $line" >> matrix.log
echo "$line"
analyze.py
#!/usr/bin/env python3
"""Aggregate runs/*/score.json (matrix runs only: <task>-<cfg>-r<rep>, no tag) -> results.json + results.md."""
import glob
import json
import os
import re
HERE = os.path.dirname(os.path.abspath(__file__))
CFGS = ["bare", "g1", "g3", "g13", "g13adv"]
TASKS = ["T1", "T2", "T3", "T4", "T5"]
rows = []
for p in sorted(glob.glob(os.path.join(HERE, "runs", "*", "score.json"))):
if re.fullmatch(r"T\d-[a-z0-9]+-r\d+", os.path.basename(os.path.dirname(p))):
rows.append(json.load(open(p)))
def mean(xs):
xs = [x for x in xs if x is not None]
return round(sum(xs) / len(xs), 4) if xs else None
def agg(rs):
if not rs:
return None
n = len(rs)
hp = sum(r["hidden_pass"] for r in rs)
ht = sum(r["hidden_pass"] + r["hidden_fail"] for r in rs)
rb = {g: sum(r["right_blocks_by_gate"][g] for r in rs) for g in ("g1", "g3")}
fb = {g: sum(r["false_blocks_by_gate"][g] for r in rs) for g in ("g1", "g3")}
return {
"runs": n, "hidden_reqs_passed": hp, "hidden_reqs_total": ht, "hidden_rate": round(hp / ht, 4) if ht else None,
"runs_fully_passing": sum(1 for r in rs if r["hidden_fail"] == 0),
"suite_green_runs": sum(1 for r in rs if r["suite_fail"] == 0),
"mean_cost_total_usd": mean([r["cost_total_usd"] for r in rs]),
"mean_cost_worker_usd": mean([r["cost_worker_usd"] for r in rs]),
"mean_cost_reviewers_usd": mean([r["cost_reviewers_usd"] for r in rs]),
"mean_cost_testwriter_usd": mean([r.get("cost_testwriter_usd") or 0 for r in rs]),
# per-gate cost: gate 1 = the test writer (its tests then run locally for free), gate 3 = the reviewers
"mean_cost_gate1_usd": mean([r.get("cost_testwriter_usd") or 0 for r in rs]),
"mean_cost_gate3_usd": mean([r["cost_reviewers_usd"] for r in rs]),
"gate_tests_total": sum(r.get("gate_tests_total") or 0 for r in rs),
"gate_tests_fail_final": sum(r.get("gate_tests_fail_final") or 0 for r in rs),
"mean_wall_s": mean([r["wall_s"] for r in rs]), "mean_turns": mean([r["turns"] for r in rs]),
"blocks_per_run": round(sum(r["blocks_total"] for r in rs) / n, 3),
"blocks_by_gate": {g: sum(r["blocks_by_gate"][g] for r in rs) for g in ("g1", "g3")},
"right_blocks": rb, "false_blocks": fb,
"false_block_rate": {g: (round(fb[g] / (rb[g] + fb[g]), 3) if rb[g] + fb[g] else None) for g in ("g1", "g3")},
"give_ups": sum(1 for r in rs if r["give_up"]), "ended_blocked": sum(1 for r in rs if r["ended_blocked"]),
"reviewer_findings": sum(r["reviewer_findings_total"] for r in rs),
"reviewer_errors": sum(r.get("reviewer_errors", 0) for r in rs),
"model_ok": all(r["model_ok"] for r in rs),
"missed_reqs": sorted(set("%s:%s" % (r["task"], q) for r in rs for q in r["failed_reqs"])),
}
by_cfg = {c: agg([r for r in rows if r["cfg"] == c]) for c in CFGS}
by_task = {t: {c: agg([r for r in rows if r["task"] == t and r["cfg"] == c]) for c in CFGS} for t in TASKS}
by_task_all = {t: agg([r for r in rows if r["task"] == t]) for t in TASKS}
def rate(c):
return (by_cfg.get(c) or {}).get("hidden_rate")
def d(a, b):
return round(a - b, 4) if a is not None and b is not None else None
answers = {
"q1_gate1_vs_bare": {"bare": rate("bare"), "g1": rate("g1"), "delta": d(rate("g1"), rate("bare"))},
"q2_gate3_over_g1": {"g1": rate("g1"), "g13": rate("g13"), "delta": d(rate("g13"), rate("g1")),
"g3_alone": rate("g3"), "g3_vs_bare": d(rate("g3"), rate("bare"))},
"q3_adv_vs_indep": {"g13": rate("g13"), "g13adv": rate("g13adv"), "delta": d(rate("g13adv"), rate("g13")),
"false_blocks_g3": {c: (by_cfg.get(c) or {}).get("false_blocks", {}).get("g3") for c in ("g13", "g13adv")}},
"q4_false_block_rate": {c: (by_cfg.get(c) or {}).get("false_block_rate") for c in CFGS},
"q4_gate1_false_blocks": {c: {"false": (by_cfg.get(c) or {}).get("false_blocks", {}).get("g1"),
"right": (by_cfg.get(c) or {}).get("right_blocks", {}).get("g1")}
for c in ("g1", "g13", "g13adv")},
"q5_cost": {c: {k: (by_cfg.get(c) or {}).get(k) for k in ("mean_cost_total_usd", "mean_cost_worker_usd",
"mean_cost_gate1_usd", "mean_cost_gate3_usd", "mean_wall_s")} for c in CFGS},
}
json.dump({"n_runs": len(rows), "by_cfg": by_cfg, "by_task": by_task, "by_task_all": by_task_all,
"answers": answers, "runs": rows}, open(os.path.join(HERE, "results.json"), "w"), indent=2)
L = ["# gate-bench results", "", "%d runs scored (matrix runs only)." % len(rows), "",
"## Per config", "",
"| cfg | runs | hidden reqs | fully passing | suite green | mean $ total (worker + g1 writer + g3 reviewers) | mean wall s | blocks/run | g1 right/false | g3 right/false | give-ups | ended blocked | model ok |",
"|---|---|---|---|---|---|---|---|---|---|---|---|---|"]
for c in CFGS:
a = by_cfg[c]
if not a:
L.append("| %s | 0 |" % c); continue
L.append("| %s | %d | %d/%d (%.0f%%) | %d | %d | %.3f (%.3f + %.3f + %.3f) | %.0f | %.2f | %d/%d | %d/%d | %d | %d | %s |" % (
c, a["runs"], a["hidden_reqs_passed"], a["hidden_reqs_total"], 100 * (a["hidden_rate"] or 0),
a["runs_fully_passing"], a["suite_green_runs"], a["mean_cost_total_usd"] or 0, a["mean_cost_worker_usd"] or 0,
a["mean_cost_testwriter_usd"] or 0, a["mean_cost_reviewers_usd"] or 0, a["mean_wall_s"] or 0, a["blocks_per_run"], a["right_blocks"]["g1"],
a["false_blocks"]["g1"], a["right_blocks"]["g3"], a["false_blocks"]["g3"], a["give_ups"], a["ended_blocked"],
a["model_ok"]))
L += ["", "## Per task (hidden reqs passed / total, per config)", "", "| task | " + " | ".join(CFGS) + " | missed reqs (all configs) |",
"|---|" + "---|" * (len(CFGS) + 1)]
for t in TASKS:
cells = []
for c in CFGS:
a = by_task[t][c]
cells.append("%d/%d, $%.2f, %.1f blk" % (a["hidden_reqs_passed"], a["hidden_reqs_total"], a["mean_cost_total_usd"] or 0,
a["blocks_per_run"]) if a else "-")
L.append("| %s | %s | %s |" % (t, " | ".join(cells), ", ".join((by_task_all[t] or {}).get("missed_reqs", []))))
L += ["", "## The video's questions", ""]
for k, v in answers.items():
L.append("- **%s**: `%s`" % (k, json.dumps(v)))
L += ["", "Gate 1 false blocks (gate 1 blocked while the hidden oracle was fully green): " + ", ".join(
"%s %s/%s" % (c, (by_cfg.get(c) or {}).get("false_blocks", {}).get("g1"),
((by_cfg.get(c) or {}).get("false_blocks", {}).get("g1") or 0) + ((by_cfg.get(c) or {}).get("right_blocks", {}).get("g1") or 0))
for c in ("g1", "g13", "g13adv")) + " (false / all g1 blocks)."]
L += ["", "Right block = the hidden oracle had at least one failing requirement at that stop; false block = the oracle "
"was fully green at that stop (the gate sent the agent back for nothing the user asked for)."]
open(os.path.join(HERE, "results.md"), "w").write("\n".join(L) + "\n")
print("\n".join(L))
results.md
# gate-bench results
50 runs scored (matrix runs only).
## Per config
| cfg | runs | hidden reqs | fully passing | suite green | mean $ total (worker + g1 writer + g3 reviewers) | mean wall s | blocks/run | g1 right/false | g3 right/false | give-ups | ended blocked | model ok |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bare | 10 | 35/50 (70%) | 3 | 10 | 0.085 (0.085 + 0.000 + 0.000) | 33 | 0.00 | 0/0 | 0/0 | 0 | 0 | True |
| g1 | 10 | 42/50 (84%) | 6 | 10 | 0.201 (0.130 + 0.071 + 0.000) | 60 | 0.60 | 3/3 | 0/0 | 0 | 0 | True |
| g3 | 10 | 34/50 (68%) | 3 | 10 | 0.151 (0.094 + 0.000 + 0.058) | 54 | 0.10 | 0/0 | 1/0 | 0 | 0 | True |
| g13 | 10 | 41/50 (82%) | 5 | 10 | 0.282 (0.153 + 0.071 + 0.057) | 86 | 1.70 | 14/1 | 2/0 | 2 | 0 | True |
| g13adv | 10 | 39/50 (78%) | 6 | 10 | 0.276 (0.157 + 0.072 + 0.048) | 86 | 1.20 | 11/1 | 0/0 | 1 | 0 | True |
## Per task (hidden reqs passed / total, per config)
| task | bare | g1 | g3 | g13 | g13adv | missed reqs (all configs) |
|---|---|---|---|---|---|---|
| T1 | 4/10, $0.08, 0.0 blk | 4/10, $0.22, 0.5 blk | 3/10, $0.16, 0.5 blk | 4/10, $0.36, 3.0 blk | 5/10, $0.31, 1.0 blk | T1:test_req2, T1:test_req3, T1:test_req4, T1:test_req5 |
| T2 | 8/10, $0.10, 0.0 blk | 8/10, $0.17, 0.0 blk | 8/10, $0.17, 0.0 blk | 8/10, $0.27, 3.0 blk | 9/10, $0.30, 3.5 blk | T2:test_req4 |
| T3 | 10/10, $0.09, 0.0 blk | 10/10, $0.21, 1.0 blk | 10/10, $0.14, 0.0 blk | 10/10, $0.26, 0.5 blk | 10/10, $0.31, 0.5 blk | |
| T4 | 5/10, $0.07, 0.0 blk | 10/10, $0.23, 0.5 blk | 5/10, $0.15, 0.0 blk | 10/10, $0.30, 1.0 blk | 5/10, $0.21, 0.0 blk | T4:test_req1, T4:test_req2, T4:test_req3, T4:test_req4, T4:test_req5 |
| T5 | 8/10, $0.08, 0.0 blk | 10/10, $0.18, 1.0 blk | 8/10, $0.12, 0.0 blk | 9/10, $0.23, 1.0 blk | 10/10, $0.25, 1.0 blk | T5:test_req4 |
## The video's questions
- **q1_gate1_vs_bare**: `{"bare": 0.7, "g1": 0.84, "delta": 0.14}`
- **q2_gate3_over_g1**: `{"g1": 0.84, "g13": 0.82, "delta": -0.02, "g3_alone": 0.68, "g3_vs_bare": -0.02}`
- **q3_adv_vs_indep**: `{"g13": 0.82, "g13adv": 0.78, "delta": -0.04, "false_blocks_g3": {"g13": 0, "g13adv": 0}}`
- **q4_false_block_rate**: `{"bare": {"g1": null, "g3": null}, "g1": {"g1": 0.5, "g3": null}, "g3": {"g1": null, "g3": 0.0}, "g13": {"g1": 0.067, "g3": 0.0}, "g13adv": {"g1": 0.083, "g3": null}}`
- **q4_gate1_false_blocks**: `{"g1": {"false": 3, "right": 3}, "g13": {"false": 1, "right": 14}, "g13adv": {"false": 1, "right": 11}}`
- **q5_cost**: `{"bare": {"mean_cost_total_usd": 0.0849, "mean_cost_worker_usd": 0.0849, "mean_cost_gate1_usd": 0.0, "mean_cost_gate3_usd": 0.0, "mean_wall_s": 32.7}, "g1": {"mean_cost_total_usd": 0.201, "mean_cost_worker_usd": 0.1302, "mean_cost_gate1_usd": 0.0709, "mean_cost_gate3_usd": 0.0, "mean_wall_s": 59.5}, "g3": {"mean_cost_total_usd": 0.1513, "mean_cost_worker_usd": 0.0938, "mean_cost_gate1_usd": 0.0, "mean_cost_gate3_usd": 0.0575, "mean_wall_s": 54.1}, "g13": {"mean_cost_total_usd": 0.2818, "mean_cost_worker_usd": 0.1534, "mean_cost_gate1_usd": 0.0712, "mean_cost_gate3_usd": 0.0573, "mean_wall_s": 86.4}, "g13adv": {"mean_cost_total_usd": 0.2763, "mean_cost_worker_usd": 0.1568, "mean_cost_gate1_usd": 0.0718, "mean_cost_gate3_usd": 0.0478, "mean_wall_s": 85.9}}`
Gate 1 false blocks (gate 1 blocked while the hidden oracle was fully green): g1 3/6, g13 1/15, g13adv 1/12 (false / all g1 blocks).
Right block = the hidden oracle had at least one failing requirement at that stop; false block = the oracle was fully green at that stop (the gate sent the agent back for nothing the user asked for).
검증과 데모:
validate.sh: 오라클이 업스트림에서 실패하는지, 훅이 작동하는지,gate.sh가 빨간 복사본을 막는지, 리뷰어가 유효한 JSON을 돌려주는지 확인해요.validate.sh
#!/usr/bin/env bash
# Validation before the matrix (SPEC § Validation). Steps 1, 3, 4 here; step 2 = the g1 red-start smoke run:
# INJECT=$PWD/inject/break.patch RUN_TAG=val2 ./run.sh g1 T1 1 (then: ./validate.sh step2)
# Usage: validate.sh [all|step1|step2|step3|step4] -> appends to validation.log
set -u
HERE="$(cd "$(dirname "$0")" && pwd)"; cd "$HERE"
. "$HERE/env.sh"
PY="$HERE/.venv/bin/python"
STEP="${1:-all}"
status=0
count() { grep -Eo "[0-9]+ $2" <<<"$1" | tail -1 | awk '{print $1}'; }
log() { echo "$*" | tee -a validation.log; }
log "=== $(date '+%F %T') validate.sh $STEP ($(claude --version))"
if [ "$STEP" = all ] || [ "$STEP" = step1 ]; then
# 1. upstream fails the hidden tests (<= 1 req passing: T5 R3 is a regression guard); reference passes hidden + suite
TMP="$(mktemp -d "${TMPDIR:-/tmp}/gb-validate.XXXXXX")"
for t in T1 T2 T3 T4 T5; do
tree="$TMP/$t"; git clone -q "$HERE/upstream" "$tree"
v=$(cd "$tree" && TINYDB_TREE="$tree" "$PY" -m pytest -v -c "$HERE/hidden/pytest.ini" --rootdir "$HERE/hidden" "$HERE/hidden/test_$t.py" 2>&1)
up_pass=$(grep -Eo "test_req[0-9]+[^ ]* PASSED" <<<"$v" | wc -l | tr -d ' ')
(cd "$tree" && git apply "$HERE/reference/$t.patch") || { log "$t: reference patch does not apply"; status=1; continue; }
h=$(cd "$tree" && TINYDB_TREE="$tree" "$PY" -m pytest -q -c "$HERE/hidden/pytest.ini" --rootdir "$HERE/hidden" "$HERE/hidden/test_$t.py" 2>&1); hrc=$?
s=$(cd "$tree" && "$PY" -m pytest -q -p no:cacheprovider --no-cov 2>&1); src=$?
ok=OK; { [ "$up_pass" -gt 1 ] || [ $hrc -ne 0 ] || [ $src -ne 0 ]; } && { ok=FAIL; status=1; }
log "step1 $t: upstream reqs passing=$up_pass reference hidden rc=$hrc ($(count "$h" passed) passed) suite rc=$src ($(count "$s" passed) passed) [$ok]"
done
rm -rf "$TMP"
fi
if [ "$STEP" = all ] || [ "$STEP" = step3 ]; then
# 3. gate.sh by hand, fake stdin. green = T1 reference + one new test (g1 passes) -> exit 0 allow;
# red = break.patch (one repo test fails) -> exit 2, pytest tail on stderr.
FAKE='{"session_id":"validate","transcript_path":"/dev/null","cwd":"x","hook_event_name":"Stop","stop_hook_active":false}'
for k in green red; do
R="$HERE/runs/_val3-$k"; W="$HERE/work/_val3-$k"; rm -rf "$R"; mkdir -p "$R"
./prep.sh g1 "$W" "$R" > /dev/null
if [ $k = green ]; then
(cd "$W" && git apply "$HERE/reference/T1.patch" && cp "$HERE/inject/test_pushpull.py" tests/)
else
(cd "$W" && git apply "$HERE/inject/break.patch")
fi
printf '{"task":"T1","cfg":"g1","rep":0,"work":"%s","name":"_val3-%s"}\n' "$W" "$k" > "$R/meta.json"; cp tasks/T1.md "$R/task.md"
echo "$FAKE" | GATE_RUN="$R" GATE_CONFIG=g1 bash gate.sh > "$R/stdout.txt" 2> "$R/stderr.txt"; rc=$?
dec=$("$PY" -c "import json;print(json.loads(open('$R/gate.jsonl').readline())['decision'])")
want=0; [ $k = red ] && want=2
ok=OK; [ $rc -ne $want ] && { ok=FAIL; status=1; }
[ $k = red ] && ! grep -q "short test summary" "$R/stderr.txt" && { ok=FAIL; status=1; }
[ $k = green ] && [ -s "$R/stderr.txt" ] && { ok=FAIL; status=1; }
log "step3 $k: exit=$rc decision=$dec stderr_bytes=$(wc -c < "$R/stderr.txt" | tr -d ' ') [$ok]"
done
fi
if [ "$STEP" = all ] || [ "$STEP" = step4 ]; then
# 4. one reviewer pair by hand on the green copy (needs step3's work/_val3-green): structured_output valid
W="$HERE/work/_val3-green"
[ -d "$W" ] || { log "step4: run step3 first"; exit 1; }
for v in indep adv; do
"$PY" lib/review.py "$W" T1 $v > "runs/_val3-green/review-$v.txt" 2>&1
s=$("$PY" - "runs/_val3-green/review-$v.txt" <<'PY'
import json, sys
t = open(sys.argv[1]).read(); i = t.find("--- reason"); d = json.loads(t[:i] if i > 0 else t)
ok = all(r["error"] is None and isinstance(r["approve_raw"], bool) for r in d["reviewers"])
ok = ok and all(f["severity"] in ("blocking", "minor") and isinstance(f["line"], int) for r in d["reviewers"] for f in r["findings"])
print("%s pass=%s blocking=%d findings=%s cost=$%.4f wall=%s" % ("OK" if ok else "FAIL", d["pass"], d["blocking_findings"],
[len(r["findings"]) for r in d["reviewers"]], d["cost_usd"], [r["wall_s"] for r in d["reviewers"]]))
PY
)
case "$s" in OK*) ;; *) status=1 ;; esac
log "step4 $v: $s"
done
fi
if [ "$STEP" = all ] || [ "$STEP" = step2 ]; then
# 2. hook fires on a red start: runs/T1-g1-r1-val2 must log a block and the stream must show the agent continuing
R="$HERE/runs/T1-g1-r1-val2"
if [ -f "$R/gate.jsonl" ]; then
s=$("$PY" - "$R" <<'PY'
import json, sys, os
r = sys.argv[1]
g = [json.loads(l) for l in open(os.path.join(r, "gate.jsonl")) if l.strip()]
st = [json.loads(l) for l in open(os.path.join(r, "stream.jsonl")) if l.startswith("{")]
blocks = [x for x in g if x["decision"] == "block"]
# index in the stream of the first hook response that blocked, then any assistant tool_use after it
txt = [json.dumps(o) for o in st]
fb = next((i for i, t in enumerate(txt) if "Stop hook feedback" in t or ('"hook_event": "Stop"' in t and '"exit_code": 2' in t)), None)
after = fb is not None and any(o.get("type") == "assistant" for o in st[fb + 1:])
print("%s blocks=%d hook_calls=%d feedback_idx=%s assistant_after=%s last=%s" % (
"OK" if blocks and after else "FAIL", len(blocks), len(g), fb, after, g[-1]["decision"] if g else None))
PY
)
case "$s" in OK*) ;; *) status=1 ;; esac
log "step2: $s"
else
log "step2: not run yet (INJECT=\$PWD/inject/break.patch RUN_TAG=val2 ./run.sh g1 T1 1)"
fi
fi
[ $status -eq 0 ] && log "VALIDATION OK" || log "VALIDATION FAILED"
exit $status
validate_writer.sh
#!/usr/bin/env bash
# Validation of the gate-1 test writer: one writer call per task (in parallel) into validation/writer/<T>/,
# then its tests on upstream and on the reference patch. Usage: validate_writer.sh [T1 ...] (reuses an existing
# file unless REWRITE=1). Appends to validation.log.
set -u
HERE="$(cd "$(dirname "$0")" && pwd)"; cd "$HERE"
. "$HERE/env.sh"
PY="$HERE/.venv/bin/python"
TASKS="${*:-T1 T2 T3 T4 T5}"
for t in $TASKS; do
D="$HERE/validation/writer/$t"; mkdir -p "$D"
if [ "${REWRITE:-0}" = 1 ] || [ ! -f "$D/gate_tests/test_$t.py" ]; then "$PY" lib/testwriter.py "$t" "$D" > "$D/writer.out" 2>&1 & fi
done
wait
TMP="$(mktemp -d "${TMPDIR:-/tmp}/gb-vw.XXXXXX")"; trap 'rm -rf "$TMP"' EXIT
echo "=== $(date '+%F %T') validate_writer.sh $TASKS" >> validation.log
for t in $TASKS; do
D="$HERE/validation/writer/$t"
git clone -q "$HERE/upstream" "$TMP/up-$t"; git clone -q "$HERE/upstream" "$TMP/ref-$t"
(cd "$TMP/ref-$t" && git apply "$HERE/reference/$t.patch")
line=$("$PY" - "$D" "$t" "$TMP/up-$t" "$TMP/ref-$t" <<'PY'
import json, os, sys
sys.path.insert(0, "lib")
from testwriter import run_gate_tests
d, t, up, ref = sys.argv[1:5]
w = json.load(open(os.path.join(d, "testwriter.json")))
if not w["ok"]:
print("%s: writer FAILED %s" % (t, [a["error"] for a in w["attempts"]])); sys.exit()
g = os.path.join(d, "gate_tests")
_, uo, up_p, up_f, _ = run_gate_tests(g, t, up)
_, ro, rp, rf, rnames = run_gate_tests(g, t, ref)
open(os.path.join(d, "on-upstream.txt"), "w").write(uo); open(os.path.join(d, "on-reference.txt"), "w").write(ro)
print("%s: writer tests: %d, pass on reference: %d, pass on upstream: %d | fail on reference: %s | writer $%.3f %ss turns=%s" % (
t, rp + rf, rp, up_p, rnames, w["cost_usd"], w["wall_s"], [a.get("turns") for a in w["attempts"]]))
PY
)
echo "$line" | tee -a validation.log
done
demo.sh
#!/usr/bin/env bash
# Interactive demo of the gate for a screen recording: a gated tinydb work copy (config g13: acceptance
# tests, then two reviewers) with T1's acceptance tests already written. Usage: ./demo.sh [T1..T5]
# Then: cd work/demo && claude --setting-sources project --permission-mode acceptEdits
# and paste the request printed below. When Claude stops, the Stop hook runs; a red acceptance test shows
# as "Stop hook feedback: ... The acceptance tests for this change fail:" and Claude keeps working.
set -euo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"; TASK="${1:-T1}"
. "$HERE/env.sh"
RUN="$HERE/runs/demo"; W="$HERE/work/demo"
rm -rf "$RUN"; mkdir -p "$RUN"
"$HERE/prep.sh" g13 "$W" "$RUN" > "$RUN/prep.txt" 2>&1
printf '{"task":"%s","cfg":"g13","rep":0,"work":"%s","name":"demo"}\n' "$TASK" "$W" > "$RUN/meta.json"
cp "$HERE/tasks/$TASK.md" "$RUN/task.md"
printf '%s\n\nWhen you are done, stop and summarise what you changed.\n' "$(cat "$HERE/tasks/$TASK.md")" > "$RUN/prompt.txt"
# Reuse the acceptance tests of a matrix run of the same task (no new writer call).
SRC=$(ls -d "$HERE"/runs/"$TASK"-g13-r1/gate_tests 2>/dev/null | head -1)
cp -R "$SRC" "$RUN/gate_tests"
echo "ready: cd $W && claude --setting-sources project --permission-mode acceptEdits"
echo "--- paste this request ---"; cat "$RUN/prompt.txt"
inject/break.patch
diff --git a/tinydb/queries.py b/tinydb/queries.py
index cdab9fb..db00a2a 100644
--- a/tinydb/queries.py
+++ b/tinydb/queries.py
@@ -322,7 +322,7 @@ class Query(QueryInstance):
:param rhs: The value to compare against
"""
return self._generate_test(
- lambda value: value >= rhs,
+ lambda value: value > rhs,
('>=', self._path, rhs)
)
inject/test_pushpull.py
from tinydb import TinyDB, Query
from tinydb.storages import MemoryStorage
from tinydb.operations import push, pull
def test_push_pull():
db = TinyDB(storage=MemoryStorage)
db.insert({'n': 1, 't': ['a']})
db.update(push('t', 'b'), Query().n == 1)
db.update(pull('t', 'a'), Query().n == 1)
assert db.all()[0]['t'] == ['b']
inject/test_pushpull.py on GitHub
문서:
README.md: 구성, 명령, 하네스가 무엇인지 설명해요.README.md
# gate-bench
What does each of Shopify's Helix gates catch when rebuilt as a Claude Code **Stop hook**, and what does it
cost? Headless Sonnet 5.5 workers implement five tinydb features (T1..T5); a Stop hook runs gate 1
(behaviour tests) and/or gate 3 (adversarial review by headless Sonnet 5.5 reviewers) every time the worker
tries to stop; hidden tests (the oracle, never visible to the agent) record the ground truth at every stop
and at the end. Built 2026-10-08 from `SPEC.md`; upstream, hidden tests, tasks and reference patches copied
from `../verify-loops-bench` (nothing there or in `../haiku-routing-bench` is modified).
## Commands (from this dir)
```bash
./validate.sh # steps 1,3,4 (+ step 2 once its run exists) -> validation.log
INJECT=$PWD/inject/break.patch RUN_TAG=val2 ./run.sh g1 T1 1 # validation step 2 (red start state)
RUN_TAG=smoke ./run.sh g13adv T1 1 # smoke (tagged runs are ignored by analyze.py)
./validate_writer.sh # gate-1 test writer on T1..T5: tests on upstream vs reference (REWRITE=1 to redo)
./gen_matrix.py # matrix.txt (50 runs, already generated)
nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 & # 3 runs in parallel (MATRIX_JOBS=3)
SKIP_DONE=1 nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 & # resume
.venv/bin/python analyze.py # results.json + results.md
.venv/bin/python lib/finalize.py runs/<run> # rescore one run from its work copy
.venv/bin/python lib/review.py work/<run> T1 adv # one gate-3 round by hand
```
Single run: `./run.sh <cfg> <task> <rep>` -> `runs/<task>-<cfg>-r<rep>/`, work copy `work/<same name>/`.
## Layout
| path | what |
|---|---|
| `SPEC.md` | the spec this harness implements |
| `upstream/` | tinydb @ 18d73a1 (git repo) |
| `hidden/` | `test_T1..T6.py` (T6 unused), `conftest.py` (`$TINYDB_TREE` or cwd first on sys.path), `pytest.ini` |
| `tasks/T1..T5.md` + `tasks/README.md` | the task prompts (verbatim) and what each hidden req checks |
| `reference/T1..T5.patch` | reference solutions (validation step 1) |
| `inject/break.patch` | `>=` -> `>` in `Query.__ge__` (one repo test red; validation steps 2-3) |
| `inject/test_pushpull.py` | one thin test, makes the T1 reference copy "green with tests touched" (step 3) |
| `configs.json` | `bare` (no hook), `g1`, `g3` (indep), `g13` (g1 then g3 indep), `g13adv` (g1 then g3 adv) |
| `env.sh` | scrubs inherited `CLAUDE*` vars, default `~/.claude` auth, `CLAUDE_CODE_DISABLE_AUTO_MEMORY=1`, venv on PATH |
| `prep.sh <cfg> <work> <run>` | fresh copy of upstream, `.claude/` in `.git/info/exclude`, hook settings for gated configs |
| `run.sh <cfg> <task> <rep>` | one run (worker + hook + `lib/finalize.py`) |
| `gate.sh` | the Stop hook (bash 3.2 wrapper: `exec .venv/bin/python -I lib/hook.py`) |
| `lib/hook.py` | gate dispatch, snapshot, hidden run, gate 1, gate 3, decision, `gate.jsonl` line |
| `lib/review.py` | gate-3 reviewers (indep / adv), schemas, prompts, reason text; CLI for a manual round |
| `lib/testwriter.py` | gate-1 test writer (before the worker) + `run_gate_tests` (path rewritten to `acceptance/`) |
| `validate_writer.sh`, `validation/writer/<T>/` | writer validation files and their runs on upstream / reference |
| `tasks/` (terse prompts, used), `tasks-explicit/` + `runs-explicit/` | the matrix uses terse prompts; see `tasks/README.md` |
| `lib/common.py` | diff incl. untracked files, suite run, hidden run (requirement level) |
| `lib/finalize.py` | post-run diff/suite/hidden, `result.json`, `score.json` |
| `gen_matrix.py`, `matrix.txt`, `matrix.sh`, `matrix-one.sh`, `matrix.log` | 50-run matrix, 3 in parallel (xargs -P), resumable |
| `analyze.py` -> `results.json`, `results.md` | per config, per task, the five questions |
| `validate.sh`, `VALIDATION.md`, `validation.log` | pre-matrix checks |
| `.venv/` | python 3.14.7, pytest 9.1.1, pytest-cov 7.1.0, pyyaml, ruff 0.16.10 (same pins as haiku-routing-bench) |
### Worker (run.sh)
`claude -p "<T-file verbatim>\n\nWhen you are done, stop and summarise what you changed." --model claude-sonnet-5-5
--output-format stream-json --verbose --include-hook-events --max-turns 60 --permission-mode acceptEdits
--allowedTools <haiku-routing-bench list> --setting-sources project --strict-mcp-config`, cwd = `work/<run>/`, stdin /dev/null.
Gated configs get `work/<run>/.claude/settings.json`:
`{"permissions":{"deny":[...see traps...]},"hooks":{"Stop":[{"hooks":[{"type":"command","command":"bash \"$CLAUDE_PROJECT_DIR/.claude/hooks/stop.sh\"","timeout":1500}]}]}}`
and `.claude/hooks/stop.sh` = `GATE_RUN=<abs run dir> GATE_CONFIG=<cfg> bash <bench>/gate.sh` (the SPEC command, one level down: see traps).
Copies of both land in the run dir (`settings.json`, `stop.sh`).
### Hook (lib/hook.py), one invocation
1. read stdin JSON; n = lines already in `gate.jsonl` + 1; blocks so far from the same file
2. `snap-<n>.diff` = `git diff HEAD` + untracked files (index never touched)
3. hidden tests on the work copy (`TINYDB_TREE` = work copy, cwd = work copy) -> `hidden-<n>.txt`; never shown to the agent
4. gate 1 (if configured) = the repo suite `python -m pytest -q -x --no-header -p no:cacheprovider --no-cov` (300 s)
-> `gate1-<n>.txt` AND the writer's acceptance tests `runs/<run>/gate_tests/test_<T>.py` run with `TINYDB_TREE` = work
copy -> `gatetests-<n>.txt`. Block when either is red. Reason: "The acceptance tests for this change fail:" + last 40
lines (failing test names + assertion output, the gate_tests path rewritten to `acceptance/`), then the suite tail if the
suite is red too; <= 1500 chars. `tests_touched` is still logged but no longer blocks. No writer file (writer failed
twice) -> suite only, logged as `gate_tests_present: false`.
5. gate 3 (if configured and gate 1 passed or absent): see below
6. decision: allow / block / give_up (a would-be block when 6 blocks already happened -> allow, logged `give_up`).
`stop_hook_active` is logged, never used. Block = exit 2 + reason on stderr. A crash of the hook itself is
logged to `gate-errors.txt` and allows (counted as `gate_crashes` in score.json).
### Gate 1 test writer (lib/testwriter.py, run.sh before the worker, configs g1/g13/g13adv)
`claude -p` Sonnet 5.5 with the reviewer isolation flags, `--tools Read,Grep,Glob --max-turns 12 --json-schema
{"file": string}`, cwd = a pristine temp copy of upstream (deleted after). Prompt = framing + `tasks/<T>.md` + the
coordinator's instruction ("Write integration-style pytest cases for this request against the tinydb API, one test per
behaviour a careful senior would expect, pushing for edge cases beyond the happy path (missing fields, wrong types,
boundaries, persistence, cache, existing behaviour unchanged). Tests only, no implementation. Return the file content.")
+ mechanics (self-contained module, imports of the new API inside each test, MemoryStorage/tmp_path). The file must
compile and contain `def test_`, else one retry. Output `runs/<run>/gate_tests/` (+ hidden's conftest.py with
`TINYDB_TREE`, pytest.ini) and `testwriter.json` (cost, turns, attempts). Fixed per-run cost, outside the worker's wall.
### Gate 3 (lib/review.py)
Each reviewer: `claude -p "<prompt>" --model claude-sonnet-5-5 --tools Read,Grep,Glob --max-turns 15 --output-format json
--json-schema <schema> --permission-mode dontAsk --setting-sources "" --settings '{"disableAllHooks":true}' --strict-mcp-config`
with `CLAUDE_CODE_DISABLE_CLAUDE_MDS=1 CLAUDE_CODE_DISABLE_AUTO_MEMORY=1`, all inherited `CLAUDE*` vars dropped, cwd = work copy,
600 s timeout. Prompt = framing + task prompt (T-file) + diff (tracked + untracked, cap 60k chars) + the SPEC's "Find defects..."
sentence. `structured_output` is read from the JSON result; `total_cost_usd`, `num_turns`, wall and models are recorded.
- `indep`: reviewers A and B, same prompt, run in parallel, no shared context. Blocking findings = union. Pass when both approve.
- `adv`: A as above; challenger B gets A's numbered findings + the diff, answers `upheld|rejected` + reason per id, may add
findings (`added_findings`). Blocking = A's blocking findings B upheld + B's added blocking findings. B always runs (same
2-call budget as indep).
- Effective `approve` = "no blocking finding" (the raw `approve` is logged as `approve_raw`; a reviewer saying approve=false with
only minor findings does not block). A reviewer call that errors approves (fail open) with `error` recorded; an errored
challenger upholds nothing.
- Reason = "Code review found N blocking defect(s):" + numbered `file:line text` + "Fix every finding, re-run the tests, then stop.", <= 1500 chars.
### Per run (`runs/<task>-<cfg>-r<rep>/`)
`stream.jsonl`, `stderr.txt`, `exit_code.txt`, `wall_s.txt` (includes hook time: the hook runs inside the session),
`gate.jsonl`, `snap-<n>.diff`, `hidden-<n>.txt`, `gate1-<n>.txt`, `hook-input-1.json`, `post.diff`, `numstat.txt`,
`suite.txt` (full suite, no -x), `hidden.txt`, `gate_tests/`, `testwriter.json`, `gatetests-<n>.txt`, `gatetests-final.txt`, `result.json`, `score.json`, `meta.json`, `task.md`, `prompt.txt`,
`settings.json` + `stop.sh` (gated configs).
`score.json`: task, cfg, rep, hidden_pass/hidden_fail/failed_reqs (**requirement level**: 5 reqs per task), suite_pass/fail,
hook_calls, blocks_total, blocks_by_gate {g1,g3}, right_blocks / false_blocks (+ `_by_gate`), give_up, ended_blocked,
cost_worker_usd, cost_reviewers_usd, cost_testwriter_usd, cost_total_usd (all three), testwriter_ok, gate_tests_total,
gate_tests_fail_final, turns, wall_s, reviewer_findings_total, reviewer_errors, gate_crashes,
model_ok (worker init model + every modelUsage key + every reviewer model == claude-sonnet-5-5), hook_events_in_stream,
hidden_first_stop.
## Matrix
5 tasks x 5 configs x 2 reps = 50 runs. Rep 1 config blocks bare, g1, g3, g13, g13adv (tasks rotated per block), rep 2 the
reverse block order. Up to 3 runs at once, each in its own work copy (`matrix-one.sh` via `xargs -P 3`, bash 3.2 has no
`wait -n`). `matrix.log`: one `start` and one result line per run.
## Traps found
- **Gate 1 as first built passed by construction** (it only re-ran the implementer's own tests: `T2-g1-r1` one allow, hidden
R4 missed). Redesigned around an independent test writer (above). The writer hedges where the terse prompt is silent
(accepts "raises or no-op"), so it cannot recover unstated requirements; per-task overlap is in VALIDATION.md.
- **A blocked agent hunts for the tests.** After a gate-1 block the smoke agent ran `cat .claude/hooks/stop.sh; find / -name
test_T1.py` (denied only because `find` is not on the allow-list; `cat` is, and `stop.sh` names the bench dir, from where
`hidden/` is one `cat` away). Gated configs' `.claude/settings.json` now carries `permissions.deny`: Read of `./.claude/**`,
the bench's `hidden/ runs/ validation/ reference/`, the two sibling benches; Bash patterns containing `.claude/hooks`,
`.claude/settings`, `../hidden`, `../runs`, `gate_tests`, the bench sub-paths, `find *`. Probed with a Haiku session: all
five escape attempts (cat, Read, ls, python open) denied, in-tree reads fine. Bash rules match command text, not a security
boundary. Bare runs have no settings file (bare r1 ran before this change; bare has no motive to hunt).
Reviewers and the writer run with `--setting-sources ""`, so these rules do not apply to them.
- **Transient `FileNotFoundError` spawning a reviewer** inside the hook (once, both reviewers of one call, wall 0.0 s;
never reproduced by hand). Reviewer calls now retry once after 3 s; the error message records `filename`.
- **`--bare` cannot authenticate here.** It reads `ANTHROPIC_API_KEY` / `apiKeyHelper` only, never OAuth or the keychain;
this account is OAuth (`~/.claude`), so a `--bare` reviewer returns `terminal_reason: api_error`, cost 0. Used the
documented equivalents instead: `--setting-sources ""` + `--settings '{"disableAllHooks":true}'` + `--strict-mcp-config` +
`CLAUDE_CODE_DISABLE_CLAUDE_MDS=1` + `CLAUDE_CODE_DISABLE_AUTO_MEMORY=1` + `--tools Read,Grep,Glob`. The reviewers' cwd is the
work copy, whose `.claude/settings.json` carries the Stop hook: without `--setting-sources ""`/`disableAllHooks` a reviewer
would run the gate recursively. Verified: a dummy Stop hook in the probe copy never fired.
- **Inherited session env.** A matrix launched from inside Claude Code inherits `CLAUDE_EFFORT=medium`,
`CLAUDE_CODE_SUBAGENT_MODEL=opus`, `CLAUDE_CODE_SESSION_ID`, `CLAUDECODE`, `CLAUDE_CODE_CHILD_SESSION`, messaging tokens.
`env.sh` unsets every `CLAUDE*` var before setting its own; reviewers get a scrubbed env too (the hook process itself
inherits the worker's `CLAUDE_*` vars, e.g. the worker's session id).
- **The feedback line leaks the hook command.** Claude Code prefixes each block with the command line:
`Stop hook feedback:\n[<command>]: <reason>`. With the SPEC command inline the agent would read `GATE_CONFIG=g13adv` and
`gate-bench/gate.sh`. The settings command is therefore `bash "$CLAUDE_PROJECT_DIR/.claude/hooks/stop.sh"`, a one-line
wrapper holding the SPEC command. The agent can still `cat` it (it lives in the work copy) but it is not pushed at it.
- **Stop hook input keys (2.1.293):** `background_tasks, cwd, effort ({"level":"medium"}), hook_event_name,
last_assistant_message, permission_mode, prompt_id, session_crons, session_id, stop_hook_active, transcript_path`.
- **8-cap resets on tool calls.** Per the docs the consecutive-continuation count resets each time Claude calls a tool, so the
built-in cap only bites an agent that answers blocks with text alone. Our give-up (6 blocks per run, counted from
`gate.jsonl`, tool calls or not) fires first in every case.
- **Red start state: the agent refuses to fix code it did not write.** Validation step 2 (T1 + an uncommitted `>=` -> `>`
edit) shows Sonnet 5.5 implementing T1, then answering 6 consecutive gate-1 blocks with "the red test is your pre-existing
uncommitted edit, not my change" without touching it, until give_up. The hook mechanics are validated; the behaviour is a
finding in itself (gate 1 cannot force a fix the agent considers out of scope).
- **pytest-cov.** The repo's `pytest.ini` adds `--cov tinydb`; gate 1 adds `--no-cov` so the 40-line tail holds the failure,
not the coverage table (and no `.coverage` file is written). `.coverage` is gitignored anyway.
- **T5 R4 is parametrized** (several test ids, one requirement). hidden counts are per requirement (5 per task), a req
passes only when all its ids pass. `validate.sh` step 1: T5 upstream passes 1 req (R3, the regression guard) as designed.
- **Diff without touching the index.** `git add -N` would change what the agent sees in `git status`; untracked files are
diffed with `git diff --no-index /dev/null <f>` instead. `.claude/` is in `.git/info/exclude`, so the hook config never
shows in any diff or in `git status`.
- **Worker Bash denials.** The 2.1.293 permission layer denies some agent commands with "Contains brace with quote character
(expansion obfuscation)" even under acceptEdits + allow-list; agents route around it (visible as `permission_denied` events).
- **Reviewers on a correct solution still flag "blocking".** On the T1 reference patch, reviewer A marked "pull does not check
the field is a list" (not required by the task) as blocking; indep therefore blocks, the adv challenger rejected it. This is
exactly what false_blocks measures (hidden all green at that stop).
- **Shared plan meter.** Workers and reviewers bill the user's plan quota; `total_cost_usd` is the list-price equivalent the
CLI reports, not a bill. Nothing else heavy should run on the account during the matrix.
- **rtk rewrites interactive `grep`/`diff`** in the launching Claude session only; scripts here call binaries directly
(`/usr/bin/diff`/`cmp` for verification diffs).
- **CLI version:** `claude --version` = 2.1.293 (Claude Code); flags checked in `claude --help`: `--json-schema`, `--bare`, `--tools`,
`--permission-mode` (choices acceptEdits, auto, bypassPermissions, manual, dontAsk, plan), `--setting-sources`,
`--include-hook-events`. `--max-turns` is accepted though not listed in `--help`.
## Matrix launch (2026-10-08 16:33:54, after the gate-1 redesign)
`cd /Users/thomas/dev/gate-bench && SKIP_DONE=1 nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 &` -> PID 71317, 3 lanes.
The 5 bare r1 runs (terse prompts) are kept and skipped; 45 runs to go. Expected wall ~40-70 min: gated runs so far take
~30-70 s of worker wall + ~35-40 s of test writer (before the worker) + 1-35 s per hook call, more with blocks.
Then `.venv/bin/python analyze.py`.
HARNESS.md
# gate-bench
What does each of Shopify's Helix gates catch when rebuilt as a Claude Code **Stop hook**, and what does it
cost? Headless Sonnet 5.5 workers implement five tinydb features (T1..T5); a Stop hook runs gate 1
(behaviour tests) and/or gate 3 (adversarial review by headless Sonnet 5.5 reviewers) every time the worker
tries to stop; hidden tests (the oracle, never visible to the agent) record the ground truth at every stop
and at the end. Built 2026-10-08 from `SPEC.md`; upstream, hidden tests, tasks and reference patches copied
from `../verify-loops-bench` (nothing there or in `../haiku-routing-bench` is modified).
## Commands (from this dir)
```bash
./validate.sh # steps 1,3,4 (+ step 2 once its run exists) -> validation.log
INJECT=$PWD/inject/break.patch RUN_TAG=val2 ./run.sh g1 T1 1 # validation step 2 (red start state)
RUN_TAG=smoke ./run.sh g13adv T1 1 # smoke (tagged runs are ignored by analyze.py)
./validate_writer.sh # gate-1 test writer on T1..T5: tests on upstream vs reference (REWRITE=1 to redo)
./gen_matrix.py # matrix.txt (50 runs, already generated)
nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 & # 3 runs in parallel (MATRIX_JOBS=3)
SKIP_DONE=1 nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 & # resume
.venv/bin/python analyze.py # results.json + results.md
.venv/bin/python lib/finalize.py runs/<run> # rescore one run from its work copy
.venv/bin/python lib/review.py work/<run> T1 adv # one gate-3 round by hand
```
Single run: `./run.sh <cfg> <task> <rep>` -> `runs/<task>-<cfg>-r<rep>/`, work copy `work/<same name>/`.
## Layout
| path | what |
|---|---|
| `SPEC.md` | the spec this harness implements |
| `upstream/` | tinydb @ 18d73a1 (git repo) |
| `hidden/` | `test_T1..T6.py` (T6 unused), `conftest.py` (`$TINYDB_TREE` or cwd first on sys.path), `pytest.ini` |
| `tasks/T1..T5.md` + `tasks/README.md` | the task prompts (verbatim) and what each hidden req checks |
| `reference/T1..T5.patch` | reference solutions (validation step 1) |
| `inject/break.patch` | `>=` -> `>` in `Query.__ge__` (one repo test red; validation steps 2-3) |
| `inject/test_pushpull.py` | one thin test, makes the T1 reference copy "green with tests touched" (step 3) |
| `configs.json` | `bare` (no hook), `g1`, `g3` (indep), `g13` (g1 then g3 indep), `g13adv` (g1 then g3 adv) |
| `env.sh` | scrubs inherited `CLAUDE*` vars, default `~/.claude` auth, `CLAUDE_CODE_DISABLE_AUTO_MEMORY=1`, venv on PATH |
| `prep.sh <cfg> <work> <run>` | fresh copy of upstream, `.claude/` in `.git/info/exclude`, hook settings for gated configs |
| `run.sh <cfg> <task> <rep>` | one run (worker + hook + `lib/finalize.py`) |
| `gate.sh` | the Stop hook (bash 3.2 wrapper: `exec .venv/bin/python -I lib/hook.py`) |
| `lib/hook.py` | gate dispatch, snapshot, hidden run, gate 1, gate 3, decision, `gate.jsonl` line |
| `lib/review.py` | gate-3 reviewers (indep / adv), schemas, prompts, reason text; CLI for a manual round |
| `lib/testwriter.py` | gate-1 test writer (before the worker) + `run_gate_tests` (path rewritten to `acceptance/`) |
| `validate_writer.sh`, `validation/writer/<T>/` | writer validation files and their runs on upstream / reference |
| `tasks/` (terse prompts, used), `tasks-explicit/` + `runs-explicit/` | the matrix uses terse prompts; see `tasks/README.md` |
| `lib/common.py` | diff incl. untracked files, suite run, hidden run (requirement level) |
| `lib/finalize.py` | post-run diff/suite/hidden, `result.json`, `score.json` |
| `gen_matrix.py`, `matrix.txt`, `matrix.sh`, `matrix-one.sh`, `matrix.log` | 50-run matrix, 3 in parallel (xargs -P), resumable |
| `analyze.py` -> `results.json`, `results.md` | per config, per task, the five questions |
| `validate.sh`, `VALIDATION.md`, `validation.log` | pre-matrix checks |
| `.venv/` | python 3.14.7, pytest 9.1.1, pytest-cov 7.1.0, pyyaml, ruff 0.16.10 (same pins as haiku-routing-bench) |
### Worker (run.sh)
`claude -p "<T-file verbatim>\n\nWhen you are done, stop and summarise what you changed." --model claude-sonnet-5-5
--output-format stream-json --verbose --include-hook-events --max-turns 60 --permission-mode acceptEdits
--allowedTools <haiku-routing-bench list> --setting-sources project --strict-mcp-config`, cwd = `work/<run>/`, stdin /dev/null.
Gated configs get `work/<run>/.claude/settings.json`:
`{"permissions":{"deny":[...see traps...]},"hooks":{"Stop":[{"hooks":[{"type":"command","command":"bash \"$CLAUDE_PROJECT_DIR/.claude/hooks/stop.sh\"","timeout":1500}]}]}}`
and `.claude/hooks/stop.sh` = `GATE_RUN=<abs run dir> GATE_CONFIG=<cfg> bash <bench>/gate.sh` (the SPEC command, one level down: see traps).
Copies of both land in the run dir (`settings.json`, `stop.sh`).
### Hook (lib/hook.py), one invocation
1. read stdin JSON; n = lines already in `gate.jsonl` + 1; blocks so far from the same file
2. `snap-<n>.diff` = `git diff HEAD` + untracked files (index never touched)
3. hidden tests on the work copy (`TINYDB_TREE` = work copy, cwd = work copy) -> `hidden-<n>.txt`; never shown to the agent
4. gate 1 (if configured) = the repo suite `python -m pytest -q -x --no-header -p no:cacheprovider --no-cov` (300 s)
-> `gate1-<n>.txt` AND the writer's acceptance tests `runs/<run>/gate_tests/test_<T>.py` run with `TINYDB_TREE` = work
copy -> `gatetests-<n>.txt`. Block when either is red. Reason: "The acceptance tests for this change fail:" + last 40
lines (failing test names + assertion output, the gate_tests path rewritten to `acceptance/`), then the suite tail if the
suite is red too; <= 1500 chars. `tests_touched` is still logged but no longer blocks. No writer file (writer failed
twice) -> suite only, logged as `gate_tests_present: false`.
5. gate 3 (if configured and gate 1 passed or absent): see below
6. decision: allow / block / give_up (a would-be block when 6 blocks already happened -> allow, logged `give_up`).
`stop_hook_active` is logged, never used. Block = exit 2 + reason on stderr. A crash of the hook itself is
logged to `gate-errors.txt` and allows (counted as `gate_crashes` in score.json).
### Gate 1 test writer (lib/testwriter.py, run.sh before the worker, configs g1/g13/g13adv)
`claude -p` Sonnet 5.5 with the reviewer isolation flags, `--tools Read,Grep,Glob --max-turns 12 --json-schema
{"file": string}`, cwd = a pristine temp copy of upstream (deleted after). Prompt = framing + `tasks/<T>.md` + the
coordinator's instruction ("Write integration-style pytest cases for this request against the tinydb API, one test per
behaviour a careful senior would expect, pushing for edge cases beyond the happy path (missing fields, wrong types,
boundaries, persistence, cache, existing behaviour unchanged). Tests only, no implementation. Return the file content.")
+ mechanics (self-contained module, imports of the new API inside each test, MemoryStorage/tmp_path). The file must
compile and contain `def test_`, else one retry. Output `runs/<run>/gate_tests/` (+ hidden's conftest.py with
`TINYDB_TREE`, pytest.ini) and `testwriter.json` (cost, turns, attempts). Fixed per-run cost, outside the worker's wall.
### Gate 3 (lib/review.py)
Each reviewer: `claude -p "<prompt>" --model claude-sonnet-5-5 --tools Read,Grep,Glob --max-turns 15 --output-format json
--json-schema <schema> --permission-mode dontAsk --setting-sources "" --settings '{"disableAllHooks":true}' --strict-mcp-config`
with `CLAUDE_CODE_DISABLE_CLAUDE_MDS=1 CLAUDE_CODE_DISABLE_AUTO_MEMORY=1`, all inherited `CLAUDE*` vars dropped, cwd = work copy,
600 s timeout. Prompt = framing + task prompt (T-file) + diff (tracked + untracked, cap 60k chars) + the SPEC's "Find defects..."
sentence. `structured_output` is read from the JSON result; `total_cost_usd`, `num_turns`, wall and models are recorded.
- `indep`: reviewers A and B, same prompt, run in parallel, no shared context. Blocking findings = union. Pass when both approve.
- `adv`: A as above; challenger B gets A's numbered findings + the diff, answers `upheld|rejected` + reason per id, may add
findings (`added_findings`). Blocking = A's blocking findings B upheld + B's added blocking findings. B always runs (same
2-call budget as indep).
- Effective `approve` = "no blocking finding" (the raw `approve` is logged as `approve_raw`; a reviewer saying approve=false with
only minor findings does not block). A reviewer call that errors approves (fail open) with `error` recorded; an errored
challenger upholds nothing.
- Reason = "Code review found N blocking defect(s):" + numbered `file:line text` + "Fix every finding, re-run the tests, then stop.", <= 1500 chars.
### Per run (`runs/<task>-<cfg>-r<rep>/`)
`stream.jsonl`, `stderr.txt`, `exit_code.txt`, `wall_s.txt` (includes hook time: the hook runs inside the session),
`gate.jsonl`, `snap-<n>.diff`, `hidden-<n>.txt`, `gate1-<n>.txt`, `hook-input-1.json`, `post.diff`, `numstat.txt`,
`suite.txt` (full suite, no -x), `hidden.txt`, `gate_tests/`, `testwriter.json`, `gatetests-<n>.txt`, `gatetests-final.txt`, `result.json`, `score.json`, `meta.json`, `task.md`, `prompt.txt`,
`settings.json` + `stop.sh` (gated configs).
`score.json`: task, cfg, rep, hidden_pass/hidden_fail/failed_reqs (**requirement level**: 5 reqs per task), suite_pass/fail,
hook_calls, blocks_total, blocks_by_gate {g1,g3}, right_blocks / false_blocks (+ `_by_gate`), give_up, ended_blocked,
cost_worker_usd, cost_reviewers_usd, cost_testwriter_usd, cost_total_usd (all three), testwriter_ok, gate_tests_total,
gate_tests_fail_final, turns, wall_s, reviewer_findings_total, reviewer_errors, gate_crashes,
model_ok (worker init model + every modelUsage key + every reviewer model == claude-sonnet-5-5), hook_events_in_stream,
hidden_first_stop.
## Matrix
5 tasks x 5 configs x 2 reps = 50 runs. Rep 1 config blocks bare, g1, g3, g13, g13adv (tasks rotated per block), rep 2 the
reverse block order. Up to 3 runs at once, each in its own work copy (`matrix-one.sh` via `xargs -P 3`, bash 3.2 has no
`wait -n`). `matrix.log`: one `start` and one result line per run.
## Traps found
- **Gate 1 as first built passed by construction** (it only re-ran the implementer's own tests: `T2-g1-r1` one allow, hidden
R4 missed). Redesigned around an independent test writer (above). The writer hedges where the terse prompt is silent
(accepts "raises or no-op"), so it cannot recover unstated requirements; per-task overlap is in VALIDATION.md.
- **A blocked agent hunts for the tests.** After a gate-1 block the smoke agent ran `cat .claude/hooks/stop.sh; find / -name
test_T1.py` (denied only because `find` is not on the allow-list; `cat` is, and `stop.sh` names the bench dir, from where
`hidden/` is one `cat` away). Gated configs' `.claude/settings.json` now carries `permissions.deny`: Read of `./.claude/**`,
the bench's `hidden/ runs/ validation/ reference/`, the two sibling benches; Bash patterns containing `.claude/hooks`,
`.claude/settings`, `../hidden`, `../runs`, `gate_tests`, the bench sub-paths, `find *`. Probed with a Haiku session: all
five escape attempts (cat, Read, ls, python open) denied, in-tree reads fine. Bash rules match command text, not a security
boundary. Bare runs have no settings file (bare r1 ran before this change; bare has no motive to hunt).
Reviewers and the writer run with `--setting-sources ""`, so these rules do not apply to them.
- **Transient `FileNotFoundError` spawning a reviewer** inside the hook (once, both reviewers of one call, wall 0.0 s;
never reproduced by hand). Reviewer calls now retry once after 3 s; the error message records `filename`.
- **`--bare` cannot authenticate here.** It reads `ANTHROPIC_API_KEY` / `apiKeyHelper` only, never OAuth or the keychain;
this account is OAuth (`~/.claude`), so a `--bare` reviewer returns `terminal_reason: api_error`, cost 0. Used the
documented equivalents instead: `--setting-sources ""` + `--settings '{"disableAllHooks":true}'` + `--strict-mcp-config` +
`CLAUDE_CODE_DISABLE_CLAUDE_MDS=1` + `CLAUDE_CODE_DISABLE_AUTO_MEMORY=1` + `--tools Read,Grep,Glob`. The reviewers' cwd is the
work copy, whose `.claude/settings.json` carries the Stop hook: without `--setting-sources ""`/`disableAllHooks` a reviewer
would run the gate recursively. Verified: a dummy Stop hook in the probe copy never fired.
- **Inherited session env.** A matrix launched from inside Claude Code inherits `CLAUDE_EFFORT=medium`,
`CLAUDE_CODE_SUBAGENT_MODEL=opus`, `CLAUDE_CODE_SESSION_ID`, `CLAUDECODE`, `CLAUDE_CODE_CHILD_SESSION`, messaging tokens.
`env.sh` unsets every `CLAUDE*` var before setting its own; reviewers get a scrubbed env too (the hook process itself
inherits the worker's `CLAUDE_*` vars, e.g. the worker's session id).
- **The feedback line leaks the hook command.** Claude Code prefixes each block with the command line:
`Stop hook feedback:\n[<command>]: <reason>`. With the SPEC command inline the agent would read `GATE_CONFIG=g13adv` and
`gate-bench/gate.sh`. The settings command is therefore `bash "$CLAUDE_PROJECT_DIR/.claude/hooks/stop.sh"`, a one-line
wrapper holding the SPEC command. The agent can still `cat` it (it lives in the work copy) but it is not pushed at it.
- **Stop hook input keys (2.1.293):** `background_tasks, cwd, effort ({"level":"medium"}), hook_event_name,
last_assistant_message, permission_mode, prompt_id, session_crons, session_id, stop_hook_active, transcript_path`.
- **8-cap resets on tool calls.** Per the docs the consecutive-continuation count resets each time Claude calls a tool, so the
built-in cap only bites an agent that answers blocks with text alone. Our give-up (6 blocks per run, counted from
`gate.jsonl`, tool calls or not) fires first in every case.
- **Red start state: the agent refuses to fix code it did not write.** Validation step 2 (T1 + an uncommitted `>=` -> `>`
edit) shows Sonnet 5.5 implementing T1, then answering 6 consecutive gate-1 blocks with "the red test is your pre-existing
uncommitted edit, not my change" without touching it, until give_up. The hook mechanics are validated; the behaviour is a
finding in itself (gate 1 cannot force a fix the agent considers out of scope).
- **pytest-cov.** The repo's `pytest.ini` adds `--cov tinydb`; gate 1 adds `--no-cov` so the 40-line tail holds the failure,
not the coverage table (and no `.coverage` file is written). `.coverage` is gitignored anyway.
- **T5 R4 is parametrized** (several test ids, one requirement). hidden counts are per requirement (5 per task), a req
passes only when all its ids pass. `validate.sh` step 1: T5 upstream passes 1 req (R3, the regression guard) as designed.
- **Diff without touching the index.** `git add -N` would change what the agent sees in `git status`; untracked files are
diffed with `git diff --no-index /dev/null <f>` instead. `.claude/` is in `.git/info/exclude`, so the hook config never
shows in any diff or in `git status`.
- **Worker Bash denials.** The 2.1.293 permission layer denies some agent commands with "Contains brace with quote character
(expansion obfuscation)" even under acceptEdits + allow-list; agents route around it (visible as `permission_denied` events).
- **Reviewers on a correct solution still flag "blocking".** On the T1 reference patch, reviewer A marked "pull does not check
the field is a list" (not required by the task) as blocking; indep therefore blocks, the adv challenger rejected it. This is
exactly what false_blocks measures (hidden all green at that stop).
- **Shared plan meter.** Workers and reviewers bill the user's plan quota; `total_cost_usd` is the list-price equivalent the
CLI reports, not a bill. Nothing else heavy should run on the account during the matrix.
- **rtk rewrites interactive `grep`/`diff`** in the launching Claude session only; scripts here call binaries directly
(`/usr/bin/diff`/`cmp` for verification diffs).
- **CLI version:** `claude --version` = 2.1.293 (Claude Code); flags checked in `claude --help`: `--json-schema`, `--bare`, `--tools`,
`--permission-mode` (choices acceptEdits, auto, bypassPermissions, manual, dontAsk, plan), `--setting-sources`,
`--include-hook-events`. `--max-turns` is accepted though not listed in `--help`.
## Matrix launch (2026-10-08 16:33:54, after the gate-1 redesign)
`cd /Users/thomas/dev/gate-bench && SKIP_DONE=1 nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 &` -> PID 71317, 3 lanes.
The 5 bare r1 runs (terse prompts) are kept and skipped; 45 runs to go. Expected wall ~40-70 min: gated runs so far take
~30-70 s of worker wall + ~35-40 s of test writer (before the worker) + 1-35 s per hook call, more with blocks.
Then `.venv/bin/python analyze.py`.
SPEC.md
# gate-bench — SPEC (written 2026-10-08 by the /plan stage of the AIDive pipeline)
Goal: measure, on a real repo, what each of Shopify's Helix gates catches when rebuilt as a Claude Code
Stop hook, and what it costs. Output feeds a YouTube video; every number must be reproducible from this dir.
## Reuse (read-only, never modify)
- `/Users/thomas/dev/verify-loops-bench/upstream/` : tinydb @ 18d73a1 (git repo). Copy it as `upstream/` here.
- `/Users/thomas/dev/verify-loops-bench/hidden/` : `test_T1.py..test_T6.py`, `conftest.py`, `pytest.ini`. Copy as `hidden/`.
Hidden tests are the ORACLE: one `test_reqN` per requirement of the task prompt. They live OUTSIDE every work copy
and the agent never sees them. Run: `cd <tree> && python -m pytest -c <bench>/hidden/pytest.ini --rootdir <bench>/hidden <bench>/hidden/test_Tn.py -q`
(conftest puts cwd or `$TINYDB_TREE` first on sys.path). Check `tasks/README.md` there for the exact invocation and validate with `validate.sh`.
- `/Users/thomas/dev/verify-loops-bench/tasks/T1.md..T5.md` : the five task prompts, typed as a developer would. Copy as `tasks/`. T6 is NOT used.
- `/Users/thomas/dev/haiku-routing-bench/{env.sh,run.sh,extract.py,prep.sh,matrix.sh}` : patterns to copy (auth via the
user's default config dir, `CLAUDE_CODE_DISABLE_AUTO_MEMORY=1`, `.venv` with pytest, stream-json extraction of
`total_cost_usd`, `num_turns`, `usage`, `modelUsage`, fresh work copy per run, `--setting-sources project --strict-mcp-config`).
- Claude Code CLI 2.1.293 is installed. Docs that define the hook contract: https://code.claude.com/docs/en/hooks
(fetch with `curl -s https://r.jina.ai/https://code.claude.com/docs/en/hooks`): a Stop hook that exits 2 blocks the
stop and its stderr is fed back; `{"decision":"block","reason":"..."}` on stdout does the same; input JSON on stdin
carries `stop_hook_active`, `session_id`, `transcript_path`; after 8 consecutive continuations Claude Code overrides
the next block (`CLAUDE_CODE_STOP_HOOK_BLOCK_CAP` raises it); a hook past `timeout` (seconds, default 600) renders no decision.
## Worker
`claude -p "<task prompt>" --model claude-sonnet-5-5 --output-format stream-json --verbose --max-turns 60
--permission-mode acceptEdits --allowedTools "<same list as haiku-routing-bench/run.sh>" --setting-sources project
--strict-mcp-config`, cwd = fresh work copy `work/<run>/` (copy of upstream, `git` clean at HEAD). The task prompt is the
T-file verbatim plus one line appended for every config (bare included): "When you are done, stop and summarise what
you changed." No mention of gates in the prompt: the gate is the hook, not the instruction.
Work copy carries `.claude/settings.json` with the Stop hook for gated configs (none for `bare`):
`{"hooks":{"Stop":[{"hooks":[{"type":"command","command":"GATE_RUN=<abs run dir> GATE_CONFIG=<cfg> bash <bench>/gate.sh","timeout":1500}]}]}}`
(check the exact settings shape in the hooks doc; `timeout` is in seconds; verify the hook fires with a smoke run before the matrix).
## Gates (gate.sh, one script, dispatches on GATE_CONFIG)
Reads stdin JSON. Logs ONE line per invocation to `$GATE_RUN/gate.jsonl`: {ts, n (invocation index), stop_hook_active,
gate1:{ran, suite_pass, suite_fail, tests_touched (bool: any file under tests/ added or modified vs HEAD), pass},
gate3:{ran, variant, reviewers:[{id, approve, findings:[{severity, file, line, text}], cost_usd, turns, wall_s}], blocking_findings, pass},
hidden:{pass, fail, failed_reqs} (ground truth at this moment, run SILENTLY on the work copy; never shown to the agent),
decision ("block"|"allow"|"give_up"), reason (the text sent back, truncated 500), wall_s}.
Give-up: after 6 blocks in one run the hook allows and logs decision "give_up" (the video needs to count it; Claude
Code's own 8-cap would otherwise hide it). Never use stop_hook_active to allow: log it only.
Order inside one invocation: gate 1 (when the config has it) first; gate 3 only when gate 1 passed or is absent.
- Gate 1 (behaviour tests): `python -m pytest -q -x --no-header -p no:cacheprovider` on the work copy (the repo's own 226 tests plus
whatever the agent wrote). Block when red (reason = last 40 lines of pytest), OR when `tests_touched` is false
(reason: "No test covers the new behaviour. Add tests for every requirement in the task, run them, then stop.").
- Gate 3 (adversarial reviews): headless reviewers, each `claude -p --bare --model claude-sonnet-5-5 --tools "Read,Grep,Glob"
--max-turns 15 --output-format json --json-schema '<schema>' --permission-mode dontAsk` (confirm each flag exists in
`claude --help`; drop `--permission-mode dontAsk` if it errors), cwd = work copy, prompt = system framing + the task
prompt + `git diff HEAD` (tracked + untracked files, cap 60k chars) + "Find defects: requirements not met, edge cases
wrong, behaviour broken, tests that do not test the requirement. Report only real defects with file and line. approve=true
only when nothing blocks." Schema: {approve: boolean, findings: [{severity: "blocking"|"minor", file, line, text}]}.
Read `structured_output` from the JSON result; record `total_cost_usd`. Set `CLAUDE_CODE_DISABLE_AUTO_MEMORY=1`.
- variant `indep` (Shopify as written): two reviewers, same prompt, independent, no shared context; union of findings;
block if any blocking finding from either; reason = the blocking findings, numbered, "Fix every finding, re-run the
tests, then stop." Pass when BOTH approve.
- variant `adv` (reviewer + challenger, arXiv 2608.18167 shape): reviewer A as above; then challenger B gets A's
findings plus the diff and must, per finding, answer upheld|rejected with a reason, and may add findings A missed;
only upheld (or B-added) blocking findings block. Pass when nothing upheld blocks.
- Reason text is what the worker sees next turn: keep it factual, under 1500 chars.
## Configs (configs.json)
- `bare`: no hook.
- `g1`: gate 1 only.
- `g3`: gate 3 indep only.
- `g13`: gate 1 then gate 3 indep (Shopify's order).
- `g13adv`: gate 1 then gate 3 adv.
Gate 2 (UI) and gate 4 (engineer) are out of scope by design: tinydb is a library, Shopify allows skipping UI review for
logic-only changes; gate 4 is the human, measured as "what reached the human": the hidden failures left at the end.
## Per run outputs (runs/<task>-<cfg>-r<rep>/)
stream.jsonl, stderr.txt, exit_code.txt, wall_s.txt, gate.jsonl, post.diff, numstat.txt, suite.txt (final repo suite),
hidden.txt (final hidden run), result.json (worker: total_cost_usd, num_turns, usage, modelUsage, is_error, subtype,
final text), score.json: {task, cfg, rep, hidden_pass, hidden_fail, failed_reqs, suite_pass, suite_fail, blocks_total,
blocks_by_gate {g1, g3}, right_blocks (hidden failing at that moment), false_blocks (hidden all passing at that moment),
give_up, ended_blocked (the LAST gate.jsonl line is a block, i.e. Claude Code's cap or the agent ended anyway),
cost_worker_usd, cost_reviewers_usd, cost_total_usd, turns, wall_s, reviewer_findings_total, model_ok}.
Snapshot at every hook call: before gate 1 runs, `git diff HEAD` of the work copy to `$GATE_RUN/snap-<n>.diff`.
## Matrix
5 tasks (T1..T5) × 5 configs × 2 reps = 50 runs. `matrix.txt` ordered: rep 1 of everything first (config-contiguous:
bare, g1, g3, g13, g13adv, tasks rotated), then rep 2 in reverse config order. `matrix.sh` runs up to 3 runs in parallel
(each in its own work copy), appends one line per run to `matrix.log`, resumable (`SKIP_DONE=1` skips runs with score.json).
`analyze.py` → `results.json` + `results.md`: per config: hidden requirements passed / total, runs fully passing, mean
cost, mean wall, blocks per run, right vs false blocks per gate, give-ups, ended_blocked; per task the same; and the
answer to the video's questions: (1) does gate 1 alone move hidden pass rate vs bare, (2) does gate 3 indep add anything
over g1, (3) does adv beat indep, (4) false-block rate per gate, (5) cost per gate.
## Validation before the matrix (VALIDATION.md + validate.sh)
1. upstream fails every hidden test file; the reference patches in `/Users/thomas/dev/verify-loops-bench/reference/`
pass hidden + suite (reuse their validate.sh logic).
2. The hook fires: one smoke run of T1 on `g1` with a deliberately failing state (apply a patch that breaks one test)
must show a block in gate.jsonl, and the stream must show the agent continuing after "Stop hook feedback".
3. gate.sh called by hand with a fake stdin on a clean work copy returns allow, on a red copy returns exit 2 with the
pytest tail on stderr.
4. One reviewer call by hand returns valid structured_output against the schema.
5. Record the exact `claude --version`, the settings.json used, and the hook input keys seen.
## Deliverables
Everything under /Users/thomas/dev/gate-bench with README.md (layout, commands, traps found). When the matrix is launched,
run it with `nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 &` and report the PID; do not wait for it to finish.
Copy the scripts (not runs/, not work/) to /Users/thomas/dev/faceless-video-pipeline/jobs/shopify-four-gates-claude-code-20261008-1529/research/code/
with a one-line README. Report: what was validated, the smoke run's gate.jsonl, the matrix PID, expected duration.
VALIDATION.md
# Validation (2026-10-08, claude 2.1.293) — raw output in validation.log
1. **Oracle.** `./validate.sh step1`: on upstream every hidden req fails (T5: 1 passing, R3 = regression guard, as designed);
each reference patch passes its hidden file (T1-T4 5 tests, T5 6 tests = 5 reqs) and the repo's 226 tests. OK.
2. **Hook fires.** `INJECT=$PWD/inject/break.patch RUN_TAG=val2 ./run.sh g1 T1 1` (T1 + an uncommitted `>=` -> `>` break):
gate.jsonl = 7 calls, 6 `block` (g1, suite red 1 failed / 33 passed under -x) then `give_up`; the stream shows
`Stop hook feedback:` user messages followed by new assistant turns (the agent continued each time; it kept
explaining the red test is the user's pre-existing edit and never touched it). `stop_hook_active` false on call 1, true after. OK.
3. **gate.sh by hand** (`./validate.sh step3`, fake stdin): green copy (T1 reference + one new test) -> exit 0, `allow`,
empty stderr; red copy (break.patch) -> exit 2, `block`, stderr = pytest tail incl. `short test summary info`. OK.
4. **Reviewer by hand** (`./validate.sh step4`, `lib/review.py work/_val3-green T1 indep|adv`): every call returned a
schema-valid `structured_output`, cost $0.03-0.05 per pair, 10-45 s per call. OK.
5. **Recorded.** `claude --version` 2.1.293. settings.json (gated configs):
`{"hooks":{"Stop":[{"hooks":[{"type":"command","command":"bash \"$CLAUDE_PROJECT_DIR/.claude/hooks/stop.sh\"","timeout":1500}]}]}}`,
`.claude/hooks/stop.sh` = `GATE_RUN=<run dir> GATE_CONFIG=<cfg> bash /Users/thomas/dev/gate-bench/gate.sh`.
Hook input keys seen: background_tasks, cwd, effort, hook_event_name, last_assistant_message, permission_mode, prompt_id,
session_crons, session_id, stop_hook_active, transcript_path. Stream hook events in the smoke run: exactly one
`hook_started`/`hook_response` pair, `hook_name: Stop` (no user/global hooks: `--setting-sources project`).
Smoke (`runs/T1-g13adv-r1-smoke`, wrapper hook): hidden 5/5, suite 238/238, 1 hook call, gate 1 green with tests touched,
adv reviewer A approve (0 findings), challenger approve, allow. Worker $0.084 + reviewers $0.034 = $0.117, 12 turns, 66 s.
`runs/T1-g13adv-r1-smoke0` = the first smoke with the inline SPEC command (same outcome, $0.195, 69 s), superseded because the
feedback prefix would leak the command line.
## Gate 1 redesign (2026-10-08, after the coordinator's review): independent test writer
Gate 1 no longer re-runs only the implementer's own tests (it passed by construction: `T2-g1-r1` one allow, hidden R4
missed, suite green). Before the worker starts, `lib/testwriter.py` (headless Sonnet 5.5, reviewer isolation flags,
`--tools Read,Grep,Glob --max-turns 12 --json-schema {file}`, cwd = a pristine temp copy of upstream) gets ONLY
`tasks/<T>.md` and writes `runs/<run>/gate_tests/test_<T>.py` (+ hidden's conftest/pytest.ini) outside the work copy.
Gate 1 = repo suite AND those tests (`TINYDB_TREE` = work copy). `tests_touched` is logged, no longer a rule.
`./validate_writer.sh` (one writer call per task, files in `validation/writer/<T>/`; reference = verify-loops-bench patches):
| task | writer tests | pass on reference | pass on upstream | fail on reference (possible false blocks, kept) | writer $ / wall |
|---|---|---|---|---|---|
| T1 | 52 | 51 | 1 | `test_pull_on_non_list_field_raises_and_leaves_data_unchanged` | 0.099 / 39 s |
| T2 | 42 | 42 | 5 | none | 0.094 / 35 s |
| T3 | 35 | 34 | 11 | `test_rename_table_non_string_target_does_not_corrupt_database` | 0.077 / 34 s |
| T4 | 48 | 47 | 7 | `test_reverse_without_sort_by_reverses_default_order` | 0.092 / 42 s |
| T5 | 26 | 25 | 7 | `test_non_integer_size_is_rejected` | 0.086 / 30 s |
Tests passing on upstream are the "existing behaviour unchanged" guards the instruction asks for (plus a few that
accept "raises" as an outcome); every task's file fails on upstream overall. Each matrix run writes its own file, so
these counts are one sample; per-run counts land in `score.json` (`gate_tests_total`, `gate_tests_fail_final`).
Overlap with the 5 hidden requirements (judgement from test names + spot reads of the assertions):
- T1: 2/5 strict (R1 order, R5 push non-list raises). R2/R4 hedged (`try: ... except (KeyError, AttributeError, TypeError)`: a raise is accepted), R3 hedged (`count('x') < 2`, first-occurrence removal passes).
- T2: 4/5 (R1, R2, R3, R5 cache inclusive vs exclusive and other bounds); R4 hedged (accepts a TypeError).
- T3: 3/5 strict (R1 ids, R2 persistence + `tables()`, R5 stale cache / old name empty / id sequence); R3/R4 use `pytest.raises(Exception)`, not KeyError/ValueError (R4 does check both tables intact).
- T4: 4/5 (R1, R3, R4 limit 0 / negative ValueError, R5 cache not polluted + Document/doc_id); R2 hedged (checks order of docs that have the field and that none are dropped, not "missing last in both directions").
- T5: 5/5 (R1, R2 restart, R3 class/subclass default, R4 non-positive ValueError at construction, R5 close flushes).
The writer hedges exactly where the terse prompt is silent: it encodes "a raise or a no-op are both fine", so gate 1
cannot recover a requirement the prompt never states (T1 R2/R4, T4 R2).
Smoke after the redesign (`runs/T1-g13adv-r1-smoke`, with the deny rules): writer 45 tests ($0.071), first stop gate 1
green (232 suite + 45/45 gate tests), adv A+B approve, allow; hidden 2/5 (R2, R4, R5 missed). Total $0.189, 68 s.
`runs/T1-g13adv-r1-smoke-a` (first redesign smoke): stop 1 blocked by gate 1 (5 gate tests red, reason starts
"The acceptance tests for this change fail:"), the agent fixed push/pull type checks, stop 2 gate 1 green, both reviewers
errored (transient FileNotFoundError on spawn, logged, fail-open allow; one retry added since). The agent also tried
`cat .claude/hooks/stop.sh; find / -name test_T1.py` after the block (denied by the allow-list) -> deny rules added.
`runs/T1-g13adv-r1-smoke-b`: second redesign smoke, clean (reviewers ran, $0.167).
월요일에 할 일
- 검사를 종료 코드 2와 stderr의 실패 내용으로 끝나는 스크립트로 쓰고, 스위트보다 긴 타임아웃을 단 Stop 명령 훅으로 연결하세요.
- 별도의 에이전트가 요청만 보고 인수 테스트를 쓰게 하고, 저장소 밖에 저장하며, 구현자가 훅과 설정과 그 테스트를 읽지 못하도록 거부하세요.
- 멈춤마다 결정과 게이트가 빨간 상태였는지를 파일에 기록해서, 작업을 통과시킨 게이트가 눈에 보이게 하세요.
- 차단 6번 뒤에 루프를 내보내는 자체 카운터를 추가하세요. Claude Code의 상한은 연속 차단 8번이고 도구를 부를 때마다 초기화돼요.
- 리뷰어는
--tools Read,Grep,Glob, 빈 설정 소스, 비활성화된 훅으로 헤드리스로 돌려서 게이트를 재귀적으로 일으키지 못하게 하세요. - 리뷰 게이트를 남긴다면 발견 사항을 장벽으로 만들지 말고 피드백 속 조언으로 돌려보내세요.
- 게이트를 믿기 전에 2주 동안 정당한 차단과 잘못된 차단을 세어 보세요.
-
--bare는 API 키만 읽는다는 점을 기억하세요. 구독 로그인에서는 오류를 돌려주므로 리뷰어는 설정 플래그로 격리하세요.
측정
results.md에서: 50번 실행, 구성당 10번, 작업 5개 곱하기 반복 2번이에요. 숨은 요구사항은 구성당 50개 기준으로 세요. 차단은 정당/잘못 형태로 적어요. 정당한 차단은 그 멈춤에서 숨은 오라클에 실패한 요구사항이 있었다는 뜻이고, 잘못된 차단은 완전히 초록이었다는 뜻이에요.
| 구성 | 통과한 숨은 요구사항 | 완전히 깨끗한 실행 | 실행당 평균 비용 | 평균 실제 시간(초) | 게이트 1 차단 정당/잘못 | 게이트 3 차단 정당/잘못 | 포기 |
|---|---|---|---|---|---|---|---|
| bare (훅 없음) | 35/50 (70%) | 3 | $0.085 | 33 | 0/0 | 0/0 | 0 |
| g1 (인수 테스트) | 42/50 (84%) | 6 | $0.201 | 60 | 3/3 | 0/0 | 0 |
| g3 (리뷰어 둘) | 34/50 (68%) | 3 | $0.151 | 54 | 0/0 | 1/0 | 0 |
| g13 (테스트, 그다음 리뷰어 둘) | 41/50 (82%) | 5 | $0.282 | 86 | 14/1 | 2/0 | 2 |
| g13adv (테스트, 그다음 리뷰어와 크리틱) | 39/50 (78%) | 6 | $0.276 | 86 | 11/1 | 0/0 | 1 |
비용은 total_cost_usd가 보고하는 대로 워커, 게이트 1 작성자, 게이트 3 리뷰어의 합이고, 청구서가 아니라 정가 환산이에요. 모든 실행이 설정된 모델을 썼어요. 게이트가 있는 실행 전체에서 게이트 1은 실제 요구사항이 실패한 채로 에이전트를 28번, 모든 요구사항이 통과한 채로 5번 되돌려 보냈어요. 게이트 3은 3번 막았고 잘못 막은 적은 없어요. 루프 세 개가 차단 6번에서 포기했고, Claude Code 자체의 상한은 한 번도 작동하지 않았어요.
요구사항 50개 중 사람에게 남은 것: 게이트 없음 15, 테스트 8, 리뷰어만 16, 테스트와 리뷰어 9, 테스트와 리뷰어, 크리틱 11이에요. 테스트가 작업을 움직인 곳: 정렬 검색은 10 중 5에서 10 중 10으로, 캐시 크기 작업은 8에서 10으로 올랐어요. push와 pull 작업은 10 중 4에 머물렀는데, 필드가 없는 문서를 어떻게 할지 요청에 적혀 있지 않았기 때문이에요.
더 나아가기
- UI 게이트. Shopify의 게이트 2는 프런트엔드 저장소에서 두 번째 모델이 설계 리뷰어 역할을 해요 s1. 우리 저장소는 라이브러리라서 건너뛰었어요. 이를 측정하기에 알맞은 곳은 이후의 프런트엔드 저장소 테스트예요.
- Ralph 루프. 조건이 성립할 때까지 에이전트를 계속 돌린다는 점은 같지만, 출구가 에이전트가 쓸 수 있는 문구에 있어요. 우리의 출구는 외부에 있어요. 에이전트의 손이 닿지 않는 스크립트와 카운터예요.
- 적대적 리뷰 논문. 요약이 아니라 표를 읽으세요. 리뷰어와 크리틱 형태 s5는 종이 위에서는 토큰값을 하는 형태지만, 우리 실행은 이 저장소에서 이득을 재현하지 못했어요. 도입하기 전에 여러분의 저장소에서 시험해 보세요. 초록 페이지: s23.
- 차단 상한과 타임아웃. 훅을 강제 정지로 믿기 전에 hooks 레퍼런스의 Stop 절을 읽고 s2, hooks 가이드의 문제 해결 절도 읽으세요 s19.
- 보고 방식을 비교해 볼 측정된 게이트: s8.
FAQ
Stop hook으로 Claude Code가 끝내는 것을 정말 막을 수 있나요?
대체로요. 종료 코드 2는 멈춤을 막고 stderr를 피드백으로 돌려주지만 s2, 연속 8번 이어지면 Claude Code가 다음 차단을 무시하고, 타임아웃된 훅은 판단을 내리지 않아요 s2. 멈춤마다 기록해서 작업을 통과시킨 순간을 확인하세요.
테스트 작성자는 왜 별도의 에이전트인가요?
게이트 1의 첫 버전은 구현자 자신의 테스트를 다시 돌렸어요. 숨은 요구사항이 실패하는데도 모든 실행을 첫 멈춤에서 통과시켰어요. 에이전트의 테스트는 자기가 만든 것을 테스트하기 때문이에요. 다시 설계한 작성자는 요청과 변경 전 저장소만 봐요.
리뷰어에 --bare를 쓰면 안 되나요?
--bare는 훅과 메모리를 건너뛰지만 s20, 구독 로그인에서는 API 키만 읽어서 여기서는 api_error를 돌려줬어요. 그래서 리뷰어는 빈 설정 소스, 비활성화된 훅, 엄격한 MCP 설정, 읽기 전용 도구로 격리했어요.
리뷰어가 멀쩡한 작업을 막은 적이 있나요?
우리 50번 실행에서는 없어요. 차단 3번에 잘못된 것은 없었어요. 문제는 그들이 통과시킨 것이었어요. 두 번의 실행에서 리뷰어 둘이 정렬 필드가 없을 때 나는 같은 크래시를 보고도 minor로 평가했고, 두 실행 모두 다섯 개 중 0개로 끝났어요.
리뷰어가 도움이 된 적이 없다는 증명인가요?
아니에요. 저장소 하나, 워커 모델 하나, 작업 다섯 개, 칸당 반복 두 번이에요. 여기서는 리뷰어가 합격률을 움직이지 못했고 테스트 게이트는 움직였다는 것만 말해 줘요.