AIDiveTL;DR y veredicto
- Helix, de Shopify, pasa cada checkpoint por cuatro gates en orden: tests de comportamiento, revisión de UI, dos revisores independientes con contexto aislado y la aprobación de un ingeniero s1. El post no da ninguna tasa de aciertos. Reconstruimos los gates 1, 3 y 4 como un Stop hook de Claude Code en un repo real y contamos.
- Nuestra matriz: 50 ejecuciones, cinco tareas de tinydb, cinco configuraciones, Sonnet 5.5 headless. Sin ningún gate, pasaron 35 de 50 requisitos ocultos (70%). Solo con el gate de tests de aceptación, 42 de 50 (84%).
- El gate de dos revisores tal como está escrito no movió la tasa de aciertos: 34 de 50 solo, 41 de 50 después de los tests. Un revisor más un crítico llegó a 39 de 50.
- Veredicto: quédate con el gate de tests, con un escritor que no sea el implementador y tests que el agente no pueda editar. Espera un falso bloqueo de cada seis, y dale al bucle un contador.
- Si mantienes el gate de dos revisores, mantenlo como consejo en el feedback, no como una barrera.
- Límites: un repo, un modelo trabajador, cinco tareas, dos repeticiones por celda. Una diferencia de dos puntos entre filas es ruido. Los 14 puntos entre no tener gate y el gate de tests no lo son.
Lo que dicen las fuentes
Helix enuncia la regla por sí mismo: si un gate falla, el agente usa el feedback para corregir la implementación y vuelve a ejecutar la comprobación, y no puede anular una comprobación fallida porque considere que el resultado es suficientemente bueno s1. El gate 3 son dos agentes revisores independientes y con contexto aislado, hay que corregir todos los hallazgos y el bucle se repite hasta que ambos aprueban s1. Un cambio de pura lógica simplemente se salta el gate de revisión de UI s1, por eso nuestro repo de librería de estilo CLI no tiene gate 2. Un análisis independiente clasifica la evidencia de Helix como caso de estudio, de primera mano y sin medir, y señala que nada valida al juez: no se reporta ninguna tasa de falsos bloqueos s6.
La propia guía de Anthropic plantea el problema: Claude se detiene cuando el trabajo parece terminado, y sin una comprobación que pueda ejecutar, «parece terminado» es la única señal disponible s3. La versión determinista es un Stop hook que ejecuta tu comprobación como un script y bloquea el final del turno hasta que pasa s3. La guía añade que un revisor en un contexto de subagente nuevo ve solo el diff y los criterios, no el razonamiento detrás del cambio s3, y que los subagentes empiezan con una ventana de contexto nueva y aislada s22.
El contrato del hook: salir con 2 en Stop impide que Claude se detenga s2, y el mensaje de stderr es lo que Claude recibe como motivo s2. Cualquier otro código de salida, el 1 incluido, no bloquea por sí solo s2. La forma JSON es decision: "block" con un reason obligatorio s2.
El gate tiene tres salidas en su propia documentación:
- El tope. Después de que los stop hooks hayan continuado el turno ocho veces seguidas, Claude Code anula el siguiente bloqueo y termina el turno. El contador se reinicia cada vez que Claude llama a una herramienta, y
CLAUDE_CODE_STOP_HOOK_BLOCK_CAPlo eleva s2 s19. - El timeout. Los hooks de tipo comando tienen 600 segundos por defecto, y un hook que alcanza su timeout se cancela y su salida se descarta, así que en la mayoría de los eventos no emite ninguna decisión s2.
- El flag.
stop_hook_activees verdadero cuando Claude Code ya está continuando por un stop hook, de modo que un script puede evitar bloquear por una condición que nunca se resolverá s2.
Un profesional advierte de que un auditor barato y con poco contexto «gets things wrong a lot» y bloquea sesiones buenas, así que no debería tener la última palabra s7. Un gate Stop medido sobre 100 paradas etiquetadas bloqueó 8 turnos de 100, 7 de ellos con razón, con la salvedad de que las etiquetas las puso un modelo, no una persona s8. Nosotros reportamos los bloqueos de la misma manera: correctos y falsos.
Los flags de scripting en los que nos apoyamos: --bare omite hooks, skills, plugins, servidores MCP, memoria automática y CLAUDE.md s20, --max-turns limita los turnos agénticos en modo print s20, --json-schema devuelve una salida validada en el campo structured_output s21, y --output-format json incluye total_cost_usd, una estimación del lado del cliente que puede diferir de tu factura s21.
Sobre el gate 3, un paper de workshop de 2026 midió la forma. En LiveCodeBench: zero-shot 77%, un solo revisor 77%, dos revisores 75% sobre 105 problemas, y 34 de 57 en el subconjunto difícil s5. El paper considera el resultado de dos revisores todavía dentro del ruido respecto a un solo revisor, self-refine y zero-shot s5. Su forma adversarial, un revisor más un crítico que audita la revisión antes de que el agente principal edite, llegó a 87% s5. En SWE-bench Verified llegó a 75.2% frente a 71.6% zero-shot, con unas 4.5x los tokens s5. Paper: s23.
El kit
El banco de pruebas es tinydb en el commit 18d73a1 con 226 tests en verde. Cinco peticiones escritas como las escribe un desarrollador (operaciones push y pull, una consulta between, el cambio de nombre de una tabla, búsqueda ordenada, un tamaño de caché de escritura), cada una con cinco requisitos ocultos que el agente nunca ve: 25 en total. El trabajador es Sonnet 5.5 headless. Todo lo que sigue es el harness y los gates. El checkout de tinydb, los tests ocultos y los prompts de las tareas no están incluidos: apunta el kit a tu propio repo y escribe tu propio oráculo.
El gate:
gate.sh: el punto de entrada del Stop hook, un wrapper de una línea que ejecutalib/hook.py.gate.sh
#!/usr/bin/env bash
# Stop hook entry point (bash 3.2 OK). All logic lives in lib/hook.py.
# Called as: GATE_RUN=<abs run dir> GATE_CONFIG=<cfg> bash <bench>/gate.sh (stdin = hook input JSON)
HERE="$(cd "$(dirname "$0")" && pwd)"
exec "$HERE/.venv/bin/python" -I "$HERE/lib/hook.py"
lib/hook.py
"""Stop hook: gate 1 (behaviour tests) and/or gate 3 (adversarial review), per GATE_CONFIG.
Env: GATE_RUN (abs run dir, holds meta.json + prompt.txt), GATE_CONFIG (key of configs.json).
stdin: the Stop hook input JSON. Block = exit 2 with the reason on stderr. Allow = exit 0, no output.
Every invocation appends ONE line to $GATE_RUN/gate.jsonl. Nothing but the reason ever reaches the agent.
"""
import json
import os
import sys
import time
import traceback
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from common import BENCH, run_hidden, run_suite, tests_touched, work_diff # noqa: E402
from testwriter import run_gate_tests # noqa: E402
GIVE_UP_AFTER = 6
def tail40(out, room=1400):
t = "\n".join(out.rstrip().splitlines()[-40:])
if len(t) > room:
t = "...\n" + t[-room:]
return t
def main():
t0 = time.time()
run = os.environ["GATE_RUN"]
cfg_name = os.environ["GATE_CONFIG"]
cfg = json.load(open(os.path.join(BENCH, "configs.json")))[cfg_name]
meta = json.load(open(os.path.join(run, "meta.json")))
work, task = meta["work"], meta["task"]
raw = sys.stdin.read()
try:
hin = json.loads(raw) if raw.strip() else {}
except Exception:
hin = {"_unparsed": raw[:500]}
log_path = os.path.join(run, "gate.jsonl")
prev = []
if os.path.exists(log_path):
prev = [json.loads(l) for l in open(log_path) if l.strip()]
n = len(prev) + 1
blocks_before = sum(1 for p in prev if p.get("decision") == "block")
# Snapshot of the work at this stop, before gate 1 runs.
open(os.path.join(run, "snap-%d.diff" % n), "w").write(work_diff(work))
# Ground truth at this moment (silent).
_, hout, hp, hf, hfailed = run_hidden(work, task)
open(os.path.join(run, "hidden-%d.txt" % n), "w").write(hout)
rec = {"ts": time.strftime("%Y-%m-%dT%H:%M:%S"), "n": n, "stop_hook_active": hin.get("stop_hook_active"),
"hook_input_keys": sorted(hin.keys()),
"gate1": {"ran": False}, "gate3": {"ran": False},
"hidden": {"pass": hp, "fail": hf, "failed_reqs": hfailed}}
reason, blocked_by = None, None
if cfg["gate1"]:
# Gate 1 = the repo suite AND the acceptance tests written before the run by the independent test writer
# (run dir gate_tests/, outside the work copy). A writer failure leaves the suite alone (logged).
rc, out, sp, sf = run_suite(work)
open(os.path.join(run, "gate1-%d.txt" % n), "w").write(out)
suite_ok = rc == 0 and sf == 0
gdir = os.path.join(run, "gate_tests")
has_gt = os.path.exists(os.path.join(gdir, "test_%s.py" % task))
g = {"ran": True, "suite_pass": sp, "suite_fail": sf, "tests_touched": tests_touched(work),
"gate_tests_present": has_gt}
gt_ok, gout = True, ""
if has_gt:
grc, gout, gp, gf, gnames = run_gate_tests(gdir, task, work)
open(os.path.join(run, "gatetests-%d.txt" % n), "w").write(gout)
gt_ok = grc == 0 and gf == 0
g.update(gate_tests_pass=gp, gate_tests_fail=gf, gate_tests_failed=gnames)
g["pass"] = suite_ok and gt_ok
rec["gate1"] = g
if not g["pass"]:
blocked_by = "g1"
parts = []
if not gt_ok:
parts.append("The acceptance tests for this change fail:\n" + tail40(gout, 1400 if suite_ok else 800))
if not suite_ok:
parts.append("The repo's test suite is red (python -m pytest -q -x):\n" + tail40(out, 1400 if gt_ok else 550))
reason = "\n\n".join(parts)
if cfg["gate3"] and blocked_by is None:
from review import gate3, reason_text
g3 = gate3(work, open(os.path.join(run, "task.md")).read(), cfg["gate3"])
rec["gate3"] = g3
if not g3["pass"]:
blocked_by = "g3"
reason = reason_text(g3)
if blocked_by is None:
decision = "allow"
elif blocks_before >= GIVE_UP_AFTER:
decision = "give_up"
else:
decision = "block"
rec["blocked_by"] = blocked_by
rec["decision"] = decision
rec["reason"] = (reason or "")[:500]
rec["reason_chars"] = len(reason or "")
rec["wall_s"] = round(time.time() - t0, 1)
with open(log_path, "a") as fh:
fh.write(json.dumps(rec) + "\n")
if n == 1:
json.dump(hin, open(os.path.join(run, "hook-input-1.json"), "w"), indent=2)
if decision == "block":
sys.stderr.write(reason)
sys.exit(2)
sys.exit(0)
if __name__ == "__main__":
try:
main()
except SystemExit:
raise
except Exception:
# A crashed gate must never deadlock a run: log and allow.
run = os.environ.get("GATE_RUN", "/tmp")
with open(os.path.join(run, "gate-errors.txt"), "a") as fh:
fh.write(traceback.format_exc() + "\n")
sys.exit(0)
lib/testwriter.py
"""Gate 1 test writer: an independent headless Sonnet 5.5 that sees ONLY the task prompt (plus read-only access to a
pristine upstream copy) and writes integration-style pytest cases. The file lives OUTSIDE the work copy, in
<out dir>/gate_tests/ next to a copy of hidden/conftest.py (TINYDB_TREE) and hidden/pytest.ini.
CLI: testwriter.py <task id> <out dir> -> writes <out dir>/gate_tests/test_<T>.py and <out dir>/testwriter.json
"""
import json
import os
import shutil
import subprocess
import sys
import tempfile
import time
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from common import BENCH, HIDDEN, PY, counts, env_clean, sh # noqa: E402
MODEL = "claude-sonnet-5-5"
CALL_TIMEOUT = 600
SCHEMA = {"type": "object", "properties": {"file": {"type": "string"}}, "required": ["file"]}
FRAMING = ("You are a senior engineer writing acceptance tests for a change request to tinydb, a small Python document "
"database. You can read the CURRENT repository (before the change) with Read, Grep and Glob to learn the API; "
"you cannot run code. The change is not implemented yet: your tests must describe the requested behaviour.")
ASK = ("Write integration-style pytest cases for this request against the tinydb API, one test per behaviour a careful "
"senior would expect, pushing for edge cases beyond the happy path (missing fields, wrong types, boundaries, "
"persistence, cache, existing behaviour unchanged). Tests only, no implementation. Return the file content.\n"
"Constraints: one self-contained pytest module (field `file`); `tinydb` is importable (the tree under test is put "
"first on sys.path); do not import anything from the repo's tests/ directory or its fixtures; use "
"tinydb.storages.MemoryStorage or pytest's tmp_path for files; import the new API inside each test, not at module "
"level, so one missing name fails its tests instead of the whole module; give every test a descriptive name.")
def write(task, out_dir, attempts=2):
gt = os.path.join(out_dir, "gate_tests")
os.makedirs(gt, exist_ok=True)
shutil.copy(os.path.join(HIDDEN, "conftest.py"), gt)
shutil.copy(os.path.join(HIDDEN, "pytest.ini"), gt)
task_prompt = open(os.path.join(BENCH, "tasks", task + ".md")).read().strip()
prompt = "%s\n\n## The request (as the developer typed it)\n%s\n\n%s" % (FRAMING, task_prompt, ASK)
rec = {"task": task, "attempts": [], "cost_usd": 0.0, "ok": False, "file": None}
pristine = tempfile.mkdtemp(prefix="gb-writer-")
try:
shutil.copytree(os.path.join(BENCH, "upstream"), os.path.join(pristine, "tinydb-repo"), symlinks=True)
cwd = os.path.join(pristine, "tinydb-repo")
for i in range(attempts):
t0 = time.time()
a = {"error": None}
args = ["claude", "-p", prompt, "--model", MODEL, "--tools", "Read,Grep,Glob", "--max-turns", "12",
"--output-format", "json", "--json-schema", json.dumps(SCHEMA), "--permission-mode", "dontAsk",
"--setting-sources", "", "--settings", '{"disableAllHooks":true}', "--strict-mcp-config"]
env = env_clean({"CLAUDE_CODE_DISABLE_AUTO_MEMORY": "1", "CLAUDE_CODE_DISABLE_CLAUDE_MDS": "1"})
try:
p = subprocess.run(args, cwd=cwd, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
stdin=subprocess.DEVNULL, timeout=CALL_TIMEOUT, env=env)
r = json.loads(p.stdout.decode("utf-8", "replace"))
a.update(cost_usd=r.get("total_cost_usd") or 0.0, turns=r.get("num_turns"),
models=sorted((r.get("modelUsage") or {}).keys()), subtype=r.get("subtype"))
so = r.get("structured_output")
src = so.get("file") if isinstance(so, dict) else None
if r.get("is_error") or not src:
a["error"] = "is_error=%s subtype=%s terminal=%s" % (r.get("is_error"), r.get("subtype"),
r.get("terminal_reason"))
else:
compile(src, "test_%s.py" % task, "exec")
if "def test_" not in src:
raise ValueError("no test functions")
open(os.path.join(gt, "test_%s.py" % task), "w").write(src)
rec["ok"], rec["file"] = True, os.path.join(gt, "test_%s.py" % task)
except subprocess.TimeoutExpired:
a["error"] = "timeout after %ss" % CALL_TIMEOUT
except Exception as e:
a["error"] = "%s: %s" % (type(e).__name__, str(e)[:300])
a["wall_s"] = round(time.time() - t0, 1)
rec["attempts"].append(a)
rec["cost_usd"] += a.get("cost_usd") or 0.0
if rec["ok"]:
break
finally:
shutil.rmtree(pristine, ignore_errors=True)
rec["cost_usd"] = round(rec["cost_usd"], 6)
rec["wall_s"] = round(sum(a["wall_s"] for a in rec["attempts"]), 1)
rec["models"] = sorted({m for a in rec["attempts"] for m in a.get("models") or []})
json.dump(rec, open(os.path.join(out_dir, "testwriter.json"), "w"), indent=2)
return rec
def run_gate_tests(gate_dir, task, tree, timeout=300):
"""Generated tests on `tree` (TINYDB_TREE = tree, cwd = tree). Returns (rc, out, passed, failed, failed_names).
Paths to the gate_tests dir are rewritten to `acceptance/` so the agent never learns where the file is."""
f = os.path.join(gate_dir, "test_%s.py" % task)
rc, out = sh([PY, "-m", "pytest", "-q", "--tb=short", "-rfE", "--no-header", "-c", os.path.join(gate_dir, "pytest.ini"),
"--rootdir", gate_dir, f], tree, timeout=timeout, env=env_clean({"TINYDB_TREE": tree}))
out = out.replace(os.path.realpath(gate_dir), "acceptance").replace(gate_dir, "acceptance")
rel = os.path.relpath(gate_dir, tree)
out = out.replace(rel, "acceptance")
import re
names = sorted(set(re.findall(r"(?:FAILED|ERROR) \S*::(test_\w+)", out)))
p, fl = counts(out)
if rc != 0 and fl == 0:
fl = max(1, len(names))
return rc, out, p, fl, names
if __name__ == "__main__":
r = write(sys.argv[1], os.path.abspath(sys.argv[2]))
print(json.dumps({k: r[k] for k in ("task", "ok", "cost_usd", "wall_s", "file")}))
lib/review.py
"""Gate 3 reviewers: headless `claude -p` calls returning schema-validated JSON.
CLI (validation step 4): review.py <work dir> <task id> [indep|adv] -> prints the gate3 record as JSON
"""
import json
import subprocess
import os
import sys
import time
from concurrent.futures import ThreadPoolExecutor
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from common import BENCH, env_clean, sh, work_diff # noqa: E402
MODEL = "claude-sonnet-5-5"
DIFF_CAP = 60000
CALL_TIMEOUT = 600 # seconds per reviewer call; the hook timeout is 1500
FINDING = {"type": "object",
"properties": {"severity": {"type": "string", "enum": ["blocking", "minor"]},
"file": {"type": "string"}, "line": {"type": "integer"}, "text": {"type": "string"}},
"required": ["severity", "file", "line", "text"]}
SCHEMA_REVIEW = {"type": "object",
"properties": {"approve": {"type": "boolean"}, "findings": {"type": "array", "items": FINDING}},
"required": ["approve", "findings"]}
SCHEMA_CHALLENGE = {"type": "object",
"properties": {
"verdicts": {"type": "array", "items": {
"type": "object",
"properties": {"id": {"type": "integer"},
"verdict": {"type": "string", "enum": ["upheld", "rejected"]},
"reason": {"type": "string"}},
"required": ["id", "verdict", "reason"]}},
"added_findings": {"type": "array", "items": FINDING},
"approve": {"type": "boolean"}},
"required": ["verdicts", "added_findings", "approve"]}
FRAMING = ("You are a senior engineer doing a pre-merge code review of a change to tinydb, a small Python "
"document database. You can read the repository with Read, Grep and Glob (cwd = the repo with the "
"change applied). You cannot run code. Below: the task the developer was given, then the full diff "
"(tracked and new files).")
ASK = ("Find defects: requirements not met, edge cases wrong, behaviour broken, tests that do not test the "
"requirement. Report only real defects with file and line. approve=true only when nothing blocks.")
def capped_diff(work):
d = work_diff(work)
if len(d) > DIFF_CAP:
d = d[:DIFF_CAP] + "\n[diff truncated at %d chars]\n" % DIFF_CAP
return d
def call(work, prompt, schema, rid):
"""One headless reviewer. Never raises: an error is recorded and counts as approve (fail open)."""
# Equivalent of --bare that still authenticates with the user's OAuth login (--bare reads
# ANTHROPIC_API_KEY only): no settings sources, hooks disabled, no MCP, no CLAUDE.md, no auto-memory.
args = ["claude", "-p", prompt, "--model", MODEL, "--tools", "Read,Grep,Glob", "--max-turns", "15",
"--output-format", "json", "--json-schema", json.dumps(schema), "--permission-mode", "dontAsk",
"--setting-sources", "", "--settings", '{"disableAllHooks":true}', "--strict-mcp-config"]
env = env_clean({"CLAUDE_CODE_DISABLE_AUTO_MEMORY": "1", "CLAUDE_CODE_DISABLE_CLAUDE_MDS": "1"})
t0 = time.time()
rec = {"id": rid, "approve": True, "findings": [], "cost_usd": 0.0, "turns": None, "wall_s": None, "error": None}
try:
try:
p = subprocess.run(args, cwd=work, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.DEVNULL,
timeout=CALL_TIMEOUT, env=env)
except OSError as e1: # seen once in a smoke run: transient FileNotFoundError on spawn -> one retry
rec["retry_after"] = "%r filename=%s" % (e1, getattr(e1, "filename", None))
time.sleep(3)
p = subprocess.run(args, cwd=work, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.DEVNULL,
timeout=CALL_TIMEOUT, env=env)
out = p.stdout.decode("utf-8", "replace")
try:
r = json.loads(out)
except Exception:
r = None
if not isinstance(r, dict):
rec["error"] = "no json (rc=%s): %s %s" % (p.returncode, out[-300:], p.stderr.decode("utf-8", "replace")[-300:])
else:
rec["cost_usd"] = r.get("total_cost_usd") or 0.0
rec["turns"] = r.get("num_turns")
rec["models"] = sorted((r.get("modelUsage") or {}).keys())
so = r.get("structured_output")
if r.get("is_error") or not isinstance(so, dict):
rec["error"] = "is_error=%s subtype=%s terminal=%s structured_output=%s" % (
r.get("is_error"), r.get("subtype"), r.get("terminal_reason"), type(so).__name__)
else:
rec["raw"] = so
except subprocess.TimeoutExpired:
rec["error"] = "timeout after %ss" % CALL_TIMEOUT
except Exception as e: # pragma: no cover
rec["error"] = "exception: %r filename=%s path_head=%s" % (e, getattr(e, "filename", None), os.environ.get("PATH", "")[:200])
rec["wall_s"] = round(time.time() - t0, 1)
return rec
def norm_findings(lst):
out = []
for f in lst or []:
if isinstance(f, dict):
out.append({"severity": f.get("severity") if f.get("severity") in ("blocking", "minor") else "minor",
"file": str(f.get("file", "")), "line": f.get("line"), "text": str(f.get("text", ""))})
return out
def review_prompt(task_prompt, diff):
return "%s\n\n## Task given to the developer\n%s\n\n## Diff (git diff HEAD, new files included)\n```diff\n%s\n```\n\n%s" % (
FRAMING, task_prompt.strip(), diff, ASK)
def challenge_prompt(task_prompt, diff, a_findings):
listing = "\n".join("[%d] (%s) %s:%s %s" % (i + 1, f["severity"], f["file"], f["line"], f["text"])
for i, f in enumerate(a_findings)) or "(reviewer A reported no findings)"
return ("%s\n\nA first reviewer (A) already reviewed this diff. You are the challenger: for EACH of A's findings "
"decide whether it is a real defect (upheld) or not (rejected: wrong, not required by the task, already "
"handled, or speculative), with a one-sentence reason grounded in the code; check the code before you "
"decide. Then add any real blocking defect A missed in added_findings (file and line). approve=true only "
"when no upheld or added finding blocks.\n\n## Task given to the developer\n%s\n\n## Diff (git diff HEAD, "
"new files included)\n```diff\n%s\n```\n\n## Reviewer A's findings (id in brackets)\n%s\n\n%s" % (
FRAMING, task_prompt.strip(), diff, listing,
"Answer every id in verdicts. Report only real defects."))
def gate3(work, task_prompt, variant):
diff = capped_diff(work)
reviewers, blocking = [], []
if variant == "indep":
prompt = review_prompt(task_prompt, diff)
with ThreadPoolExecutor(max_workers=2) as ex:
recs = list(ex.map(lambda rid: call(work, prompt, SCHEMA_REVIEW, rid), ["A", "B"]))
for rec in recs:
raw = rec.pop("raw", None) or {}
rec["findings"] = norm_findings(raw.get("findings"))
rec["approve_raw"] = raw.get("approve", True) if not rec["error"] else None
rb = [f for f in rec["findings"] if f["severity"] == "blocking"]
rec["approve"] = not rb # effective verdict = no blocking finding (an errored call approves)
blocking += [dict(f, by=rec["id"]) for f in rb]
reviewers.append(rec)
elif variant == "adv":
a = call(work, review_prompt(task_prompt, diff), SCHEMA_REVIEW, "A")
raw = a.pop("raw", None) or {}
a["findings"] = norm_findings(raw.get("findings"))
a["approve_raw"] = raw.get("approve", True) if not a["error"] else None
a["approve"] = not [f for f in a["findings"] if f["severity"] == "blocking"]
b = call(work, challenge_prompt(task_prompt, diff, a["findings"]), SCHEMA_CHALLENGE, "B")
rawb = b.pop("raw", None) or {}
verdicts = {}
for v in rawb.get("verdicts") or []:
if isinstance(v, dict) and isinstance(v.get("id"), int):
verdicts[v["id"]] = {"verdict": v.get("verdict"), "reason": str(v.get("reason", ""))}
b["verdicts"] = [dict(id=k, **verdicts[k]) for k in sorted(verdicts)]
b["findings"] = norm_findings(rawb.get("added_findings"))
b["approve_raw"] = rawb.get("approve", True) if not b["error"] else None
# Only upheld findings of A block. An errored challenger upholds nothing (fail open, logged).
for i, f in enumerate(a["findings"]):
v = verdicts.get(i + 1)
f["challenge"] = v["verdict"] if v else ("error" if b["error"] else "unanswered")
if f["severity"] == "blocking" and v and v["verdict"] == "upheld":
blocking.append(dict(f, by="A", upheld_by="B"))
blocking += [dict(f, by="B") for f in b["findings"] if f["severity"] == "blocking"]
b["approve"] = not blocking
reviewers = [a, b]
else:
raise ValueError(variant)
return {"ran": True, "variant": variant, "reviewers": reviewers, "blocking_findings": len(blocking),
"blocking": blocking, "pass": not blocking,
"cost_usd": round(sum(r.get("cost_usd") or 0 for r in reviewers), 6), "diff_chars": len(diff)}
def reason_text(g3):
lines = ["Code review found %d blocking defect(s):" % g3["blocking_findings"]]
for i, f in enumerate(g3["blocking"]):
lines.append("%d. %s:%s %s" % (i + 1, f["file"], f["line"], " ".join(f["text"].split())))
tail = "Fix every finding, re-run the tests, then stop."
body = "\n".join(lines)
room = 1500 - len(tail) - 2
if len(body) > room:
body = body[:room - 4] + " ..."
return body + "\n" + tail
if __name__ == "__main__":
work, task = sys.argv[1], sys.argv[2]
variant = sys.argv[3] if len(sys.argv) > 3 else "indep"
tp = open(os.path.join(BENCH, "tasks", task + ".md")).read()
g3 = gate3(os.path.abspath(work), tp, variant)
print(json.dumps(g3, indent=2))
if not g3["pass"]:
print("--- reason ---\n" + reason_text(g3))
lib/common.py
"""Shared helpers: work-copy diff, pytest runs and parsing. Used by hook.py, review.py, finalize.py."""
import os
import re
import subprocess
BENCH = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
PY = os.path.join(BENCH, ".venv", "bin", "python")
HIDDEN = os.path.join(BENCH, "hidden")
def sh(args, cwd, timeout=None, env=None):
"""Run a command, capture stdout+stderr together. Returns (rc, text). rc=124 on timeout."""
try:
p = subprocess.run(args, cwd=cwd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
stdin=subprocess.DEVNULL, timeout=timeout, env=env)
return p.returncode, p.stdout.decode("utf-8", "replace")
except subprocess.TimeoutExpired as e:
out = (e.stdout or b"").decode("utf-8", "replace")
return 124, out + "\n[timed out after %ss]" % timeout
def work_diff(work):
"""`git diff HEAD` of tracked files plus every untracked (non-ignored) file as a new-file diff.
Never touches the index (no `git add -N`), so the agent's `git status` is unaffected."""
_, d = sh(["git", "diff", "HEAD", "--no-color", "--", ".", ":(exclude).coverage"], work)
_, others = sh(["git", "ls-files", "--others", "--exclude-standard"], work)
for f in [l for l in others.splitlines() if l.strip()]:
_, nd = sh(["git", "diff", "--no-color", "--no-index", "--", "/dev/null", f], work)
d += nd
return d
def tests_touched(work):
_, mod = sh(["git", "diff", "HEAD", "--name-only", "--", "tests"], work)
_, new = sh(["git", "ls-files", "--others", "--exclude-standard", "--", "tests"], work)
return bool(mod.strip() or new.strip())
def counts(txt):
"""(passed, failed+errors) from a pytest summary line."""
p = re.findall(r"(\d+) passed", txt)
f = re.findall(r"(\d+) failed", txt)
e = re.findall(r"(\d+) errors?\b", txt)
return (int(p[-1]) if p else 0), (int(f[-1]) if f else 0) + (int(e[-1]) if e else 0)
def env_clean(extra=None):
env = {k: v for k, v in os.environ.items() if not k.startswith("CLAUDE") and not k.startswith("GATE_")}
env["PATH"] = os.path.join(BENCH, ".venv", "bin") + os.pathsep + env.get("PATH", "/usr/bin:/bin")
env["PYTHONDONTWRITEBYTECODE"] = "1"
env.update(extra or {})
return env
def run_suite(work, timeout=300):
"""Gate 1 / final suite: the repo's own pytest run (226 tests + whatever the agent wrote).
--no-cov: the repo's pytest.ini adds --cov; the coverage table would fill the 40-line tail."""
rc, out = sh([PY, "-m", "pytest", "-q", "-x", "--no-header", "-p", "no:cacheprovider", "--no-cov"],
work, timeout=timeout, env=env_clean())
p, f = counts(out)
if rc != 0 and f == 0:
f = 1 # collection error, timeout, crash: count as red
return rc, out, p, f
def run_hidden(work, task, timeout=300):
"""Ground truth: hidden tests on the work copy (cwd = work copy, TINYDB_TREE set). Output never reaches the agent."""
rc, out = sh([PY, "-m", "pytest", "-q", "-rA", "-c", os.path.join(HIDDEN, "pytest.ini"), "--rootdir", HIDDEN,
os.path.join(HIDDEN, "test_%s.py" % task)],
work, timeout=timeout, env=env_clean({"TINYDB_TREE": work}))
# Requirement level (T5 R4 is parametrized: one req = several test ids). A req passes iff none of its
# test ids failed or errored; a run that collected nothing counts every req as failed.
reqs = sorted(set(re.findall(r"^def (test_req\d+)", open(os.path.join(HIDDEN, "test_%s.py" % task)).read(), re.M)))
failed = sorted(set(re.findall(r"(?:FAILED|ERROR) \S+::(test_req\d+)", out)))
p_tests, f_tests = counts(out)
if rc != 0 and not failed:
failed = list(reqs)
if p_tests == 0 and f_tests == 0:
failed = list(reqs)
return rc, out, len(reqs) - len(failed), len(failed), failed
lib/finalize.py
"""After the worker exits: final diff, suite, hidden run, result.json and score.json. Prints one line.
Usage: finalize.py <run dir> (re-runnable: rescoring an existing run only reads/rewrites files in it)"""
import json
import os
import re
import sys
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from common import PY, env_clean, run_hidden, sh, work_diff # noqa: E402
from testwriter import run_gate_tests # noqa: E402
MODEL = "claude-sonnet-5-5"
out = os.path.abspath(sys.argv[1])
meta = json.load(open(os.path.join(out, "meta.json")))
work, task, cfg, rep = meta["work"], meta["task"], meta["cfg"], meta["rep"]
def w(name, txt):
open(os.path.join(out, name), "w").write(txt)
def rd(name, default=""):
try:
return open(os.path.join(out, name)).read().strip()
except Exception:
return default
# Final state of the work copy
w("post.diff", work_diff(work))
_, ns = sh(["git", "diff", "HEAD", "--numstat", "--", ".", ":(exclude).coverage"], work)
_, others = sh(["git", "ls-files", "--others", "--exclude-standard"], work)
for f in [l for l in others.splitlines() if l.strip()]:
try:
ns += "%d\t0\t%s\n" % (sum(1 for _ in open(os.path.join(work, f), errors="replace")), f)
except Exception:
pass
w("numstat.txt", ns)
rc, suite = sh([PY, "-m", "pytest", "-q", "--no-header", "-p", "no:cacheprovider", "--no-cov"], work,
timeout=300, env=env_clean())
w("suite.txt", suite)
sp = int((re.findall(r"(\d+) passed", suite) or [0])[-1])
sf = int((re.findall(r"(\d+) failed", suite) or [0])[-1]) + int((re.findall(r"(\d+) errors?\b", suite) or [0])[-1])
if rc != 0 and sf == 0:
sf = 1
_, hout, hp, hf, hfailed = run_hidden(work, task)
w("hidden.txt", hout)
# Worker result line
lines = []
for raw in open(os.path.join(out, "stream.jsonl"), errors="replace") if os.path.exists(os.path.join(out, "stream.jsonl")) else []:
raw = raw.strip()
if raw.startswith("{"):
try:
lines.append(json.loads(raw))
except Exception:
pass
result = next((o for o in reversed(lines) if o.get("type") == "result"), None) or {}
init = next((o for o in lines if o.get("type") == "system" and o.get("subtype") == "init"), {})
res = {k: result.get(k) for k in ("total_cost_usd", "num_turns", "usage", "modelUsage", "is_error", "subtype",
"stop_reason", "terminal_reason", "duration_ms")}
res["final_text"] = (result.get("result") or "")[-3000:]
res["has_result_line"] = bool(result)
json.dump(res, open(os.path.join(out, "result.json"), "w"), indent=2)
# Hook events seen in the stream (which hooks ran)
hook_events = sorted({"%s:%s" % (o.get("hook_event") or o.get("hook_event_name") or "?", o.get("hook_name") or "")
for o in lines if o.get("type") == "system" and str(o.get("subtype", "")).startswith("hook")})
# Gate log
gl = [json.loads(l) for l in open(os.path.join(out, "gate.jsonl")) if l.strip()] if os.path.exists(os.path.join(out, "gate.jsonl")) else []
blocks = [g for g in gl if g.get("decision") == "block"]
by_gate = {"g1": sum(1 for g in blocks if g.get("blocked_by") == "g1"),
"g3": sum(1 for g in blocks if g.get("blocked_by") == "g3")}
right = {k: sum(1 for g in blocks if g.get("blocked_by") == k and g["hidden"]["fail"] > 0) for k in ("g1", "g3")}
false = {k: sum(1 for g in blocks if g.get("blocked_by") == k and g["hidden"]["fail"] == 0) for k in ("g1", "g3")}
rev_cost, rev_findings, rev_errors, rev_models = 0.0, 0, 0, set()
for g in gl:
for r in (g.get("gate3") or {}).get("reviewers") or []:
rev_cost += r.get("cost_usd") or 0
rev_findings += len(r.get("findings") or [])
rev_errors += 1 if r.get("error") else 0
rev_models |= set(r.get("models") or [])
# Gate-1 test writer (gated-1 configs only): cost and the final state of its tests on the work copy
tw = {}
try:
tw = json.load(open(os.path.join(out, "testwriter.json")))
except Exception:
pass
cost_tw = tw.get("cost_usd") or 0.0
rev_models |= set(tw.get("models") or [])
gt_total = gt_fail = None
gdir = os.path.join(out, "gate_tests")
if os.path.exists(os.path.join(gdir, "test_%s.py" % task)):
_, gout, gp, gf, _ = run_gate_tests(gdir, task, work)
w("gatetests-final.txt", gout)
gt_total, gt_fail = gp + gf, gf
mu = result.get("modelUsage") or {}
cw = result.get("total_cost_usd") or 0.0
s = {
"task": task, "cfg": cfg, "rep": rep, "run": meta["name"],
"hidden_pass": hp, "hidden_fail": hf, "failed_reqs": hfailed,
"suite_pass": sp, "suite_fail": sf,
"hook_calls": len(gl), "blocks_total": len(blocks), "blocks_by_gate": by_gate,
"right_blocks": sum(right.values()), "false_blocks": sum(false.values()),
"right_blocks_by_gate": right, "false_blocks_by_gate": false,
"give_up": any(g.get("decision") == "give_up" for g in gl),
"ended_blocked": bool(gl) and gl[-1].get("decision") == "block",
"cost_worker_usd": round(cw, 6), "cost_reviewers_usd": round(rev_cost, 6), "cost_testwriter_usd": round(cost_tw, 6),
"cost_total_usd": round(cw + rev_cost + cost_tw, 6),
"testwriter_ok": tw.get("ok") if tw else None, "gate_tests_total": gt_total, "gate_tests_fail_final": gt_fail,
"turns": result.get("num_turns"), "wall_s": int(rd("wall_s.txt", "0") or 0),
"reviewer_findings_total": rev_findings, "reviewer_errors": rev_errors,
"gate_crashes": rd("gate-errors.txt").count("Traceback"),
"model_ok": init.get("model") == MODEL and bool(mu) and set(mu) <= {MODEL} and rev_models <= {MODEL},
"worker_models": sorted(mu), "reviewer_models": sorted(rev_models),
"is_error": result.get("is_error"), "subtype": result.get("subtype"), "exit_code": rd("exit_code.txt"),
"has_result_line": bool(result), "claude_code_version": init.get("claude_code_version"),
"hook_events_in_stream": hook_events,
"hidden_first_stop": gl[0]["hidden"] if gl else None,
}
json.dump(s, open(os.path.join(out, "score.json"), "w"), indent=2)
print("%s: hidden %d/%d suite %d/%d blocks=%d (g1 %d, g3 %d; right %d, false %d) give_up=%s ended_blocked=%s "
"gate_tests_fail=%s/%s cost=$%.3f (worker %.3f + rev %.3f + writer %.3f) turns=%s wall=%ss model_ok=%s" % (
s["run"], hp, hp + hf, sp, sp + sf, len(blocks), by_gate["g1"], by_gate["g3"], s["right_blocks"],
s["false_blocks"], s["give_up"], s["ended_blocked"], gt_fail, gt_total, s["cost_total_usd"], cw, rev_cost, cost_tw, s["turns"],
s["wall_s"], s["model_ok"]))
La configuración de .claude se describe en HARNESS.md: un settings.json con un Stop hook de tipo comando (timeout 1500) que llama a un stop.sh de una línea, más una lista permissions.deny que bloquea la lectura del hook, de la configuración y de los tests ocultos. Los revisores y el escritor se ejecutan con una fuente de configuración vacía y los hooks desactivados, de modo que el gate nunca recurre a sí mismo.
HARNESS.md
# gate-bench
What does each of Shopify's Helix gates catch when rebuilt as a Claude Code **Stop hook**, and what does it
cost? Headless Sonnet 5.5 workers implement five tinydb features (T1..T5); a Stop hook runs gate 1
(behaviour tests) and/or gate 3 (adversarial review by headless Sonnet 5.5 reviewers) every time the worker
tries to stop; hidden tests (the oracle, never visible to the agent) record the ground truth at every stop
and at the end. Built 2026-10-08 from `SPEC.md`; upstream, hidden tests, tasks and reference patches copied
from `../verify-loops-bench` (nothing there or in `../haiku-routing-bench` is modified).
## Commands (from this dir)
```bash
./validate.sh # steps 1,3,4 (+ step 2 once its run exists) -> validation.log
INJECT=$PWD/inject/break.patch RUN_TAG=val2 ./run.sh g1 T1 1 # validation step 2 (red start state)
RUN_TAG=smoke ./run.sh g13adv T1 1 # smoke (tagged runs are ignored by analyze.py)
./validate_writer.sh # gate-1 test writer on T1..T5: tests on upstream vs reference (REWRITE=1 to redo)
./gen_matrix.py # matrix.txt (50 runs, already generated)
nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 & # 3 runs in parallel (MATRIX_JOBS=3)
SKIP_DONE=1 nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 & # resume
.venv/bin/python analyze.py # results.json + results.md
.venv/bin/python lib/finalize.py runs/<run> # rescore one run from its work copy
.venv/bin/python lib/review.py work/<run> T1 adv # one gate-3 round by hand
```
Single run: `./run.sh <cfg> <task> <rep>` -> `runs/<task>-<cfg>-r<rep>/`, work copy `work/<same name>/`.
## Layout
| path | what |
|---|---|
| `SPEC.md` | the spec this harness implements |
| `upstream/` | tinydb @ 18d73a1 (git repo) |
| `hidden/` | `test_T1..T6.py` (T6 unused), `conftest.py` (`$TINYDB_TREE` or cwd first on sys.path), `pytest.ini` |
| `tasks/T1..T5.md` + `tasks/README.md` | the task prompts (verbatim) and what each hidden req checks |
| `reference/T1..T5.patch` | reference solutions (validation step 1) |
| `inject/break.patch` | `>=` -> `>` in `Query.__ge__` (one repo test red; validation steps 2-3) |
| `inject/test_pushpull.py` | one thin test, makes the T1 reference copy "green with tests touched" (step 3) |
| `configs.json` | `bare` (no hook), `g1`, `g3` (indep), `g13` (g1 then g3 indep), `g13adv` (g1 then g3 adv) |
| `env.sh` | scrubs inherited `CLAUDE*` vars, default `~/.claude` auth, `CLAUDE_CODE_DISABLE_AUTO_MEMORY=1`, venv on PATH |
| `prep.sh <cfg> <work> <run>` | fresh copy of upstream, `.claude/` in `.git/info/exclude`, hook settings for gated configs |
| `run.sh <cfg> <task> <rep>` | one run (worker + hook + `lib/finalize.py`) |
| `gate.sh` | the Stop hook (bash 3.2 wrapper: `exec .venv/bin/python -I lib/hook.py`) |
| `lib/hook.py` | gate dispatch, snapshot, hidden run, gate 1, gate 3, decision, `gate.jsonl` line |
| `lib/review.py` | gate-3 reviewers (indep / adv), schemas, prompts, reason text; CLI for a manual round |
| `lib/testwriter.py` | gate-1 test writer (before the worker) + `run_gate_tests` (path rewritten to `acceptance/`) |
| `validate_writer.sh`, `validation/writer/<T>/` | writer validation files and their runs on upstream / reference |
| `tasks/` (terse prompts, used), `tasks-explicit/` + `runs-explicit/` | the matrix uses terse prompts; see `tasks/README.md` |
| `lib/common.py` | diff incl. untracked files, suite run, hidden run (requirement level) |
| `lib/finalize.py` | post-run diff/suite/hidden, `result.json`, `score.json` |
| `gen_matrix.py`, `matrix.txt`, `matrix.sh`, `matrix-one.sh`, `matrix.log` | 50-run matrix, 3 in parallel (xargs -P), resumable |
| `analyze.py` -> `results.json`, `results.md` | per config, per task, the five questions |
| `validate.sh`, `VALIDATION.md`, `validation.log` | pre-matrix checks |
| `.venv/` | python 3.14.7, pytest 9.1.1, pytest-cov 7.1.0, pyyaml, ruff 0.16.10 (same pins as haiku-routing-bench) |
### Worker (run.sh)
`claude -p "<T-file verbatim>\n\nWhen you are done, stop and summarise what you changed." --model claude-sonnet-5-5
--output-format stream-json --verbose --include-hook-events --max-turns 60 --permission-mode acceptEdits
--allowedTools <haiku-routing-bench list> --setting-sources project --strict-mcp-config`, cwd = `work/<run>/`, stdin /dev/null.
Gated configs get `work/<run>/.claude/settings.json`:
`{"permissions":{"deny":[...see traps...]},"hooks":{"Stop":[{"hooks":[{"type":"command","command":"bash \"$CLAUDE_PROJECT_DIR/.claude/hooks/stop.sh\"","timeout":1500}]}]}}`
and `.claude/hooks/stop.sh` = `GATE_RUN=<abs run dir> GATE_CONFIG=<cfg> bash <bench>/gate.sh` (the SPEC command, one level down: see traps).
Copies of both land in the run dir (`settings.json`, `stop.sh`).
### Hook (lib/hook.py), one invocation
1. read stdin JSON; n = lines already in `gate.jsonl` + 1; blocks so far from the same file
2. `snap-<n>.diff` = `git diff HEAD` + untracked files (index never touched)
3. hidden tests on the work copy (`TINYDB_TREE` = work copy, cwd = work copy) -> `hidden-<n>.txt`; never shown to the agent
4. gate 1 (if configured) = the repo suite `python -m pytest -q -x --no-header -p no:cacheprovider --no-cov` (300 s)
-> `gate1-<n>.txt` AND the writer's acceptance tests `runs/<run>/gate_tests/test_<T>.py` run with `TINYDB_TREE` = work
copy -> `gatetests-<n>.txt`. Block when either is red. Reason: "The acceptance tests for this change fail:" + last 40
lines (failing test names + assertion output, the gate_tests path rewritten to `acceptance/`), then the suite tail if the
suite is red too; <= 1500 chars. `tests_touched` is still logged but no longer blocks. No writer file (writer failed
twice) -> suite only, logged as `gate_tests_present: false`.
5. gate 3 (if configured and gate 1 passed or absent): see below
6. decision: allow / block / give_up (a would-be block when 6 blocks already happened -> allow, logged `give_up`).
`stop_hook_active` is logged, never used. Block = exit 2 + reason on stderr. A crash of the hook itself is
logged to `gate-errors.txt` and allows (counted as `gate_crashes` in score.json).
### Gate 1 test writer (lib/testwriter.py, run.sh before the worker, configs g1/g13/g13adv)
`claude -p` Sonnet 5.5 with the reviewer isolation flags, `--tools Read,Grep,Glob --max-turns 12 --json-schema
{"file": string}`, cwd = a pristine temp copy of upstream (deleted after). Prompt = framing + `tasks/<T>.md` + the
coordinator's instruction ("Write integration-style pytest cases for this request against the tinydb API, one test per
behaviour a careful senior would expect, pushing for edge cases beyond the happy path (missing fields, wrong types,
boundaries, persistence, cache, existing behaviour unchanged). Tests only, no implementation. Return the file content.")
+ mechanics (self-contained module, imports of the new API inside each test, MemoryStorage/tmp_path). The file must
compile and contain `def test_`, else one retry. Output `runs/<run>/gate_tests/` (+ hidden's conftest.py with
`TINYDB_TREE`, pytest.ini) and `testwriter.json` (cost, turns, attempts). Fixed per-run cost, outside the worker's wall.
### Gate 3 (lib/review.py)
Each reviewer: `claude -p "<prompt>" --model claude-sonnet-5-5 --tools Read,Grep,Glob --max-turns 15 --output-format json
--json-schema <schema> --permission-mode dontAsk --setting-sources "" --settings '{"disableAllHooks":true}' --strict-mcp-config`
with `CLAUDE_CODE_DISABLE_CLAUDE_MDS=1 CLAUDE_CODE_DISABLE_AUTO_MEMORY=1`, all inherited `CLAUDE*` vars dropped, cwd = work copy,
600 s timeout. Prompt = framing + task prompt (T-file) + diff (tracked + untracked, cap 60k chars) + the SPEC's "Find defects..."
sentence. `structured_output` is read from the JSON result; `total_cost_usd`, `num_turns`, wall and models are recorded.
- `indep`: reviewers A and B, same prompt, run in parallel, no shared context. Blocking findings = union. Pass when both approve.
- `adv`: A as above; challenger B gets A's numbered findings + the diff, answers `upheld|rejected` + reason per id, may add
findings (`added_findings`). Blocking = A's blocking findings B upheld + B's added blocking findings. B always runs (same
2-call budget as indep).
- Effective `approve` = "no blocking finding" (the raw `approve` is logged as `approve_raw`; a reviewer saying approve=false with
only minor findings does not block). A reviewer call that errors approves (fail open) with `error` recorded; an errored
challenger upholds nothing.
- Reason = "Code review found N blocking defect(s):" + numbered `file:line text` + "Fix every finding, re-run the tests, then stop.", <= 1500 chars.
### Per run (`runs/<task>-<cfg>-r<rep>/`)
`stream.jsonl`, `stderr.txt`, `exit_code.txt`, `wall_s.txt` (includes hook time: the hook runs inside the session),
`gate.jsonl`, `snap-<n>.diff`, `hidden-<n>.txt`, `gate1-<n>.txt`, `hook-input-1.json`, `post.diff`, `numstat.txt`,
`suite.txt` (full suite, no -x), `hidden.txt`, `gate_tests/`, `testwriter.json`, `gatetests-<n>.txt`, `gatetests-final.txt`, `result.json`, `score.json`, `meta.json`, `task.md`, `prompt.txt`,
`settings.json` + `stop.sh` (gated configs).
`score.json`: task, cfg, rep, hidden_pass/hidden_fail/failed_reqs (**requirement level**: 5 reqs per task), suite_pass/fail,
hook_calls, blocks_total, blocks_by_gate {g1,g3}, right_blocks / false_blocks (+ `_by_gate`), give_up, ended_blocked,
cost_worker_usd, cost_reviewers_usd, cost_testwriter_usd, cost_total_usd (all three), testwriter_ok, gate_tests_total,
gate_tests_fail_final, turns, wall_s, reviewer_findings_total, reviewer_errors, gate_crashes,
model_ok (worker init model + every modelUsage key + every reviewer model == claude-sonnet-5-5), hook_events_in_stream,
hidden_first_stop.
## Matrix
5 tasks x 5 configs x 2 reps = 50 runs. Rep 1 config blocks bare, g1, g3, g13, g13adv (tasks rotated per block), rep 2 the
reverse block order. Up to 3 runs at once, each in its own work copy (`matrix-one.sh` via `xargs -P 3`, bash 3.2 has no
`wait -n`). `matrix.log`: one `start` and one result line per run.
## Traps found
- **Gate 1 as first built passed by construction** (it only re-ran the implementer's own tests: `T2-g1-r1` one allow, hidden
R4 missed). Redesigned around an independent test writer (above). The writer hedges where the terse prompt is silent
(accepts "raises or no-op"), so it cannot recover unstated requirements; per-task overlap is in VALIDATION.md.
- **A blocked agent hunts for the tests.** After a gate-1 block the smoke agent ran `cat .claude/hooks/stop.sh; find / -name
test_T1.py` (denied only because `find` is not on the allow-list; `cat` is, and `stop.sh` names the bench dir, from where
`hidden/` is one `cat` away). Gated configs' `.claude/settings.json` now carries `permissions.deny`: Read of `./.claude/**`,
the bench's `hidden/ runs/ validation/ reference/`, the two sibling benches; Bash patterns containing `.claude/hooks`,
`.claude/settings`, `../hidden`, `../runs`, `gate_tests`, the bench sub-paths, `find *`. Probed with a Haiku session: all
five escape attempts (cat, Read, ls, python open) denied, in-tree reads fine. Bash rules match command text, not a security
boundary. Bare runs have no settings file (bare r1 ran before this change; bare has no motive to hunt).
Reviewers and the writer run with `--setting-sources ""`, so these rules do not apply to them.
- **Transient `FileNotFoundError` spawning a reviewer** inside the hook (once, both reviewers of one call, wall 0.0 s;
never reproduced by hand). Reviewer calls now retry once after 3 s; the error message records `filename`.
- **`--bare` cannot authenticate here.** It reads `ANTHROPIC_API_KEY` / `apiKeyHelper` only, never OAuth or the keychain;
this account is OAuth (`~/.claude`), so a `--bare` reviewer returns `terminal_reason: api_error`, cost 0. Used the
documented equivalents instead: `--setting-sources ""` + `--settings '{"disableAllHooks":true}'` + `--strict-mcp-config` +
`CLAUDE_CODE_DISABLE_CLAUDE_MDS=1` + `CLAUDE_CODE_DISABLE_AUTO_MEMORY=1` + `--tools Read,Grep,Glob`. The reviewers' cwd is the
work copy, whose `.claude/settings.json` carries the Stop hook: without `--setting-sources ""`/`disableAllHooks` a reviewer
would run the gate recursively. Verified: a dummy Stop hook in the probe copy never fired.
- **Inherited session env.** A matrix launched from inside Claude Code inherits `CLAUDE_EFFORT=medium`,
`CLAUDE_CODE_SUBAGENT_MODEL=opus`, `CLAUDE_CODE_SESSION_ID`, `CLAUDECODE`, `CLAUDE_CODE_CHILD_SESSION`, messaging tokens.
`env.sh` unsets every `CLAUDE*` var before setting its own; reviewers get a scrubbed env too (the hook process itself
inherits the worker's `CLAUDE_*` vars, e.g. the worker's session id).
- **The feedback line leaks the hook command.** Claude Code prefixes each block with the command line:
`Stop hook feedback:\n[<command>]: <reason>`. With the SPEC command inline the agent would read `GATE_CONFIG=g13adv` and
`gate-bench/gate.sh`. The settings command is therefore `bash "$CLAUDE_PROJECT_DIR/.claude/hooks/stop.sh"`, a one-line
wrapper holding the SPEC command. The agent can still `cat` it (it lives in the work copy) but it is not pushed at it.
- **Stop hook input keys (2.1.293):** `background_tasks, cwd, effort ({"level":"medium"}), hook_event_name,
last_assistant_message, permission_mode, prompt_id, session_crons, session_id, stop_hook_active, transcript_path`.
- **8-cap resets on tool calls.** Per the docs the consecutive-continuation count resets each time Claude calls a tool, so the
built-in cap only bites an agent that answers blocks with text alone. Our give-up (6 blocks per run, counted from
`gate.jsonl`, tool calls or not) fires first in every case.
- **Red start state: the agent refuses to fix code it did not write.** Validation step 2 (T1 + an uncommitted `>=` -> `>`
edit) shows Sonnet 5.5 implementing T1, then answering 6 consecutive gate-1 blocks with "the red test is your pre-existing
uncommitted edit, not my change" without touching it, until give_up. The hook mechanics are validated; the behaviour is a
finding in itself (gate 1 cannot force a fix the agent considers out of scope).
- **pytest-cov.** The repo's `pytest.ini` adds `--cov tinydb`; gate 1 adds `--no-cov` so the 40-line tail holds the failure,
not the coverage table (and no `.coverage` file is written). `.coverage` is gitignored anyway.
- **T5 R4 is parametrized** (several test ids, one requirement). hidden counts are per requirement (5 per task), a req
passes only when all its ids pass. `validate.sh` step 1: T5 upstream passes 1 req (R3, the regression guard) as designed.
- **Diff without touching the index.** `git add -N` would change what the agent sees in `git status`; untracked files are
diffed with `git diff --no-index /dev/null <f>` instead. `.claude/` is in `.git/info/exclude`, so the hook config never
shows in any diff or in `git status`.
- **Worker Bash denials.** The 2.1.293 permission layer denies some agent commands with "Contains brace with quote character
(expansion obfuscation)" even under acceptEdits + allow-list; agents route around it (visible as `permission_denied` events).
- **Reviewers on a correct solution still flag "blocking".** On the T1 reference patch, reviewer A marked "pull does not check
the field is a list" (not required by the task) as blocking; indep therefore blocks, the adv challenger rejected it. This is
exactly what false_blocks measures (hidden all green at that stop).
- **Shared plan meter.** Workers and reviewers bill the user's plan quota; `total_cost_usd` is the list-price equivalent the
CLI reports, not a bill. Nothing else heavy should run on the account during the matrix.
- **rtk rewrites interactive `grep`/`diff`** in the launching Claude session only; scripts here call binaries directly
(`/usr/bin/diff`/`cmp` for verification diffs).
- **CLI version:** `claude --version` = 2.1.293 (Claude Code); flags checked in `claude --help`: `--json-schema`, `--bare`, `--tools`,
`--permission-mode` (choices acceptEdits, auto, bypassPermissions, manual, dontAsk, plan), `--setting-sources`,
`--include-hook-events`. `--max-turns` is accepted though not listed in `--help`.
## Matrix launch (2026-10-08 16:33:54, after the gate-1 redesign)
`cd /Users/thomas/dev/gate-bench && SKIP_DONE=1 nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 &` -> PID 71317, 3 lanes.
The 5 bare r1 runs (terse prompts) are kept and skipped; 45 runs to go. Expected wall ~40-70 min: gated runs so far take
~30-70 s of worker wall + ~35-40 s of test writer (before the worker) + 1-35 s per hook call, more with blocks.
Then `.venv/bin/python analyze.py`.
El runner:
run.sh: una ejecución, trabajador más hook más puntuación.run.sh
#!/usr/bin/env bash
# One run: fresh work copy, headless Sonnet 5.5 worker, Stop hook per config, final scoring.
# Usage: run.sh <cfg> <task> <rep> cfg: bare|g1|g3|g13|g13adv task: T1..T5
# RUN_TAG=x -> runs/<task>-<cfg>-r<rep>-x (smoke runs stay out of the matrix)
# INJECT=<patch> -> git apply it to the work copy before the worker starts (validation: red start state)
set -uo pipefail
CFG="$1"; TASK="$2"; REP="${3:-1}"
HERE="$(cd "$(dirname "$0")" && pwd)"
. "$HERE/env.sh"
PY="$HERE/.venv/bin/python"
MAX_TURNS="${BENCH_MAX_TURNS:-60}"
NAME="${TASK}-${CFG}-r${REP}${RUN_TAG:+-$RUN_TAG}"
OUT="$HERE/runs/$NAME"; W="$HERE/work/$NAME"
rm -rf "$OUT"; mkdir -p "$OUT"
"$HERE/prep.sh" "$CFG" "$W" "$OUT" > "$OUT/prep.txt" 2>&1 || { echo "$NAME: prep failed"; cat "$OUT/prep.txt"; exit 2; }
if [ -n "${INJECT:-}" ]; then ( cd "$W" && git apply "$INJECT" ) || { echo "$NAME: inject failed"; exit 2; }; fi
cp "$HERE/tasks/$TASK.md" "$OUT/task.md"
printf '%s\n\nWhen you are done, stop and summarise what you changed.\n' "$(cat "$HERE/tasks/$TASK.md")" > "$OUT/prompt.txt"
printf '{"task":"%s","cfg":"%s","rep":%s,"work":"%s","name":"%s"}\n' "$TASK" "$CFG" "$REP" "$W" "$NAME" > "$OUT/meta.json"
# Gate 1 configs: the independent test writer runs BEFORE the worker (fixed per-run cost), sees only tasks/<T>.md,
# writes $OUT/gate_tests/test_<T>.py outside the work copy.
if "$PY" -c "import json,sys; sys.exit(0 if json.load(open('$HERE/configs.json'))['$CFG']['gate1'] else 1)"; then
"$PY" "$HERE/lib/testwriter.py" "$TASK" "$OUT" > "$OUT/testwriter.out" 2>&1
fi
[ -f "$W/.claude/settings.json" ] && cp "$W/.claude/settings.json" "$OUT/settings.json" && cp "$W/.claude/hooks/stop.sh" "$OUT/stop.sh"
# Same allow-list as haiku-routing-bench/run.sh (verify-loops-bench's + Bash(ruff:*)).
TOOLS="Read,Edit,Write,MultiEdit,Glob,Grep,Skill,TodoWrite,Bash(uv:*),Bash(python:*),Bash(python3:*),Bash(pytest:*),Bash(ruff:*),Bash(git status:*),Bash(git diff:*),Bash(git log:*),Bash(ls:*),Bash(cat:*),Bash(mkdir:*),Bash(mktemp:*),Bash(head:*),Bash(tail:*),Bash(grep:*),Bash(wc:*)"
cd "$W"
START=$(date +%s)
claude -p "$(cat "$OUT/prompt.txt")" --model claude-sonnet-5-5 \
--output-format stream-json --verbose --include-hook-events --max-turns "$MAX_TURNS" \
--permission-mode acceptEdits --allowedTools "$TOOLS" \
--setting-sources project --strict-mcp-config \
< /dev/null > "$OUT/stream.jsonl" 2> "$OUT/stderr.txt"
echo "$?" > "$OUT/exit_code.txt"
END=$(date +%s)
echo "$((END-START))" > "$OUT/wall_s.txt"
"$PY" "$HERE/lib/finalize.py" "$OUT"
prep.sh
#!/usr/bin/env bash
# Fresh work copy of upstream tinydb (git clean at HEAD) + the Stop hook settings for gated configs.
# Usage: prep.sh <cfg> <work dir> <run dir>
set -euo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"
CFG="$1"; W="$2"; RUN="$3"
rm -rf "$W"; mkdir -p "$(dirname "$W")"
cp -R "$HERE/upstream" "$W"
cd "$W"
/usr/bin/find . -name __pycache__ -type d -prune -exec rm -rf {} +
rm -f .coverage
# .claude/ is excluded locally (not committed, not in .gitignore): `git status` stays clean,
# the hook config never shows up in the diffs the reviewers and the scorer read.
echo ".claude/" >> .git/info/exclude
git update-index -q --refresh || true
if [ "$CFG" != "bare" ]; then
# The settings command is a neutral wrapper: Claude Code prefixes every "Stop hook feedback" with the
# hook's command line, so the SPEC command (GATE_RUN=... GATE_CONFIG=<cfg> bash <bench>/gate.sh) would
# show the agent the config name and the bench path. The wrapper holds that exact command instead.
mkdir -p .claude/hooks
printf '#!/bin/bash\nGATE_RUN=%s GATE_CONFIG=%s bash %s/gate.sh\n' "$RUN" "$CFG" "$HERE" > .claude/hooks/stop.sh
"$HERE/.venv/bin/python" - "$HERE" > .claude/settings.json <<'PY'
import json
import sys
bench = sys.argv[1]
cmd = 'bash "$CLAUDE_PROJECT_DIR/.claude/hooks/stop.sh"'
# Once a gate blocks, the agent hunts for the tests behind it (seen in a smoke: `cat .claude/hooks/stop.sh;
# find / -name test_T1.py`). The oracle and the gate tests live under the bench: deny the usual forms.
# Read rules cover Read/Grep/Glob; Bash rules match the command text (not a security boundary).
deny = ["Read(./.claude/**)", "Read(/%s/hidden/**)" % bench, "Read(/%s/runs/**)" % bench,
"Read(/%s/validation/**)" % bench, "Read(/%s/reference/**)" % bench,
"Read(//Users/thomas/dev/verify-loops-bench/**)", "Read(//Users/thomas/dev/haiku-routing-bench/**)",
"Bash(*.claude/hooks*)", "Bash(*.claude/settings*)", "Bash(*gate-bench/hidden*)", "Bash(*../hidden*)",
"Bash(*gate-bench/runs*)", "Bash(*../runs*)", "Bash(*gate_tests*)", "Bash(*gate-bench/reference*)",
"Bash(*gate-bench/validation*)", "Bash(*verify-loops-bench*)", "Bash(*haiku-routing-bench*)", "Bash(find *)"]
print(json.dumps({"permissions": {"deny": deny},
"hooks": {"Stop": [{"hooks": [{"type": "command", "command": cmd, "timeout": 1500}]}]}}, indent=2))
PY
fi
test -z "$(git status --porcelain)" || { echo "work copy not clean"; git status --porcelain; exit 1; }
env.sh
# Sourced by run.sh, gate.sh and validate.sh.
# 1. Scrub every CLAUDE* variable inherited from a launching Claude Code session
# (CLAUDE_EFFORT, CLAUDE_CODE_SUBAGENT_MODEL, CLAUDE_CODE_SESSION_ID, CLAUDECODE,
# CLAUDE_CODE_CHILD_SESSION, ...): they would silently change the worker's effort,
# subagent model and session bookkeeping.
for _v in $(env | sed -n 's/^\(CLAUDE[A-Za-z0-9_]*\)=.*/\1/p'); do unset "$_v"; done
unset _v
# 2. Auth: the user's default config dir (~/.claude, OAuth). See README traps (--bare cannot auth).
unset CLAUDE_CONFIG_DIR
# 3. Fresh work copies at the same path must not share auto-memory across reps.
export CLAUDE_CODE_DISABLE_AUTO_MEMORY=1
# 4. No bytecode in the shared hidden/ dir when 3 runs execute hidden tests in parallel.
export PYTHONDONTWRITEBYTECODE=1
# 5. The bench venv first: `python -m pytest` inside the work copy uses it.
GATE_BENCH="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
export GATE_BENCH
export PATH="$GATE_BENCH/.venv/bin:$PATH"
configs.json
{
"bare": {"gate1": false, "gate3": null, "desc": "no hook"},
"g1": {"gate1": true, "gate3": null, "desc": "gate 1 (behaviour tests) only"},
"g3": {"gate1": false, "gate3": "indep", "desc": "gate 3, two independent reviewers"},
"g13": {"gate1": true, "gate3": "indep", "desc": "gate 1 then gate 3 indep (Shopify order)"},
"g13adv": {"gate1": true, "gate3": "adv", "desc": "gate 1 then gate 3 reviewer + challenger"}
}
gen_matrix.py
#!/usr/bin/env python3
"""Write matrix.txt: "<cfg> <task> <rep>" lines. Rep 1: configs bare,g1,g3,g13,g13adv as contiguous blocks,
tasks rotated per block; rep 2: same blocks in reverse config order (ABBA over time)."""
CFGS = ["bare", "g1", "g3", "g13", "g13adv"]
TASKS = ["T1", "T2", "T3", "T4", "T5"]
lines = []
for rep, order in ((1, CFGS), (2, CFGS[::-1])):
lines.append("# rep %d" % rep)
for i, c in enumerate(order):
k = (i + rep - 1) % len(TASKS)
for t in TASKS[k:] + TASKS[:k]:
lines.append("%s %s %d" % (c, t, rep))
open("matrix.txt", "w").write("\n".join(lines) + "\n")
print(sum(1 for l in lines if not l.startswith("#")), "runs")
matrix.txt
# rep 1
bare T1 1
bare T2 1
bare T3 1
bare T4 1
bare T5 1
g1 T2 1
g1 T3 1
g1 T4 1
g1 T5 1
g1 T1 1
g3 T3 1
g3 T4 1
g3 T5 1
g3 T1 1
g3 T2 1
g13 T4 1
g13 T5 1
g13 T1 1
g13 T2 1
g13 T3 1
g13adv T5 1
g13adv T1 1
g13adv T2 1
g13adv T3 1
g13adv T4 1
# rep 2
g13adv T2 2
g13adv T3 2
g13adv T4 2
g13adv T5 2
g13adv T1 2
g13 T3 2
g13 T4 2
g13 T5 2
g13 T1 2
g13 T2 2
g3 T4 2
g3 T5 2
g3 T1 2
g3 T2 2
g3 T3 2
g1 T5 2
g1 T1 2
g1 T2 2
g1 T3 2
g1 T4 2
bare T1 2
bare T2 2
bare T3 2
bare T4 2
bare T5 2
matrix.sh
#!/usr/bin/env bash
# Run an ordered list of "<cfg> <task> <rep>" lines, up to MATRIX_JOBS (default 3) at a time, each run in
# its own work copy. Blank lines and #comments skipped. Appends to matrix.log. Resumable: SKIP_DONE=1.
# Usage: nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 &
set -uo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"; cd "$HERE"
LIST="${1:-matrix.txt}"; JOBS="${MATRIX_JOBS:-3}"
mkdir "$HERE/.matrix.lock" 2>/dev/null || { echo "another matrix.sh is running (remove .matrix.lock if stale)" >&2; exit 1; }
trap 'rmdir "$HERE/.matrix.lock"' EXIT
echo "=== $(date '+%F %T') start $LIST jobs=$JOBS pid=$$" >> matrix.log
grep -vE '^[[:space:]]*(#|$)' "$LIST" | xargs -P "$JOBS" -L 1 ./matrix-one.sh
echo "=== $(date '+%F %T') done $LIST" >> matrix.log
matrix-one.sh
#!/usr/bin/env bash
# One matrix cell (called by matrix.sh through xargs). Args: <cfg> <task> <rep>
HERE="$(cd "$(dirname "$0")" && pwd)"; cd "$HERE"
CFG="$1"; TASK="$2"; REP="$3"
if [ "${SKIP_DONE:-0}" = 1 ] && [ -f "runs/${TASK}-${CFG}-r${REP}/score.json" ]; then
echo "$(date '+%F %T') skip ${TASK}-${CFG}-r${REP}" >> matrix.log; exit 0
fi
echo "$(date '+%F %T') start ${TASK}-${CFG}-r${REP}" >> matrix.log
line=$(./run.sh "$CFG" "$TASK" "$REP" < /dev/null 2>&1 | tail -1)
echo "$(date '+%F %T') $line" >> matrix.log
echo "$line"
analyze.py
#!/usr/bin/env python3
"""Aggregate runs/*/score.json (matrix runs only: <task>-<cfg>-r<rep>, no tag) -> results.json + results.md."""
import glob
import json
import os
import re
HERE = os.path.dirname(os.path.abspath(__file__))
CFGS = ["bare", "g1", "g3", "g13", "g13adv"]
TASKS = ["T1", "T2", "T3", "T4", "T5"]
rows = []
for p in sorted(glob.glob(os.path.join(HERE, "runs", "*", "score.json"))):
if re.fullmatch(r"T\d-[a-z0-9]+-r\d+", os.path.basename(os.path.dirname(p))):
rows.append(json.load(open(p)))
def mean(xs):
xs = [x for x in xs if x is not None]
return round(sum(xs) / len(xs), 4) if xs else None
def agg(rs):
if not rs:
return None
n = len(rs)
hp = sum(r["hidden_pass"] for r in rs)
ht = sum(r["hidden_pass"] + r["hidden_fail"] for r in rs)
rb = {g: sum(r["right_blocks_by_gate"][g] for r in rs) for g in ("g1", "g3")}
fb = {g: sum(r["false_blocks_by_gate"][g] for r in rs) for g in ("g1", "g3")}
return {
"runs": n, "hidden_reqs_passed": hp, "hidden_reqs_total": ht, "hidden_rate": round(hp / ht, 4) if ht else None,
"runs_fully_passing": sum(1 for r in rs if r["hidden_fail"] == 0),
"suite_green_runs": sum(1 for r in rs if r["suite_fail"] == 0),
"mean_cost_total_usd": mean([r["cost_total_usd"] for r in rs]),
"mean_cost_worker_usd": mean([r["cost_worker_usd"] for r in rs]),
"mean_cost_reviewers_usd": mean([r["cost_reviewers_usd"] for r in rs]),
"mean_cost_testwriter_usd": mean([r.get("cost_testwriter_usd") or 0 for r in rs]),
# per-gate cost: gate 1 = the test writer (its tests then run locally for free), gate 3 = the reviewers
"mean_cost_gate1_usd": mean([r.get("cost_testwriter_usd") or 0 for r in rs]),
"mean_cost_gate3_usd": mean([r["cost_reviewers_usd"] for r in rs]),
"gate_tests_total": sum(r.get("gate_tests_total") or 0 for r in rs),
"gate_tests_fail_final": sum(r.get("gate_tests_fail_final") or 0 for r in rs),
"mean_wall_s": mean([r["wall_s"] for r in rs]), "mean_turns": mean([r["turns"] for r in rs]),
"blocks_per_run": round(sum(r["blocks_total"] for r in rs) / n, 3),
"blocks_by_gate": {g: sum(r["blocks_by_gate"][g] for r in rs) for g in ("g1", "g3")},
"right_blocks": rb, "false_blocks": fb,
"false_block_rate": {g: (round(fb[g] / (rb[g] + fb[g]), 3) if rb[g] + fb[g] else None) for g in ("g1", "g3")},
"give_ups": sum(1 for r in rs if r["give_up"]), "ended_blocked": sum(1 for r in rs if r["ended_blocked"]),
"reviewer_findings": sum(r["reviewer_findings_total"] for r in rs),
"reviewer_errors": sum(r.get("reviewer_errors", 0) for r in rs),
"model_ok": all(r["model_ok"] for r in rs),
"missed_reqs": sorted(set("%s:%s" % (r["task"], q) for r in rs for q in r["failed_reqs"])),
}
by_cfg = {c: agg([r for r in rows if r["cfg"] == c]) for c in CFGS}
by_task = {t: {c: agg([r for r in rows if r["task"] == t and r["cfg"] == c]) for c in CFGS} for t in TASKS}
by_task_all = {t: agg([r for r in rows if r["task"] == t]) for t in TASKS}
def rate(c):
return (by_cfg.get(c) or {}).get("hidden_rate")
def d(a, b):
return round(a - b, 4) if a is not None and b is not None else None
answers = {
"q1_gate1_vs_bare": {"bare": rate("bare"), "g1": rate("g1"), "delta": d(rate("g1"), rate("bare"))},
"q2_gate3_over_g1": {"g1": rate("g1"), "g13": rate("g13"), "delta": d(rate("g13"), rate("g1")),
"g3_alone": rate("g3"), "g3_vs_bare": d(rate("g3"), rate("bare"))},
"q3_adv_vs_indep": {"g13": rate("g13"), "g13adv": rate("g13adv"), "delta": d(rate("g13adv"), rate("g13")),
"false_blocks_g3": {c: (by_cfg.get(c) or {}).get("false_blocks", {}).get("g3") for c in ("g13", "g13adv")}},
"q4_false_block_rate": {c: (by_cfg.get(c) or {}).get("false_block_rate") for c in CFGS},
"q4_gate1_false_blocks": {c: {"false": (by_cfg.get(c) or {}).get("false_blocks", {}).get("g1"),
"right": (by_cfg.get(c) or {}).get("right_blocks", {}).get("g1")}
for c in ("g1", "g13", "g13adv")},
"q5_cost": {c: {k: (by_cfg.get(c) or {}).get(k) for k in ("mean_cost_total_usd", "mean_cost_worker_usd",
"mean_cost_gate1_usd", "mean_cost_gate3_usd", "mean_wall_s")} for c in CFGS},
}
json.dump({"n_runs": len(rows), "by_cfg": by_cfg, "by_task": by_task, "by_task_all": by_task_all,
"answers": answers, "runs": rows}, open(os.path.join(HERE, "results.json"), "w"), indent=2)
L = ["# gate-bench results", "", "%d runs scored (matrix runs only)." % len(rows), "",
"## Per config", "",
"| cfg | runs | hidden reqs | fully passing | suite green | mean $ total (worker + g1 writer + g3 reviewers) | mean wall s | blocks/run | g1 right/false | g3 right/false | give-ups | ended blocked | model ok |",
"|---|---|---|---|---|---|---|---|---|---|---|---|---|"]
for c in CFGS:
a = by_cfg[c]
if not a:
L.append("| %s | 0 |" % c); continue
L.append("| %s | %d | %d/%d (%.0f%%) | %d | %d | %.3f (%.3f + %.3f + %.3f) | %.0f | %.2f | %d/%d | %d/%d | %d | %d | %s |" % (
c, a["runs"], a["hidden_reqs_passed"], a["hidden_reqs_total"], 100 * (a["hidden_rate"] or 0),
a["runs_fully_passing"], a["suite_green_runs"], a["mean_cost_total_usd"] or 0, a["mean_cost_worker_usd"] or 0,
a["mean_cost_testwriter_usd"] or 0, a["mean_cost_reviewers_usd"] or 0, a["mean_wall_s"] or 0, a["blocks_per_run"], a["right_blocks"]["g1"],
a["false_blocks"]["g1"], a["right_blocks"]["g3"], a["false_blocks"]["g3"], a["give_ups"], a["ended_blocked"],
a["model_ok"]))
L += ["", "## Per task (hidden reqs passed / total, per config)", "", "| task | " + " | ".join(CFGS) + " | missed reqs (all configs) |",
"|---|" + "---|" * (len(CFGS) + 1)]
for t in TASKS:
cells = []
for c in CFGS:
a = by_task[t][c]
cells.append("%d/%d, $%.2f, %.1f blk" % (a["hidden_reqs_passed"], a["hidden_reqs_total"], a["mean_cost_total_usd"] or 0,
a["blocks_per_run"]) if a else "-")
L.append("| %s | %s | %s |" % (t, " | ".join(cells), ", ".join((by_task_all[t] or {}).get("missed_reqs", []))))
L += ["", "## The video's questions", ""]
for k, v in answers.items():
L.append("- **%s**: `%s`" % (k, json.dumps(v)))
L += ["", "Gate 1 false blocks (gate 1 blocked while the hidden oracle was fully green): " + ", ".join(
"%s %s/%s" % (c, (by_cfg.get(c) or {}).get("false_blocks", {}).get("g1"),
((by_cfg.get(c) or {}).get("false_blocks", {}).get("g1") or 0) + ((by_cfg.get(c) or {}).get("right_blocks", {}).get("g1") or 0))
for c in ("g1", "g13", "g13adv")) + " (false / all g1 blocks)."]
L += ["", "Right block = the hidden oracle had at least one failing requirement at that stop; false block = the oracle "
"was fully green at that stop (the gate sent the agent back for nothing the user asked for)."]
open(os.path.join(HERE, "results.md"), "w").write("\n".join(L) + "\n")
print("\n".join(L))
results.md
# gate-bench results
50 runs scored (matrix runs only).
## Per config
| cfg | runs | hidden reqs | fully passing | suite green | mean $ total (worker + g1 writer + g3 reviewers) | mean wall s | blocks/run | g1 right/false | g3 right/false | give-ups | ended blocked | model ok |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bare | 10 | 35/50 (70%) | 3 | 10 | 0.085 (0.085 + 0.000 + 0.000) | 33 | 0.00 | 0/0 | 0/0 | 0 | 0 | True |
| g1 | 10 | 42/50 (84%) | 6 | 10 | 0.201 (0.130 + 0.071 + 0.000) | 60 | 0.60 | 3/3 | 0/0 | 0 | 0 | True |
| g3 | 10 | 34/50 (68%) | 3 | 10 | 0.151 (0.094 + 0.000 + 0.058) | 54 | 0.10 | 0/0 | 1/0 | 0 | 0 | True |
| g13 | 10 | 41/50 (82%) | 5 | 10 | 0.282 (0.153 + 0.071 + 0.057) | 86 | 1.70 | 14/1 | 2/0 | 2 | 0 | True |
| g13adv | 10 | 39/50 (78%) | 6 | 10 | 0.276 (0.157 + 0.072 + 0.048) | 86 | 1.20 | 11/1 | 0/0 | 1 | 0 | True |
## Per task (hidden reqs passed / total, per config)
| task | bare | g1 | g3 | g13 | g13adv | missed reqs (all configs) |
|---|---|---|---|---|---|---|
| T1 | 4/10, $0.08, 0.0 blk | 4/10, $0.22, 0.5 blk | 3/10, $0.16, 0.5 blk | 4/10, $0.36, 3.0 blk | 5/10, $0.31, 1.0 blk | T1:test_req2, T1:test_req3, T1:test_req4, T1:test_req5 |
| T2 | 8/10, $0.10, 0.0 blk | 8/10, $0.17, 0.0 blk | 8/10, $0.17, 0.0 blk | 8/10, $0.27, 3.0 blk | 9/10, $0.30, 3.5 blk | T2:test_req4 |
| T3 | 10/10, $0.09, 0.0 blk | 10/10, $0.21, 1.0 blk | 10/10, $0.14, 0.0 blk | 10/10, $0.26, 0.5 blk | 10/10, $0.31, 0.5 blk | |
| T4 | 5/10, $0.07, 0.0 blk | 10/10, $0.23, 0.5 blk | 5/10, $0.15, 0.0 blk | 10/10, $0.30, 1.0 blk | 5/10, $0.21, 0.0 blk | T4:test_req1, T4:test_req2, T4:test_req3, T4:test_req4, T4:test_req5 |
| T5 | 8/10, $0.08, 0.0 blk | 10/10, $0.18, 1.0 blk | 8/10, $0.12, 0.0 blk | 9/10, $0.23, 1.0 blk | 10/10, $0.25, 1.0 blk | T5:test_req4 |
## The video's questions
- **q1_gate1_vs_bare**: `{"bare": 0.7, "g1": 0.84, "delta": 0.14}`
- **q2_gate3_over_g1**: `{"g1": 0.84, "g13": 0.82, "delta": -0.02, "g3_alone": 0.68, "g3_vs_bare": -0.02}`
- **q3_adv_vs_indep**: `{"g13": 0.82, "g13adv": 0.78, "delta": -0.04, "false_blocks_g3": {"g13": 0, "g13adv": 0}}`
- **q4_false_block_rate**: `{"bare": {"g1": null, "g3": null}, "g1": {"g1": 0.5, "g3": null}, "g3": {"g1": null, "g3": 0.0}, "g13": {"g1": 0.067, "g3": 0.0}, "g13adv": {"g1": 0.083, "g3": null}}`
- **q4_gate1_false_blocks**: `{"g1": {"false": 3, "right": 3}, "g13": {"false": 1, "right": 14}, "g13adv": {"false": 1, "right": 11}}`
- **q5_cost**: `{"bare": {"mean_cost_total_usd": 0.0849, "mean_cost_worker_usd": 0.0849, "mean_cost_gate1_usd": 0.0, "mean_cost_gate3_usd": 0.0, "mean_wall_s": 32.7}, "g1": {"mean_cost_total_usd": 0.201, "mean_cost_worker_usd": 0.1302, "mean_cost_gate1_usd": 0.0709, "mean_cost_gate3_usd": 0.0, "mean_wall_s": 59.5}, "g3": {"mean_cost_total_usd": 0.1513, "mean_cost_worker_usd": 0.0938, "mean_cost_gate1_usd": 0.0, "mean_cost_gate3_usd": 0.0575, "mean_wall_s": 54.1}, "g13": {"mean_cost_total_usd": 0.2818, "mean_cost_worker_usd": 0.1534, "mean_cost_gate1_usd": 0.0712, "mean_cost_gate3_usd": 0.0573, "mean_wall_s": 86.4}, "g13adv": {"mean_cost_total_usd": 0.2763, "mean_cost_worker_usd": 0.1568, "mean_cost_gate1_usd": 0.0718, "mean_cost_gate3_usd": 0.0478, "mean_wall_s": 85.9}}`
Gate 1 false blocks (gate 1 blocked while the hidden oracle was fully green): g1 3/6, g13 1/15, g13adv 1/12 (false / all g1 blocks).
Right block = the hidden oracle had at least one failing requirement at that stop; false block = the oracle was fully green at that stop (the gate sent the agent back for nothing the user asked for).
Validación y demo:
validate.sh: comprueba que el oráculo falla sobre upstream, que el hook se dispara, quegate.shbloquea una copia en rojo y que un revisor devuelve JSON válido.validate.sh
#!/usr/bin/env bash
# Validation before the matrix (SPEC § Validation). Steps 1, 3, 4 here; step 2 = the g1 red-start smoke run:
# INJECT=$PWD/inject/break.patch RUN_TAG=val2 ./run.sh g1 T1 1 (then: ./validate.sh step2)
# Usage: validate.sh [all|step1|step2|step3|step4] -> appends to validation.log
set -u
HERE="$(cd "$(dirname "$0")" && pwd)"; cd "$HERE"
. "$HERE/env.sh"
PY="$HERE/.venv/bin/python"
STEP="${1:-all}"
status=0
count() { grep -Eo "[0-9]+ $2" <<<"$1" | tail -1 | awk '{print $1}'; }
log() { echo "$*" | tee -a validation.log; }
log "=== $(date '+%F %T') validate.sh $STEP ($(claude --version))"
if [ "$STEP" = all ] || [ "$STEP" = step1 ]; then
# 1. upstream fails the hidden tests (<= 1 req passing: T5 R3 is a regression guard); reference passes hidden + suite
TMP="$(mktemp -d "${TMPDIR:-/tmp}/gb-validate.XXXXXX")"
for t in T1 T2 T3 T4 T5; do
tree="$TMP/$t"; git clone -q "$HERE/upstream" "$tree"
v=$(cd "$tree" && TINYDB_TREE="$tree" "$PY" -m pytest -v -c "$HERE/hidden/pytest.ini" --rootdir "$HERE/hidden" "$HERE/hidden/test_$t.py" 2>&1)
up_pass=$(grep -Eo "test_req[0-9]+[^ ]* PASSED" <<<"$v" | wc -l | tr -d ' ')
(cd "$tree" && git apply "$HERE/reference/$t.patch") || { log "$t: reference patch does not apply"; status=1; continue; }
h=$(cd "$tree" && TINYDB_TREE="$tree" "$PY" -m pytest -q -c "$HERE/hidden/pytest.ini" --rootdir "$HERE/hidden" "$HERE/hidden/test_$t.py" 2>&1); hrc=$?
s=$(cd "$tree" && "$PY" -m pytest -q -p no:cacheprovider --no-cov 2>&1); src=$?
ok=OK; { [ "$up_pass" -gt 1 ] || [ $hrc -ne 0 ] || [ $src -ne 0 ]; } && { ok=FAIL; status=1; }
log "step1 $t: upstream reqs passing=$up_pass reference hidden rc=$hrc ($(count "$h" passed) passed) suite rc=$src ($(count "$s" passed) passed) [$ok]"
done
rm -rf "$TMP"
fi
if [ "$STEP" = all ] || [ "$STEP" = step3 ]; then
# 3. gate.sh by hand, fake stdin. green = T1 reference + one new test (g1 passes) -> exit 0 allow;
# red = break.patch (one repo test fails) -> exit 2, pytest tail on stderr.
FAKE='{"session_id":"validate","transcript_path":"/dev/null","cwd":"x","hook_event_name":"Stop","stop_hook_active":false}'
for k in green red; do
R="$HERE/runs/_val3-$k"; W="$HERE/work/_val3-$k"; rm -rf "$R"; mkdir -p "$R"
./prep.sh g1 "$W" "$R" > /dev/null
if [ $k = green ]; then
(cd "$W" && git apply "$HERE/reference/T1.patch" && cp "$HERE/inject/test_pushpull.py" tests/)
else
(cd "$W" && git apply "$HERE/inject/break.patch")
fi
printf '{"task":"T1","cfg":"g1","rep":0,"work":"%s","name":"_val3-%s"}\n' "$W" "$k" > "$R/meta.json"; cp tasks/T1.md "$R/task.md"
echo "$FAKE" | GATE_RUN="$R" GATE_CONFIG=g1 bash gate.sh > "$R/stdout.txt" 2> "$R/stderr.txt"; rc=$?
dec=$("$PY" -c "import json;print(json.loads(open('$R/gate.jsonl').readline())['decision'])")
want=0; [ $k = red ] && want=2
ok=OK; [ $rc -ne $want ] && { ok=FAIL; status=1; }
[ $k = red ] && ! grep -q "short test summary" "$R/stderr.txt" && { ok=FAIL; status=1; }
[ $k = green ] && [ -s "$R/stderr.txt" ] && { ok=FAIL; status=1; }
log "step3 $k: exit=$rc decision=$dec stderr_bytes=$(wc -c < "$R/stderr.txt" | tr -d ' ') [$ok]"
done
fi
if [ "$STEP" = all ] || [ "$STEP" = step4 ]; then
# 4. one reviewer pair by hand on the green copy (needs step3's work/_val3-green): structured_output valid
W="$HERE/work/_val3-green"
[ -d "$W" ] || { log "step4: run step3 first"; exit 1; }
for v in indep adv; do
"$PY" lib/review.py "$W" T1 $v > "runs/_val3-green/review-$v.txt" 2>&1
s=$("$PY" - "runs/_val3-green/review-$v.txt" <<'PY'
import json, sys
t = open(sys.argv[1]).read(); i = t.find("--- reason"); d = json.loads(t[:i] if i > 0 else t)
ok = all(r["error"] is None and isinstance(r["approve_raw"], bool) for r in d["reviewers"])
ok = ok and all(f["severity"] in ("blocking", "minor") and isinstance(f["line"], int) for r in d["reviewers"] for f in r["findings"])
print("%s pass=%s blocking=%d findings=%s cost=$%.4f wall=%s" % ("OK" if ok else "FAIL", d["pass"], d["blocking_findings"],
[len(r["findings"]) for r in d["reviewers"]], d["cost_usd"], [r["wall_s"] for r in d["reviewers"]]))
PY
)
case "$s" in OK*) ;; *) status=1 ;; esac
log "step4 $v: $s"
done
fi
if [ "$STEP" = all ] || [ "$STEP" = step2 ]; then
# 2. hook fires on a red start: runs/T1-g1-r1-val2 must log a block and the stream must show the agent continuing
R="$HERE/runs/T1-g1-r1-val2"
if [ -f "$R/gate.jsonl" ]; then
s=$("$PY" - "$R" <<'PY'
import json, sys, os
r = sys.argv[1]
g = [json.loads(l) for l in open(os.path.join(r, "gate.jsonl")) if l.strip()]
st = [json.loads(l) for l in open(os.path.join(r, "stream.jsonl")) if l.startswith("{")]
blocks = [x for x in g if x["decision"] == "block"]
# index in the stream of the first hook response that blocked, then any assistant tool_use after it
txt = [json.dumps(o) for o in st]
fb = next((i for i, t in enumerate(txt) if "Stop hook feedback" in t or ('"hook_event": "Stop"' in t and '"exit_code": 2' in t)), None)
after = fb is not None and any(o.get("type") == "assistant" for o in st[fb + 1:])
print("%s blocks=%d hook_calls=%d feedback_idx=%s assistant_after=%s last=%s" % (
"OK" if blocks and after else "FAIL", len(blocks), len(g), fb, after, g[-1]["decision"] if g else None))
PY
)
case "$s" in OK*) ;; *) status=1 ;; esac
log "step2: $s"
else
log "step2: not run yet (INJECT=\$PWD/inject/break.patch RUN_TAG=val2 ./run.sh g1 T1 1)"
fi
fi
[ $status -eq 0 ] && log "VALIDATION OK" || log "VALIDATION FAILED"
exit $status
validate_writer.sh
#!/usr/bin/env bash
# Validation of the gate-1 test writer: one writer call per task (in parallel) into validation/writer/<T>/,
# then its tests on upstream and on the reference patch. Usage: validate_writer.sh [T1 ...] (reuses an existing
# file unless REWRITE=1). Appends to validation.log.
set -u
HERE="$(cd "$(dirname "$0")" && pwd)"; cd "$HERE"
. "$HERE/env.sh"
PY="$HERE/.venv/bin/python"
TASKS="${*:-T1 T2 T3 T4 T5}"
for t in $TASKS; do
D="$HERE/validation/writer/$t"; mkdir -p "$D"
if [ "${REWRITE:-0}" = 1 ] || [ ! -f "$D/gate_tests/test_$t.py" ]; then "$PY" lib/testwriter.py "$t" "$D" > "$D/writer.out" 2>&1 & fi
done
wait
TMP="$(mktemp -d "${TMPDIR:-/tmp}/gb-vw.XXXXXX")"; trap 'rm -rf "$TMP"' EXIT
echo "=== $(date '+%F %T') validate_writer.sh $TASKS" >> validation.log
for t in $TASKS; do
D="$HERE/validation/writer/$t"
git clone -q "$HERE/upstream" "$TMP/up-$t"; git clone -q "$HERE/upstream" "$TMP/ref-$t"
(cd "$TMP/ref-$t" && git apply "$HERE/reference/$t.patch")
line=$("$PY" - "$D" "$t" "$TMP/up-$t" "$TMP/ref-$t" <<'PY'
import json, os, sys
sys.path.insert(0, "lib")
from testwriter import run_gate_tests
d, t, up, ref = sys.argv[1:5]
w = json.load(open(os.path.join(d, "testwriter.json")))
if not w["ok"]:
print("%s: writer FAILED %s" % (t, [a["error"] for a in w["attempts"]])); sys.exit()
g = os.path.join(d, "gate_tests")
_, uo, up_p, up_f, _ = run_gate_tests(g, t, up)
_, ro, rp, rf, rnames = run_gate_tests(g, t, ref)
open(os.path.join(d, "on-upstream.txt"), "w").write(uo); open(os.path.join(d, "on-reference.txt"), "w").write(ro)
print("%s: writer tests: %d, pass on reference: %d, pass on upstream: %d | fail on reference: %s | writer $%.3f %ss turns=%s" % (
t, rp + rf, rp, up_p, rnames, w["cost_usd"], w["wall_s"], [a.get("turns") for a in w["attempts"]]))
PY
)
echo "$line" | tee -a validation.log
done
demo.sh
#!/usr/bin/env bash
# Interactive demo of the gate for a screen recording: a gated tinydb work copy (config g13: acceptance
# tests, then two reviewers) with T1's acceptance tests already written. Usage: ./demo.sh [T1..T5]
# Then: cd work/demo && claude --setting-sources project --permission-mode acceptEdits
# and paste the request printed below. When Claude stops, the Stop hook runs; a red acceptance test shows
# as "Stop hook feedback: ... The acceptance tests for this change fail:" and Claude keeps working.
set -euo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"; TASK="${1:-T1}"
. "$HERE/env.sh"
RUN="$HERE/runs/demo"; W="$HERE/work/demo"
rm -rf "$RUN"; mkdir -p "$RUN"
"$HERE/prep.sh" g13 "$W" "$RUN" > "$RUN/prep.txt" 2>&1
printf '{"task":"%s","cfg":"g13","rep":0,"work":"%s","name":"demo"}\n' "$TASK" "$W" > "$RUN/meta.json"
cp "$HERE/tasks/$TASK.md" "$RUN/task.md"
printf '%s\n\nWhen you are done, stop and summarise what you changed.\n' "$(cat "$HERE/tasks/$TASK.md")" > "$RUN/prompt.txt"
# Reuse the acceptance tests of a matrix run of the same task (no new writer call).
SRC=$(ls -d "$HERE"/runs/"$TASK"-g13-r1/gate_tests 2>/dev/null | head -1)
cp -R "$SRC" "$RUN/gate_tests"
echo "ready: cd $W && claude --setting-sources project --permission-mode acceptEdits"
echo "--- paste this request ---"; cat "$RUN/prompt.txt"
inject/break.patch
diff --git a/tinydb/queries.py b/tinydb/queries.py
index cdab9fb..db00a2a 100644
--- a/tinydb/queries.py
+++ b/tinydb/queries.py
@@ -322,7 +322,7 @@ class Query(QueryInstance):
:param rhs: The value to compare against
"""
return self._generate_test(
- lambda value: value >= rhs,
+ lambda value: value > rhs,
('>=', self._path, rhs)
)
inject/test_pushpull.py
from tinydb import TinyDB, Query
from tinydb.storages import MemoryStorage
from tinydb.operations import push, pull
def test_push_pull():
db = TinyDB(storage=MemoryStorage)
db.insert({'n': 1, 't': ['a']})
db.update(push('t', 'b'), Query().n == 1)
db.update(pull('t', 'a'), Query().n == 1)
assert db.all()[0]['t'] == ['b']
inject/test_pushpull.py on GitHub
Documentación:
README.md: estructura, comandos y qué es el harness.README.md
# gate-bench
What does each of Shopify's Helix gates catch when rebuilt as a Claude Code **Stop hook**, and what does it
cost? Headless Sonnet 5.5 workers implement five tinydb features (T1..T5); a Stop hook runs gate 1
(behaviour tests) and/or gate 3 (adversarial review by headless Sonnet 5.5 reviewers) every time the worker
tries to stop; hidden tests (the oracle, never visible to the agent) record the ground truth at every stop
and at the end. Built 2026-10-08 from `SPEC.md`; upstream, hidden tests, tasks and reference patches copied
from `../verify-loops-bench` (nothing there or in `../haiku-routing-bench` is modified).
## Commands (from this dir)
```bash
./validate.sh # steps 1,3,4 (+ step 2 once its run exists) -> validation.log
INJECT=$PWD/inject/break.patch RUN_TAG=val2 ./run.sh g1 T1 1 # validation step 2 (red start state)
RUN_TAG=smoke ./run.sh g13adv T1 1 # smoke (tagged runs are ignored by analyze.py)
./validate_writer.sh # gate-1 test writer on T1..T5: tests on upstream vs reference (REWRITE=1 to redo)
./gen_matrix.py # matrix.txt (50 runs, already generated)
nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 & # 3 runs in parallel (MATRIX_JOBS=3)
SKIP_DONE=1 nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 & # resume
.venv/bin/python analyze.py # results.json + results.md
.venv/bin/python lib/finalize.py runs/<run> # rescore one run from its work copy
.venv/bin/python lib/review.py work/<run> T1 adv # one gate-3 round by hand
```
Single run: `./run.sh <cfg> <task> <rep>` -> `runs/<task>-<cfg>-r<rep>/`, work copy `work/<same name>/`.
## Layout
| path | what |
|---|---|
| `SPEC.md` | the spec this harness implements |
| `upstream/` | tinydb @ 18d73a1 (git repo) |
| `hidden/` | `test_T1..T6.py` (T6 unused), `conftest.py` (`$TINYDB_TREE` or cwd first on sys.path), `pytest.ini` |
| `tasks/T1..T5.md` + `tasks/README.md` | the task prompts (verbatim) and what each hidden req checks |
| `reference/T1..T5.patch` | reference solutions (validation step 1) |
| `inject/break.patch` | `>=` -> `>` in `Query.__ge__` (one repo test red; validation steps 2-3) |
| `inject/test_pushpull.py` | one thin test, makes the T1 reference copy "green with tests touched" (step 3) |
| `configs.json` | `bare` (no hook), `g1`, `g3` (indep), `g13` (g1 then g3 indep), `g13adv` (g1 then g3 adv) |
| `env.sh` | scrubs inherited `CLAUDE*` vars, default `~/.claude` auth, `CLAUDE_CODE_DISABLE_AUTO_MEMORY=1`, venv on PATH |
| `prep.sh <cfg> <work> <run>` | fresh copy of upstream, `.claude/` in `.git/info/exclude`, hook settings for gated configs |
| `run.sh <cfg> <task> <rep>` | one run (worker + hook + `lib/finalize.py`) |
| `gate.sh` | the Stop hook (bash 3.2 wrapper: `exec .venv/bin/python -I lib/hook.py`) |
| `lib/hook.py` | gate dispatch, snapshot, hidden run, gate 1, gate 3, decision, `gate.jsonl` line |
| `lib/review.py` | gate-3 reviewers (indep / adv), schemas, prompts, reason text; CLI for a manual round |
| `lib/testwriter.py` | gate-1 test writer (before the worker) + `run_gate_tests` (path rewritten to `acceptance/`) |
| `validate_writer.sh`, `validation/writer/<T>/` | writer validation files and their runs on upstream / reference |
| `tasks/` (terse prompts, used), `tasks-explicit/` + `runs-explicit/` | the matrix uses terse prompts; see `tasks/README.md` |
| `lib/common.py` | diff incl. untracked files, suite run, hidden run (requirement level) |
| `lib/finalize.py` | post-run diff/suite/hidden, `result.json`, `score.json` |
| `gen_matrix.py`, `matrix.txt`, `matrix.sh`, `matrix-one.sh`, `matrix.log` | 50-run matrix, 3 in parallel (xargs -P), resumable |
| `analyze.py` -> `results.json`, `results.md` | per config, per task, the five questions |
| `validate.sh`, `VALIDATION.md`, `validation.log` | pre-matrix checks |
| `.venv/` | python 3.14.7, pytest 9.1.1, pytest-cov 7.1.0, pyyaml, ruff 0.16.10 (same pins as haiku-routing-bench) |
### Worker (run.sh)
`claude -p "<T-file verbatim>\n\nWhen you are done, stop and summarise what you changed." --model claude-sonnet-5-5
--output-format stream-json --verbose --include-hook-events --max-turns 60 --permission-mode acceptEdits
--allowedTools <haiku-routing-bench list> --setting-sources project --strict-mcp-config`, cwd = `work/<run>/`, stdin /dev/null.
Gated configs get `work/<run>/.claude/settings.json`:
`{"permissions":{"deny":[...see traps...]},"hooks":{"Stop":[{"hooks":[{"type":"command","command":"bash \"$CLAUDE_PROJECT_DIR/.claude/hooks/stop.sh\"","timeout":1500}]}]}}`
and `.claude/hooks/stop.sh` = `GATE_RUN=<abs run dir> GATE_CONFIG=<cfg> bash <bench>/gate.sh` (the SPEC command, one level down: see traps).
Copies of both land in the run dir (`settings.json`, `stop.sh`).
### Hook (lib/hook.py), one invocation
1. read stdin JSON; n = lines already in `gate.jsonl` + 1; blocks so far from the same file
2. `snap-<n>.diff` = `git diff HEAD` + untracked files (index never touched)
3. hidden tests on the work copy (`TINYDB_TREE` = work copy, cwd = work copy) -> `hidden-<n>.txt`; never shown to the agent
4. gate 1 (if configured) = the repo suite `python -m pytest -q -x --no-header -p no:cacheprovider --no-cov` (300 s)
-> `gate1-<n>.txt` AND the writer's acceptance tests `runs/<run>/gate_tests/test_<T>.py` run with `TINYDB_TREE` = work
copy -> `gatetests-<n>.txt`. Block when either is red. Reason: "The acceptance tests for this change fail:" + last 40
lines (failing test names + assertion output, the gate_tests path rewritten to `acceptance/`), then the suite tail if the
suite is red too; <= 1500 chars. `tests_touched` is still logged but no longer blocks. No writer file (writer failed
twice) -> suite only, logged as `gate_tests_present: false`.
5. gate 3 (if configured and gate 1 passed or absent): see below
6. decision: allow / block / give_up (a would-be block when 6 blocks already happened -> allow, logged `give_up`).
`stop_hook_active` is logged, never used. Block = exit 2 + reason on stderr. A crash of the hook itself is
logged to `gate-errors.txt` and allows (counted as `gate_crashes` in score.json).
### Gate 1 test writer (lib/testwriter.py, run.sh before the worker, configs g1/g13/g13adv)
`claude -p` Sonnet 5.5 with the reviewer isolation flags, `--tools Read,Grep,Glob --max-turns 12 --json-schema
{"file": string}`, cwd = a pristine temp copy of upstream (deleted after). Prompt = framing + `tasks/<T>.md` + the
coordinator's instruction ("Write integration-style pytest cases for this request against the tinydb API, one test per
behaviour a careful senior would expect, pushing for edge cases beyond the happy path (missing fields, wrong types,
boundaries, persistence, cache, existing behaviour unchanged). Tests only, no implementation. Return the file content.")
+ mechanics (self-contained module, imports of the new API inside each test, MemoryStorage/tmp_path). The file must
compile and contain `def test_`, else one retry. Output `runs/<run>/gate_tests/` (+ hidden's conftest.py with
`TINYDB_TREE`, pytest.ini) and `testwriter.json` (cost, turns, attempts). Fixed per-run cost, outside the worker's wall.
### Gate 3 (lib/review.py)
Each reviewer: `claude -p "<prompt>" --model claude-sonnet-5-5 --tools Read,Grep,Glob --max-turns 15 --output-format json
--json-schema <schema> --permission-mode dontAsk --setting-sources "" --settings '{"disableAllHooks":true}' --strict-mcp-config`
with `CLAUDE_CODE_DISABLE_CLAUDE_MDS=1 CLAUDE_CODE_DISABLE_AUTO_MEMORY=1`, all inherited `CLAUDE*` vars dropped, cwd = work copy,
600 s timeout. Prompt = framing + task prompt (T-file) + diff (tracked + untracked, cap 60k chars) + the SPEC's "Find defects..."
sentence. `structured_output` is read from the JSON result; `total_cost_usd`, `num_turns`, wall and models are recorded.
- `indep`: reviewers A and B, same prompt, run in parallel, no shared context. Blocking findings = union. Pass when both approve.
- `adv`: A as above; challenger B gets A's numbered findings + the diff, answers `upheld|rejected` + reason per id, may add
findings (`added_findings`). Blocking = A's blocking findings B upheld + B's added blocking findings. B always runs (same
2-call budget as indep).
- Effective `approve` = "no blocking finding" (the raw `approve` is logged as `approve_raw`; a reviewer saying approve=false with
only minor findings does not block). A reviewer call that errors approves (fail open) with `error` recorded; an errored
challenger upholds nothing.
- Reason = "Code review found N blocking defect(s):" + numbered `file:line text` + "Fix every finding, re-run the tests, then stop.", <= 1500 chars.
### Per run (`runs/<task>-<cfg>-r<rep>/`)
`stream.jsonl`, `stderr.txt`, `exit_code.txt`, `wall_s.txt` (includes hook time: the hook runs inside the session),
`gate.jsonl`, `snap-<n>.diff`, `hidden-<n>.txt`, `gate1-<n>.txt`, `hook-input-1.json`, `post.diff`, `numstat.txt`,
`suite.txt` (full suite, no -x), `hidden.txt`, `gate_tests/`, `testwriter.json`, `gatetests-<n>.txt`, `gatetests-final.txt`, `result.json`, `score.json`, `meta.json`, `task.md`, `prompt.txt`,
`settings.json` + `stop.sh` (gated configs).
`score.json`: task, cfg, rep, hidden_pass/hidden_fail/failed_reqs (**requirement level**: 5 reqs per task), suite_pass/fail,
hook_calls, blocks_total, blocks_by_gate {g1,g3}, right_blocks / false_blocks (+ `_by_gate`), give_up, ended_blocked,
cost_worker_usd, cost_reviewers_usd, cost_testwriter_usd, cost_total_usd (all three), testwriter_ok, gate_tests_total,
gate_tests_fail_final, turns, wall_s, reviewer_findings_total, reviewer_errors, gate_crashes,
model_ok (worker init model + every modelUsage key + every reviewer model == claude-sonnet-5-5), hook_events_in_stream,
hidden_first_stop.
## Matrix
5 tasks x 5 configs x 2 reps = 50 runs. Rep 1 config blocks bare, g1, g3, g13, g13adv (tasks rotated per block), rep 2 the
reverse block order. Up to 3 runs at once, each in its own work copy (`matrix-one.sh` via `xargs -P 3`, bash 3.2 has no
`wait -n`). `matrix.log`: one `start` and one result line per run.
## Traps found
- **Gate 1 as first built passed by construction** (it only re-ran the implementer's own tests: `T2-g1-r1` one allow, hidden
R4 missed). Redesigned around an independent test writer (above). The writer hedges where the terse prompt is silent
(accepts "raises or no-op"), so it cannot recover unstated requirements; per-task overlap is in VALIDATION.md.
- **A blocked agent hunts for the tests.** After a gate-1 block the smoke agent ran `cat .claude/hooks/stop.sh; find / -name
test_T1.py` (denied only because `find` is not on the allow-list; `cat` is, and `stop.sh` names the bench dir, from where
`hidden/` is one `cat` away). Gated configs' `.claude/settings.json` now carries `permissions.deny`: Read of `./.claude/**`,
the bench's `hidden/ runs/ validation/ reference/`, the two sibling benches; Bash patterns containing `.claude/hooks`,
`.claude/settings`, `../hidden`, `../runs`, `gate_tests`, the bench sub-paths, `find *`. Probed with a Haiku session: all
five escape attempts (cat, Read, ls, python open) denied, in-tree reads fine. Bash rules match command text, not a security
boundary. Bare runs have no settings file (bare r1 ran before this change; bare has no motive to hunt).
Reviewers and the writer run with `--setting-sources ""`, so these rules do not apply to them.
- **Transient `FileNotFoundError` spawning a reviewer** inside the hook (once, both reviewers of one call, wall 0.0 s;
never reproduced by hand). Reviewer calls now retry once after 3 s; the error message records `filename`.
- **`--bare` cannot authenticate here.** It reads `ANTHROPIC_API_KEY` / `apiKeyHelper` only, never OAuth or the keychain;
this account is OAuth (`~/.claude`), so a `--bare` reviewer returns `terminal_reason: api_error`, cost 0. Used the
documented equivalents instead: `--setting-sources ""` + `--settings '{"disableAllHooks":true}'` + `--strict-mcp-config` +
`CLAUDE_CODE_DISABLE_CLAUDE_MDS=1` + `CLAUDE_CODE_DISABLE_AUTO_MEMORY=1` + `--tools Read,Grep,Glob`. The reviewers' cwd is the
work copy, whose `.claude/settings.json` carries the Stop hook: without `--setting-sources ""`/`disableAllHooks` a reviewer
would run the gate recursively. Verified: a dummy Stop hook in the probe copy never fired.
- **Inherited session env.** A matrix launched from inside Claude Code inherits `CLAUDE_EFFORT=medium`,
`CLAUDE_CODE_SUBAGENT_MODEL=opus`, `CLAUDE_CODE_SESSION_ID`, `CLAUDECODE`, `CLAUDE_CODE_CHILD_SESSION`, messaging tokens.
`env.sh` unsets every `CLAUDE*` var before setting its own; reviewers get a scrubbed env too (the hook process itself
inherits the worker's `CLAUDE_*` vars, e.g. the worker's session id).
- **The feedback line leaks the hook command.** Claude Code prefixes each block with the command line:
`Stop hook feedback:\n[<command>]: <reason>`. With the SPEC command inline the agent would read `GATE_CONFIG=g13adv` and
`gate-bench/gate.sh`. The settings command is therefore `bash "$CLAUDE_PROJECT_DIR/.claude/hooks/stop.sh"`, a one-line
wrapper holding the SPEC command. The agent can still `cat` it (it lives in the work copy) but it is not pushed at it.
- **Stop hook input keys (2.1.293):** `background_tasks, cwd, effort ({"level":"medium"}), hook_event_name,
last_assistant_message, permission_mode, prompt_id, session_crons, session_id, stop_hook_active, transcript_path`.
- **8-cap resets on tool calls.** Per the docs the consecutive-continuation count resets each time Claude calls a tool, so the
built-in cap only bites an agent that answers blocks with text alone. Our give-up (6 blocks per run, counted from
`gate.jsonl`, tool calls or not) fires first in every case.
- **Red start state: the agent refuses to fix code it did not write.** Validation step 2 (T1 + an uncommitted `>=` -> `>`
edit) shows Sonnet 5.5 implementing T1, then answering 6 consecutive gate-1 blocks with "the red test is your pre-existing
uncommitted edit, not my change" without touching it, until give_up. The hook mechanics are validated; the behaviour is a
finding in itself (gate 1 cannot force a fix the agent considers out of scope).
- **pytest-cov.** The repo's `pytest.ini` adds `--cov tinydb`; gate 1 adds `--no-cov` so the 40-line tail holds the failure,
not the coverage table (and no `.coverage` file is written). `.coverage` is gitignored anyway.
- **T5 R4 is parametrized** (several test ids, one requirement). hidden counts are per requirement (5 per task), a req
passes only when all its ids pass. `validate.sh` step 1: T5 upstream passes 1 req (R3, the regression guard) as designed.
- **Diff without touching the index.** `git add -N` would change what the agent sees in `git status`; untracked files are
diffed with `git diff --no-index /dev/null <f>` instead. `.claude/` is in `.git/info/exclude`, so the hook config never
shows in any diff or in `git status`.
- **Worker Bash denials.** The 2.1.293 permission layer denies some agent commands with "Contains brace with quote character
(expansion obfuscation)" even under acceptEdits + allow-list; agents route around it (visible as `permission_denied` events).
- **Reviewers on a correct solution still flag "blocking".** On the T1 reference patch, reviewer A marked "pull does not check
the field is a list" (not required by the task) as blocking; indep therefore blocks, the adv challenger rejected it. This is
exactly what false_blocks measures (hidden all green at that stop).
- **Shared plan meter.** Workers and reviewers bill the user's plan quota; `total_cost_usd` is the list-price equivalent the
CLI reports, not a bill. Nothing else heavy should run on the account during the matrix.
- **rtk rewrites interactive `grep`/`diff`** in the launching Claude session only; scripts here call binaries directly
(`/usr/bin/diff`/`cmp` for verification diffs).
- **CLI version:** `claude --version` = 2.1.293 (Claude Code); flags checked in `claude --help`: `--json-schema`, `--bare`, `--tools`,
`--permission-mode` (choices acceptEdits, auto, bypassPermissions, manual, dontAsk, plan), `--setting-sources`,
`--include-hook-events`. `--max-turns` is accepted though not listed in `--help`.
## Matrix launch (2026-10-08 16:33:54, after the gate-1 redesign)
`cd /Users/thomas/dev/gate-bench && SKIP_DONE=1 nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 &` -> PID 71317, 3 lanes.
The 5 bare r1 runs (terse prompts) are kept and skipped; 45 runs to go. Expected wall ~40-70 min: gated runs so far take
~30-70 s of worker wall + ~35-40 s of test writer (before the worker) + 1-35 s per hook call, more with blocks.
Then `.venv/bin/python analyze.py`.
HARNESS.md
# gate-bench
What does each of Shopify's Helix gates catch when rebuilt as a Claude Code **Stop hook**, and what does it
cost? Headless Sonnet 5.5 workers implement five tinydb features (T1..T5); a Stop hook runs gate 1
(behaviour tests) and/or gate 3 (adversarial review by headless Sonnet 5.5 reviewers) every time the worker
tries to stop; hidden tests (the oracle, never visible to the agent) record the ground truth at every stop
and at the end. Built 2026-10-08 from `SPEC.md`; upstream, hidden tests, tasks and reference patches copied
from `../verify-loops-bench` (nothing there or in `../haiku-routing-bench` is modified).
## Commands (from this dir)
```bash
./validate.sh # steps 1,3,4 (+ step 2 once its run exists) -> validation.log
INJECT=$PWD/inject/break.patch RUN_TAG=val2 ./run.sh g1 T1 1 # validation step 2 (red start state)
RUN_TAG=smoke ./run.sh g13adv T1 1 # smoke (tagged runs are ignored by analyze.py)
./validate_writer.sh # gate-1 test writer on T1..T5: tests on upstream vs reference (REWRITE=1 to redo)
./gen_matrix.py # matrix.txt (50 runs, already generated)
nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 & # 3 runs in parallel (MATRIX_JOBS=3)
SKIP_DONE=1 nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 & # resume
.venv/bin/python analyze.py # results.json + results.md
.venv/bin/python lib/finalize.py runs/<run> # rescore one run from its work copy
.venv/bin/python lib/review.py work/<run> T1 adv # one gate-3 round by hand
```
Single run: `./run.sh <cfg> <task> <rep>` -> `runs/<task>-<cfg>-r<rep>/`, work copy `work/<same name>/`.
## Layout
| path | what |
|---|---|
| `SPEC.md` | the spec this harness implements |
| `upstream/` | tinydb @ 18d73a1 (git repo) |
| `hidden/` | `test_T1..T6.py` (T6 unused), `conftest.py` (`$TINYDB_TREE` or cwd first on sys.path), `pytest.ini` |
| `tasks/T1..T5.md` + `tasks/README.md` | the task prompts (verbatim) and what each hidden req checks |
| `reference/T1..T5.patch` | reference solutions (validation step 1) |
| `inject/break.patch` | `>=` -> `>` in `Query.__ge__` (one repo test red; validation steps 2-3) |
| `inject/test_pushpull.py` | one thin test, makes the T1 reference copy "green with tests touched" (step 3) |
| `configs.json` | `bare` (no hook), `g1`, `g3` (indep), `g13` (g1 then g3 indep), `g13adv` (g1 then g3 adv) |
| `env.sh` | scrubs inherited `CLAUDE*` vars, default `~/.claude` auth, `CLAUDE_CODE_DISABLE_AUTO_MEMORY=1`, venv on PATH |
| `prep.sh <cfg> <work> <run>` | fresh copy of upstream, `.claude/` in `.git/info/exclude`, hook settings for gated configs |
| `run.sh <cfg> <task> <rep>` | one run (worker + hook + `lib/finalize.py`) |
| `gate.sh` | the Stop hook (bash 3.2 wrapper: `exec .venv/bin/python -I lib/hook.py`) |
| `lib/hook.py` | gate dispatch, snapshot, hidden run, gate 1, gate 3, decision, `gate.jsonl` line |
| `lib/review.py` | gate-3 reviewers (indep / adv), schemas, prompts, reason text; CLI for a manual round |
| `lib/testwriter.py` | gate-1 test writer (before the worker) + `run_gate_tests` (path rewritten to `acceptance/`) |
| `validate_writer.sh`, `validation/writer/<T>/` | writer validation files and their runs on upstream / reference |
| `tasks/` (terse prompts, used), `tasks-explicit/` + `runs-explicit/` | the matrix uses terse prompts; see `tasks/README.md` |
| `lib/common.py` | diff incl. untracked files, suite run, hidden run (requirement level) |
| `lib/finalize.py` | post-run diff/suite/hidden, `result.json`, `score.json` |
| `gen_matrix.py`, `matrix.txt`, `matrix.sh`, `matrix-one.sh`, `matrix.log` | 50-run matrix, 3 in parallel (xargs -P), resumable |
| `analyze.py` -> `results.json`, `results.md` | per config, per task, the five questions |
| `validate.sh`, `VALIDATION.md`, `validation.log` | pre-matrix checks |
| `.venv/` | python 3.14.7, pytest 9.1.1, pytest-cov 7.1.0, pyyaml, ruff 0.16.10 (same pins as haiku-routing-bench) |
### Worker (run.sh)
`claude -p "<T-file verbatim>\n\nWhen you are done, stop and summarise what you changed." --model claude-sonnet-5-5
--output-format stream-json --verbose --include-hook-events --max-turns 60 --permission-mode acceptEdits
--allowedTools <haiku-routing-bench list> --setting-sources project --strict-mcp-config`, cwd = `work/<run>/`, stdin /dev/null.
Gated configs get `work/<run>/.claude/settings.json`:
`{"permissions":{"deny":[...see traps...]},"hooks":{"Stop":[{"hooks":[{"type":"command","command":"bash \"$CLAUDE_PROJECT_DIR/.claude/hooks/stop.sh\"","timeout":1500}]}]}}`
and `.claude/hooks/stop.sh` = `GATE_RUN=<abs run dir> GATE_CONFIG=<cfg> bash <bench>/gate.sh` (the SPEC command, one level down: see traps).
Copies of both land in the run dir (`settings.json`, `stop.sh`).
### Hook (lib/hook.py), one invocation
1. read stdin JSON; n = lines already in `gate.jsonl` + 1; blocks so far from the same file
2. `snap-<n>.diff` = `git diff HEAD` + untracked files (index never touched)
3. hidden tests on the work copy (`TINYDB_TREE` = work copy, cwd = work copy) -> `hidden-<n>.txt`; never shown to the agent
4. gate 1 (if configured) = the repo suite `python -m pytest -q -x --no-header -p no:cacheprovider --no-cov` (300 s)
-> `gate1-<n>.txt` AND the writer's acceptance tests `runs/<run>/gate_tests/test_<T>.py` run with `TINYDB_TREE` = work
copy -> `gatetests-<n>.txt`. Block when either is red. Reason: "The acceptance tests for this change fail:" + last 40
lines (failing test names + assertion output, the gate_tests path rewritten to `acceptance/`), then the suite tail if the
suite is red too; <= 1500 chars. `tests_touched` is still logged but no longer blocks. No writer file (writer failed
twice) -> suite only, logged as `gate_tests_present: false`.
5. gate 3 (if configured and gate 1 passed or absent): see below
6. decision: allow / block / give_up (a would-be block when 6 blocks already happened -> allow, logged `give_up`).
`stop_hook_active` is logged, never used. Block = exit 2 + reason on stderr. A crash of the hook itself is
logged to `gate-errors.txt` and allows (counted as `gate_crashes` in score.json).
### Gate 1 test writer (lib/testwriter.py, run.sh before the worker, configs g1/g13/g13adv)
`claude -p` Sonnet 5.5 with the reviewer isolation flags, `--tools Read,Grep,Glob --max-turns 12 --json-schema
{"file": string}`, cwd = a pristine temp copy of upstream (deleted after). Prompt = framing + `tasks/<T>.md` + the
coordinator's instruction ("Write integration-style pytest cases for this request against the tinydb API, one test per
behaviour a careful senior would expect, pushing for edge cases beyond the happy path (missing fields, wrong types,
boundaries, persistence, cache, existing behaviour unchanged). Tests only, no implementation. Return the file content.")
+ mechanics (self-contained module, imports of the new API inside each test, MemoryStorage/tmp_path). The file must
compile and contain `def test_`, else one retry. Output `runs/<run>/gate_tests/` (+ hidden's conftest.py with
`TINYDB_TREE`, pytest.ini) and `testwriter.json` (cost, turns, attempts). Fixed per-run cost, outside the worker's wall.
### Gate 3 (lib/review.py)
Each reviewer: `claude -p "<prompt>" --model claude-sonnet-5-5 --tools Read,Grep,Glob --max-turns 15 --output-format json
--json-schema <schema> --permission-mode dontAsk --setting-sources "" --settings '{"disableAllHooks":true}' --strict-mcp-config`
with `CLAUDE_CODE_DISABLE_CLAUDE_MDS=1 CLAUDE_CODE_DISABLE_AUTO_MEMORY=1`, all inherited `CLAUDE*` vars dropped, cwd = work copy,
600 s timeout. Prompt = framing + task prompt (T-file) + diff (tracked + untracked, cap 60k chars) + the SPEC's "Find defects..."
sentence. `structured_output` is read from the JSON result; `total_cost_usd`, `num_turns`, wall and models are recorded.
- `indep`: reviewers A and B, same prompt, run in parallel, no shared context. Blocking findings = union. Pass when both approve.
- `adv`: A as above; challenger B gets A's numbered findings + the diff, answers `upheld|rejected` + reason per id, may add
findings (`added_findings`). Blocking = A's blocking findings B upheld + B's added blocking findings. B always runs (same
2-call budget as indep).
- Effective `approve` = "no blocking finding" (the raw `approve` is logged as `approve_raw`; a reviewer saying approve=false with
only minor findings does not block). A reviewer call that errors approves (fail open) with `error` recorded; an errored
challenger upholds nothing.
- Reason = "Code review found N blocking defect(s):" + numbered `file:line text` + "Fix every finding, re-run the tests, then stop.", <= 1500 chars.
### Per run (`runs/<task>-<cfg>-r<rep>/`)
`stream.jsonl`, `stderr.txt`, `exit_code.txt`, `wall_s.txt` (includes hook time: the hook runs inside the session),
`gate.jsonl`, `snap-<n>.diff`, `hidden-<n>.txt`, `gate1-<n>.txt`, `hook-input-1.json`, `post.diff`, `numstat.txt`,
`suite.txt` (full suite, no -x), `hidden.txt`, `gate_tests/`, `testwriter.json`, `gatetests-<n>.txt`, `gatetests-final.txt`, `result.json`, `score.json`, `meta.json`, `task.md`, `prompt.txt`,
`settings.json` + `stop.sh` (gated configs).
`score.json`: task, cfg, rep, hidden_pass/hidden_fail/failed_reqs (**requirement level**: 5 reqs per task), suite_pass/fail,
hook_calls, blocks_total, blocks_by_gate {g1,g3}, right_blocks / false_blocks (+ `_by_gate`), give_up, ended_blocked,
cost_worker_usd, cost_reviewers_usd, cost_testwriter_usd, cost_total_usd (all three), testwriter_ok, gate_tests_total,
gate_tests_fail_final, turns, wall_s, reviewer_findings_total, reviewer_errors, gate_crashes,
model_ok (worker init model + every modelUsage key + every reviewer model == claude-sonnet-5-5), hook_events_in_stream,
hidden_first_stop.
## Matrix
5 tasks x 5 configs x 2 reps = 50 runs. Rep 1 config blocks bare, g1, g3, g13, g13adv (tasks rotated per block), rep 2 the
reverse block order. Up to 3 runs at once, each in its own work copy (`matrix-one.sh` via `xargs -P 3`, bash 3.2 has no
`wait -n`). `matrix.log`: one `start` and one result line per run.
## Traps found
- **Gate 1 as first built passed by construction** (it only re-ran the implementer's own tests: `T2-g1-r1` one allow, hidden
R4 missed). Redesigned around an independent test writer (above). The writer hedges where the terse prompt is silent
(accepts "raises or no-op"), so it cannot recover unstated requirements; per-task overlap is in VALIDATION.md.
- **A blocked agent hunts for the tests.** After a gate-1 block the smoke agent ran `cat .claude/hooks/stop.sh; find / -name
test_T1.py` (denied only because `find` is not on the allow-list; `cat` is, and `stop.sh` names the bench dir, from where
`hidden/` is one `cat` away). Gated configs' `.claude/settings.json` now carries `permissions.deny`: Read of `./.claude/**`,
the bench's `hidden/ runs/ validation/ reference/`, the two sibling benches; Bash patterns containing `.claude/hooks`,
`.claude/settings`, `../hidden`, `../runs`, `gate_tests`, the bench sub-paths, `find *`. Probed with a Haiku session: all
five escape attempts (cat, Read, ls, python open) denied, in-tree reads fine. Bash rules match command text, not a security
boundary. Bare runs have no settings file (bare r1 ran before this change; bare has no motive to hunt).
Reviewers and the writer run with `--setting-sources ""`, so these rules do not apply to them.
- **Transient `FileNotFoundError` spawning a reviewer** inside the hook (once, both reviewers of one call, wall 0.0 s;
never reproduced by hand). Reviewer calls now retry once after 3 s; the error message records `filename`.
- **`--bare` cannot authenticate here.** It reads `ANTHROPIC_API_KEY` / `apiKeyHelper` only, never OAuth or the keychain;
this account is OAuth (`~/.claude`), so a `--bare` reviewer returns `terminal_reason: api_error`, cost 0. Used the
documented equivalents instead: `--setting-sources ""` + `--settings '{"disableAllHooks":true}'` + `--strict-mcp-config` +
`CLAUDE_CODE_DISABLE_CLAUDE_MDS=1` + `CLAUDE_CODE_DISABLE_AUTO_MEMORY=1` + `--tools Read,Grep,Glob`. The reviewers' cwd is the
work copy, whose `.claude/settings.json` carries the Stop hook: without `--setting-sources ""`/`disableAllHooks` a reviewer
would run the gate recursively. Verified: a dummy Stop hook in the probe copy never fired.
- **Inherited session env.** A matrix launched from inside Claude Code inherits `CLAUDE_EFFORT=medium`,
`CLAUDE_CODE_SUBAGENT_MODEL=opus`, `CLAUDE_CODE_SESSION_ID`, `CLAUDECODE`, `CLAUDE_CODE_CHILD_SESSION`, messaging tokens.
`env.sh` unsets every `CLAUDE*` var before setting its own; reviewers get a scrubbed env too (the hook process itself
inherits the worker's `CLAUDE_*` vars, e.g. the worker's session id).
- **The feedback line leaks the hook command.** Claude Code prefixes each block with the command line:
`Stop hook feedback:\n[<command>]: <reason>`. With the SPEC command inline the agent would read `GATE_CONFIG=g13adv` and
`gate-bench/gate.sh`. The settings command is therefore `bash "$CLAUDE_PROJECT_DIR/.claude/hooks/stop.sh"`, a one-line
wrapper holding the SPEC command. The agent can still `cat` it (it lives in the work copy) but it is not pushed at it.
- **Stop hook input keys (2.1.293):** `background_tasks, cwd, effort ({"level":"medium"}), hook_event_name,
last_assistant_message, permission_mode, prompt_id, session_crons, session_id, stop_hook_active, transcript_path`.
- **8-cap resets on tool calls.** Per the docs the consecutive-continuation count resets each time Claude calls a tool, so the
built-in cap only bites an agent that answers blocks with text alone. Our give-up (6 blocks per run, counted from
`gate.jsonl`, tool calls or not) fires first in every case.
- **Red start state: the agent refuses to fix code it did not write.** Validation step 2 (T1 + an uncommitted `>=` -> `>`
edit) shows Sonnet 5.5 implementing T1, then answering 6 consecutive gate-1 blocks with "the red test is your pre-existing
uncommitted edit, not my change" without touching it, until give_up. The hook mechanics are validated; the behaviour is a
finding in itself (gate 1 cannot force a fix the agent considers out of scope).
- **pytest-cov.** The repo's `pytest.ini` adds `--cov tinydb`; gate 1 adds `--no-cov` so the 40-line tail holds the failure,
not the coverage table (and no `.coverage` file is written). `.coverage` is gitignored anyway.
- **T5 R4 is parametrized** (several test ids, one requirement). hidden counts are per requirement (5 per task), a req
passes only when all its ids pass. `validate.sh` step 1: T5 upstream passes 1 req (R3, the regression guard) as designed.
- **Diff without touching the index.** `git add -N` would change what the agent sees in `git status`; untracked files are
diffed with `git diff --no-index /dev/null <f>` instead. `.claude/` is in `.git/info/exclude`, so the hook config never
shows in any diff or in `git status`.
- **Worker Bash denials.** The 2.1.293 permission layer denies some agent commands with "Contains brace with quote character
(expansion obfuscation)" even under acceptEdits + allow-list; agents route around it (visible as `permission_denied` events).
- **Reviewers on a correct solution still flag "blocking".** On the T1 reference patch, reviewer A marked "pull does not check
the field is a list" (not required by the task) as blocking; indep therefore blocks, the adv challenger rejected it. This is
exactly what false_blocks measures (hidden all green at that stop).
- **Shared plan meter.** Workers and reviewers bill the user's plan quota; `total_cost_usd` is the list-price equivalent the
CLI reports, not a bill. Nothing else heavy should run on the account during the matrix.
- **rtk rewrites interactive `grep`/`diff`** in the launching Claude session only; scripts here call binaries directly
(`/usr/bin/diff`/`cmp` for verification diffs).
- **CLI version:** `claude --version` = 2.1.293 (Claude Code); flags checked in `claude --help`: `--json-schema`, `--bare`, `--tools`,
`--permission-mode` (choices acceptEdits, auto, bypassPermissions, manual, dontAsk, plan), `--setting-sources`,
`--include-hook-events`. `--max-turns` is accepted though not listed in `--help`.
## Matrix launch (2026-10-08 16:33:54, after the gate-1 redesign)
`cd /Users/thomas/dev/gate-bench && SKIP_DONE=1 nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 &` -> PID 71317, 3 lanes.
The 5 bare r1 runs (terse prompts) are kept and skipped; 45 runs to go. Expected wall ~40-70 min: gated runs so far take
~30-70 s of worker wall + ~35-40 s of test writer (before the worker) + 1-35 s per hook call, more with blocks.
Then `.venv/bin/python analyze.py`.
SPEC.md
# gate-bench — SPEC (written 2026-10-08 by the /plan stage of the AIDive pipeline)
Goal: measure, on a real repo, what each of Shopify's Helix gates catches when rebuilt as a Claude Code
Stop hook, and what it costs. Output feeds a YouTube video; every number must be reproducible from this dir.
## Reuse (read-only, never modify)
- `/Users/thomas/dev/verify-loops-bench/upstream/` : tinydb @ 18d73a1 (git repo). Copy it as `upstream/` here.
- `/Users/thomas/dev/verify-loops-bench/hidden/` : `test_T1.py..test_T6.py`, `conftest.py`, `pytest.ini`. Copy as `hidden/`.
Hidden tests are the ORACLE: one `test_reqN` per requirement of the task prompt. They live OUTSIDE every work copy
and the agent never sees them. Run: `cd <tree> && python -m pytest -c <bench>/hidden/pytest.ini --rootdir <bench>/hidden <bench>/hidden/test_Tn.py -q`
(conftest puts cwd or `$TINYDB_TREE` first on sys.path). Check `tasks/README.md` there for the exact invocation and validate with `validate.sh`.
- `/Users/thomas/dev/verify-loops-bench/tasks/T1.md..T5.md` : the five task prompts, typed as a developer would. Copy as `tasks/`. T6 is NOT used.
- `/Users/thomas/dev/haiku-routing-bench/{env.sh,run.sh,extract.py,prep.sh,matrix.sh}` : patterns to copy (auth via the
user's default config dir, `CLAUDE_CODE_DISABLE_AUTO_MEMORY=1`, `.venv` with pytest, stream-json extraction of
`total_cost_usd`, `num_turns`, `usage`, `modelUsage`, fresh work copy per run, `--setting-sources project --strict-mcp-config`).
- Claude Code CLI 2.1.293 is installed. Docs that define the hook contract: https://code.claude.com/docs/en/hooks
(fetch with `curl -s https://r.jina.ai/https://code.claude.com/docs/en/hooks`): a Stop hook that exits 2 blocks the
stop and its stderr is fed back; `{"decision":"block","reason":"..."}` on stdout does the same; input JSON on stdin
carries `stop_hook_active`, `session_id`, `transcript_path`; after 8 consecutive continuations Claude Code overrides
the next block (`CLAUDE_CODE_STOP_HOOK_BLOCK_CAP` raises it); a hook past `timeout` (seconds, default 600) renders no decision.
## Worker
`claude -p "<task prompt>" --model claude-sonnet-5-5 --output-format stream-json --verbose --max-turns 60
--permission-mode acceptEdits --allowedTools "<same list as haiku-routing-bench/run.sh>" --setting-sources project
--strict-mcp-config`, cwd = fresh work copy `work/<run>/` (copy of upstream, `git` clean at HEAD). The task prompt is the
T-file verbatim plus one line appended for every config (bare included): "When you are done, stop and summarise what
you changed." No mention of gates in the prompt: the gate is the hook, not the instruction.
Work copy carries `.claude/settings.json` with the Stop hook for gated configs (none for `bare`):
`{"hooks":{"Stop":[{"hooks":[{"type":"command","command":"GATE_RUN=<abs run dir> GATE_CONFIG=<cfg> bash <bench>/gate.sh","timeout":1500}]}]}}`
(check the exact settings shape in the hooks doc; `timeout` is in seconds; verify the hook fires with a smoke run before the matrix).
## Gates (gate.sh, one script, dispatches on GATE_CONFIG)
Reads stdin JSON. Logs ONE line per invocation to `$GATE_RUN/gate.jsonl`: {ts, n (invocation index), stop_hook_active,
gate1:{ran, suite_pass, suite_fail, tests_touched (bool: any file under tests/ added or modified vs HEAD), pass},
gate3:{ran, variant, reviewers:[{id, approve, findings:[{severity, file, line, text}], cost_usd, turns, wall_s}], blocking_findings, pass},
hidden:{pass, fail, failed_reqs} (ground truth at this moment, run SILENTLY on the work copy; never shown to the agent),
decision ("block"|"allow"|"give_up"), reason (the text sent back, truncated 500), wall_s}.
Give-up: after 6 blocks in one run the hook allows and logs decision "give_up" (the video needs to count it; Claude
Code's own 8-cap would otherwise hide it). Never use stop_hook_active to allow: log it only.
Order inside one invocation: gate 1 (when the config has it) first; gate 3 only when gate 1 passed or is absent.
- Gate 1 (behaviour tests): `python -m pytest -q -x --no-header -p no:cacheprovider` on the work copy (the repo's own 226 tests plus
whatever the agent wrote). Block when red (reason = last 40 lines of pytest), OR when `tests_touched` is false
(reason: "No test covers the new behaviour. Add tests for every requirement in the task, run them, then stop.").
- Gate 3 (adversarial reviews): headless reviewers, each `claude -p --bare --model claude-sonnet-5-5 --tools "Read,Grep,Glob"
--max-turns 15 --output-format json --json-schema '<schema>' --permission-mode dontAsk` (confirm each flag exists in
`claude --help`; drop `--permission-mode dontAsk` if it errors), cwd = work copy, prompt = system framing + the task
prompt + `git diff HEAD` (tracked + untracked files, cap 60k chars) + "Find defects: requirements not met, edge cases
wrong, behaviour broken, tests that do not test the requirement. Report only real defects with file and line. approve=true
only when nothing blocks." Schema: {approve: boolean, findings: [{severity: "blocking"|"minor", file, line, text}]}.
Read `structured_output` from the JSON result; record `total_cost_usd`. Set `CLAUDE_CODE_DISABLE_AUTO_MEMORY=1`.
- variant `indep` (Shopify as written): two reviewers, same prompt, independent, no shared context; union of findings;
block if any blocking finding from either; reason = the blocking findings, numbered, "Fix every finding, re-run the
tests, then stop." Pass when BOTH approve.
- variant `adv` (reviewer + challenger, arXiv 2608.18167 shape): reviewer A as above; then challenger B gets A's
findings plus the diff and must, per finding, answer upheld|rejected with a reason, and may add findings A missed;
only upheld (or B-added) blocking findings block. Pass when nothing upheld blocks.
- Reason text is what the worker sees next turn: keep it factual, under 1500 chars.
## Configs (configs.json)
- `bare`: no hook.
- `g1`: gate 1 only.
- `g3`: gate 3 indep only.
- `g13`: gate 1 then gate 3 indep (Shopify's order).
- `g13adv`: gate 1 then gate 3 adv.
Gate 2 (UI) and gate 4 (engineer) are out of scope by design: tinydb is a library, Shopify allows skipping UI review for
logic-only changes; gate 4 is the human, measured as "what reached the human": the hidden failures left at the end.
## Per run outputs (runs/<task>-<cfg>-r<rep>/)
stream.jsonl, stderr.txt, exit_code.txt, wall_s.txt, gate.jsonl, post.diff, numstat.txt, suite.txt (final repo suite),
hidden.txt (final hidden run), result.json (worker: total_cost_usd, num_turns, usage, modelUsage, is_error, subtype,
final text), score.json: {task, cfg, rep, hidden_pass, hidden_fail, failed_reqs, suite_pass, suite_fail, blocks_total,
blocks_by_gate {g1, g3}, right_blocks (hidden failing at that moment), false_blocks (hidden all passing at that moment),
give_up, ended_blocked (the LAST gate.jsonl line is a block, i.e. Claude Code's cap or the agent ended anyway),
cost_worker_usd, cost_reviewers_usd, cost_total_usd, turns, wall_s, reviewer_findings_total, model_ok}.
Snapshot at every hook call: before gate 1 runs, `git diff HEAD` of the work copy to `$GATE_RUN/snap-<n>.diff`.
## Matrix
5 tasks (T1..T5) × 5 configs × 2 reps = 50 runs. `matrix.txt` ordered: rep 1 of everything first (config-contiguous:
bare, g1, g3, g13, g13adv, tasks rotated), then rep 2 in reverse config order. `matrix.sh` runs up to 3 runs in parallel
(each in its own work copy), appends one line per run to `matrix.log`, resumable (`SKIP_DONE=1` skips runs with score.json).
`analyze.py` → `results.json` + `results.md`: per config: hidden requirements passed / total, runs fully passing, mean
cost, mean wall, blocks per run, right vs false blocks per gate, give-ups, ended_blocked; per task the same; and the
answer to the video's questions: (1) does gate 1 alone move hidden pass rate vs bare, (2) does gate 3 indep add anything
over g1, (3) does adv beat indep, (4) false-block rate per gate, (5) cost per gate.
## Validation before the matrix (VALIDATION.md + validate.sh)
1. upstream fails every hidden test file; the reference patches in `/Users/thomas/dev/verify-loops-bench/reference/`
pass hidden + suite (reuse their validate.sh logic).
2. The hook fires: one smoke run of T1 on `g1` with a deliberately failing state (apply a patch that breaks one test)
must show a block in gate.jsonl, and the stream must show the agent continuing after "Stop hook feedback".
3. gate.sh called by hand with a fake stdin on a clean work copy returns allow, on a red copy returns exit 2 with the
pytest tail on stderr.
4. One reviewer call by hand returns valid structured_output against the schema.
5. Record the exact `claude --version`, the settings.json used, and the hook input keys seen.
## Deliverables
Everything under /Users/thomas/dev/gate-bench with README.md (layout, commands, traps found). When the matrix is launched,
run it with `nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 &` and report the PID; do not wait for it to finish.
Copy the scripts (not runs/, not work/) to /Users/thomas/dev/faceless-video-pipeline/jobs/shopify-four-gates-claude-code-20261008-1529/research/code/
with a one-line README. Report: what was validated, the smoke run's gate.jsonl, the matrix PID, expected duration.
VALIDATION.md
# Validation (2026-10-08, claude 2.1.293) — raw output in validation.log
1. **Oracle.** `./validate.sh step1`: on upstream every hidden req fails (T5: 1 passing, R3 = regression guard, as designed);
each reference patch passes its hidden file (T1-T4 5 tests, T5 6 tests = 5 reqs) and the repo's 226 tests. OK.
2. **Hook fires.** `INJECT=$PWD/inject/break.patch RUN_TAG=val2 ./run.sh g1 T1 1` (T1 + an uncommitted `>=` -> `>` break):
gate.jsonl = 7 calls, 6 `block` (g1, suite red 1 failed / 33 passed under -x) then `give_up`; the stream shows
`Stop hook feedback:` user messages followed by new assistant turns (the agent continued each time; it kept
explaining the red test is the user's pre-existing edit and never touched it). `stop_hook_active` false on call 1, true after. OK.
3. **gate.sh by hand** (`./validate.sh step3`, fake stdin): green copy (T1 reference + one new test) -> exit 0, `allow`,
empty stderr; red copy (break.patch) -> exit 2, `block`, stderr = pytest tail incl. `short test summary info`. OK.
4. **Reviewer by hand** (`./validate.sh step4`, `lib/review.py work/_val3-green T1 indep|adv`): every call returned a
schema-valid `structured_output`, cost $0.03-0.05 per pair, 10-45 s per call. OK.
5. **Recorded.** `claude --version` 2.1.293. settings.json (gated configs):
`{"hooks":{"Stop":[{"hooks":[{"type":"command","command":"bash \"$CLAUDE_PROJECT_DIR/.claude/hooks/stop.sh\"","timeout":1500}]}]}}`,
`.claude/hooks/stop.sh` = `GATE_RUN=<run dir> GATE_CONFIG=<cfg> bash /Users/thomas/dev/gate-bench/gate.sh`.
Hook input keys seen: background_tasks, cwd, effort, hook_event_name, last_assistant_message, permission_mode, prompt_id,
session_crons, session_id, stop_hook_active, transcript_path. Stream hook events in the smoke run: exactly one
`hook_started`/`hook_response` pair, `hook_name: Stop` (no user/global hooks: `--setting-sources project`).
Smoke (`runs/T1-g13adv-r1-smoke`, wrapper hook): hidden 5/5, suite 238/238, 1 hook call, gate 1 green with tests touched,
adv reviewer A approve (0 findings), challenger approve, allow. Worker $0.084 + reviewers $0.034 = $0.117, 12 turns, 66 s.
`runs/T1-g13adv-r1-smoke0` = the first smoke with the inline SPEC command (same outcome, $0.195, 69 s), superseded because the
feedback prefix would leak the command line.
## Gate 1 redesign (2026-10-08, after the coordinator's review): independent test writer
Gate 1 no longer re-runs only the implementer's own tests (it passed by construction: `T2-g1-r1` one allow, hidden R4
missed, suite green). Before the worker starts, `lib/testwriter.py` (headless Sonnet 5.5, reviewer isolation flags,
`--tools Read,Grep,Glob --max-turns 12 --json-schema {file}`, cwd = a pristine temp copy of upstream) gets ONLY
`tasks/<T>.md` and writes `runs/<run>/gate_tests/test_<T>.py` (+ hidden's conftest/pytest.ini) outside the work copy.
Gate 1 = repo suite AND those tests (`TINYDB_TREE` = work copy). `tests_touched` is logged, no longer a rule.
`./validate_writer.sh` (one writer call per task, files in `validation/writer/<T>/`; reference = verify-loops-bench patches):
| task | writer tests | pass on reference | pass on upstream | fail on reference (possible false blocks, kept) | writer $ / wall |
|---|---|---|---|---|---|
| T1 | 52 | 51 | 1 | `test_pull_on_non_list_field_raises_and_leaves_data_unchanged` | 0.099 / 39 s |
| T2 | 42 | 42 | 5 | none | 0.094 / 35 s |
| T3 | 35 | 34 | 11 | `test_rename_table_non_string_target_does_not_corrupt_database` | 0.077 / 34 s |
| T4 | 48 | 47 | 7 | `test_reverse_without_sort_by_reverses_default_order` | 0.092 / 42 s |
| T5 | 26 | 25 | 7 | `test_non_integer_size_is_rejected` | 0.086 / 30 s |
Tests passing on upstream are the "existing behaviour unchanged" guards the instruction asks for (plus a few that
accept "raises" as an outcome); every task's file fails on upstream overall. Each matrix run writes its own file, so
these counts are one sample; per-run counts land in `score.json` (`gate_tests_total`, `gate_tests_fail_final`).
Overlap with the 5 hidden requirements (judgement from test names + spot reads of the assertions):
- T1: 2/5 strict (R1 order, R5 push non-list raises). R2/R4 hedged (`try: ... except (KeyError, AttributeError, TypeError)`: a raise is accepted), R3 hedged (`count('x') < 2`, first-occurrence removal passes).
- T2: 4/5 (R1, R2, R3, R5 cache inclusive vs exclusive and other bounds); R4 hedged (accepts a TypeError).
- T3: 3/5 strict (R1 ids, R2 persistence + `tables()`, R5 stale cache / old name empty / id sequence); R3/R4 use `pytest.raises(Exception)`, not KeyError/ValueError (R4 does check both tables intact).
- T4: 4/5 (R1, R3, R4 limit 0 / negative ValueError, R5 cache not polluted + Document/doc_id); R2 hedged (checks order of docs that have the field and that none are dropped, not "missing last in both directions").
- T5: 5/5 (R1, R2 restart, R3 class/subclass default, R4 non-positive ValueError at construction, R5 close flushes).
The writer hedges exactly where the terse prompt is silent: it encodes "a raise or a no-op are both fine", so gate 1
cannot recover a requirement the prompt never states (T1 R2/R4, T4 R2).
Smoke after the redesign (`runs/T1-g13adv-r1-smoke`, with the deny rules): writer 45 tests ($0.071), first stop gate 1
green (232 suite + 45/45 gate tests), adv A+B approve, allow; hidden 2/5 (R2, R4, R5 missed). Total $0.189, 68 s.
`runs/T1-g13adv-r1-smoke-a` (first redesign smoke): stop 1 blocked by gate 1 (5 gate tests red, reason starts
"The acceptance tests for this change fail:"), the agent fixed push/pull type checks, stop 2 gate 1 green, both reviewers
errored (transient FileNotFoundError on spawn, logged, fail-open allow; one retry added since). The agent also tried
`cat .claude/hooks/stop.sh; find / -name test_T1.py` after the block (denied by the allow-list) -> deny rules added.
`runs/T1-g13adv-r1-smoke-b`: second redesign smoke, clean (reviewers ran, $0.167).
Haz esto el lunes
- Escribe la comprobación como un script que sale con 2 y deja el fallo en stderr, y conéctala como un Stop hook de tipo comando con un timeout mayor que tu suite.
- Haz que un agente aparte escriba los tests de aceptación solo a partir de la petición, guárdalos fuera del repo y deniega al implementador la lectura del hook, de la configuración y de esos tests.
- Registra cada parada en un archivo con la decisión y si el gate estaba en rojo, para que un gate que dejó pasar trabajo sea visible.
- Añade tu propio contador que deje salir al bucle tras seis bloqueos. El tope de Claude Code son ocho bloqueos consecutivos y se reinicia en cada llamada a una herramienta.
- Ejecuta los revisores en headless con
--tools Read,Grep,Glob, una fuente de configuración vacía y los hooks desactivados, para que no puedan disparar el gate de forma recursiva. - Si mantienes un gate de revisión, devuelve sus hallazgos como consejo en el feedback en lugar de convertirlos en una barrera.
- Cuenta los bloqueos correctos y los falsos durante dos semanas antes de fiarte del gate.
- Recuerda que
--baresolo lee una clave de API: con un login de suscripción devuelve un error, así que aísla a los revisores con flags de configuración.
Mediciones
Desde results.md: 50 ejecuciones, 10 por configuración, 5 tareas por 2 repeticiones. Los requisitos ocultos se cuentan sobre 50 por configuración. Los bloqueos se escriben correctos/falsos: un bloqueo correcto significa que el oráculo oculto tenía un requisito fallando en esa parada, un bloqueo falso significa que estaba totalmente en verde.
| Configuración | Requisitos ocultos superados | Ejecuciones totalmente limpias | Coste medio por ejecución | Segundos de reloj medios | Bloqueos gate 1 correctos/falsos | Bloqueos gate 3 correctos/falsos | Rendiciones |
|---|---|---|---|---|---|---|---|
| bare (sin hook) | 35/50 (70%) | 3 | $0.085 | 33 | 0/0 | 0/0 | 0 |
| g1 (tests de aceptación) | 42/50 (84%) | 6 | $0.201 | 60 | 3/3 | 0/0 | 0 |
| g3 (dos revisores) | 34/50 (68%) | 3 | $0.151 | 54 | 0/0 | 1/0 | 0 |
| g13 (tests, luego dos revisores) | 41/50 (82%) | 5 | $0.282 | 86 | 14/1 | 2/0 | 2 |
| g13adv (tests, luego revisor más crítico) | 39/50 (78%) | 6 | $0.276 | 86 | 11/1 | 0/0 | 1 |
El coste es la suma del trabajador, el escritor del gate 1 y los revisores del gate 3, tal como lo reporta total_cost_usd, un equivalente a precio de lista y no una factura. Cada ejecución llevó el modelo configurado. En las ejecuciones con gate, el gate 1 devolvió al agente 28 veces con un requisito real fallando y 5 veces con todos los requisitos superados. El gate 3 bloqueó 3 veces y nunca por error. Tres bucles se rindieron a los seis bloqueos, y el tope propio de Claude Code nunca saltó.
Pendientes para el humano sobre 50 requisitos: 15 sin gate, 8 con los tests, 16 con los revisores solos, 9 con tests y revisores, 11 con tests y el revisor más crítico. Donde los tests movieron una tarea: la búsqueda ordenada pasó de 5 de 10 a 10 de 10 y la tarea del tamaño de caché de 8 a 10. La tarea de push y pull se quedó en 4 de 10, porque la petición nunca decía qué hacer con un documento al que le falta el campo.
Para ir más lejos
- El gate de UI. El gate 2 de Shopify hace que un segundo modelo actúe como revisor de diseño en un repo de front-end s1. Lo omitimos porque el repo es una librería. Una prueba posterior en un repo de front-end es el lugar para medirlo.
- El bucle Ralph. También mantiene a un agente en marcha hasta que se cumple una condición, con la salida en una frase que el agente puede escribir. Nuestra salida es externa: un script y un contador fuera del alcance del agente.
- El paper de revisión adversarial. Lee las tablas, no el resumen: la forma de revisor más crítico s5 es la que en el papel rentabiliza sus tokens, y nuestra ejecución no reprodujo una mejora en este repo, así que pruébala en el tuyo antes de adoptarla. Página del abstract: s23.
- El tope de bloqueos y el timeout. Lee la sección Stop de la referencia de hooks antes de depender de un hook como parada firme s2, y la sección de troubleshooting de la guía de hooks s19.
- Un gate medido con el que comparar tu forma de reportar: s8.
FAQ
¿Puede un Stop hook impedir de verdad que Claude Code termine?
Casi siempre. Salir con 2 impide la parada y devuelve stderr s2, pero tras ocho continuaciones consecutivas Claude Code anula el siguiente bloqueo, y un hook que agota su timeout no emite ninguna decisión s2. Registra cada parada para ver cuándo dejó pasar trabajo.
¿Por qué el escritor de tests es un agente aparte?
Nuestra primera versión del gate 1 volvía a ejecutar los tests del propio implementador. Dejaba pasar todas las ejecuciones en su primera parada mientras un requisito oculto fallaba, porque los tests de un agente prueban lo que ese agente construyó. El escritor rediseñado ve solo la petición y el repo previo al cambio.
¿Por qué no usar --bare para los revisores?
Omite hooks y memoria s20, pero con un login de suscripción solo lee una clave de API y aquí devolvió un api_error. Aislamos a los revisores con una fuente de configuración vacía, hooks desactivados, una configuración MCP estricta y herramientas de solo lectura.
¿Bloquearon alguna vez los revisores un trabajo bueno?
No en nuestras 50 ejecuciones: 3 bloqueos, ninguno erróneo. El coste fue lo que dejaron pasar. En dos ejecuciones ambos revisores vieron el mismo fallo con un campo de ordenación ausente y lo calificaron de menor, y las dos ejecuciones terminaron en cero de cinco.
¿Demuestra esto que los revisores nunca ayudan?
No. Es un repo, un modelo trabajador, cinco tareas y dos repeticiones por celda. Dice que los revisores no movieron la tasa de aciertos aquí y que el gate de tests sí.