# flow-bench: Superpowers loop vs Matt Pocock's Main Flow, one feature, one repo Built 2026-10-09 from `SPEC.md` (the fixed decisions are implemented as written; deviations and their reasons are listed under "Decisions taken where SPEC left a choice"). Worker = headless `claude-sonnet-5-5` (`claude -p`, OAuth from `~/.claude`), Claude Code 2.1.294. ## Commands (from this dir) ```bash ./validate.sh # SPEC validation steps 1-5 -> validation.log (3-5 are smoke runs, RUN_TAG=smoke) ./validate.sh 1 2 # only the static steps (reference patch, untouched upstream) RUN_TAG=x ./run.sh # one run -> runs/-r-x/, work copy work// ; tagged runs are ignored by analyze.py nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 & # the 10-run matrix, 2 lanes (MATRIX_JOBS=2) SKIP_DONE=1 nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 & # resume python3 analyze.py # results.json + results.md python3 lib/finalize.py runs/ # rescore one run (restores node_modules into the work copy if it was removed) python3 lib/reaccount.py runs/ # recompute turns.jsonl accounting from the saved streams (then finalize) ``` ## Layout | path | what | |---|---| | `upstream/` | **ledgerly**, the invented repo: Node 20 + TS + vitest + pnpm, 18 source files (domain / store / http layers), 34 tests in 7 files, 4 commits (HEAD 980d542), `test` + `typecheck` scripts, `node_modules` installed (pnpm 11) | | `tasks/F1.md` | the task, verbatim (3 lines, the way a dev types it) | | `po/F1-brief.md` | the product owner's full intent: the 5 requirements in plain words, preferences, constraints (never the tests) | | `hidden/` | `test_F1.test.ts` (the oracle: `test_req1..5`), its own `vitest.config.ts` (alias `@work` -> `$WORK`) and `node_modules` (vitest 2.1.9). Runs from OUTSIDE the work copy | | `reference/F1.patch` | reference solution: hidden 5/5, suite 39/39, tsc green | | `configs.json` | `bare`, `sp`, `mp`, `both` | | `packs/` | `superpowers/` (plugin root of obra/superpowers v6.4.2), `mattpocock-skills/.claude/skills/` (27 skills of mattpocock/skills v1.3.1 installed by `npx skills add`), `VERSIONS.md`, `mattpocock-install.log`; `*-src/` = the clones (gitignored) | | `env.sh` | copied from an earlier bench of the same kind: scrubs inherited `CLAUDE*` vars, `CLAUDE_CODE_DISABLE_AUTO_MEMORY=1` | | `prep.sh ` | fresh work copy + the pack fixtures of the config + `.claude/settings.json`; prints the base commit | | `run.sh ` | prep -> `lib/driver.py` -> `lib/finalize.py` | | `lib/driver.py` | the turn loop (flows below), caps, `turns.jsonl` | | `lib/accounting.py` | per-turn and per-stage tokens / USD / wall / tools / Skill events / hook events from one turn's stream | | `lib/po.py` | the product-owner simulator | | `lib/finalize.py` | diff vs base, suite, typecheck, hidden oracle, stages -> `score.json`, `stages.json` | | `analyze.py` | `results.json`, `results.md` | | `validate.sh`, `validation.log`, `validation/routing-probes/` | pre-matrix checks (+ 6 one-turn probes: plugin wiring vs project wiring) | | `matrix.txt`, `matrix.sh`, `matrix-one.sh`, `matrix.log` | 10 runs, 2 at a time | ### The feature and the oracle `tasks/F1.md`: partial payments, `POST /invoices/:id/payments` with an amount in cents, `amountPaid`/`balance` on invoices, flip to `paid` at zero balance, keep `/pay` working, "we have prod data in the JSON store". The 5 hidden requirements (one `it` each) and the codebase precedent that makes each one a senior's expectation: | req | checks | precedent in the repo | |---|---|---| | R1 edge cases | payments accumulate; status stays non-`paid` until the exact remainder; `paidAt` set; `/reports/outstanding` sums balances, not totals | the report sums `total` of open invoices | | R2 validation | 0, negative, 12.5, "100", null, missing -> 400/422; overpayment (of the total, then of the balance) -> 400/409/422; state unchanged | `assertCents`, "discount cannot exceed the subtotal" | | R3 migration of existing data | a schemaVersion-1 file: legacy `paid` invoice shows amountPaid=total, balance 0; open shows 0/total; payments persist across a reload; `/pay` leaves balance 0 | `schemaVersion` + `src/store/migrations.ts` (v0->v1), README house rule | | R4 error path | unknown id 404; draft 409; void 409; nothing recorded | `markPaid` throws `ConflictError` on draft/void | | R5 a test | a `*.test.ts` changed or added since the base commit mentions `payments` | 34 existing tests | Lenient where the brief would be a guess (status codes for overpayment accept 400/409/422, payment response 2xx, response body never inspected: state is read back with GET). ## Configs and how each pack is wired (`prep.sh`) - Every config: `cp -R upstream`, `.claude/` in `.git/info/exclude`, `.claude/settings.json` with a `permissions.deny` list (Read of `hidden/ reference/ runs/ po/ validation/` of this bench; Bash commands containing `hidden`, `reference/F1`, `F1-brief`, `/runs/`, `../`, and `find`). Bare has nothing else: no skills, no hooks. - `sp`: the plugin root is copied to `.claude/plugins/superpowers/` (the hook script reads `../skills/using-superpowers/SKILL.md` relative to itself), the 15 skills to `.claude/skills//`, and `hooks/hooks.json` is reproduced in `.claude/settings.json` verbatim (`SessionStart`, matcher `startup|clear|compact`, `"shell": "bash"`, `"async": false`) with the command prefixed by `CLAUDE_PLUGIN_ROOT="$CLAUDE_PROJECT_DIR/.claude/plugins/superpowers"`: Claude Code sets that variable for plugin hooks, and `hooks/session-start` uses it to pick the `hookSpecificOutput.additionalContext` output shape. Superpowers ships no other hook (`hooks-cursor.json` is Cursor's). - `mp`: the 27 skills to `.claude/skills/`. Pocock ships **no hook** (verified: no settings or hooks file in the install; `git-guardrails-claude-code`, a skill that writes hooks, is outside the promoted set and was trimmed). The pack's per-repo precondition `/setup-matt-pocock-skills` is materialised once: `docs/agents/issue-tracker.md` (local markdown under `.scratch/`), `docs/agents/domain.md`, `docs/agents/triage-labels.md` from the skill's own seed templates, and a `CLAUDE.md` "## Agent skills" block, committed as "chore: configure agent skills (local tracker)" so the diff only shows the feature. Base commit for mp/both = that commit. - `both`: sp + mp in the same `.claude/` (no skill-name collision between the two packs), driven with the mp flow. ## Flows (`lib/driver.py`) Every worker turn: `claude -p [--resume ] --model claude-sonnet-5-5 --output-format stream-json --verbose --include-hook-events --include-partial-messages --max-turns 60 --permission-mode acceptEdits --allowedTools --disallowedTools AskUserQuestion,EnterPlanMode,ExitPlanMode,EnterWorktree,ExitWorktree,WebFetch,WebSearch --setting-sources project --strict-mcp-config`, cwd = work copy, stdin /dev/null, scrubbed env. Allow-list: `Read,Edit,Write,MultiEdit,Glob,Grep,Skill,TodoWrite,Task,Agent`, Bash `node npm pnpm npx vitest npx tsc tsc git status|diff|log|add|commit|checkout -b ls cat mkdir head tail grep wc gh`. - `bare`: turn 1 = task + "When you are done, stop and summarise what you changed." - `sp`: turn 1 = the task alone (the SessionStart hook is supposed to route it). - `mp` / `both`: `/grill-with-docs ` -> `/to-spec` -> `/to-tickets` -> `/implement the tickets in .scratch//issues/, all of them, blockers first` -> `/code-review since ` -> `/pr`. - After every turn the PO reads the last assistant message and classifies it: `question` (its reply is the next prompt), `waiting` (driver sends "go on" / "go on, proceed with the plan"; mp: stays in the stage), `done` (bare/sp: stop; mp: next slash command; after `/pr`: stop). If the PO call fails twice the SPEC heuristic (ends on "?" or an options list) decides. - Caps: 12 driver turns, 60 `--max-turns` per claude turn (a turn that hits it gets "go on"), 40 min wall per run (the claude process is killed at the deadline). Caps hit are recorded in `driver.json` / `score.json`, never discarded. ### PO simulator (`lib/po.py`) Headless `claude-sonnet-5-5`, `--tools ""`, `--setting-sources ""`, `--settings '{"disableAllHooks":true}'`, `--strict-mcp-config`, `CLAUDE_CODE_DISABLE_CLAUDE_MDS=1`, `CLAUDE_CODE_DISABLE_AUTO_MEMORY=1`, cwd = an empty temp dir, `--json-schema {kind, reply}`. Prompt = role + `po/F1-brief.md` + flow note + stage + the last message. Rules: terse, answer every open question at once, approve what matches the brief, correct only contradictions, "your call" where the brief is silent, **answer only what was asked** (first smoke: the PO volunteered the migration requirement nobody had asked about; the rule was added before any counted run), how to finish a branch: "keep it on the current branch, no PR". mp rule: a stage whose output is delivered and that only asks "shall I proceed / start?" is `done` (first mp smoke: the PO's "yes, go" after the grilling recap made the worker start implementing inside `/grill-with-docs`; that smoke was killed and rerun). PO cost is recorded apart (`usd_po`), never in the worker's USD. ## Accounting (the per-turn / per-stage numbers) Checked on 2.1.294 with probes before writing `lib/accounting.py`: - `result.total_cost_usd` and `result.modelUsage` are **cumulative over a resumed session** and **include subagents**: per-turn USD and tokens = delta against the previous turn's result line. - `result.usage` = main thread of the invocation only (a Task subagent's tokens are missing from it): not used. - `assistant` events carry the usage snapshot of the message START (`output_tokens` partial, e.g. 3 for a 783-token message): useless for output. `--include-partial-messages` adds `stream_event` `message_start` / `message_delta` with the exact per-API-call usage, main thread only. - Sonnet 5.5 list prices derived from two probe turns and matched to the cent: $2/M input, $10/M output, $0.20/M cache read, $4/M cache write (1 h TTL, what Claude Code uses here); 5-min writes priced $2.5/M (unseen). In every turn without subagents, the per-call priced sum equals the `total_cost_usd` delta exactly (`pricing_check` in `turns.jsonl`). - Stage = slash command of the turn (mp/both, exact) or the skill in force (sp: switches at each main-thread `Skill` tool_use; turn 1 starts at `(none)`). Inside a turn, main-thread calls are attributed exactly; the subagent remainder (turn delta minus main calls) goes to the stage that spawned the Task, weighted by the subagents' snapshot tokens. Wall per stage = receive-time gaps of the stdout lines (`turn-NN.times`) attributed to the stage in force. - `analyze.py` also groups sp skills into phases (brainstorming / writing-plans / execution incl. TDD / verification / review / finishing). - Tokens reported = input + output + cache read + cache write. Cache reads dominate (re-reading the context each call). ### Per run (`runs/-r/`) `meta.json`, `prep.txt`, `settings.json` (copy of the work copy's), per turn `turn-NN.stream.jsonl` (raw stdout, every line), `turn-NN.times`, `turn-NN.stderr`, `turn-NN.last.txt`, `po-NN.json`; `turns.jsonl` (one accounting record per turn: prompt, source, wall, subtype, USD, tokens per model, segments, tools, Skill events, hook events, files, denials, PO answer), `driver.json` (end reason, caps), `post.diff`, `numstat.txt`, `suite.txt`, `typecheck.txt`, `hidden.txt`, `stages.json`, `score.json`. Work copies stay in `work/` without `node_modules` (49 MB each; finalize removes it, `KEEP_NODE_MODULES=1` keeps it). ## Decisions taken where SPEC left a choice - **Implement stage = `/implement`, not `/implement-spec`.** v1.3.1 ships both. `/implement-spec` requires implementer subagents "each in its own worktree on its own branch" plus merger subagents: `git worktree`, `git merge`, `git branch` are outside the SPEC Bash allow-list, so it could not run as designed. `/implement` is sent once with the ticket directory ("all of them, blockers first"), since one session cannot `/clear` between tickets as `ask-matt` suggests. - `/code-review` gets `since ` (the skill reviews "the diff since a fixed point"); Pocock's `code-review` replaces Claude Code's bundled `code-review` skill of the same name in the session (init lists it once, the Skill event loads `.claude/skills/code-review`). - `/retro` not run (optional in SPEC). - `both` uses the mp flow: the stacking question is whether the Superpowers SessionStart routing hijacks the slash commands. - Reps / order: `matrix.txt` (rep 1 of every config first), 2 lanes. ## Traps met - **Superpowers routing is stochastic headless.** On turn 1 the hook always fires, but the model picks `test-driven-development` instead of `brainstorming` part of the time. Six one-turn probes on the F1 prompt (`validation/routing-probes/`): real plugin wiring (`--plugin-dir packs/superpowers`, namespaced `superpowers:*`) 1/3 brainstorming, 2/3 TDD; project wiring (this bench) 2/3 brainstorming, 1/3 TDD. So the project wiring is not the cause; the first sp smoke went TDD-only (one turn, hidden 5/5). A second sp smoke (`smoke2`) was run to see the brainstorming path end to end. - **The SessionStart hook fires only on turn 1.** Its matcher is `startup|clear|compact`; every later turn is `--resume` (source `resume`), so the injection is not repeated; it stays in the transcript. - **User plugins do not leak** with `--setting-sources project`: the init event lists only Claude Code's built-in plugins (`cc-plugin-agents-md`, `cc-plugin-telemetry`, `cc-plugin-plugin-authoring`), never the user's Superpowers 6.4.1; bare smoke: no hook event, no Skill event. Built-in skills (`verify`, `debug`, `simplify`, `code-review`, ...) are still listed in every config. - **`--bare` is never used** (cannot authenticate on this OAuth account; see the previous bench). - **Read-only shell commands bypass the allow-list.** Claude Code auto-approves read-only commands (`git ls-files`, `sed -n`, pipes of read-only commands) whatever `--allowedTools` says; `python3` edits are denied (the bare smoke had 4 `permission_denials`, all `python3` heredoc edits; the worker fell back to Edit). `cat > /tmp/x < hidden 5/5, suite 39/39 green, tsc green: PASS. 2. upstream unchanged -> hidden 0/5, suite 34/34 green: PASS. 3. `bare-r1-smoke` -> 2 driver turns (the PO answered one closing question), hidden 5/5, $0.52, 125 s; isolation: no hook event, no Skill event, only built-in plugins in init: PASS. 4. `sp-r1-smoke` -> hook fired and injected `using-superpowers` (PASS) but the model went straight to `test-driven-development` (brainstorming FAIL in that run). Retry `sp-r1-smoke2`: hook fired, `brainstorming` on turn 1, PO approved the design, then `test-driven-development`: PASS. Routing probes: see Traps. 5. `mp-r1-smoke` -> turn 1 loads `grilling` + `domain-modeling` (what `/grill-with-docs` says to do): PASS. The run then used all 12 driver turns (grill 4, to-spec 2, to-tickets 2, implement 2, code-review 2) and never reached `/pr`. Fixes made during validation, before any counted run: the PO volunteering unasked requirements; the PO telling the worker "yes, go" at the end of the grilling (the worker started implementing inside `/grill-with-docs`; that first mp smoke was killed at turn 4 and rerun). ## Matrix launch `nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 &` at 2026-10-09 10:41:33, 2 lanes, order in `matrix.txt`. Done 11:06:29 (10 runs, 25 min). ### Reruns of the mp-flow runs (harness fixes, `matrix.log`) - `both-r1` (first): the PO answered the grilling recap's branch question with an approval and the worker coded inside `/grill-with-docs` (turn 4, 12 src/test files). Kept as `runs/both-r1-pofault` (tag `pofault`, ignored by analyze). PO rule added (mp flow: before `/implement`, end with "Don't start implementing yet." when the worker offers to build). - `both-r1` (second): the PO did not apply that rule and the worker coded inside `/grill-with-docs` again (turn 3, after "say so and I'll start with failing tests"). The rule was made deterministic in `lib/driver.py`: in `/grill-with-docs`, `/to-spec`, `/to-tickets`, the driver appends "Don't start implementing yet." to every PO answer. Since that changes the conditions of all mp-flow runs, mp-r1..r3 and both-r1 were moved to `runs/*-po1` (tag `po1`; mp-r1..r3-po1 never coded early, hidden 5/5 each, $2.22 to $2.57) and rerun with `matrix-mp2.txt` (11:18 to 11:40). The counted mp/both runs are the reruns. - sp stage labels: brainstorming's SKILL.md ends at the approved design, but in sp-r1 and sp-r3 the model then coded with no new Skill call, so "skill in force" would bill the whole build to brainstorming. `lib/accounting.py` now switches the stage to `implementation (no skill)` at the first main-thread Edit/Write under `src/` or `test/` while only a design skill (`(none)`, `using-superpowers`, `brainstorming`, `writing-plans`) is in force; the sp runs were re-accounted from their saved streams with `lib/reaccount.py` (tokens, USD and wall totals unchanged, only the split). ## Results `results.md` / `results.json` (`python3 analyze.py`), reading in `FINDINGS.md`. --- # Round 2 (2026-10-09 afternoon): `spd`, feature F2, isolation fix, full rerun ## Isolation leak found and fixed (all round-1 runs superseded) The first `spd` smoke said "the most capable model your CLAUDE.md permits for subagents": the workers were reading CLAUDE.md files. `--setting-sources project` does keep `~/.claude/CLAUDE.md` out (probe in an empty dir outside the repo: "NONE"), but Claude Code also loads **every CLAUDE.md of the parent directories**, and the work copies lived under the host repository, so every round-1 session carried that repository's CLAUDE.md and what it imports (a probe in a round-1 work copy showed the host rules present; the round-1 mp code reviews even cited them). Fix: `run.sh` builds the work copy in `/private/tmp/flow-bench/` (outside any CLAUDE.md tree) and archives it to `work/` (without node_modules) after finalize; probe in an mp copy there: only the pack's own "## Agent skills" CLAUDE.md block appears. Every round-1 run was renamed `runs/*-leak` (tag `leak`, ignored by analyze) and the whole matrix rerun (`matrix2.txt`). The short path also removes the "pocock-skins" typo surface (see Traps). ## `spd`: Superpowers, driven Same project install as `sp` (skills in `.claude/skills/`, SessionStart hook). Installed in the project, the skills resolve **without the plugin namespace**: init `slash_commands` lists `brainstorming`, `writing-plans`, `executing-plans` (with `--plugin-dir` they are `superpowers:brainstorming`, ...; see `validation/routing-probes/`), so the driver types `/brainstorming ` -> `/writing-plans` -> `/executing-plans`. writing-plans ends by asking "Subagent-driven or Native?": the PO answers Native (added to both briefs: "If asked to choose how to execute a written plan: Native (inline, in this session) ... working on main is fine: you consent"). Before `/executing-plans` the driver appends "Don't start implementing yet." to PO answers (same rule as mp). A stage whose output is delivered and that only asks for approval/go is `done` -> next command; after `/executing-plans` the run ends when the PO says done (executing-plans itself runs the final whole-branch review and finishing-a-development-branch). - Allow-list: the SPEC list + `Bash(*scripts/task-start*)`, `task-done`, `sdd-workspace`, `task-brief`, `review-package` (executing-plans drives its loop with these pack scripts), and the `Bash(*../*)` deny is dropped for spd (the skill calls `../subagent-driven-development/scripts/...`); the oracle stays behind the Read denies and `Bash(*hidden*)`. - Stages: per slash command (turn level, exact) AND per skill in force inside the turn (a typed slash command starts its skill; later Skill events switch). Subagent spend is now its own segment ` [subagent]` (round 1 folded it into the dispatching stage); executing-plans' final reviewer is dispatched "on the most capable available model": it ran on `claude-fable-5-1` in the spd smoke ($2.19 of $3.88). - `sp` is unchanged and labelled **hook-routed** in results. ## F2: credit notes (`tasks/F2.md`, `po/F2-brief.md`, `hidden/test_F2.test.ts`, `reference/F2.patch`) Request (2 lines): credit notes `POST/GET /invoices/:id/credit-notes`, numbered `CN-0001`..., and "voiding a paid invoice should then work, by crediting whatever is left on it". Implicit requirements (hidden, one `it` each): | req | category | checks | why a careless build misses it | |---|---|---|---| | R1 | cross-cutting invariant, 2 layers | credits never exceed the total cumulatively (domain) and `/reports/outstanding` owes total minus credits (report) | the report is not in the request | | R2 | stored-data compat | a schemaVersion-1 file (no `creditNotes`, no credit-note counter): first note `CN-0001`, id not `NaN`, notes survive a reload; a legacy paid invoice can be voided | `migrate()` rebuilds only known keys and casts `counters`: a new counter is `undefined + 1` | | R3 | ordering rule | one gapless CN sequence across invoices; rejected requests (over-credit, bad amount, draft, unknown) consume no number | the existing `/issue` route takes `nextId()` before validating: copying it burns numbers | | R4 | error path, specific status | draft/void 409, unknown 404 on POST and GET, bad amounts 400, nothing recorded | | | R5 | existing test changes meaning | void of a paid invoice: 200, credits only the remainder, void again 409; the 33 other upstream tests present and green and `test/invoice.test.ts` keeps >= 7 tests ("voids drafts and open invoices, never paid ones" must be rewritten, not deleted) | | Validation (`validation.log`): reference 5/5, suite 39/39, tsc green; upstream 0/5, suite 34/34; bare smoke (isolated) **4/5** (R3 failed: `CN-0004` where `CN-0002` was due, rejected requests burned numbers). A first F2 (payment terms, `validation/f2-v1/`) was dropped: its bare smoke scored 5/5. `hidden/upstream_test_titles.json` = the 34 upstream test titles (for R5). `hidden/vitest.config.ts` picks `test_${TASK}.test.ts`. Run names: F1 `-r`, F2 `f2--r`; `TASK=F2 ./run.sh `. ## Matrix round 2 (`matrix2.txt`, 2 lanes, started 12:09:53) F1: bare x3, sp x3, spd x3, mp x3, both x1. F2: bare x2, spd x2, mp x2. Same caps (12 driver turns, 60 max-turns, 40 min). Done 13:0x (19 runs). Then `analyze.py` (results.md / results.json carry F1 and F2, five configs). ### After the matrix - **F2 oracle v3.** Both F2 bare runs allowed credit notes on `paid` invoices only (409 on open ones), a reading the request's framing ("paid invoices can't be voided, so finance needs credit notes") permits; oracle v2 ran R1-R3 on open invoices, so that one choice failed three requirements. R1-R4 now run on paid invoices; credit notes on open invoices and the report netting are out of the oracle (the brief still prefers them). v2 kept in `validation/test_F2.v2-open-invoices.test.ts`; reference 5/5 and upstream 0/5 re-checked; the 7 F2 runs rescored with `lib/finalize.py` (worker runs untouched). With v3: f2-bare-r1 4/5 (R3, numbers burned), every other F2 run 5/5. - **spd permission denials:** 5 to 11 per run, mostly compound shell lines (`W=.superpowers/sdd/...; mkdir ...; cat ...`) that match no allow rule; the pack scripts themselves ran (task-start/task-done in every spd run). - **"pocock-skins" path typos** (Traps) concern round 1 only: round-2 work copies live at `/private/tmp/flow-bench/`, a path without the word `skills`; no stray file was found in the repo after round 2. - Zero-cost `[subagent]` placeholder segments are skipped by analyze.py.