AIDiveTL;DR और verdict
- एक ही feature, एक काल्पनिक repo पर, bare Claude Code session, Superpowers और Matt Pocock के Main Flow से बनाया गया, headless Sonnet 5.5, 19 गिने हुए runs। Tokens, समय और list-price लागत हर run पर नहीं, हर stage पर पढ़ी गई हैं।
- Stage map: Superpowers रखिए, hook-routed। जब request धुंधली हो तो Pocock का शुरुआती हिस्सा, grill-with-docs, to-spec, to-tickets, उधार लीजिए। executing-plans के reviewer की model वाली पंक्ति बदलिए, क्योंकि दोनों packs के बीच का लगभग पूरा फ़र्क वही एक stage है।
- Feature 1 (partial payments), प्रति configuration 3 runs के औसत: bare $0.343 और 84.5 s, अपने hook के भरोसे Superpowers $0.502 और 118.1 s, पूरे loop से चलाया Superpowers $3.738 और 607.367 s, Main Flow $2.043 और 439.033 s। दोनों packs stack करके एक run: $2.326। इन सभी runs ने 5 में से 5 hidden tests पास किए।
- चलाए गए Superpowers loop के भीतर, कुल $3.738 में से executing-plans $3.307 रहा, और उसमें $2.192 अकेला एक reviewer subagent था जिसे skill सबसे सक्षम model पर भेजती है। Brainstorming $0.166 रही, writing-plans $0.266।
- Feature 2 (credit notes), कठिन: bare $0.413 और दो runs में एक चूक, Superpowers driven $3.213, Main Flow $2.067, बाद के दोनों 5 में से 5 पर। 19 runs में छूटी इकलौती requirement बिना gap वाला credit note number थी, जो एक bare run ने भेजी।
- सीमाएँ: एक काल्पनिक repo, एक worker model, दो features, एक scripted product owner, हर cell में छोटा n। यहाँ packs को quality ने नहीं, लागत ने अलग किया।
Sources क्या कहते हैं
Run के समय के versions: Superpowers v6.4.2 पर है, जो 2026-09-25 को release हुआ s20, और mattpocock/skills v1.3.1 पर है, जो 2026-10-04 को release हुआ s26। Superpowers में 15 skills हैं; Pocock का repo 27 shipped skills गिनाता है s1 s3। 4 साझा, 10 सिर्फ़ Superpowers की, 12 सिर्फ़ Pocock की वाली पुरानी गिनती Superpowers 6.0.3 और Pocock 1.0.1 पर की गई थी, 14 बनाम 17 skills के साथ, इसलिए वह पुरानी पड़ चुकी है s6।
दोनों packs उलटे दांव हैं। Superpowers एक SessionStart hook लगाता है जो startup, clear और compact पर using-superpowers skill को EXTREMELY_IMPORTANT tag में लपेटकर inject करता है s23। उसका text: "If you think there is even a 1% chance a skill might apply to what you are doing, you ABSOLUTELY MUST invoke the skill." s22 वह यह भी कहता है कि किसी भी response या action से पहले, clarifying questions समेत, relevant skills invoke करो s22। Pocock के pack में कोई session hook नहीं: "There is no session hook; nothing gets injected at startup." s6 Repo tree में भी कोई hooks file नहीं है s3।
Skill files छोटी हैं। grill-with-docs 35 शब्दों की है, और वह बस Skill tool को दो बार call करती है, grilling और domain-modeling के लिए s27। implement 103 शब्दों की है, उसमें disable-model-invocation है, और वह code-review call करके खत्म होती है s28। Matt ने जो 63% का आँकड़ा बताया वह skill descriptions के token cost में कमी है, किसी run की नहीं s34 s35।
Main Flow, जैसा router skill उसे बिछाती है, एक pull request step से गुज़रता है और एक retro पर खत्म होता है जो loop को बंद करता है s29। Overview page Main Flow में 6 skills गिनाता है और /pr को reference skills में रखता है s1। अपने v1.3 notes में Matt लिखते हैं: "The agent loop is worse than a deterministic loop. But it is a good way to start with AFK work" s2। वे खुरदरे किनारे भी गिनाते हैं: "worktrees do not remove collisions, they move them to merge time" s2। v1.3 CONTEXT.md का नाम बदलकर GLOSSARY.md और CONTEXT-MAP.md का नाम GLOSSARY-MAP.md कर देता है s2 s26।
Install के रास्ते। README एक plugin command और एक installer command देता है, claude plugin install mattpocock-skills@claude-plugins-official और npx skills@latest add mattpocock/skills -a <agent>, और कहता है कि हर agent के लिए एक चुनिए, क्योंकि दोनों install करने पर हर skill दो बार आती है s3। Marketplace वाली copy पिछड़ी है: आधिकारिक Claude plugin marketplace mattpocock-skills को 1.2.3 पर pin रखता है, और bump का अनुरोध अब भी खुला है s32। हमारी जाँच में upstream main उस pin से 71 commits आगे था, और pinned copy में implement-spec, pr और retro नहीं हैं s31 s32। इसी पुरानेपन की एक user report Pocock के repo पर खुली है s31, और 2026-10-08 को Matt ने X पर cached version bump करने को कहा: "All my users are stuck on v1.2." s33 उसी जाँच में marketplace ने Superpowers को 6.4.1 पर pin रखा, v6.4.2 से एक patch पीछे s20।
Superpowers की अपनी तरफ़, 6.4.1 ने executing-plans को native, inline mode के साथ दोबारा बनाया, "the cheapest way to run a plan" s21, और 6.4.2 ने writing-plans को हल्का किया s20। लोग जो खुले issues गिनाते हैं: skill install करने के बाद सुस्ती s24, और एक Codex subagent-driven run जिसने पूरा 5 घंटे का budget खा लिया, जो Superpowers 5.0.7 और Codex पर रिपोर्ट हुआ, किसी 6.4 release पर नहीं s25। /implement की token लागत पर Pocock का issue एक user चर्चा है, जिसे Matt ने skill की विफलता नहीं, चर्चा मानकर बंद किया s30।
लोग बिना नापे क्या कहते हैं। एक switcher ने पाया कि Pocock का Q&A Superpowers की brainstorming से कम tokens में "with no worse results" चला s10। एक और लिखता है कि अनिवार्य orchestration सुरक्षा जाल नहीं रहती, कर बन जाती है s9। एक Hacker News thread Superpowers को बिल्कुल इस्तेमाल न करने के मुकाबले benchmarks माँगता है, और साथ में यह बचाव भी है कि वह evals से बना है s13। Grilling पर एक thread छोटे implementation से पहले "5 hours wayfinding and spec reading" बताता है, और एक commenter को नए models के साथ grilling "feels like a hinderance" लगती है s7। दो packs को routers की तरह stack करने पर एक post दावा करती है कि उनके slash commands और session hooks टकराते हैं s11। इनमें से कोई भी per-stage token आँकड़े नहीं देता, और यही वह खाली जगह है जिसे यह bench भरता है।
Bench
Bench ledgerly नाम का एक छोटा काल्पनिक TypeScript invoicing API है: एक domain layer, schema migrations वाला JSON file store, एक HTTP layer, 34 पास होते tests, 4 commits। हर feature request तीन पंक्तियों की है, ठीक वैसे लिखी जैसे कोई developer लिखता है। एक senior engineer पाँच चीज़ें उम्मीद करता है जो request कभी नहीं बताती, और वही पाँच hidden tests हैं जो worker कभी नहीं देखता। Worker headless Sonnet 5.5 है। एक दूसरा headless model product owner बनता है, उसके पास कोई tools नहीं, वह एक निजी brief पढ़ता है और worker के सवालों के छोटे जवाब देता है। हर turn tokens, list-price लागत, wall समय, tool calls और skill invocations के साथ लॉग होता है। लागत CLI का total_cost_usd है, यानी list-price के बराबर, बिल नहीं। इसमें कुछ भी किसी की असली code पर नहीं चलाया गया, इसलिए इसे अपने repo के लिए template मानिए।
README.md
# flow-bench: Superpowers loop vs Matt Pocock's Main Flow, one feature, one repo
Built 2026-10-09 from `SPEC.md` (the fixed decisions are implemented as written; deviations and their reasons are
listed under "Decisions taken where SPEC left a choice"). Worker = headless `claude-sonnet-5-5` (`claude -p`, OAuth
from `~/.claude`), Claude Code 2.1.294.
## Commands (from this dir)
```bash
./validate.sh # SPEC validation steps 1-5 -> validation.log (3-5 are smoke runs, RUN_TAG=smoke)
./validate.sh 1 2 # only the static steps (reference patch, untouched upstream)
RUN_TAG=x ./run.sh <cfg> <rep> # one run -> runs/<cfg>-r<rep>-x/, work copy work/<same>/ ; tagged runs are ignored by analyze.py
nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 & # the 10-run matrix, 2 lanes (MATRIX_JOBS=2)
SKIP_DONE=1 nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 & # resume
python3 analyze.py # results.json + results.md
python3 lib/finalize.py runs/<run> # rescore one run (restores node_modules into the work copy if it was removed)
python3 lib/reaccount.py runs/<run> # recompute turns.jsonl accounting from the saved streams (then finalize)
```
## Layout
| path | what |
|---|---|
| `upstream/` | **ledgerly**, the invented repo: Node 20 + TS + vitest + pnpm, 18 source files (domain / store / http layers), 34 tests in 7 files, 4 commits (HEAD 980d542), `test` + `typecheck` scripts, `node_modules` installed (pnpm 11) |
| `tasks/F1.md` | the task, verbatim (3 lines, the way a dev types it) |
| `po/F1-brief.md` | the product owner's full intent: the 5 requirements in plain words, preferences, constraints (never the tests) |
| `hidden/` | `test_F1.test.ts` (the oracle: `test_req1..5`), its own `vitest.config.ts` (alias `@work` -> `$WORK`) and `node_modules` (vitest 2.1.9). Runs from OUTSIDE the work copy |
| `reference/F1.patch` | reference solution: hidden 5/5, suite 39/39, tsc green |
| `configs.json` | `bare`, `sp`, `mp`, `both` |
| `packs/` | `superpowers/` (plugin root of obra/superpowers v6.4.2), `mattpocock-skills/.claude/skills/` (27 skills of mattpocock/skills v1.3.1 installed by `npx skills add`), `VERSIONS.md`, `mattpocock-install.log`; `*-src/` = the clones (gitignored) |
| `env.sh` | copied from an earlier bench of the same kind: scrubs inherited `CLAUDE*` vars, `CLAUDE_CODE_DISABLE_AUTO_MEMORY=1` |
| `prep.sh <cfg> <work>` | fresh work copy + the pack fixtures of the config + `.claude/settings.json`; prints the base commit |
| `run.sh <cfg> <rep>` | prep -> `lib/driver.py` -> `lib/finalize.py` |
| `lib/driver.py` | the turn loop (flows below), caps, `turns.jsonl` |
| `lib/accounting.py` | per-turn and per-stage tokens / USD / wall / tools / Skill events / hook events from one turn's stream |
| `lib/po.py` | the product-owner simulator |
| `lib/finalize.py` | diff vs base, suite, typecheck, hidden oracle, stages -> `score.json`, `stages.json` |
| `analyze.py` | `results.json`, `results.md` |
| `validate.sh`, `validation.log`, `validation/routing-probes/` | pre-matrix checks (+ 6 one-turn probes: plugin wiring vs project wiring) |
| `matrix.txt`, `matrix.sh`, `matrix-one.sh`, `matrix.log` | 10 runs, 2 at a time |
### The feature and the oracle
`tasks/F1.md`: partial payments, `POST /invoices/:id/payments` with an amount in cents, `amountPaid`/`balance` on
invoices, flip to `paid` at zero balance, keep `/pay` working, "we have prod data in the JSON store".
The 5 hidden requirements (one `it` each) and the codebase precedent that makes each one a senior's expectation:
| req | checks | precedent in the repo |
|---|---|---|
| R1 edge cases | payments accumulate; status stays non-`paid` until the exact remainder; `paidAt` set; `/reports/outstanding` sums balances, not totals | the report sums `total` of open invoices |
| R2 validation | 0, negative, 12.5, "100", null, missing -> 400/422; overpayment (of the total, then of the balance) -> 400/409/422; state unchanged | `assertCents`, "discount cannot exceed the subtotal" |
| R3 migration of existing data | a schemaVersion-1 file: legacy `paid` invoice shows amountPaid=total, balance 0; open shows 0/total; payments persist across a reload; `/pay` leaves balance 0 | `schemaVersion` + `src/store/migrations.ts` (v0->v1), README house rule |
| R4 error path | unknown id 404; draft 409; void 409; nothing recorded | `markPaid` throws `ConflictError` on draft/void |
| R5 a test | a `*.test.ts` changed or added since the base commit mentions `payments` | 34 existing tests |
Lenient where the brief would be a guess (status codes for overpayment accept 400/409/422, payment response 2xx,
response body never inspected: state is read back with GET).
## Configs and how each pack is wired (`prep.sh`)
- Every config: `cp -R upstream`, `.claude/` in `.git/info/exclude`, `.claude/settings.json` with a `permissions.deny`
list (Read of `hidden/ reference/ runs/ po/ validation/` of this bench; Bash commands containing `hidden`,
`reference/F1`, `F1-brief`, `/runs/`, `../`, and `find`). Bare has nothing else: no skills, no hooks.
- `sp`: the plugin root is copied to `.claude/plugins/superpowers/` (the hook script reads
`../skills/using-superpowers/SKILL.md` relative to itself), the 15 skills to `.claude/skills/<name>/`, and
`hooks/hooks.json` is reproduced in `.claude/settings.json` verbatim (`SessionStart`, matcher
`startup|clear|compact`, `"shell": "bash"`, `"async": false`) with the command prefixed by
`CLAUDE_PLUGIN_ROOT="$CLAUDE_PROJECT_DIR/.claude/plugins/superpowers"`: Claude Code sets that variable for plugin hooks,
and `hooks/session-start` uses it to pick the `hookSpecificOutput.additionalContext` output shape. Superpowers ships no
other hook (`hooks-cursor.json` is Cursor's).
- `mp`: the 27 skills to `.claude/skills/`. Pocock ships **no hook** (verified: no settings or hooks file in the install;
`git-guardrails-claude-code`, a skill that writes hooks, is outside the promoted set and was trimmed). The pack's
per-repo precondition `/setup-matt-pocock-skills` is materialised once: `docs/agents/issue-tracker.md` (local
markdown under `.scratch/`), `docs/agents/domain.md`, `docs/agents/triage-labels.md` from the skill's own seed
templates, and a `CLAUDE.md` "## Agent skills" block, committed as "chore: configure agent skills (local tracker)"
so the diff only shows the feature. Base commit for mp/both = that commit.
- `both`: sp + mp in the same `.claude/` (no skill-name collision between the two packs), driven with the mp flow.
## Flows (`lib/driver.py`)
Every worker turn: `claude -p <prompt> [--resume <sid>] --model claude-sonnet-5-5 --output-format stream-json --verbose
--include-hook-events --include-partial-messages --max-turns 60 --permission-mode acceptEdits --allowedTools <SPEC list>
--disallowedTools AskUserQuestion,EnterPlanMode,ExitPlanMode,EnterWorktree,ExitWorktree,WebFetch,WebSearch
--setting-sources project --strict-mcp-config`, cwd = work copy, stdin /dev/null, scrubbed env.
Allow-list: `Read,Edit,Write,MultiEdit,Glob,Grep,Skill,TodoWrite,Task,Agent`, Bash `node npm pnpm npx vitest npx tsc tsc
git status|diff|log|add|commit|checkout -b ls cat mkdir head tail grep wc gh`.
- `bare`: turn 1 = task + "When you are done, stop and summarise what you changed."
- `sp`: turn 1 = the task alone (the SessionStart hook is supposed to route it).
- `mp` / `both`: `/grill-with-docs <task>` -> `/to-spec` -> `/to-tickets` -> `/implement the tickets in
.scratch/<slug>/issues/, all of them, blockers first` -> `/code-review since <base sha>` -> `/pr`.
- After every turn the PO reads the last assistant message and classifies it: `question` (its reply is the next
prompt), `waiting` (driver sends "go on" / "go on, proceed with the plan"; mp: stays in the stage), `done` (bare/sp:
stop; mp: next slash command; after `/pr`: stop). If the PO call fails twice the SPEC heuristic (ends on "?" or an
options list) decides.
- Caps: 12 driver turns, 60 `--max-turns` per claude turn (a turn that hits it gets "go on"), 40 min wall per run
(the claude process is killed at the deadline). Caps hit are recorded in `driver.json` / `score.json`, never discarded.
### PO simulator (`lib/po.py`)
Headless `claude-sonnet-5-5`, `--tools ""`, `--setting-sources ""`, `--settings '{"disableAllHooks":true}'`,
`--strict-mcp-config`, `CLAUDE_CODE_DISABLE_CLAUDE_MDS=1`, `CLAUDE_CODE_DISABLE_AUTO_MEMORY=1`, cwd = an empty temp dir,
`--json-schema {kind, reply}`. Prompt = role + `po/F1-brief.md` + flow note + stage + the last message. Rules: terse,
answer every open question at once, approve what matches the brief, correct only contradictions, "your call" where
the brief is silent, **answer only what was asked** (first smoke: the PO volunteered the migration requirement nobody
had asked about; the rule was added before any counted run), how to finish a branch: "keep it on the current branch,
no PR". mp rule: a stage whose output is delivered and that only asks "shall I proceed / start?" is `done` (first mp
smoke: the PO's "yes, go" after the grilling recap made the worker start implementing inside `/grill-with-docs`; that
smoke was killed and rerun). PO cost is recorded apart (`usd_po`), never in the worker's USD.
## Accounting (the per-turn / per-stage numbers)
Checked on 2.1.294 with probes before writing `lib/accounting.py`:
- `result.total_cost_usd` and `result.modelUsage` are **cumulative over a resumed session** and **include subagents**:
per-turn USD and tokens = delta against the previous turn's result line.
- `result.usage` = main thread of the invocation only (a Task subagent's tokens are missing from it): not used.
- `assistant` events carry the usage snapshot of the message START (`output_tokens` partial, e.g. 3 for a 783-token
message): useless for output. `--include-partial-messages` adds `stream_event` `message_start` / `message_delta`
with the exact per-API-call usage, main thread only.
- Sonnet 5.5 list prices derived from two probe turns and matched to the cent: $2/M input, $10/M output, $0.20/M cache
read, $4/M cache write (1 h TTL, what Claude Code uses here); 5-min writes priced $2.5/M (unseen). In every turn without
subagents, the per-call priced sum equals the `total_cost_usd` delta exactly (`pricing_check` in `turns.jsonl`).
- Stage = slash command of the turn (mp/both, exact) or the skill in force (sp: switches at each main-thread `Skill`
tool_use; turn 1 starts at `(none)`). Inside a turn, main-thread calls are attributed exactly; the subagent remainder
(turn delta minus main calls) goes to the stage that spawned the Task, weighted by the subagents' snapshot tokens.
Wall per stage = receive-time gaps of the stdout lines (`turn-NN.times`) attributed to the stage in force.
- `analyze.py` also groups sp skills into phases (brainstorming / writing-plans / execution incl. TDD /
verification / review / finishing).
- Tokens reported = input + output + cache read + cache write. Cache reads dominate (re-reading the context each call).
### Per run (`runs/<cfg>-r<rep>/`)
`meta.json`, `prep.txt`, `settings.json` (copy of the work copy's), per turn `turn-NN.stream.jsonl` (raw stdout, every
line), `turn-NN.times`, `turn-NN.stderr`, `turn-NN.last.txt`, `po-NN.json`; `turns.jsonl` (one accounting record per
turn: prompt, source, wall, subtype, USD, tokens per model, segments, tools, Skill events, hook events, files, denials,
PO answer), `driver.json` (end reason, caps), `post.diff`, `numstat.txt`, `suite.txt`, `typecheck.txt`, `hidden.txt`,
`stages.json`, `score.json`. Work copies stay in `work/` without `node_modules` (49 MB each; finalize removes it,
`KEEP_NODE_MODULES=1` keeps it).
## Decisions taken where SPEC left a choice
- **Implement stage = `/implement`, not `/implement-spec`.** v1.3.1 ships both. `/implement-spec` requires implementer
subagents "each in its own worktree on its own branch" plus merger subagents: `git worktree`, `git merge`, `git branch`
are outside the SPEC Bash allow-list, so it could not run as designed. `/implement` is sent once with the ticket
directory ("all of them, blockers first"), since one session cannot `/clear` between tickets as `ask-matt` suggests.
- `/code-review` gets `since <base sha>` (the skill reviews "the diff since a fixed point"); Pocock's `code-review`
replaces Claude Code's bundled `code-review` skill of the same name in the session (init lists it once, the Skill
event loads `.claude/skills/code-review`).
- `/retro` not run (optional in SPEC).
- `both` uses the mp flow: the stacking question is whether the Superpowers SessionStart routing hijacks the slash
commands.
- Reps / order: `matrix.txt` (rep 1 of every config first), 2 lanes.
## Traps met
- **Superpowers routing is stochastic headless.** On turn 1 the hook always fires, but the model picks
`test-driven-development` instead of `brainstorming` part of the time. Six one-turn probes on the F1 prompt
(`validation/routing-probes/`): real plugin wiring (`--plugin-dir packs/superpowers`, namespaced
`superpowers:*`) 1/3 brainstorming, 2/3 TDD; project wiring (this bench) 2/3 brainstorming, 1/3 TDD. So the project
wiring is not the cause; the first sp smoke went TDD-only (one turn, hidden 5/5). A second sp smoke (`smoke2`) was run
to see the brainstorming path end to end.
- **The SessionStart hook fires only on turn 1.** Its matcher is `startup|clear|compact`; every later turn is
`--resume` (source `resume`), so the injection is not repeated; it stays in the transcript.
- **User plugins do not leak** with `--setting-sources project`: the init event lists only Claude Code's built-in
plugins (`cc-plugin-agents-md`, `cc-plugin-telemetry`, `cc-plugin-plugin-authoring`), never the user's Superpowers
6.4.1; bare smoke: no hook event, no Skill event. Built-in skills (`verify`, `debug`, `simplify`, `code-review`, ...)
are still listed in every config.
- **`--bare` is never used** (cannot authenticate on this OAuth account; see the previous bench).
- **Read-only shell commands bypass the allow-list.** Claude Code auto-approves read-only commands (`git ls-files`,
`sed -n`, pipes of read-only commands) whatever `--allowedTools` says; `python3` edits are denied (the bare smoke had 4
`permission_denials`, all `python3` heredoc edits; the worker fell back to Edit). `cat > /tmp/x <<EOF` matches
`Bash(cat:*)`: the allow-list is a prefix match, not a sandbox.
- **pnpm 11** refuses to install without approving esbuild's build script: `pnpm-workspace.yaml` with
`allowBuilds: {esbuild: true}` (pnpm 11 no longer reads the `pnpm` field of package.json). No `packages:` key, so it
is not a monorepo signal for Pocock's setup.
- **`npx skills add` ignores `--skill a,b` and `--skill "a b"`** ("No matching skills found"); repeated `--skill x`
flags installed every skill of the repo (37). Trimmed to the plugin's 27 (`packs/VERSIONS.md`). Pinning works with
the `https://github.com/mattpocock/skills/tree/v1.3.1` source (main is past the release).
- **mattpocock skills are mostly user-invoked** (`disable-model-invocation: true` on grill-with-docs, to-spec,
to-tickets, implement, implement-spec, ...): reachable as slash commands only; `/grill-with-docs` itself is one line
("Call the Skill tool twice, for grilling and domain-modeling"), so the proof that it resolved is those two Skill
events on turn 1.
- **The model mistypes the bench path.** Sonnet 5.5 regularly writes absolute paths with
`superpowers-vs-matt-pocock-skins-...` (or `skips`) instead of `skills`: 0 to 10 tool calls per run (0 to 7% of calls;
e.g. mp-r3-po1 10/145, mp-r2 0/124, sp-r2 3/48, bare-r1 0/28), in every config. Edits on the wrong path fail and are
retried (a small, roughly proportional token tax); **Writes succeed outside the work copy** because `Write` in
`--allowedTools` is not path-limited: the runs left stray files under a mistyped copy of the bench path (a spec
copy, an `x`) and three 1-byte placeholder files, deleted after the matrix
(nothing else in the repo was touched; `git status` checked). A future bench should put work copies on a short
path without the word `skills` and deny `Write(//**)` outside the work copy.
- **Shared plan meter.** Workers and PO bill the user's plan quota; USD = the CLI's list-price equivalent.
- **rtk** rewrites interactive `diff`/`grep` in the launching session only; scripts call binaries directly.
## Validation (2026-10-09, `validation.log`)
1. `reference/F1.patch` -> hidden 5/5, suite 39/39 green, tsc green: PASS.
2. upstream unchanged -> hidden 0/5, suite 34/34 green: PASS.
3. `bare-r1-smoke` -> 2 driver turns (the PO answered one closing question), hidden 5/5, $0.52, 125 s; isolation: no
hook event, no Skill event, only built-in plugins in init: PASS.
4. `sp-r1-smoke` -> hook fired and injected `using-superpowers` (PASS) but the model went straight to
`test-driven-development` (brainstorming FAIL in that run). Retry `sp-r1-smoke2`: hook fired, `brainstorming` on
turn 1, PO approved the design, then `test-driven-development`: PASS. Routing probes: see Traps.
5. `mp-r1-smoke` -> turn 1 loads `grilling` + `domain-modeling` (what `/grill-with-docs` says to do): PASS. The run then
used all 12 driver turns (grill 4, to-spec 2, to-tickets 2, implement 2, code-review 2) and never reached `/pr`.
Fixes made during validation, before any counted run: the PO volunteering unasked requirements; the PO telling the
worker "yes, go" at the end of the grilling (the worker started implementing inside `/grill-with-docs`; that first mp
smoke was killed at turn 4 and rerun).
## Matrix launch
`nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 &` at 2026-10-09 10:41:33, 2 lanes, order in `matrix.txt`.
Done 11:06:29 (10 runs, 25 min).
### Reruns of the mp-flow runs (harness fixes, `matrix.log`)
- `both-r1` (first): the PO answered the grilling recap's branch question with an approval and the worker coded inside
`/grill-with-docs` (turn 4, 12 src/test files). Kept as `runs/both-r1-pofault` (tag `pofault`, ignored by analyze).
PO rule added (mp flow: before `/implement`, end with "Don't start implementing yet." when the worker offers to build).
- `both-r1` (second): the PO did not apply that rule and the worker coded inside `/grill-with-docs` again (turn 3, after
"say so and I'll start with failing tests"). The rule was made deterministic in `lib/driver.py`: in `/grill-with-docs`,
`/to-spec`, `/to-tickets`, the driver appends "Don't start implementing yet." to every PO answer. Since that changes
the conditions of all mp-flow runs, mp-r1..r3 and both-r1 were moved to `runs/*-po1` (tag `po1`; mp-r1..r3-po1 never
coded early, hidden 5/5 each, $2.22 to $2.57) and rerun with `matrix-mp2.txt` (11:18 to 11:40). The counted mp/both
runs are the reruns.
- sp stage labels: brainstorming's SKILL.md ends at the approved design, but in sp-r1 and sp-r3 the model then coded with
no new Skill call, so "skill in force" would bill the whole build to brainstorming. `lib/accounting.py` now switches the
stage to `implementation (no skill)` at the first main-thread Edit/Write under `src/` or `test/` while only a design
skill (`(none)`, `using-superpowers`, `brainstorming`, `writing-plans`) is in force; the sp runs were re-accounted from
their saved streams with `lib/reaccount.py` (tokens, USD and wall totals unchanged, only the split).
## Results
`results.md` / `results.json` (`python3 analyze.py`), reading in `FINDINGS.md`.
---
# Round 2 (2026-10-09 afternoon): `spd`, feature F2, isolation fix, full rerun
## Isolation leak found and fixed (all round-1 runs superseded)
The first `spd` smoke said "the most capable model your CLAUDE.md permits for subagents": the workers were reading
CLAUDE.md files. `--setting-sources project` does keep `~/.claude/CLAUDE.md` out (probe in an empty dir outside the
repo: "NONE"), but Claude Code also loads **every CLAUDE.md of the parent directories**, and the work copies lived under
the host repository, so every round-1 session carried that repository's CLAUDE.md and what it imports
(a probe in a round-1 work copy showed the host rules present; the round-1 mp code
reviews even cited them). Fix: `run.sh` builds the work copy in
`/private/tmp/flow-bench/<run>` (outside any CLAUDE.md tree) and archives it to `work/<run>` (without node_modules)
after finalize; probe in an mp copy there: only the pack's own "## Agent skills" CLAUDE.md block appears.
Every round-1 run was renamed `runs/*-leak` (tag `leak`, ignored by analyze) and the whole matrix rerun
(`matrix2.txt`). The short path also removes the "pocock-skins" typo surface (see Traps).
## `spd`: Superpowers, driven
Same project install as `sp` (skills in `.claude/skills/`, SessionStart hook). Installed in the project, the skills
resolve **without the plugin namespace**: init `slash_commands` lists `brainstorming`, `writing-plans`,
`executing-plans` (with `--plugin-dir` they are `superpowers:brainstorming`, ...; see `validation/routing-probes/`),
so the driver types `/brainstorming <task>` -> `/writing-plans` -> `/executing-plans`. writing-plans ends by asking
"Subagent-driven or Native?": the PO answers Native (added to both briefs: "If asked to choose how to execute a written
plan: Native (inline, in this session) ... working on main is fine: you consent"). Before `/executing-plans` the driver
appends "Don't start implementing yet." to PO answers (same rule as mp). A stage whose output is delivered and that
only asks for approval/go is `done` -> next command; after `/executing-plans` the run ends when the PO says done
(executing-plans itself runs the final whole-branch review and finishing-a-development-branch).
- Allow-list: the SPEC list + `Bash(*scripts/task-start*)`, `task-done`, `sdd-workspace`, `task-brief`,
`review-package` (executing-plans drives its loop with these pack scripts), and the `Bash(*../*)` deny is dropped for
spd (the skill calls `../subagent-driven-development/scripts/...`); the oracle stays behind the Read denies and
`Bash(*hidden*)`.
- Stages: per slash command (turn level, exact) AND per skill in force inside the turn (a typed slash command starts
its skill; later Skill events switch). Subagent spend is now its own segment `<dispatching stage> [subagent]`
(round 1 folded it into the dispatching stage); executing-plans' final reviewer is dispatched "on the most capable
available model": it ran on `claude-fable-5-1` in the spd smoke ($2.19 of $3.88).
- `sp` is unchanged and labelled **hook-routed** in results.
## F2: credit notes (`tasks/F2.md`, `po/F2-brief.md`, `hidden/test_F2.test.ts`, `reference/F2.patch`)
Request (2 lines): credit notes `POST/GET /invoices/:id/credit-notes`, numbered `CN-0001`..., and "voiding a paid
invoice should then work, by crediting whatever is left on it". Implicit requirements (hidden, one `it` each):
| req | category | checks | why a careless build misses it |
|---|---|---|---|
| R1 | cross-cutting invariant, 2 layers | credits never exceed the total cumulatively (domain) and `/reports/outstanding` owes total minus credits (report) | the report is not in the request |
| R2 | stored-data compat | a schemaVersion-1 file (no `creditNotes`, no credit-note counter): first note `CN-0001`, id not `NaN`, notes survive a reload; a legacy paid invoice can be voided | `migrate()` rebuilds only known keys and casts `counters`: a new counter is `undefined + 1` |
| R3 | ordering rule | one gapless CN sequence across invoices; rejected requests (over-credit, bad amount, draft, unknown) consume no number | the existing `/issue` route takes `nextId()` before validating: copying it burns numbers |
| R4 | error path, specific status | draft/void 409, unknown 404 on POST and GET, bad amounts 400, nothing recorded | |
| R5 | existing test changes meaning | void of a paid invoice: 200, credits only the remainder, void again 409; the 33 other upstream tests present and green and `test/invoice.test.ts` keeps >= 7 tests ("voids drafts and open invoices, never paid ones" must be rewritten, not deleted) | |
Validation (`validation.log`): reference 5/5, suite 39/39, tsc green; upstream 0/5, suite 34/34; bare smoke (isolated)
**4/5** (R3 failed: `CN-0004` where `CN-0002` was due, rejected requests burned numbers). A first F2 (payment terms,
`validation/f2-v1/`) was dropped: its bare smoke scored 5/5.
`hidden/upstream_test_titles.json` = the 34 upstream test titles (for R5). `hidden/vitest.config.ts` picks
`test_${TASK}.test.ts`. Run names: F1 `<cfg>-r<rep>`, F2 `f2-<cfg>-r<rep>`; `TASK=F2 ./run.sh <cfg> <rep>`.
## Matrix round 2 (`matrix2.txt`, 2 lanes, started 12:09:53)
F1: bare x3, sp x3, spd x3, mp x3, both x1. F2: bare x2, spd x2, mp x2. Same caps (12 driver turns, 60 max-turns,
40 min).
Done 13:0x (19 runs). Then `analyze.py` (results.md / results.json carry F1 and F2, five configs).
### After the matrix
- **F2 oracle v3.** Both F2 bare runs allowed credit notes on `paid` invoices only (409 on open ones), a reading the
request's framing ("paid invoices can't be voided, so finance needs credit notes") permits; oracle v2 ran R1-R3 on
open invoices, so that one choice failed three requirements. R1-R4 now run on paid invoices; credit notes on open
invoices and the report netting are out of the oracle (the brief still prefers them). v2 kept in
`validation/test_F2.v2-open-invoices.test.ts`; reference 5/5 and upstream 0/5 re-checked; the 7 F2 runs rescored
with `lib/finalize.py` (worker runs untouched). With v3: f2-bare-r1 4/5 (R3, numbers burned), every other F2 run 5/5.
- **spd permission denials:** 5 to 11 per run, mostly compound shell lines (`W=.superpowers/sdd/...; mkdir ...; cat ...`)
that match no allow rule; the pack scripts themselves ran (task-start/task-done in every spd run).
- **"pocock-skins" path typos** (Traps) concern round 1 only: round-2 work copies live at `/private/tmp/flow-bench/<run>`,
a path without the word `skills`; no stray file was found in the repo after round 2.
- Zero-cost `[subagent]` placeholder segments are skipped by analyze.py.
SPEC.md
# flow-bench - Superpowers loop vs Matt Pocock's Main Flow, same feature, one repo
Question the video must answer: on one small invented repo, for one bounded brownfield feature,
what does each pack cost PER STAGE (tokens, wall-clock, USD) and what does it deliver (hidden
acceptance tests passed, repo suite green, diff size, bugs shipped), against a bare Claude Code run?
Secondary question: does stacking both packs in one project break (hook / slash conflicts)?
## Fixed decisions (2026-10-09)
- Worker model: `claude-sonnet-5-5`, headless (`claude -p`), OAuth from `~/.claude` (see traps below).
- The repo `upstream/`: INVENTED, built here, a small TypeScript project (about 12-20 source files,
vitest, pnpm or npm, a green suite of ~25-40 tests): a plausible brownfield service, e.g. an
invoicing API or a booking CLI with a store layer, a domain layer and an HTTP or CLI layer. It is a
git repo with a clean history (3-6 commits), a short README and a `package.json` with
`test` and `typecheck` scripts. No framework exotica; plain Node 20 + TS + vitest.
- ONE feature task `tasks/F1.md`: a terse developer request (headline ask + one or two constraints,
2-4 lines, the way a dev types it), whose honest completion needs 5 requirements that a senior
would expect (edge cases, a validation, a migration of existing data, an error path, a test).
`hidden/test_F1.test.ts` is the oracle (5 requirement tests, never visible to the worker, run
from OUTSIDE the work copy against it). `reference/F1.patch` passes all 5 and the repo suite.
- Configs (`configs.json`):
- `bare`: no skills, no hooks.
- `sp`: obra/superpowers installed in the PROJECT (not the user's plugin): clone the repo at the
latest release tag (record it), expose its skills under `<work>/.claude/skills/` and its
SessionStart hook + any hooks it ships under `<work>/.claude/settings.json` exactly the way the
plugin wires them (read its `hooks/` and `plugin.json`; reproduce, don't improvise). Verify in a
smoke run that the `using-superpowers` text actually appears in the session (stream-json shows
the hook output) and that `brainstorming` is invoked.
- `mp`: mattpocock/skills installed in the project via the official installer
(`npx skills add mattpocock/skills` or the README's exact command, latest release, record the
version); verify the Main Flow slash commands resolve (`/grill-with-docs`, `/to-spec`,
`/to-tickets`, `/implement` or `/implement-spec`, `/code-review`, `/pr`, `/retro`; use the names
the installed version really has).
- `both`: `sp` + `mp` together (one or two runs only: the stacking question).
- Flows (the driver `run.sh <cfg> <rep>` / `lib/driver.py`):
- `bare`: one session, the task prompt, "when done, stop and summarise".
- `sp`: one session; turn 1 = the task prompt (the hook is supposed to route into brainstorming).
Then the Superpowers loop as it unfolds (brainstorming → writing-plans → executing-plans →
review); the driver only answers questions and says "go on / proceed with the plan" when the
assistant ends a turn waiting. Record which skills were invoked and when (Skill tool events).
- `mp`: one session, one stage per driver turn, in this order: `/grill-with-docs <task>` →
`/to-spec` → `/to-tickets` → the implement stage (`/implement` or `/implement-spec`, whichever
v1.3 ships for the loop over tickets) → `/code-review` → `/pr`. (`/retro` optional, one run.)
The driver answers questions between turns.
- Every turn is `claude -p … --resume <session_id> --output-format stream-json --verbose
--include-hook-events`; the first turn creates the session. Per TURN: tokens in/out/cache
read/cache write, USD (`result.total_cost_usd` deltas), wall-clock, tool calls, Skill
invocations, files touched. Per STAGE: the sum of its turns. A "stage" for `sp` is the skill in
force (from the Skill events), for `mp` the slash command of the turn.
- **Product-owner simulator** (`lib/po.py`): both packs interrogate the user. A headless
`claude-sonnet-5-5` with NO tools, `--setting-sources ""`, `CLAUDE_CODE_DISABLE_CLAUDE_MDS=1`,
reads `po/F1-brief.md` (the full intent, the 5 requirements in plain words, preferences and
constraints; never the tests) and answers the worker's questions tersely, as a busy dev would.
When the worker ends a turn without a question, the PO replies "go on" (or the next slash
command for `mp`). Hard caps: 12 driver turns per run, 60 `--max-turns` per claude turn, 40 min
wall per run. A run that hits a cap is recorded as such, not discarded.
- Reps: 3 per config for `bare`, `sp`, `mp`; 1 for `both`. 10 runs. Parallel 2.
- Scoring (`lib/finalize.py`): hidden 0-5, repo suite pass/fail, typecheck pass/fail, `git diff
--stat` lines added/removed, files touched, tests added, plus the per-stage table. `analyze.py`
→ `results.json` + `results.md` (per config: mean/median per metric; per stage: tokens, USD, wall).
- Validation BEFORE the matrix (`validate.sh`, logged): (1) `reference/F1.patch` → hidden 5/5 and
suite green; (2) upstream unchanged → hidden 0/5 (or the honest number, recorded) and suite green;
(3) one `bare` smoke run end to end with the PO loop exercised; (4) one `sp` smoke proving the
hook fired and brainstorming ran; (5) one `mp` smoke proving `/grill-with-docs` resolved. A
smoke run is tagged (`RUN_TAG=smoke`) and ignored by analyze.py.
## Traps already known (from a previous bench of the same kind)
- `claude -p --bare` cannot authenticate on an OAuth (Max plan) account: never use `--bare`. Use
`--setting-sources project` for workers so the user's own plugins (Superpowers 6.4.1 is installed
at user scope on this machine) never leak into a `bare` or `mp` run. VERIFY this isolation in the
smoke: a `bare` run must show no Superpowers hook output and no Skill events.
- Scrub inherited `CLAUDE*` env vars (copy `env.sh` from the previous bench), set
`CLAUDE_CODE_DISABLE_AUTO_MEMORY=1`.
- Fresh work copy per run at a unique path; `.claude/` excluded via `.git/info/exclude` so diffs
stay about the feature.
- A worker may try to read the hidden tests: the hidden dir lives outside the work copy and the
allow-list has no `find`/`ls` outside cwd.
- Allowed tools: Read,Edit,Write,MultiEdit,Glob,Grep,Skill,TodoWrite,Task,AskUserQuestion is NOT
available headless (questions come as plain text; the driver detects a turn that ends on a
question mark or an explicit "?"/options list and routes it to the PO); Bash limited to
node/npm/pnpm/npx vitest/tsc/git status|diff|log|add|commit|checkout -b/ls/cat/mkdir/head/tail/
grep/wc. Pocock's `/pr` wants `gh`: allow `Bash(gh:*)` but with no remote the PR step records
its output text as the artefact (no network push).
## Deliverables
`README.md` (how to run, layout, every trap met), `SPEC.md` (this), `upstream/`, `tasks/F1.md`,
`po/F1-brief.md`, `hidden/`, `reference/`, `configs.json`, `env.sh`, `prep.sh`, `run.sh`,
`validate.sh`, `matrix.sh`, `lib/{driver.py,po.py,finalize.py}`, `analyze.py`, `validation.log`,
`runs/<cfg>-r<rep>/` (stream.jsonl per turn, stages.json, score.json), `results.json`,
`results.md`, and `FINDINGS.md`: the honest reading in 300 words, every number with its run name.
FINDINGS.md
# flow-bench findings (2026-10-09, round 2: isolated work copies, Sonnet 5.5 headless, 19 counted runs)
**F1 (partial payments): quality did not separate.** All 13 F1 runs hidden 5/5, suite and tsc green.
Cost per run: bare $0.27 (bare-r2) to $0.47 (bare-r1); sp, hook-routed, $0.44 (sp-r1) to $0.54 (sp-r2); mp $1.80
(mp-r2) to $2.22 (mp-r1); spd, Superpowers driven, $3.39 (spd-r2) to $4.41 (spd-r3).
Wall: mp 365 to 480 s (mp-r2, mp-r3), spd 520 to 701 s (spd-r2, spd-r3).
**Where the money goes.** mp-r2 (complete flow): grill $0.16, spec $0.11, tickets $0.09, implement $0.96, code-review
$0.45, pr $0.04 (F1 means: implement $1.13, code-review $0.44). spd: brainstorming $0.17, writing-plans $0.27,
executing-plans $3.31 (F1 means). Inside executing-plans the build itself (TDD in session) costs $0.97 on average; the
rest is the one whole-branch reviewer the skill dispatches "on the most capable model", which ran on claude-fable-5-1
in all five spd runs: $2.01 of spd-r1's $3.41, $2.52 of spd-r3's $4.41, $1.57 of f2-spd-r2's $3.00.
**Hook vs driven.** Left to its hook, Superpowers ran brainstorming in sp-r1, sp-r2 and sp-r3 and then coded; writing-
plans ran 0 times in 3. Driven by slash commands, the full loop ran in every spd run and cost 7.4 times more (F1 means: spd $3.74, sp $0.50). The cost is the final review, not the planning.
**F2 (credit notes) separated, barely.** f2-bare-r1 shipped one missing requirement, R3: credit-note numbers are not
gapless. It took the number before validating, like the existing `/issue` route, so rejected requests burned numbers
(CN-0004 where CN-0002 was due; the F2 bare smoke failed the same way). f2-bare-r2, f2-spd-r1, f2-spd-r2, f2-mp-r1 and
f2-mp-r2 scored 5/5. Costs: bare $0.38/$0.45, mp $2.18/$1.95, spd $3.42/$3.00 (r1/r2). Caveat: the first F2 oracle
also required credit notes on open invoices; both bare runs allowed paid invoices only, which the request's wording
permits, so that check was removed and the F2 runs rescored.
**Stacking broke nothing.** both-r1: hook injected, no Superpowers skill invoked, $2.33, inside the mp range.
**Caps.** mp-r1, mp-r3 and f2-mp-r1 hit the 12-driver-turn cap before /pr. spd never needed more than 8 turns.
**Caveats.** Round-1 runs read this repo's CLAUDE.md (parent-directory loading), so they were discarded and rerun outside the repo. A simulated PO answers each round in one message. The SessionStart hook fires on turn 1 only.
/implement covers every ticket in one turn. 5 to 11 compound shell lines per spd run were denied. PO cost is excluded.
Repo और tasks:
upstream/package.json
{
"name": "ledgerly",
"version": "0.4.0",
"private": true,
"description": "Small invoicing API: customers, invoices, reports. JSON-file store.",
"type": "module",
"engines": {
"node": ">=20"
},
"scripts": {
"test": "vitest run",
"typecheck": "tsc --noEmit"
},
"devDependencies": {
"@types/node": "^20.14.0",
"typescript": "^5.6.0",
"vitest": "^2.1.0"
}
}
upstream/package.json on GitHub
upstream/pnpm-lock.yaml
lockfileVersion: '9.0'
settings:
autoInstallPeers: true
excludeLinksFromLockfile: false
importers:
.:
devDependencies:
'@types/node':
specifier: ^20.14.0
version: 20.19.43
typescript:
specifier: ^5.6.0
version: 5.9.3
vitest:
specifier: ^2.1.0
version: 2.1.9(@types/node@20.19.43)
packages:
'@esbuild/aix-ppc64@0.21.5':
resolution: {integrity: sha512-1SDgH6ZSPTlggy1yI6+Dbkiz8xzpHJEVAlF/AM1tHPLsf5STom9rwtjE4hKAF20FfXXNTFqEYXyJNWh1GiZedQ==}
engines: {node: '>=12'}
cpu: [ppc64]
os: [aix]
'@esbuild/android-arm64@0.21.5':
resolution: {integrity: sha512-c0uX9VAUBQ7dTDCjq+wdyGLowMdtR/GoC2U5IYk/7D1H1JYC0qseD7+11iMP2mRLN9RcCMRcjC4YMclCzGwS/A==}
engines: {node: '>=12'}
cpu: [arm64]
os: [android]
'@esbuild/android-arm@0.21.5':
resolution: {integrity: sha512-vCPvzSjpPHEi1siZdlvAlsPxXl7WbOVUBBAowWug4rJHb68Ox8KualB+1ocNvT5fjv6wpkX6o/iEpbDrf68zcg==}
engines: {node: '>=12'}
cpu: [arm]
os: [android]
'@esbuild/android-x64@0.21.5':
resolution: {integrity: sha512-D7aPRUUNHRBwHxzxRvp856rjUHRFW1SdQATKXH2hqA0kAZb1hKmi02OpYRacl0TxIGz/ZmXWlbZgjwWYaCakTA==}
engines: {node: '>=12'}
cpu: [x64]
os: [android]
'@esbuild/darwin-arm64@0.21.5':
resolution: {integrity: sha512-DwqXqZyuk5AiWWf3UfLiRDJ5EDd49zg6O9wclZ7kUMv2WRFr4HKjXp/5t8JZ11QbQfUS6/cRCKGwYhtNAY88kQ==}
engines: {node: '>=12'}
cpu: [arm64]
os: [darwin]
'@esbuild/darwin-x64@0.21.5':
resolution: {integrity: sha512-se/JjF8NlmKVG4kNIuyWMV/22ZaerB+qaSi5MdrXtd6R08kvs2qCN4C09miupktDitvh8jRFflwGFBQcxZRjbw==}
engines: {node: '>=12'}
cpu: [x64]
os: [darwin]
'@esbuild/freebsd-arm64@0.21.5':
resolution: {integrity: sha512-5JcRxxRDUJLX8JXp/wcBCy3pENnCgBR9bN6JsY4OmhfUtIHe3ZW0mawA7+RDAcMLrMIZaf03NlQiX9DGyB8h4g==}
engines: {node: '>=12'}
cpu: [arm64]
os: [freebsd]
'@esbuild/freebsd-x64@0.21.5':
resolution: {integrity: sha512-J95kNBj1zkbMXtHVH29bBriQygMXqoVQOQYA+ISs0/2l3T9/kj42ow2mpqerRBxDJnmkUDCaQT/dfNXWX/ZZCQ==}
engines: {node: '>=12'}
cpu: [x64]
os: [freebsd]
'@esbuild/linux-arm64@0.21.5':
resolution: {integrity: sha512-ibKvmyYzKsBeX8d8I7MH/TMfWDXBF3db4qM6sy+7re0YXya+K1cem3on9XgdT2EQGMu4hQyZhan7TeQ8XkGp4Q==}
engines: {node: '>=12'}
cpu: [arm64]
os: [linux]
'@esbuild/linux-arm@0.21.5':
resolution: {integrity: sha512-bPb5AHZtbeNGjCKVZ9UGqGwo8EUu4cLq68E95A53KlxAPRmUyYv2D6F0uUI65XisGOL1hBP5mTronbgo+0bFcA==}
engines: {node: '>=12'}
cpu: [arm]
os: [linux]
'@esbuild/linux-ia32@0.21.5':
resolution: {integrity: sha512-YvjXDqLRqPDl2dvRODYmmhz4rPeVKYvppfGYKSNGdyZkA01046pLWyRKKI3ax8fbJoK5QbxblURkwK/MWY18Tg==}
engines: {node: '>=12'}
cpu: [ia32]
os: [linux]
'@esbuild/linux-loong64@0.21.5':
resolution: {integrity: sha512-uHf1BmMG8qEvzdrzAqg2SIG/02+4/DHB6a9Kbya0XDvwDEKCoC8ZRWI5JJvNdUjtciBGFQ5PuBlpEOXQj+JQSg==}
engines: {node: '>=12'}
cpu: [loong64]
os: [linux]
'@esbuild/linux-mips64el@0.21.5':
resolution: {integrity: sha512-IajOmO+KJK23bj52dFSNCMsz1QP1DqM6cwLUv3W1QwyxkyIWecfafnI555fvSGqEKwjMXVLokcV5ygHW5b3Jbg==}
engines: {node: '>=12'}
cpu: [mips64el]
os: [linux]
'@esbuild/linux-ppc64@0.21.5':
resolution: {integrity: sha512-1hHV/Z4OEfMwpLO8rp7CvlhBDnjsC3CttJXIhBi+5Aj5r+MBvy4egg7wCbe//hSsT+RvDAG7s81tAvpL2XAE4w==}
engines: {node: '>=12'}
cpu: [ppc64]
os: [linux]
'@esbuild/linux-riscv64@0.21.5':
resolution: {integrity: sha512-2HdXDMd9GMgTGrPWnJzP2ALSokE/0O5HhTUvWIbD3YdjME8JwvSCnNGBnTThKGEB91OZhzrJ4qIIxk/SBmyDDA==}
engines: {node: '>=12'}
cpu: [riscv64]
os: [linux]
'@esbuild/linux-s390x@0.21.5':
resolution: {integrity: sha512-zus5sxzqBJD3eXxwvjN1yQkRepANgxE9lgOW2qLnmr8ikMTphkjgXu1HR01K4FJg8h1kEEDAqDcZQtbrRnB41A==}
engines: {node: '>=12'}
cpu: [s390x]
os: [linux]
'@esbuild/linux-x64@0.21.5':
resolution: {integrity: sha512-1rYdTpyv03iycF1+BhzrzQJCdOuAOtaqHTWJZCWvijKD2N5Xu0TtVC8/+1faWqcP9iBCWOmjmhoH94dH82BxPQ==}
engines: {node: '>=12'}
cpu: [x64]
os: [linux]
'@esbuild/netbsd-x64@0.21.5':
resolution: {integrity: sha512-Woi2MXzXjMULccIwMnLciyZH4nCIMpWQAs049KEeMvOcNADVxo0UBIQPfSmxB3CWKedngg7sWZdLvLczpe0tLg==}
engines: {node: '>=12'}
cpu: [x64]
os: [netbsd]
'@esbuild/openbsd-x64@0.21.5':
resolution: {integrity: sha512-HLNNw99xsvx12lFBUwoT8EVCsSvRNDVxNpjZ7bPn947b8gJPzeHWyNVhFsaerc0n3TsbOINvRP2byTZ5LKezow==}
engines: {node: '>=12'}
cpu: [x64]
os: [openbsd]
'@esbuild/sunos-x64@0.21.5':
resolution: {integrity: sha512-6+gjmFpfy0BHU5Tpptkuh8+uw3mnrvgs+dSPQXQOv3ekbordwnzTVEb4qnIvQcYXq6gzkyTnoZ9dZG+D4garKg==}
engines: {node: '>=12'}
cpu: [x64]
os: [sunos]
'@esbuild/win32-arm64@0.21.5':
resolution: {integrity: sha512-Z0gOTd75VvXqyq7nsl93zwahcTROgqvuAcYDUr+vOv8uHhNSKROyU961kgtCD1e95IqPKSQKH7tBTslnS3tA8A==}
engines: {node: '>=12'}
cpu: [arm64]
os: [win32]
'@esbuild/win32-ia32@0.21.5':
resolution: {integrity: sha512-SWXFF1CL2RVNMaVs+BBClwtfZSvDgtL//G/smwAc5oVK/UPu2Gu9tIaRgFmYFFKrmg3SyAjSrElf0TiJ1v8fYA==}
engines: {node: '>=12'}
cpu: [ia32]
os: [win32]
'@esbuild/win32-x64@0.21.5':
resolution: {integrity: sha512-tQd/1efJuzPC6rCFwEvLtci/xNFcTZknmXs98FYDfGE4wP9ClFV98nyKrzJKVPMhdDnjzLhdUyMX4PsQAPjwIw==}
engines: {node: '>=12'}
cpu: [x64]
os: [win32]
'@jridgewell/sourcemap-codec@1.6.0':
resolution: {integrity: sha512-T7jf+5zgsZHwNJ4lvQ7/aezbyk0nNX+zJVWpmHA7VYsEx7a7qr5Rg5IbtJFqkgze5Y2sruq1RUY8Q837Od7iFw==}
'@napi-rs/lzma-linux-x64-gnu@1.5.1':
resolution: {integrity: sha512-oTXEIha4SsuXdTA4Iyskj0kpdx2yVXdhd75c2v3xGrHFfVMsbhTPZU/nMPL4sWKo4pBHm3aucLaqGlF696dTyQ==}
engines: {node: ^22.20 || ^24.12 || >=25}
cpu: [x64]
os: [linux]
libc: [glibc]
'@rollup/rollup-android-arm-eabi@4.64.2':
resolution: {integrity: sha512-gjNDUwMswogkw3fNqV00u54uZlgmInuw5EU/1TeqQ6K0bEagoGvqjPdKzy112lKyhe7PkGudZ/kSQHsc9qroIg==}
cpu: [arm]
os: [android]
'@rollup/rollup-android-arm64@4.64.2':
resolution: {integrity: sha512-YkAW8q7p5SCLSaGWaUkmsRnBEb3MGA1pB5nI/8k0lAK5fZsQKRBqkdR1h4UU9p8u3c7ZGdUF2Zcx7mK+E/pUKg==}
cpu: [arm64]
os: [android]
'@rollup/rollup-darwin-arm64@4.64.2':
resolution: {integrity: sha512-hNmuIHv+LnrP/+q/44vsmIMWfoh7X2YXOp5FLtjGuvqN52VdtFpOpnshTc1k28tWv1wE/0D5gLjsozBUowEGbQ==}
cpu: [arm64]
os: [darwin]
'@rollup/rollup-darwin-x64@4.64.2':
resolution: {integrity: sha512-B0t5CoxgF2KL7R5FHAsyN8yVEQCSIkh4XVs8MCDjuBLNjEa8l3zd5tVKaFMv53LfUtHs9p2rnqeavPTOunV+Ww==}
cpu: [x64]
os: [darwin]
'@rollup/rollup-freebsd-arm64@4.64.2':
resolution: {integrity: sha512-QQdYnfwrPmc1DoPlddjFvQPzUHFbStKXsHrG6EBYsWHhBVUCjsCRdcfGGXGAvhDpDtTzKK4vy1UrGFJ45rQaiw==}
cpu: [arm64]
os: [freebsd]
'@rollup/rollup-freebsd-x64@4.64.2':
resolution: {integrity: sha512-1xRxtGWif6BB58VSfMavi0Yfs2M4x6CtRp4o7ScY4W7AGlgvf6GSPKdDNdw7TX3tMiQwZY5W/mvyrOJTq/7AKA==}
cpu: [x64]
os: [freebsd]
'@rollup/rollup-linux-arm-gnueabihf@4.64.2':
resolution: {integrity: sha512-CXFLEt384BUwwOLR9DURWNHl1bQOVLd1ibOee+yKBJm/fZtvrSAOZ0JbxjAxszVMIovYMwFMS+oP+Q1aPcx8Eg==}
cpu: [arm]
os: [linux]
libc: [glibc]
'@rollup/rollup-linux-arm-musleabihf@4.64.2':
resolution: {integrity: sha512-T1dorY0WHm5mMNpMHFl5rYRZ02gLSZcr6J45r66tYb3S3R3DyIRGiGWVdLtF+zOQjratNJaObl9jYumueRiL1A==}
cpu: [arm]
os: [linux]
libc: [musl]
'@rollup/rollup-linux-arm64-gnu@4.64.2':
resolution: {integrity: sha512-ia6bfaCIc0ldp6bvs9oEkFrJVJKNUMoiDF0+LBZsQAZg+vE0znXnzGqH+ew/FYswE30cruAWPTO3dPw7keaVkQ==}
cpu: [arm64]
os: [linux]
libc: [glibc]
'@rollup/rollup-linux-arm64-musl@4.64.2':
resolution: {integrity: sha512-mB/gwfBfXtT0EBpbSssqKCqMFuX8b21Zy2TGoIhehSJEssZ5ksybxymwp1BkEM5Nczh/AAxpmh5q+MkhtundGg==}
cpu: [arm64]
os: [linux]
libc: [musl]
'@rollup/rollup-linux-loong64-gnu@4.64.2':
resolution: {integrity: sha512-VPd0a+XlHoNRI9jIPRHww2UKhDkD6XjbtcKYXZijAnhJS1jLLbiNtVl6a5LyEfNN+eyiyWcJOZXPzBo69oTsew==}
cpu: [loong64]
os: [linux]
libc: [glibc]
'@rollup/rollup-linux-loong64-musl@4.64.2':
resolution: {integrity: sha512-10N1I6oBpERqzGtHKcCLrEpHlbXSBbMK9TURHvqgWbIFZuU3oJROxJ0H2nHfq96TVI0XJHQ9qzolJMFd2tz0LA==}
cpu: [loong64]
os: [linux]
libc: [musl]
'@rollup/rollup-linux-ppc64-gnu@4.64.2':
resolution: {integrity: sha512-dhJrLVZ3wNNhwCmy9/VTg6Pq6J8a8eZ0n15dVRlbSWcDb+8V/1mDfbnvis+vt8W6Pbxzv6gUbTEl4piOT3bbUQ==}
cpu: [ppc64]
os: [linux]
libc: [glibc]
'@rollup/rollup-linux-ppc64-musl@4.64.2':
resolution: {integrity: sha512-tPd2QuwxkTxgju5eARyb3RdgBWWi/O9dn/NFH0VkvGbLzoQeXrH6qMABtW8IODb5lUeBzB3sy2U7qmI4MtQ72g==}
cpu: [ppc64]
os: [linux]
libc: [musl]
'@rollup/rollup-linux-riscv64-gnu@4.64.2':
resolution: {integrity: sha512-njtDdPvSQWI9CDhIPPxsiEMeHayCfu7wwcbSQ7c2kYPhGVviL+WGVhq6Jfj/b+VuhSTkFuQ0z4ZFbFjkWwBFmg==}
cpu: [riscv64]
os: [linux]
libc: [glibc]
'@rollup/rollup-linux-riscv64-musl@4.64.2':
resolution: {integrity: sha512-EbOBo3dqCj9ecEV9Z7zvulB6AieMP2ztnKQYGVOSN0pzIyOu+3JH2YJZnLmbPeVvfn6yP6Qi7NbBM27EzGbj2w==}
cpu: [riscv64]
os: [linux]
libc: [musl]
'@rollup/rollup-linux-s390x-gnu@4.64.2':
resolution: {integrity: sha512-8wLt2Q5n4qmfiE2Gk8V6KNLSf+oJuBX7Kdw5joFDtnZIZvirS0hLzBc79OiJjV71oYDmBUZLNy6pDgvylo4qiQ==}
cpu: [s390x]
os: [linux]
libc: [glibc]
'@rollup/rollup-linux-x64-gnu@4.64.2':
resolution: {integrity: sha512-aIChbkhC4d33bxkVP0LSi2/2lUSGRXE6k4Qa0G7tgpgTOK7kxHTixmx5qC3SKUJ6yUMznHBtQoWaG4QMqLUnhw==}
cpu: [x64]
os: [linux]
libc: [glibc]
'@rollup/rollup-linux-x64-musl@4.64.2':
resolution: {integrity: sha512-gQ+TkVZRfjZVcl/UOAxxfWw+sHjzIXoieNwxEyJlD3tRBmBhR8ZDq11w+QEgQDcpslEB+fhN/OEpaTsDoIIHhQ==}
cpu: [x64]
os: [linux]
libc: [musl]
'@rollup/rollup-openbsd-x64@4.64.2':
resolution: {integrity: sha512-sUaJpMKS2JBCvUk8fYimG7aMq2jHVT4kw/uVHxPolKLrMmkKy0KXirmZyZehdNpL7PffqAlPAWUELQcTiD8ChA==}
cpu: [x64]
os: [openbsd]
'@rollup/rollup-openharmony-arm64@4.64.2':
resolution: {integrity: sha512-j1RdJWb0CrlSuwt12OPjLucPlwfLtA/KMOKVvh5rXSJEtl/WDoIGR3CSykK/jLGTo9RqNfW5Ub6kY5Z5ccYl9w==}
cpu: [arm64]
os: [openharmony]
'@rollup/rollup-win32-arm64-msvc@4.64.2':
resolution: {integrity: sha512-i3r04aHCIY3eggTBLUp/4IxiBjDpAgQaK89Ue2nMJ2MKuIXX5JfoeZjHw+kPjGGNVDlhi8FT8xzuGMJZe3aehg==}
cpu: [arm64]
os: [win32]
'@rollup/rollup-win32-ia32-msvc@4.64.2':
resolution: {integrity: sha512-kCQ15h/uwNMGdN8gjXBZGf+aGE+ODzHfbi90/cAm4+Q89WoMiLODNoj8HoGZmYx87HPxfu5CBNPsZ+ZG4uIAoA==}
cpu: [ia32]
os: [win32]
'@rollup/rollup-win32-x64-gnu@4.64.2':
resolution: {integrity: sha512-yYAuySSs+0X2gSMJPw/Q4jHsHZHyMq1dpAE+XnN790Jg1WuReUAD/+j+e4vuDvS4PXJ4PA4NCzTR4OYtDf18/g==}
cpu: [x64]
os: [win32]
'@rollup/rollup-win32-x64-msvc@4.64.2':
resolution: {integrity: sha512-JY1rQdWOJrKutORqKO1uzrHXV86jFiQzfLFKx2s8S1CjLXL9fC5LOIvk0q7Z7PXJvsByt+sdoE1Q2a9gBr4gLQ==}
cpu: [x64]
os: [win32]
'@types/estree@1.0.9':
resolution: {integrity: sha512-GhdPgy1el4/ImP05X05Uw4cw2/M93BCUmnEvWZNStlCzEKME4Fkk+YpoA5OiHNQmoS7Cafb8Xa3Pya8m1Qrzeg==}
'@types/node@20.19.43':
resolution: {integrity: sha512-6oYBAi5ikg4Pl+kGsoYtawUMBT2zZMCvPNF7pVLnHZfd1zf38DRiWn/gT01RYCdUqkv7Fhr+C9ot4/tb+2sVvA==}
'@vitest/expect@2.1.9':
resolution: {integrity: sha512-UJCIkTBenHeKT1TTlKMJWy1laZewsRIzYighyYiJKZreqtdxSos/S1t+ktRMQWu2CKqaarrkeszJx1cgC5tGZw==}
'@vitest/mocker@2.1.9':
resolution: {integrity: sha512-tVL6uJgoUdi6icpxmdrn5YNo3g3Dxv+IHJBr0GXHaEdTcw3F+cPKnsXFhli6nO+f/6SDKPHEK1UN+k+TQv0Ehg==}
peerDependencies:
msw: ^2.4.9
vite: ^5.0.0
peerDependenciesMeta:
msw:
optional: true
vite:
optional: true
'@vitest/pretty-format@2.1.9':
resolution: {integrity: sha512-KhRIdGV2U9HOUzxfiHmY8IFHTdqtOhIzCpd8WRdJiE7D/HUcZVD0EgQCVjm+Q9gkUXWgBvMmTtZgIG48wq7sOQ==}
'@vitest/runner@2.1.9':
resolution: {integrity: sha512-ZXSSqTFIrzduD63btIfEyOmNcBmQvgOVsPNPe0jYtESiXkhd8u2erDLnMxmGrDCwHCCHE7hxwRDCT3pt0esT4g==}
'@vitest/snapshot@2.1.9':
resolution: {integrity: sha512-oBO82rEjsxLNJincVhLhaxxZdEtV0EFHMK5Kmx5sJ6H9L183dHECjiefOAdnqpIgT5eZwT04PoggUnW88vOBNQ==}
'@vitest/spy@2.1.9':
resolution: {integrity: sha512-E1B35FwzXXTs9FHNK6bDszs7mtydNi5MIfUWpceJ8Xbfb1gBMscAnwLbEu+B44ed6W3XjL9/ehLPHR1fkf1KLQ==}
'@vitest/utils@2.1.9':
resolution: {integrity: sha512-v0psaMSkNJ3A2NMrUEHFRzJtDPFn+/VWZ5WxImB21T9fjucJRmS7xCS3ppEnARb9y11OAzaD+P2Ps+b+BGX5iQ==}
assertion-error@2.0.1:
resolution: {integrity: sha512-Izi8RQcffqCeNVgFigKli1ssklIbpHnCYc6AknXGYoB6grJqyeby7jv12JUQgmTAnIDnbck1uxksT4dzN3PWBA==}
engines: {node: '>=12'}
cac@6.7.14:
resolution: {integrity: sha512-b6Ilus+c3RrdDk+JhLKUAQfzzgLEPy6wcXqS7f/xe1EETvsDP6GORG7SFuOs6cID5YkqchW/LXZbX5bc8j7ZcQ==}
engines: {node: '>=8'}
chai@5.3.3:
resolution: {integrity: sha512-4zNhdJD/iOjSH0A05ea+Ke6MU5mmpQcbQsSOkgdaUMJ9zTlDTD/GYlwohmIE2u0gaxHYiVHEn1Fw9mZ/ktJWgw==}
engines: {node: '>=18'}
check-error@2.1.3:
resolution: {integrity: sha512-PAJdDJusoxnwm1VwW07VWwUN1sl7smmC3OKggvndJFadxxDRyFJBX/ggnu/KE4kQAB7a3Dp8f/YXC1FlUprWmA==}
engines: {node: '>= 16'}
debug@4.4.3:
resolution: {integrity: sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA==}
engines: {node: '>=6.0'}
peerDependencies:
supports-color: '*'
peerDependenciesMeta:
supports-color:
optional: true
deep-eql@5.0.2:
resolution: {integrity: sha512-h5k/5U50IJJFpzfL6nO9jaaumfjO/f2NjK/oYB2Djzm4p9L+3T9qWpZqZ2hAbLPuuYq9wrU08WQyBTL5GbPk5Q==}
engines: {node: '>=6'}
es-module-lexer@1.7.0:
resolution: {integrity: sha512-jEQoCwk8hyb2AZziIOLhDqpm5+2ww5uIE6lkO/6jcOCusfk6LhMHpXXfBLXTZ7Ydyt0j4VoUQv6uGNYbdW+kBA==}
esbuild@0.21.5:
resolution: {integrity: sha512-mg3OPMV4hXywwpoDxu3Qda5xCKQi+vCTZq8S9J/EpkhB2HzKXq4SNFZE3+NK93JYxc8VMSep+lOUSC/RVKaBqw==}
engines: {node: '>=12'}
hasBin: true
estree-walker@3.0.3:
resolution: {integrity: sha512-7RUKfXgSMMkzt6ZuXmqapOurLGPPfgj6l9uRZ7lRGolvk0y2yocc35LdcxKC5PQZdn2DMqioAQ2NoWcrTKmm6g==}
expect-type@1.4.0:
resolution: {integrity: sha512-KfYbmpRm0VbLjEvVa9yGwCi9GI34xvi7A/HXYWQO65CSD2u3MczUJSuwXKFIxlGsgBQizV9q5J9NHj4VG0n+pA==}
engines: {node: '>=12.0.0'}
fsevents@2.3.3:
resolution: {integrity: sha512-5xoDfX+fL7faATnagmWPpbFtwh/R77WmMMqqHGS65C3vvB0YHrgF+B1YmZ3441tMj5n63k0212XNoJwzlhffQw==}
engines: {node: ^8.16.0 || ^10.6.0 || >=11.0.0}
os: [darwin]
loupe@3.2.1:
resolution: {integrity: sha512-CdzqowRJCeLU72bHvWqwRBBlLcMEtIvGrlvef74kMnV2AolS9Y8xUv1I0U/MNAWMhBlKIoyuEgoJ0t/bbwHbLQ==}
magic-string@0.30.21:
resolution: {integrity: sha512-vd2F4YUyEXKGcLHoq+TEyCjxueSeHnFxyyjNp80yg0XV4vUhnDer/lvvlqM/arB5bXQN5K2/3oinyCRyx8T2CQ==}
ms@2.1.3:
resolution: {integrity: sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==}
nanoid@3.3.20:
resolution: {integrity: sha512-uKdg2G3GNCKQn9byYOpxbGqrT2fGO5KRt5J/8b3pok8rT6qxGWF6hxMyJiEYtAf+FVyYuD9hRaDqX5uPFYJ4ZQ==}
engines: {node: ^10 || ^12 || ^13.7 || ^14 || >=15.0.1}
hasBin: true
pathe@1.1.2:
resolution: {integrity: sha512-whLdWMYL2TwI08hn8/ZqAbrVemu0LNaNNJZX73O6qaIdCTfXutsLhMkjdENX0qhsQ9uIimo4/aQOmXkoon2nDQ==}
pathval@2.0.1:
resolution: {integrity: sha512-//nshmD55c46FuFw26xV/xFAaB5HF9Xdap7HJBBnrKdAd6/GxDBaNA1870O79+9ueg61cZLSVc+OaFlfmObYVQ==}
engines: {node: '>= 14.16'}
picocolors@1.1.1:
resolution: {integrity: sha512-xceH2snhtb5M9liqDsmEw56le376mTZkEX/jEb/RxNFyegNul7eNslCXP9FDj/Lcu0X8KEyMceP2ntpaHrDEVA==}
postcss@8.5.29:
resolution: {integrity: sha512-49cGhUbXj8Qenv0iTMxA1cFBzxXoctpC9Ujd77t1WcbJIr6nF/eI7g/8MgxrYldFRuAXvja7xQRwavoW7kgrxQ==}
engines: {node: ^10 || ^12 || >=14}
rollup@4.64.2:
resolution: {integrity: sha512-RxXm0XVVV1hFI3dYe12ruaOgsYQ725lUcUWeMSQqkF9tTxVErY+fy2Ets6vWhItL9MZMSNZGbvZFc7rH/VHp2Q==}
engines: {node: '>=18.0.0', npm: '>=8.0.0'}
hasBin: true
siginfo@2.0.0:
resolution: {integrity: sha512-ybx0WO1/8bSBLEWXZvEd7gMW3Sn3JFlW3TvX1nREbDLRNQNaeNN8WK0meBwPdAaOI7TtRRRJn/Es1zhrrCHu7g==}
source-map-js@1.2.2:
resolution: {integrity: sha512-KGj/8Y43x35aZVDtt+J4mK1hoLGHULMYfSkODJNQjNDC3oW1PqPoxMwo0pLUsWM/UEGzON/NxeHywEfNXNP3Vw==}
engines: {node: '>=0.10.0'}
stackback@0.0.2:
resolution: {integrity: sha512-1XMJE5fQo1jGH6Y/7ebnwPOBEkIEnT4QF32d5R1+VXdXveM0IBMJt8zfaxX1P3QhVwrYe+576+jkANtSS2mBbw==}
std-env@3.10.0:
resolution: {integrity: sha512-5GS12FdOZNliM5mAOxFRg7Ir0pWz8MdpYm6AY6VPkGpbA7ZzmbzNcBJQ0GPvvyWgcY7QAhCgf9Uy89I03faLkg==}
tinybench@2.9.0:
resolution: {integrity: sha512-0+DUvqWMValLmha6lr4kD8iAMK1HzV0/aKnCtWb9v9641TnP/MFb7Pc2bxoxQjTXAErryXVgUOfv2YqNllqGeg==}
tinyexec@0.3.2:
resolution: {integrity: sha512-KQQR9yN7R5+OSwaK0XQoj22pwHoTlgYqmUscPYoknOoWCWfj/5/ABTMRi69FrKU5ffPVh5QcFikpWJI/P1ocHA==}
tinypool@1.1.1:
resolution: {integrity: sha512-Zba82s87IFq9A9XmjiX5uZA/ARWDrB03OHlq+Vw1fSdt0I+4/Kutwy8BP4Y/y/aORMo61FQ0vIb5j44vSo5Pkg==}
engines: {node: ^18.0.0 || >=20.0.0}
tinyrainbow@1.2.0:
resolution: {integrity: sha512-weEDEq7Z5eTHPDh4xjX789+fHfF+P8boiFB+0vbWzpbnbsEr/GRaohi/uMKxg8RZMXnl1ItAi/IUHWMsjDV7kQ==}
engines: {node: '>=14.0.0'}
tinyspy@3.0.2:
resolution: {integrity: sha512-n1cw8k1k0x4pgA2+9XrOkFydTerNcJ1zWCO5Nn9scWHTD+5tp8dghT2x1uduQePZTZgd3Tupf+x9BxJjeJi77Q==}
engines: {node: '>=14.0.0'}
typescript@5.9.3:
resolution: {integrity: sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw==}
engines: {node: '>=14.17'}
hasBin: true
undici-types@6.21.0:
resolution: {integrity: sha512-iwDZqg0QAGrg9Rav5H4n0M64c3mkR59cJ6wQp+7C4nI0gsmExaedaYLNO44eT4AtBBwjbTiGPMlt2Md0T9H9JQ==}
vite-node@2.1.9:
resolution: {integrity: sha512-AM9aQ/IPrW/6ENLQg3AGY4K1N2TGZdR5e4gu/MmmR2xR3Ll1+dib+nook92g4TV3PXVyeyxdWwtaCAiUL0hMxA==}
engines: {node: ^18.0.0 || >=20.0.0}
hasBin: true
vite@5.4.21:
resolution: {integrity: sha512-o5a9xKjbtuhY6Bi5S3+HvbRERmouabWbyUcpXXUA1u+GNUKoROi9byOJ8M0nHbHYHkYICiMlqxkg1KkYmm25Sw==}
engines: {node: ^18.0.0 || >=20.0.0}
hasBin: true
peerDependencies:
'@types/node': ^18.0.0 || >=20.0.0
less: '*'
lightningcss: ^1.21.0
sass: '*'
sass-embedded: '*'
stylus: '*'
sugarss: '*'
terser: ^5.4.0
peerDependenciesMeta:
'@types/node':
optional: true
less:
optional: true
lightningcss:
optional: true
sass:
optional: true
sass-embedded:
optional: true
stylus:
optional: true
sugarss:
optional: true
terser:
optional: true
vitest@2.1.9:
resolution: {integrity: sha512-MSmPM9REYqDGBI8439mA4mWhV5sKmDlBKWIYbA3lRb2PTHACE0mgKwA8yQ2xq9vxDTuk4iPrECBAEW2aoFXY0Q==}
engines: {node: ^18.0.0 || >=20.0.0}
hasBin: true
peerDependencies:
'@edge-runtime/vm': '*'
'@types/node': ^18.0.0 || >=20.0.0
'@vitest/browser': 2.1.9
'@vitest/ui': 2.1.9
happy-dom: '*'
jsdom: '*'
peerDependenciesMeta:
'@edge-runtime/vm':
optional: true
'@types/node':
optional: true
'@vitest/browser':
optional: true
'@vitest/ui':
optional: true
happy-dom:
optional: true
jsdom:
optional: true
why-is-node-running@2.3.0:
resolution: {integrity: sha512-hUrmaWBdVDcxvYqnyh09zunKzROWjbZTiNy8dBEjkS7ehEDQibXJ7XvlmtbwuTclUiIyN+CyXQD4Vmko8fNm8w==}
engines: {node: '>=8'}
hasBin: true
snapshots:
'@esbuild/aix-ppc64@0.21.5':
optional: true
'@esbuild/android-arm64@0.21.5':
optional: true
'@esbuild/android-arm@0.21.5':
optional: true
'@esbuild/android-x64@0.21.5':
optional: true
'@esbuild/darwin-arm64@0.21.5':
optional: true
'@esbuild/darwin-x64@0.21.5':
optional: true
'@esbuild/freebsd-arm64@0.21.5':
optional: true
'@esbuild/freebsd-x64@0.21.5':
optional: true
'@esbuild/linux-arm64@0.21.5':
optional: true
'@esbuild/linux-arm@0.21.5':
optional: true
'@esbuild/linux-ia32@0.21.5':
optional: true
'@esbuild/linux-loong64@0.21.5':
optional: true
'@esbuild/linux-mips64el@0.21.5':
optional: true
'@esbuild/linux-ppc64@0.21.5':
optional: true
'@esbuild/linux-riscv64@0.21.5':
optional: true
'@esbuild/linux-s390x@0.21.5':
optional: true
'@esbuild/linux-x64@0.21.5':
optional: true
'@esbuild/netbsd-x64@0.21.5':
optional: true
'@esbuild/openbsd-x64@0.21.5':
optional: true
'@esbuild/sunos-x64@0.21.5':
optional: true
'@esbuild/win32-arm64@0.21.5':
optional: true
'@esbuild/win32-ia32@0.21.5':
optional: true
'@esbuild/win32-x64@0.21.5':
optional: true
'@jridgewell/sourcemap-codec@1.6.0': {}
'@napi-rs/lzma-linux-x64-gnu@1.5.1':
optional: true
'@rollup/rollup-android-arm-eabi@4.64.2':
optional: true
'@rollup/rollup-android-arm64@4.64.2':
optional: true
'@rollup/rollup-darwin-arm64@4.64.2':
optional: true
'@rollup/rollup-darwin-x64@4.64.2':
optional: true
'@rollup/rollup-freebsd-arm64@4.64.2':
optional: true
'@rollup/rollup-freebsd-x64@4.64.2':
optional: true
'@rollup/rollup-linux-arm-gnueabihf@4.64.2':
optional: true
'@rollup/rollup-linux-arm-musleabihf@4.64.2':
optional: true
'@rollup/rollup-linux-arm64-gnu@4.64.2':
optional: true
'@rollup/rollup-linux-arm64-musl@4.64.2':
optional: true
'@rollup/rollup-linux-loong64-gnu@4.64.2':
optional: true
'@rollup/rollup-linux-loong64-musl@4.64.2':
optional: true
'@rollup/rollup-linux-ppc64-gnu@4.64.2':
optional: true
'@rollup/rollup-linux-ppc64-musl@4.64.2':
optional: true
'@rollup/rollup-linux-riscv64-gnu@4.64.2':
optional: true
'@rollup/rollup-linux-riscv64-musl@4.64.2':
optional: true
'@rollup/rollup-linux-s390x-gnu@4.64.2':
optional: true
'@rollup/rollup-linux-x64-gnu@4.64.2':
optional: true
'@rollup/rollup-linux-x64-musl@4.64.2':
optional: true
'@rollup/rollup-openbsd-x64@4.64.2':
optional: true
'@rollup/rollup-openharmony-arm64@4.64.2':
optional: true
'@rollup/rollup-win32-arm64-msvc@4.64.2':
optional: true
'@rollup/rollup-win32-ia32-msvc@4.64.2':
optional: true
'@rollup/rollup-win32-x64-gnu@4.64.2':
optional: true
'@rollup/rollup-win32-x64-msvc@4.64.2':
optional: true
'@types/estree@1.0.9': {}
'@types/node@20.19.43':
dependencies:
undici-types: 6.21.0
'@vitest/expect@2.1.9':
dependencies:
'@vitest/spy': 2.1.9
'@vitest/utils': 2.1.9
chai: 5.3.3
tinyrainbow: 1.2.0
'@vitest/mocker@2.1.9(vite@5.4.21(@types/node@20.19.43))':
dependencies:
'@vitest/spy': 2.1.9
estree-walker: 3.0.3
magic-string: 0.30.21
optionalDependencies:
vite: 5.4.21(@types/node@20.19.43)
'@vitest/pretty-format@2.1.9':
dependencies:
tinyrainbow: 1.2.0
'@vitest/runner@2.1.9':
dependencies:
'@vitest/utils': 2.1.9
pathe: 1.1.2
'@vitest/snapshot@2.1.9':
dependencies:
'@vitest/pretty-format': 2.1.9
magic-string: 0.30.21
pathe: 1.1.2
'@vitest/spy@2.1.9':
dependencies:
tinyspy: 3.0.2
'@vitest/utils@2.1.9':
dependencies:
'@vitest/pretty-format': 2.1.9
loupe: 3.2.1
tinyrainbow: 1.2.0
assertion-error@2.0.1: {}
cac@6.7.14: {}
chai@5.3.3:
dependencies:
assertion-error: 2.0.1
check-error: 2.1.3
deep-eql: 5.0.2
loupe: 3.2.1
pathval: 2.0.1
check-error@2.1.3: {}
debug@4.4.3:
dependencies:
ms: 2.1.3
deep-eql@5.0.2: {}
es-module-lexer@1.7.0: {}
esbuild@0.21.5:
optionalDependencies:
'@esbuild/aix-ppc64': 0.21.5
'@esbuild/android-arm': 0.21.5
'@esbuild/android-arm64': 0.21.5
'@esbuild/android-x64': 0.21.5
'@esbuild/darwin-arm64': 0.21.5
'@esbuild/darwin-x64': 0.21.5
'@esbuild/freebsd-arm64': 0.21.5
'@esbuild/freebsd-x64': 0.21.5
'@esbuild/linux-arm': 0.21.5
'@esbuild/linux-arm64': 0.21.5
'@esbuild/linux-ia32': 0.21.5
'@esbuild/linux-loong64': 0.21.5
'@esbuild/linux-mips64el': 0.21.5
'@esbuild/linux-ppc64': 0.21.5
'@esbuild/linux-riscv64': 0.21.5
'@esbuild/linux-s390x': 0.21.5
'@esbuild/linux-x64': 0.21.5
'@esbuild/netbsd-x64': 0.21.5
'@esbuild/openbsd-x64': 0.21.5
'@esbuild/sunos-x64': 0.21.5
'@esbuild/win32-arm64': 0.21.5
'@esbuild/win32-ia32': 0.21.5
'@esbuild/win32-x64': 0.21.5
estree-walker@3.0.3:
dependencies:
'@types/estree': 1.0.9
expect-type@1.4.0: {}
fsevents@2.3.3:
optional: true
loupe@3.2.1: {}
magic-string@0.30.21:
dependencies:
'@jridgewell/sourcemap-codec': 1.6.0
ms@2.1.3: {}
nanoid@3.3.20: {}
pathe@1.1.2: {}
pathval@2.0.1: {}
picocolors@1.1.1: {}
postcss@8.5.29:
dependencies:
nanoid: 3.3.20
picocolors: 1.1.1
source-map-js: 1.2.2
rollup@4.64.2:
dependencies:
'@types/estree': 1.0.9
optionalDependencies:
'@napi-rs/lzma-linux-x64-gnu': 1.5.1
'@rollup/rollup-android-arm-eabi': 4.64.2
'@rollup/rollup-android-arm64': 4.64.2
'@rollup/rollup-darwin-arm64': 4.64.2
'@rollup/rollup-darwin-x64': 4.64.2
'@rollup/rollup-freebsd-arm64': 4.64.2
'@rollup/rollup-freebsd-x64': 4.64.2
'@rollup/rollup-linux-arm-gnueabihf': 4.64.2
'@rollup/rollup-linux-arm-musleabihf': 4.64.2
'@rollup/rollup-linux-arm64-gnu': 4.64.2
'@rollup/rollup-linux-arm64-musl': 4.64.2
'@rollup/rollup-linux-loong64-gnu': 4.64.2
'@rollup/rollup-linux-loong64-musl': 4.64.2
'@rollup/rollup-linux-ppc64-gnu': 4.64.2
'@rollup/rollup-linux-ppc64-musl': 4.64.2
'@rollup/rollup-linux-riscv64-gnu': 4.64.2
'@rollup/rollup-linux-riscv64-musl': 4.64.2
'@rollup/rollup-linux-s390x-gnu': 4.64.2
'@rollup/rollup-linux-x64-gnu': 4.64.2
'@rollup/rollup-linux-x64-musl': 4.64.2
'@rollup/rollup-openbsd-x64': 4.64.2
'@rollup/rollup-openharmony-arm64': 4.64.2
'@rollup/rollup-win32-arm64-msvc': 4.64.2
'@rollup/rollup-win32-ia32-msvc': 4.64.2
'@rollup/rollup-win32-x64-gnu': 4.64.2
'@rollup/rollup-win32-x64-msvc': 4.64.2
fsevents: 2.3.3
siginfo@2.0.0: {}
source-map-js@1.2.2: {}
stackback@0.0.2: {}
std-env@3.10.0: {}
tinybench@2.9.0: {}
tinyexec@0.3.2: {}
tinypool@1.1.1: {}
tinyrainbow@1.2.0: {}
tinyspy@3.0.2: {}
typescript@5.9.3: {}
undici-types@6.21.0: {}
vite-node@2.1.9(@types/node@20.19.43):
dependencies:
cac: 6.7.14
debug: 4.4.3
es-module-lexer: 1.7.0
pathe: 1.1.2
vite: 5.4.21(@types/node@20.19.43)
transitivePeerDependencies:
- '@types/node'
- less
- lightningcss
- sass
- sass-embedded
- stylus
- sugarss
- supports-color
- terser
vite@5.4.21(@types/node@20.19.43):
dependencies:
esbuild: 0.21.5
postcss: 8.5.29
rollup: 4.64.2
optionalDependencies:
'@types/node': 20.19.43
fsevents: 2.3.3
vitest@2.1.9(@types/node@20.19.43):
dependencies:
'@vitest/expect': 2.1.9
'@vitest/mocker': 2.1.9(vite@5.4.21(@types/node@20.19.43))
'@vitest/pretty-format': 2.1.9
'@vitest/runner': 2.1.9
'@vitest/snapshot': 2.1.9
'@vitest/spy': 2.1.9
'@vitest/utils': 2.1.9
chai: 5.3.3
debug: 4.4.3
expect-type: 1.4.0
magic-string: 0.30.21
pathe: 1.1.2
std-env: 3.10.0
tinybench: 2.9.0
tinyexec: 0.3.2
tinypool: 1.1.1
tinyrainbow: 1.2.0
vite: 5.4.21(@types/node@20.19.43)
vite-node: 2.1.9(@types/node@20.19.43)
why-is-node-running: 2.3.0
optionalDependencies:
'@types/node': 20.19.43
transitivePeerDependencies:
- less
- lightningcss
- msw
- sass
- sass-embedded
- stylus
- sugarss
- supports-color
- terser
why-is-node-running@2.3.0:
dependencies:
siginfo: 2.0.0
stackback: 0.0.2
upstream/pnpm-lock.yaml on GitHub
upstream/pnpm-workspace.yaml
allowBuilds:
esbuild: true
upstream/pnpm-workspace.yaml on GitHub
upstream/README.md
# ledgerly
Small invoicing API used by the back office: customers, invoices, an outstanding-balance report.
Plain Node 20 + TypeScript, no framework. Data lives in one JSON file (`LEDGERLY_DATA`, default `data/ledgerly.json`).
```bash
pnpm install
pnpm test # vitest
pnpm typecheck # tsc --noEmit
```
## Layout
- `src/domain/` - invoices, customers, money, validation, domain errors. No I/O.
- `src/store/` - the JSON-file store, repositories, and the schema migrations applied on load.
- `src/http/` - a tiny router, request handlers, error-to-status mapping. `createApp()` returns a
`handle(method, path, body)` function that the tests call directly; `src/server.ts` binds it to `node:http`.
## Conventions
- Money is always an integer number of cents. Never floats.
- Domain functions return new objects; they never mutate their input.
- Validation failures throw `ValidationError` (HTTP 400), missing records `NotFoundError` (404),
illegal state transitions `ConflictError` (409).
- The stored file carries a `schemaVersion`. Any change to the stored shape bumps it and adds a
migration in `src/store/migrations.ts`: production files are migrated on load, never by hand.
## Invoice lifecycle
`draft` → (`issue`) → `open` → (`pay`) → `paid`; `draft`/`open` → (`void`) → `void`.
upstream/src/domain/customer.ts
import { ValidationError } from "./errors";
export type Customer = {
id: string;
name: string;
email: string;
createdAt: string;
};
export type NewCustomerInput = { name: string; email: string };
const EMAIL = /^[^\s@]+@[^\s@]+\.[^\s@]+$/;
export function parseNewCustomer(body: unknown): NewCustomerInput {
if (typeof body !== "object" || body === null) {
throw new ValidationError("body must be an object");
}
const { name, email } = body as Record<string, unknown>;
if (typeof name !== "string" || name.trim() === "") {
throw new ValidationError("name is required", "name");
}
if (typeof email !== "string" || !EMAIL.test(email)) {
throw new ValidationError("email is invalid", "email");
}
return { name: name.trim(), email: email.toLowerCase() };
}
upstream/src/domain/customer.ts on GitHub
upstream/src/domain/errors.ts
export class DomainError extends Error {
constructor(message: string) {
super(message);
this.name = new.target.name;
}
}
/** Input that can never be valid (bad shape, out of range). Maps to HTTP 400. */
export class ValidationError extends DomainError {
constructor(
message: string,
readonly field?: string,
) {
super(message);
}
}
/** A record that does not exist. Maps to HTTP 404. */
export class NotFoundError extends DomainError {
constructor(entity: string, id: string) {
super(`${entity} ${id} not found`);
}
}
/** A valid request that the current state forbids (e.g. paying a void invoice). Maps to HTTP 409. */
export class ConflictError extends DomainError {}
upstream/src/domain/errors.ts on GitHub
upstream/src/domain/invoice.ts
import { ConflictError } from "./errors";
import { type Cents, type Currency, sum } from "./money";
export type InvoiceStatus = "draft" | "open" | "paid" | "void";
export type LineItem = {
description: string;
quantity: number;
/** cents */
unitPrice: Cents;
};
export type Invoice = {
id: string;
/** Assigned when the invoice is issued (INV-0001, ...). */
number: string | null;
customerId: string;
currency: Currency;
lines: LineItem[];
/** Flat discount in cents, never above the subtotal. */
discount: Cents;
status: InvoiceStatus;
createdAt: string;
issuedAt: string | null;
dueDate: string | null;
paidAt: string | null;
voidedAt: string | null;
};
export type InvoiceTotals = { subtotal: Cents; discount: Cents; total: Cents };
export function lineTotal(line: LineItem): Cents {
return line.quantity * line.unitPrice;
}
export function computeTotals(invoice: Pick<Invoice, "lines" | "discount">): InvoiceTotals {
const subtotal = sum(invoice.lines.map(lineTotal));
return { subtotal, discount: invoice.discount, total: subtotal - invoice.discount };
}
export function issue(invoice: Invoice, number: string, now: Date): Invoice {
if (invoice.status !== "draft") {
throw new ConflictError(`invoice ${invoice.id} is ${invoice.status}, only drafts can be issued`);
}
return { ...invoice, status: "open", number, issuedAt: now.toISOString() };
}
export function markPaid(invoice: Invoice, now: Date): Invoice {
if (invoice.status !== "open") {
throw new ConflictError(`invoice ${invoice.id} is ${invoice.status}, only open invoices can be paid`);
}
return { ...invoice, status: "paid", paidAt: now.toISOString() };
}
export function voidInvoice(invoice: Invoice, now: Date): Invoice {
if (invoice.status === "paid" || invoice.status === "void") {
throw new ConflictError(`invoice ${invoice.id} is ${invoice.status} and cannot be voided`);
}
return { ...invoice, status: "void", voidedAt: now.toISOString() };
}
export function isOverdue(invoice: Invoice, now: Date): boolean {
return invoice.status === "open" && invoice.dueDate !== null && invoice.dueDate < now.toISOString().slice(0, 10);
}
upstream/src/domain/invoice.ts on GitHub
upstream/src/domain/money.ts
import { ValidationError } from "./errors";
export const CURRENCIES = ["EUR", "USD", "GBP"] as const;
export type Currency = (typeof CURRENCIES)[number];
/** Amounts are integer cents. */
export type Cents = number;
export function isCurrency(value: unknown): value is Currency {
return typeof value === "string" && (CURRENCIES as readonly string[]).includes(value);
}
export function assertCents(value: unknown, field: string, opts: { min?: number } = {}): Cents {
if (typeof value !== "number" || !Number.isInteger(value)) {
throw new ValidationError(`${field} must be an integer number of cents`, field);
}
const min = opts.min ?? 0;
if (value < min) {
throw new ValidationError(`${field} must be >= ${min}`, field);
}
return value;
}
export function formatCents(amount: Cents, currency: Currency): string {
const sign = amount < 0 ? "-" : "";
const abs = Math.abs(amount);
return `${sign}${Math.floor(abs / 100)}.${String(abs % 100).padStart(2, "0")} ${currency}`;
}
export function sum(amounts: Cents[]): Cents {
return amounts.reduce((acc, a) => acc + a, 0);
}
upstream/src/domain/money.ts on GitHub
upstream/src/domain/validation.ts
import { ValidationError } from "./errors";
import { computeTotals, type LineItem } from "./invoice";
import { assertCents, type Currency, isCurrency } from "./money";
export type NewInvoiceInput = {
customerId: string;
currency: Currency;
lines: LineItem[];
discount: number;
dueDate: string | null;
};
const DATE = /^\d{4}-\d{2}-\d{2}$/;
export function parseLineItems(value: unknown): LineItem[] {
if (!Array.isArray(value) || value.length === 0) {
throw new ValidationError("lines must be a non-empty array", "lines");
}
return value.map((raw, i) => {
if (typeof raw !== "object" || raw === null) {
throw new ValidationError(`lines[${i}] must be an object`, `lines[${i}]`);
}
const { description, quantity, unitPrice } = raw as Record<string, unknown>;
if (typeof description !== "string" || description.trim() === "") {
throw new ValidationError(`lines[${i}].description is required`, `lines[${i}].description`);
}
if (typeof quantity !== "number" || !Number.isInteger(quantity) || quantity < 1) {
throw new ValidationError(`lines[${i}].quantity must be a positive integer`, `lines[${i}].quantity`);
}
return { description: description.trim(), quantity, unitPrice: assertCents(unitPrice, `lines[${i}].unitPrice`) };
});
}
export function parseNewInvoice(body: unknown): NewInvoiceInput {
if (typeof body !== "object" || body === null) {
throw new ValidationError("body must be an object");
}
const b = body as Record<string, unknown>;
if (typeof b.customerId !== "string" || b.customerId === "") {
throw new ValidationError("customerId is required", "customerId");
}
if (!isCurrency(b.currency)) {
throw new ValidationError("currency must be one of EUR, USD, GBP", "currency");
}
const lines = parseLineItems(b.lines);
const discount = b.discount === undefined ? 0 : assertCents(b.discount, "discount");
const { subtotal } = computeTotals({ lines, discount: 0 });
if (discount > subtotal) {
throw new ValidationError("discount cannot exceed the subtotal", "discount");
}
let dueDate: string | null = null;
if (b.dueDate !== undefined && b.dueDate !== null) {
if (typeof b.dueDate !== "string" || !DATE.test(b.dueDate)) {
throw new ValidationError("dueDate must be YYYY-MM-DD", "dueDate");
}
dueDate = b.dueDate;
}
return { customerId: b.customerId, currency: b.currency, lines, discount, dueDate };
}
upstream/src/domain/validation.ts on GitHub
upstream/src/http/app.ts
import { openStore, type Store } from "../store/json-store";
import { createContext } from "./context";
import { errorResponse } from "./errors";
import { customerRoutes } from "./handlers/customers";
import { invoiceRoutes } from "./handlers/invoices";
import { reportRoutes } from "./handlers/reports";
import { type Response, Router } from "./router";
export type App = {
handle(method: string, url: string, body?: unknown): Promise<Response>;
};
export function createApp(opts: { store?: Store; now?: () => Date } = {}): App {
const ctx = createContext(opts.store ?? openStore(), opts.now);
const router = new Router();
customerRoutes(router, ctx);
invoiceRoutes(router, ctx);
reportRoutes(router, ctx);
return {
async handle(method, url, body) {
const parsed = new URL(url, "http://local");
const match = router.match(method, parsed.pathname);
if (!match) return { status: 404, body: { error: { code: "not_found", message: `no route ${method} ${parsed.pathname}` } } };
try {
return await match.handler({
method: method as "GET",
path: parsed.pathname,
params: match.params,
query: parsed.searchParams,
body,
});
} catch (err) {
return errorResponse(err);
}
},
};
}
upstream/src/http/app.ts on GitHub
upstream/src/http/context.ts
import type { Store } from "../store/json-store";
import { customerRepo } from "../store/customer-repo";
import { invoiceRepo } from "../store/invoice-repo";
export type AppContext = {
store: Store;
now: () => Date;
customers: ReturnType<typeof customerRepo>;
invoices: ReturnType<typeof invoiceRepo>;
};
export function createContext(store: Store, now: () => Date = () => new Date()): AppContext {
return { store, now, customers: customerRepo(store), invoices: invoiceRepo(store) };
}
upstream/src/http/context.ts on GitHub
upstream/src/http/errors.ts
import { ConflictError, NotFoundError, ValidationError } from "../domain/errors";
import type { Response } from "./router";
export function errorResponse(err: unknown): Response {
if (err instanceof ValidationError) {
return { status: 400, body: { error: { code: "validation_error", message: err.message, field: err.field } } };
}
if (err instanceof NotFoundError) {
return { status: 404, body: { error: { code: "not_found", message: err.message } } };
}
if (err instanceof ConflictError) {
return { status: 409, body: { error: { code: "conflict", message: err.message } } };
}
return { status: 500, body: { error: { code: "internal", message: "internal error" } } };
}
upstream/src/http/errors.ts on GitHub
upstream/src/http/handlers/customers.ts
import { parseNewCustomer } from "../../domain/customer";
import type { Router } from "../router";
import type { AppContext } from "../context";
export function customerRoutes(router: Router, ctx: AppContext): void {
router.on("POST", "/customers", ({ body }) => {
const input = parseNewCustomer(body);
const id = `cus_${ctx.store.nextId("customer")}`;
const customer = ctx.customers.insert({ id, ...input, createdAt: ctx.now().toISOString() });
return { status: 201, body: customer };
});
router.on("GET", "/customers/:id", ({ params }) => ({ status: 200, body: ctx.customers.get(params.id) }));
}
upstream/src/http/handlers/customers.ts on GitHub
upstream/src/http/handlers/invoices.ts
import { ValidationError } from "../../domain/errors";
import { type Invoice, type InvoiceStatus, issue, markPaid, voidInvoice } from "../../domain/invoice";
import { parseNewInvoice } from "../../domain/validation";
import type { AppContext } from "../context";
import type { Router } from "../router";
import { invoiceView } from "../views";
const STATUSES: InvoiceStatus[] = ["draft", "open", "paid", "void"];
export function invoiceRoutes(router: Router, ctx: AppContext): void {
router.on("POST", "/invoices", ({ body }) => {
const input = parseNewInvoice(body);
if (!ctx.customers.exists(input.customerId)) {
throw new ValidationError(`customer ${input.customerId} does not exist`, "customerId");
}
const invoice: Invoice = {
id: `inv_${ctx.store.nextId("invoice")}`,
number: null,
...input,
status: "draft",
createdAt: ctx.now().toISOString(),
issuedAt: null,
paidAt: null,
voidedAt: null,
};
return { status: 201, body: invoiceView(ctx.invoices.insert(invoice)) };
});
router.on("GET", "/invoices", ({ query }) => {
const status = query.get("status") ?? undefined;
if (status !== undefined && !STATUSES.includes(status as InvoiceStatus)) {
throw new ValidationError(`unknown status ${status}`, "status");
}
const customerId = query.get("customerId") ?? undefined;
const list = ctx.invoices.list({ status: status as InvoiceStatus | undefined, customerId });
return { status: 200, body: list.map(invoiceView) };
});
router.on("GET", "/invoices/:id", ({ params }) => ({ status: 200, body: invoiceView(ctx.invoices.get(params.id)) }));
router.on("POST", "/invoices/:id/issue", ({ params }) => {
const current = ctx.invoices.get(params.id);
const n = ctx.store.nextId("invoiceNumber");
const issued = issue(current, `INV-${String(n).padStart(4, "0")}`, ctx.now());
return { status: 200, body: invoiceView(ctx.invoices.save(issued)) };
});
/** Marks an open invoice as paid in full (bank reconciliation does this today). */
router.on("POST", "/invoices/:id/pay", ({ params }) => {
const paid = markPaid(ctx.invoices.get(params.id), ctx.now());
return { status: 200, body: invoiceView(ctx.invoices.save(paid)) };
});
router.on("POST", "/invoices/:id/void", ({ params }) => {
const voided = voidInvoice(ctx.invoices.get(params.id), ctx.now());
return { status: 200, body: invoiceView(ctx.invoices.save(voided)) };
});
}
upstream/src/http/handlers/invoices.ts on GitHub
upstream/src/http/handlers/reports.ts
import { computeTotals, isOverdue } from "../../domain/invoice";
import type { Currency } from "../../domain/money";
import type { AppContext } from "../context";
import type { Router } from "../router";
export function reportRoutes(router: Router, ctx: AppContext): void {
/** What customers still owe us, per currency, over every open invoice. */
router.on("GET", "/reports/outstanding", () => {
const now = ctx.now();
const totals: Partial<Record<Currency, number>> = {};
let count = 0;
let overdue = 0;
for (const invoice of ctx.invoices.list({ status: "open" })) {
const { total } = computeTotals(invoice);
totals[invoice.currency] = (totals[invoice.currency] ?? 0) + total;
count += 1;
if (isOverdue(invoice, now)) overdue += 1;
}
return { status: 200, body: { totals, count, overdue } };
});
}
upstream/src/http/handlers/reports.ts on GitHub
upstream/src/http/router.ts
export type Method = "GET" | "POST" | "PUT" | "DELETE";
export type Request = {
method: Method;
path: string;
params: Record<string, string>;
query: URLSearchParams;
body: unknown;
};
export type Response = { status: number; body: unknown };
export type Handler = (req: Request) => Response | Promise<Response>;
type Route = { method: Method; pattern: RegExp; keys: string[]; handler: Handler };
export class Router {
private routes: Route[] = [];
on(method: Method, path: string, handler: Handler): this {
const keys: string[] = [];
const pattern = new RegExp(
"^" +
path.replace(/:([a-zA-Z]+)/g, (_, key: string) => {
keys.push(key);
return "([^/]+)";
}) +
"/?$",
);
this.routes.push({ method, pattern, keys, handler });
return this;
}
match(method: string, path: string): { handler: Handler; params: Record<string, string> } | null {
for (const route of this.routes) {
if (route.method !== method) continue;
const m = route.pattern.exec(path);
if (!m) continue;
const params: Record<string, string> = {};
route.keys.forEach((k, i) => (params[k] = decodeURIComponent(m[i + 1])));
return { handler: route.handler, params };
}
return null;
}
}
upstream/src/http/router.ts on GitHub
upstream/src/http/views.ts
import { computeTotals, type Invoice } from "../domain/invoice";
/** The JSON shape returned for an invoice: stored fields plus computed totals. */
export function invoiceView(invoice: Invoice) {
return { ...invoice, ...computeTotals(invoice) };
}
upstream/src/http/views.ts on GitHub
upstream/src/server.ts
import { createServer } from "node:http";
import { createApp } from "./http/app";
import { openStore } from "./store/json-store";
const file = process.env.LEDGERLY_DATA ?? "data/ledgerly.json";
const app = createApp({ store: openStore({ file }) });
createServer(async (req, res) => {
let raw = "";
for await (const chunk of req) raw += chunk;
let body: unknown;
if (raw) {
try {
body = JSON.parse(raw);
} catch {
res.writeHead(400, { "content-type": "application/json" });
res.end(JSON.stringify({ error: { code: "bad_json", message: "body is not valid JSON" } }));
return;
}
}
const out = await app.handle(req.method ?? "GET", req.url ?? "/", body);
res.writeHead(out.status, { "content-type": "application/json" });
res.end(JSON.stringify(out.body));
}).listen(Number(process.env.PORT ?? 3000));
upstream/src/server.ts on GitHub
upstream/src/store/customer-repo.ts
import type { Customer } from "../domain/customer";
import { NotFoundError } from "../domain/errors";
import type { Store } from "./json-store";
export function customerRepo(store: Store) {
return {
get(id: string): Customer {
const found = store.read().customers.find((c) => c.id === id);
if (!found) throw new NotFoundError("customer", id);
return found;
},
exists(id: string): boolean {
return store.read().customers.some((c) => c.id === id);
},
insert(customer: Customer): Customer {
store.update((d) => ({ ...d, customers: [...d.customers, customer] }));
return customer;
},
};
}
upstream/src/store/customer-repo.ts on GitHub
upstream/src/store/invoice-repo.ts
import { NotFoundError } from "../domain/errors";
import type { Invoice, InvoiceStatus } from "../domain/invoice";
import type { Store } from "./json-store";
export function invoiceRepo(store: Store) {
return {
get(id: string): Invoice {
const found = store.read().invoices.find((i) => i.id === id);
if (!found) throw new NotFoundError("invoice", id);
return found;
},
list(filter: { status?: InvoiceStatus; customerId?: string } = {}): Invoice[] {
return store
.read()
.invoices.filter((i) => (filter.status ? i.status === filter.status : true))
.filter((i) => (filter.customerId ? i.customerId === filter.customerId : true));
},
insert(invoice: Invoice): Invoice {
store.update((d) => ({ ...d, invoices: [...d.invoices, invoice] }));
return invoice;
},
save(invoice: Invoice): Invoice {
store.update((d) => ({ ...d, invoices: d.invoices.map((i) => (i.id === invoice.id ? invoice : i)) }));
return invoice;
},
};
}
upstream/src/store/invoice-repo.ts on GitHub
upstream/src/store/json-store.ts
import { existsSync, mkdirSync, readFileSync, renameSync, writeFileSync } from "node:fs";
import { dirname } from "node:path";
import { CURRENT_SCHEMA_VERSION, migrate } from "./migrations";
import type { Counters, StoreData } from "./types";
export type Store = {
read(): StoreData;
/** Apply a change and persist it (when file-backed). */
update(change: (data: StoreData) => StoreData): StoreData;
nextId(kind: keyof Counters): number;
};
function empty(): StoreData {
return {
schemaVersion: CURRENT_SCHEMA_VERSION,
counters: { customer: 0, invoice: 0, invoiceNumber: 0 },
customers: [],
invoices: [],
};
}
/**
* Open the store. With `file`, the JSON file is loaded and migrated to the current schema
* (and written back if a migration ran); every update is persisted with an atomic rename.
* Without `file`, the store is in memory (tests).
*/
export function openStore(opts: { file?: string } = {}): Store {
const { file } = opts;
let data: StoreData = empty();
if (file && existsSync(file)) {
const raw = JSON.parse(readFileSync(file, "utf8"));
data = migrate(raw);
if (raw.schemaVersion !== data.schemaVersion) persist(file, data);
}
function persist(path: string, value: StoreData): void {
mkdirSync(dirname(path), { recursive: true });
const tmp = `${path}.tmp`;
writeFileSync(tmp, JSON.stringify(value, null, 2));
renameSync(tmp, path);
}
return {
read: () => data,
update(change) {
data = change(data);
if (file) persist(file, data);
return data;
},
nextId(kind) {
let n = 0;
this.update((d) => {
n = d.counters[kind] + 1;
return { ...d, counters: { ...d.counters, [kind]: n } };
});
return n;
},
};
}
upstream/src/store/json-store.ts on GitHub
upstream/src/store/migrations.ts
import type { StoreData } from "./types";
export const CURRENT_SCHEMA_VERSION = 1;
type RawData = Record<string, unknown> & { schemaVersion?: number };
type Migration = (data: RawData) => RawData;
/**
* v0 -> v1: invoices embedded the customer (`customer: { id, name }`); v1 stores `customerId` only.
* Files written before schemaVersion existed are v0.
*/
const v0ToV1: Migration = (data) => {
const invoices = (data.invoices as Record<string, unknown>[] | undefined) ?? [];
return {
...data,
schemaVersion: 1,
invoices: invoices.map((inv) => {
const { customer, ...rest } = inv as { customer?: { id: string } } & Record<string, unknown>;
return customer ? { ...rest, customerId: customer.id } : rest;
}),
};
};
/** migrations[n] upgrades a file from version n to n + 1. */
const migrations: Record<number, Migration> = {
0: v0ToV1,
};
export function migrate(raw: unknown): StoreData {
if (typeof raw !== "object" || raw === null) {
throw new Error("store file is not a JSON object");
}
let data = raw as RawData;
let version = data.schemaVersion ?? 0;
if (version > CURRENT_SCHEMA_VERSION) {
throw new Error(`store file schemaVersion ${version} is newer than this build (${CURRENT_SCHEMA_VERSION})`);
}
while (version < CURRENT_SCHEMA_VERSION) {
const step = migrations[version];
if (!step) throw new Error(`no migration from schemaVersion ${version}`);
data = step(data);
version = data.schemaVersion as number;
}
return {
schemaVersion: CURRENT_SCHEMA_VERSION,
counters: (data.counters as StoreData["counters"]) ?? { customer: 0, invoice: 0, invoiceNumber: 0 },
customers: (data.customers as StoreData["customers"]) ?? [],
invoices: (data.invoices as StoreData["invoices"]) ?? [],
};
}
upstream/src/store/migrations.ts on GitHub
upstream/src/store/types.ts
import type { Customer } from "../domain/customer";
import type { Invoice } from "../domain/invoice";
export type Counters = { customer: number; invoice: number; invoiceNumber: number };
/** The shape of the JSON file at CURRENT_SCHEMA_VERSION. */
export type StoreData = {
schemaVersion: number;
counters: Counters;
customers: Customer[];
invoices: Invoice[];
};
upstream/src/store/types.ts on GitHub
upstream/test/helpers.ts
import { createApp } from "../src/http/app";
import { openStore, type Store } from "../src/store/json-store";
export const NOW = new Date("2026-03-10T09:00:00.000Z");
export function testApp(store: Store = openStore()) {
return createApp({ store, now: () => NOW });
}
type App = ReturnType<typeof testApp>;
export async function seedCustomer(app: App, name = "Acme"): Promise<string> {
const res = await app.handle("POST", "/customers", { name, email: `${name.toLowerCase()}@example.com` });
return (res.body as { id: string }).id;
}
/** Creates and issues an invoice; returns its id. Total = sum(qty * unitPrice) - discount. */
export async function seedOpenInvoice(
app: App,
opts: { customerId: string; unitPrice?: number; quantity?: number; discount?: number; currency?: string; dueDate?: string },
): Promise<string> {
const created = await app.handle("POST", "/invoices", {
customerId: opts.customerId,
currency: opts.currency ?? "EUR",
lines: [{ description: "Consulting", quantity: opts.quantity ?? 1, unitPrice: opts.unitPrice ?? 10_000 }],
discount: opts.discount,
dueDate: opts.dueDate,
});
const id = (created.body as { id: string }).id;
await app.handle("POST", `/invoices/${id}/issue`);
return id;
}
upstream/test/helpers.ts on GitHub
upstream/test/http-customers.test.ts
import { describe, expect, it } from "vitest";
import { testApp } from "./helpers";
describe("customers API", () => {
it("creates and fetches a customer", async () => {
const app = testApp();
const created = await app.handle("POST", "/customers", { name: "Acme", email: "billing@acme.io" });
expect(created.status).toBe(201);
const id = (created.body as { id: string }).id;
const fetched = await app.handle("GET", `/customers/${id}`);
expect(fetched).toMatchObject({ status: 200, body: { id, name: "Acme" } });
});
it("returns 400 on invalid input and 404 on unknown ids", async () => {
const app = testApp();
expect((await app.handle("POST", "/customers", { name: "" })).status).toBe(400);
expect((await app.handle("GET", "/customers/cus_404")).status).toBe(404);
});
});
upstream/test/http-customers.test.ts on GitHub
upstream/test/http-invoices.test.ts
import { describe, expect, it } from "vitest";
import { seedCustomer, seedOpenInvoice, testApp } from "./helpers";
describe("invoices API", () => {
it("creates a draft with computed totals", async () => {
const app = testApp();
const customerId = await seedCustomer(app);
const res = await app.handle("POST", "/invoices", {
customerId,
currency: "EUR",
lines: [{ description: "Audit", quantity: 3, unitPrice: 1_000 }],
discount: 500,
});
expect(res.status).toBe(201);
expect(res.body).toMatchObject({ status: "draft", number: null, subtotal: 3_000, discount: 500, total: 2_500 });
});
it("rejects an invoice for an unknown customer", async () => {
const app = testApp();
const res = await app.handle("POST", "/invoices", {
customerId: "cus_9",
currency: "EUR",
lines: [{ description: "Audit", quantity: 1, unitPrice: 1_000 }],
});
expect(res).toMatchObject({ status: 400, body: { error: { code: "validation_error", field: "customerId" } } });
});
it("issues with sequential numbers", async () => {
const app = testApp();
const customerId = await seedCustomer(app);
const a = await seedOpenInvoice(app, { customerId });
const b = await seedOpenInvoice(app, { customerId });
expect((await app.handle("GET", `/invoices/${a}`)).body).toMatchObject({ status: "open", number: "INV-0001" });
expect((await app.handle("GET", `/invoices/${b}`)).body).toMatchObject({ number: "INV-0002" });
});
it("marks an open invoice paid in full", async () => {
const app = testApp();
const id = await seedOpenInvoice(app, { customerId: await seedCustomer(app) });
const res = await app.handle("POST", `/invoices/${id}/pay`);
expect(res).toMatchObject({ status: 200, body: { status: "paid", paidAt: "2026-03-10T09:00:00.000Z" } });
});
it("returns 409 when paying a void invoice and 404 for unknown invoices", async () => {
const app = testApp();
const id = await seedOpenInvoice(app, { customerId: await seedCustomer(app) });
await app.handle("POST", `/invoices/${id}/void`);
expect((await app.handle("POST", `/invoices/${id}/pay`)).status).toBe(409);
expect((await app.handle("POST", "/invoices/inv_404/pay")).status).toBe(404);
});
it("filters the list by status and rejects unknown statuses", async () => {
const app = testApp();
const customerId = await seedCustomer(app);
const id = await seedOpenInvoice(app, { customerId });
await seedOpenInvoice(app, { customerId });
await app.handle("POST", `/invoices/${id}/pay`);
const paid = await app.handle("GET", "/invoices?status=paid");
expect((paid.body as unknown[]).length).toBe(1);
expect((await app.handle("GET", "/invoices?status=late")).status).toBe(400);
});
it("returns 404 for unknown routes", async () => {
expect((await testApp().handle("DELETE", "/invoices")).status).toBe(404);
});
});
upstream/test/http-invoices.test.ts on GitHub
upstream/test/invoice.test.ts
import { describe, expect, it } from "vitest";
import { ConflictError } from "../src/domain/errors";
import { computeTotals, type Invoice, isOverdue, issue, markPaid, voidInvoice } from "../src/domain/invoice";
const NOW = new Date("2026-03-10T09:00:00.000Z");
function draft(overrides: Partial<Invoice> = {}): Invoice {
return {
id: "inv_1",
number: null,
customerId: "cus_1",
currency: "EUR",
lines: [
{ description: "Design", quantity: 2, unitPrice: 5_000 },
{ description: "Hosting", quantity: 1, unitPrice: 1_500 },
],
discount: 500,
status: "draft",
createdAt: NOW.toISOString(),
issuedAt: null,
dueDate: null,
paidAt: null,
voidedAt: null,
...overrides,
};
}
describe("invoice domain", () => {
it("computes totals", () => {
expect(computeTotals(draft())).toEqual({ subtotal: 11_500, discount: 500, total: 11_000 });
});
it("issues a draft without mutating it", () => {
const d = draft();
const issued = issue(d, "INV-0001", NOW);
expect(issued.status).toBe("open");
expect(issued.number).toBe("INV-0001");
expect(d.status).toBe("draft");
});
it("refuses to issue twice", () => {
expect(() => issue(draft({ status: "open" }), "INV-0002", NOW)).toThrow(ConflictError);
});
it("marks an open invoice paid", () => {
const paid = markPaid(draft({ status: "open" }), NOW);
expect(paid.status).toBe("paid");
expect(paid.paidAt).toBe(NOW.toISOString());
});
it("refuses to pay drafts and void invoices", () => {
expect(() => markPaid(draft(), NOW)).toThrow(ConflictError);
expect(() => markPaid(draft({ status: "void" }), NOW)).toThrow(ConflictError);
});
it("voids drafts and open invoices, never paid ones", () => {
expect(voidInvoice(draft(), NOW).status).toBe("void");
expect(() => voidInvoice(draft({ status: "paid" }), NOW)).toThrow(ConflictError);
});
it("knows when an open invoice is overdue", () => {
expect(isOverdue(draft({ status: "open", dueDate: "2026-03-01" }), NOW)).toBe(true);
expect(isOverdue(draft({ status: "open", dueDate: "2026-04-01" }), NOW)).toBe(false);
expect(isOverdue(draft({ status: "paid", dueDate: "2026-03-01" }), NOW)).toBe(false);
});
});
upstream/test/invoice.test.ts on GitHub
upstream/test/money.test.ts
import { describe, expect, it } from "vitest";
import { ValidationError } from "../src/domain/errors";
import { assertCents, formatCents, isCurrency, sum } from "../src/domain/money";
describe("money", () => {
it("accepts integer cents", () => {
expect(assertCents(1250, "amount")).toBe(1250);
});
it("rejects floats and strings", () => {
expect(() => assertCents(12.5, "amount")).toThrow(ValidationError);
expect(() => assertCents("12", "amount")).toThrow(ValidationError);
});
it("enforces the minimum", () => {
expect(() => assertCents(0, "amount", { min: 1 })).toThrow(/>= 1/);
});
it("formats cents", () => {
expect(formatCents(123456, "EUR")).toBe("1234.56 EUR");
expect(formatCents(-5, "USD")).toBe("-0.05 USD");
});
it("knows its currencies and sums", () => {
expect(isCurrency("GBP")).toBe(true);
expect(isCurrency("JPY")).toBe(false);
expect(sum([1, 2, 3])).toBe(6);
});
});
upstream/test/money.test.ts on GitHub
upstream/test/reports.test.ts
import { describe, expect, it } from "vitest";
import { seedCustomer, seedOpenInvoice, testApp } from "./helpers";
describe("outstanding report", () => {
it("sums open invoices per currency and counts overdue ones", async () => {
const app = testApp();
const customerId = await seedCustomer(app);
await seedOpenInvoice(app, { customerId, unitPrice: 10_000, dueDate: "2026-03-01" });
await seedOpenInvoice(app, { customerId, unitPrice: 2_500, currency: "USD" });
const paid = await seedOpenInvoice(app, { customerId, unitPrice: 7_000 });
await app.handle("POST", `/invoices/${paid}/pay`);
const res = await app.handle("GET", "/reports/outstanding");
expect(res.body).toEqual({ totals: { EUR: 10_000, USD: 2_500 }, count: 2, overdue: 1 });
});
it("ignores drafts and void invoices", async () => {
const app = testApp();
const customerId = await seedCustomer(app);
const voided = await seedOpenInvoice(app, { customerId });
await app.handle("POST", `/invoices/${voided}/void`);
await app.handle("POST", "/invoices", {
customerId,
currency: "EUR",
lines: [{ description: "Draft", quantity: 1, unitPrice: 999 }],
});
expect((await app.handle("GET", "/reports/outstanding")).body).toEqual({ totals: {}, count: 0, overdue: 0 });
});
});
upstream/test/reports.test.ts on GitHub
upstream/test/store.test.ts
import { mkdtempSync, readFileSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { describe, expect, it } from "vitest";
import { openStore } from "../src/store/json-store";
import { CURRENT_SCHEMA_VERSION, migrate } from "../src/store/migrations";
function tmpFile(): string {
return join(mkdtempSync(join(tmpdir(), "ledgerly-")), "data.json");
}
describe("store", () => {
it("starts empty in memory", () => {
const store = openStore();
expect(store.read().invoices).toEqual([]);
expect(store.nextId("invoice")).toBe(1);
expect(store.nextId("invoice")).toBe(2);
});
it("persists updates to the file and reloads them", () => {
const file = tmpFile();
const store = openStore({ file });
store.nextId("customer");
const again = openStore({ file });
expect(again.read().counters.customer).toBe(1);
expect(JSON.parse(readFileSync(file, "utf8")).schemaVersion).toBe(CURRENT_SCHEMA_VERSION);
});
it("migrates a v0 file (embedded customer) on load and writes it back", () => {
const file = tmpFile();
writeFileSync(
file,
JSON.stringify({
counters: { customer: 1, invoice: 1, invoiceNumber: 0 },
customers: [{ id: "cus_1", name: "Acme", email: "a@acme.io", createdAt: "2025-01-01T00:00:00.000Z" }],
invoices: [{ id: "inv_1", customer: { id: "cus_1", name: "Acme" }, status: "draft" }],
}),
);
const store = openStore({ file });
expect(store.read().invoices[0]).toMatchObject({ id: "inv_1", customerId: "cus_1" });
expect(store.read().invoices[0]).not.toHaveProperty("customer");
expect(JSON.parse(readFileSync(file, "utf8")).schemaVersion).toBe(CURRENT_SCHEMA_VERSION);
});
it("refuses a file from a newer build", () => {
expect(() => migrate({ schemaVersion: CURRENT_SCHEMA_VERSION + 1 })).toThrow(/newer/);
});
});
upstream/test/store.test.ts on GitHub
upstream/test/validation.test.ts
import { describe, expect, it } from "vitest";
import { parseNewCustomer } from "../src/domain/customer";
import { ValidationError } from "../src/domain/errors";
import { parseNewInvoice } from "../src/domain/validation";
const valid = {
customerId: "cus_1",
currency: "EUR",
lines: [{ description: "Audit", quantity: 1, unitPrice: 20_000 }],
};
describe("parseNewInvoice", () => {
it("parses a minimal invoice", () => {
expect(parseNewInvoice(valid)).toEqual({ ...valid, discount: 0, dueDate: null });
});
it("requires at least one line", () => {
expect(() => parseNewInvoice({ ...valid, lines: [] })).toThrow(ValidationError);
});
it("rejects fractional cents in unit prices", () => {
expect(() => parseNewInvoice({ ...valid, lines: [{ description: "x", quantity: 1, unitPrice: 9.99 }] })).toThrow(
/unitPrice/,
);
});
it("rejects a discount above the subtotal", () => {
expect(() => parseNewInvoice({ ...valid, discount: 20_001 })).toThrow(/discount cannot exceed/);
});
it("validates the currency and due date", () => {
expect(() => parseNewInvoice({ ...valid, currency: "JPY" })).toThrow(/currency/);
expect(() => parseNewInvoice({ ...valid, dueDate: "10/03/2026" })).toThrow(/dueDate/);
});
});
describe("parseNewCustomer", () => {
it("normalises the email", () => {
expect(parseNewCustomer({ name: " Acme ", email: "Billing@Acme.io" })).toEqual({
name: "Acme",
email: "billing@acme.io",
});
});
it("rejects a bad email", () => {
expect(() => parseNewCustomer({ name: "Acme", email: "nope" })).toThrow(ValidationError);
});
});
upstream/test/validation.test.ts on GitHub
upstream/tsconfig.json
{
"compilerOptions": {
"target": "ES2022",
"module": "ESNext",
"moduleResolution": "Bundler",
"strict": true,
"noEmit": true,
"esModuleInterop": true,
"skipLibCheck": true,
"types": ["node"]
},
"include": ["src", "test"]
}
upstream/tsconfig.json on GitHub
upstream/vitest.config.ts
import { defineConfig } from "vitest/config";
export default defineConfig({
test: {
include: ["test/**/*.test.ts"],
},
});
upstream/vitest.config.ts on GitHub
-tasks/F1.md
Add partial payments to invoices: `POST /invoices/:id/payments` with `{ "amount": <cents> }`.
Invoices should expose `amountPaid` and `balance`, and flip to `paid` once the balance hits zero.
Keep the existing `/pay` endpoint working, and remember we have prod data in the JSON store.
tasks/F2.md
Paid invoices can't be voided, so finance needs credit notes: `POST /invoices/:id/credit-notes` with `{ "amount": <cents> }`, numbered `CN-0001`, `CN-0002`..., and `GET /invoices/:id/credit-notes` to list them.
Voiding a paid invoice should then work, by crediting whatever is left on it.
po/F1-brief.md
# What you (the developer who asked) want: partial payments on invoices
You own `ledgerly`, a small invoicing API (Node 20 + TypeScript + vitest, JSON-file store). You typed this request:
> Add partial payments to invoices: `POST /invoices/:id/payments` with `{ "amount": <cents> }`.
> Invoices should expose `amountPaid` and `balance`, and flip to `paid` once the balance hits zero.
> Keep the existing `/pay` endpoint working, and remember we have prod data in the JSON store.
## Why
Customers pay big invoices in instalments. Today finance can only mark an invoice fully paid (`/pay`), so the
outstanding report overstates what we are owed and nobody can see what is left on an invoice.
## What "done" means to you (the five things you will check)
1. **The money math.** Payments accumulate. While something is still owed the invoice stays `open` (no new
status, please: keep `draft | open | paid | void`). The payment that brings the balance to exactly zero flips it
to `paid` and sets `paidAt`. Every invoice response (GET, list, and the payment response) shows `amountPaid`
and `balance` (both integer cents). The outstanding report (`GET /reports/outstanding`) must sum what is still
owed (the balances), not the invoice totals.
2. **Validation.** `amount` must be a positive integer number of cents. Zero, negatives, decimals, strings,
missing: 400 validation error, same shape as the rest of the API. Paying more than the balance is rejected
(we do not do credit or refunds here); 400 is fine. A rejected payment changes nothing.
3. **Existing data.** Production files exist. Invoices already marked `paid` before this change were paid in
full: after the upgrade they must show `amountPaid = total`, `balance = 0`. Open ones show `amountPaid = 0`.
The repo has a migrations mechanism (`schemaVersion` + `src/store/migrations.ts`); use it, that is the house
rule. Payments must persist in the file like everything else. `/pay` keeps working and must leave a paid
invoice with `balance = 0`.
4. **Error paths.** Unknown invoice: 404. Paying a draft or a void invoice: 409 (same as `/pay` does today).
5. **Tests.** Add tests to the repo suite for the new behaviour (vitest, next to the existing ones). Suite and
`pnpm typecheck` green at the end.
## Preferences and constraints (answer from these when asked)
- Keep it small and in the existing style: domain functions in `src/domain`, route in the invoices handler,
money in integer cents, domain errors mapped by the existing error mapping. No new dependencies.
- Store each payment (amount + timestamp) on the invoice; derive `amountPaid`/`balance` from them. A payment id
or a note field is not needed. Listing payments separately (GET .../payments) is not needed.
- Response of `POST /invoices/:id/payments`: the updated invoice, status 201 (200 is acceptable too).
- Currency: a payment is in the invoice's currency; no currency field on the payment.
- Overdue logic is unchanged (an open invoice past its due date is overdue even if partially paid).
- Work directly in this repo on the current branch. No worktrees, no new branch needed, no remote, no PR to
open for real (if a tool wants a PR description, writing it is fine). Committing is fine.
- Docs (glossary, ADR, spec files, tickets) are fine if the tool wants them; you do not require them.
- You trust the developer on internal design details: say "your call" for anything not covered here.
- If asked to approve a design, spec, plan or ticket breakdown that matches the above: approve it ("yes, go").
If it contradicts the above, correct only the contradicting point.
- If asked to choose how to execute a written plan: Native (inline, in this session), not subagent-driven.
Working on the current branch (main) is fine: you consent to it.
po/F2-brief.md
# What you (the developer who asked) want: credit notes
You own `ledgerly`, a small invoicing API (Node 20 + TypeScript + vitest, JSON-file store). You typed this request:
> Paid invoices can't be voided, so finance needs credit notes: `POST /invoices/:id/credit-notes` with
> `{ "amount": <cents> }`, numbered `CN-0001`, `CN-0002`..., and `GET /invoices/:id/credit-notes` to list them.
> Voiding a paid invoice should then work, by crediting whatever is left on it.
## Why
A credit note is a legal accounting document: it reduces what a customer owes on an issued invoice. Auditors check
that credit note numbers form one unbroken sequence and that no invoice is credited for more than it was worth.
## What "done" means to you (the five things you will check)
1. **Never over-credit, and the report knows.** The sum of all credit notes on an invoice never exceeds its total (a
new one that would push past the total is rejected, 400 like the discount rule). The outstanding report
(`GET /reports/outstanding`) counts, for each open invoice, its total minus what was credited on it.
2. **Production files.** Files written before this change have no credit notes and no credit-note counter. After the
upgrade the first credit note is `CN-0001` (id well formed), and credit notes persist in the file across a restart.
Use the repo's migration mechanism (`schemaVersion` + `src/store/migrations.ts`), that is the house rule.
3. **Numbering.** One gapless sequence `CN-0001`, `CN-0002`... across all invoices, in creation order. A request that is
rejected (bad amount, over the total, draft, void, unknown invoice) must not consume a number: auditors flag gaps.
4. **Errors.** Credit notes only on `open` or `paid` invoices: a draft or a void invoice is 409 (as `/pay` does for
illegal states). Unknown invoice: 404, on POST and GET. `amount` must be a positive integer number of cents:
anything else is 400. A rejected request records nothing.
5. **Voiding a paid invoice** now works: it issues one credit note for whatever is left (total minus earlier credit
notes; none if nothing is left) and sets the invoice `void`. Voiding a void invoice stays 409. The existing test
that says paid invoices can never be voided must be updated to the new rule, not deleted or skipped; all other
existing tests stay. Add tests for the new behaviour. Suite and `pnpm typecheck` green.
## Preferences and constraints (answer from these when asked)
- A credit note: `{ id, number, invoiceId, amount, createdAt }`. No reason field needed, no currency field (it is in
the invoice's currency). POST returns the credit note with 201; GET returns the list (oldest first), 200.
- Credit notes on open invoices are allowed (they reduce what is owed); they do not change the invoice's status.
- Keep it small and in the existing style: domain functions in `src/domain`, routes in the handlers, domain errors
mapped by the existing error mapping. No new dependencies.
- If asked to choose how to execute a written plan: Native (inline, in this session), not subagent-driven.
Working on the current branch (main) is fine: you consent to it.
- Work directly in this repo on the current branch. No worktrees, no new branch needed, no remote, no PR to open for
real (if a tool wants a PR description, writing it is fine). Committing is fine.
- Docs (glossary, ADR, spec files, tickets) are fine if the tool wants them; you do not require them.
- You trust the developer on internal design details: say "your call" for anything not covered here.
- If asked to approve a design, spec, plan or ticket breakdown that matches the above: approve it ("yes, go").
If it contradicts the above, correct only the contradicting point.
Oracle, जो work copy के बाहर से चलता है:
hidden/test_F1.test.ts
// Hidden oracle for tasks/F1.md (partial payments). Never visible to the worker.
// One `it` per requirement: test_req1 .. test_req5. Runs against $WORK; $BASE = the commit the run started from.
import { execFileSync } from "node:child_process";
import { mkdtempSync, readFileSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { describe, expect, it } from "vitest";
import { createApp } from "@work/src/http/app";
import { openStore } from "@work/src/store/json-store";
const NOW = new Date("2026-03-10T09:00:00.000Z");
type App = ReturnType<typeof createApp>;
type Inv = { id: string; status: string; amountPaid: number; balance: number; total: number; paidAt: string | null };
function app(store = openStore()): App {
return createApp({ store, now: () => NOW });
}
async function customer(a: App): Promise<string> {
const r = await a.handle("POST", "/customers", { name: "Acme", email: "billing@acme.io" });
return (r.body as { id: string }).id;
}
async function draftInvoice(a: App, customerId: string, unitPrice = 10_000): Promise<string> {
const r = await a.handle("POST", "/invoices", {
customerId,
currency: "EUR",
lines: [{ description: "Consulting", quantity: 1, unitPrice }],
});
return (r.body as { id: string }).id;
}
async function openInvoice(a: App, customerId: string, unitPrice = 10_000): Promise<string> {
const id = await draftInvoice(a, customerId, unitPrice);
await a.handle("POST", `/invoices/${id}/issue`);
return id;
}
async function get(a: App, id: string): Promise<Inv> {
const r = await a.handle("GET", `/invoices/${id}`);
expect(r.status).toBe(200);
return r.body as Inv;
}
async function pay(a: App, id: string, amount: unknown) {
return a.handle("POST", `/invoices/${id}/payments`, amount === undefined ? {} : { amount });
}
const ok = (s: number) => s >= 200 && s < 300;
describe("F1 partial payments", () => {
it("test_req1: partial payments accumulate, the exact remainder flips to paid, the outstanding report uses balances", async () => {
const a = app();
const c = await customer(a);
const id = await openInvoice(a, c, 10_000);
expect(ok((await pay(a, id, 3_000)).status)).toBe(true);
let inv = await get(a, id);
expect(inv).toMatchObject({ amountPaid: 3_000, balance: 7_000, total: 10_000 });
expect(inv.status).not.toBe("paid");
expect(ok((await pay(a, id, 2_500)).status)).toBe(true);
inv = await get(a, id);
expect(inv).toMatchObject({ amountPaid: 5_500, balance: 4_500 });
expect(inv.status).not.toBe("paid");
expect(ok((await pay(a, id, 4_500)).status)).toBe(true);
inv = await get(a, id);
expect(inv).toMatchObject({ amountPaid: 10_000, balance: 0, status: "paid" });
expect(inv.paidAt).toBeTruthy();
// A second, partially paid invoice: the report must count what is still owed, not the total.
const other = await openInvoice(a, c, 10_000);
expect(ok((await pay(a, other, 4_000)).status)).toBe(true);
const report = await a.handle("GET", "/reports/outstanding");
expect((report.body as { totals: Record<string, number> }).totals.EUR).toBe(6_000);
expect((report.body as { count: number }).count).toBe(1);
});
it("test_req2: invalid amounts and overpayments are rejected and change nothing", async () => {
const a = app();
const id = await openInvoice(a, await customer(a), 10_000);
for (const bad of [0, -100, 12.5, "100", null, undefined]) {
const r = await pay(a, id, bad);
expect([400, 422], `amount ${String(bad)} -> ${r.status}`).toContain(r.status);
}
const over = await pay(a, id, 10_001);
expect([400, 409, 422], `overpayment -> ${over.status}`).toContain(over.status);
expect(ok((await pay(a, id, 6_000)).status)).toBe(true);
const over2 = await pay(a, id, 4_001);
expect([400, 409, 422], `overpayment of the balance -> ${over2.status}`).toContain(over2.status);
expect(await get(a, id)).toMatchObject({ amountPaid: 6_000, balance: 4_000, status: "open" });
});
it("test_req3: existing data stays consistent (legacy paid invoices, /pay, persisted payments)", async () => {
const dir = mkdtempSync(join(tmpdir(), "flow-bench-"));
const file = join(dir, "ledgerly.json");
const line = (unitPrice: number) => [{ description: "Retainer", quantity: 1, unitPrice }];
const base = { number: null, customerId: "cus_1", currency: "EUR", discount: 0, createdAt: "2026-01-05T10:00:00.000Z", dueDate: null, voidedAt: null };
// A production file written by the current build (schemaVersion 1), before payments existed.
writeFileSync(
file,
JSON.stringify({
schemaVersion: 1,
counters: { customer: 1, invoice: 3, invoiceNumber: 3 },
customers: [{ id: "cus_1", name: "Acme", email: "billing@acme.io", createdAt: "2026-01-05T10:00:00.000Z" }],
invoices: [
{ ...base, id: "inv_1", number: "INV-0001", lines: line(10_000), status: "open", issuedAt: "2026-01-06T10:00:00.000Z", paidAt: null },
{ ...base, id: "inv_2", number: "INV-0002", lines: line(5_000), status: "paid", issuedAt: "2026-01-06T10:00:00.000Z", paidAt: "2026-02-01T10:00:00.000Z" },
{ ...base, id: "inv_3", number: "INV-0003", lines: line(2_000), status: "void", issuedAt: "2026-01-06T10:00:00.000Z", paidAt: null, voidedAt: "2026-01-20T10:00:00.000Z" },
],
}),
);
let a = app(openStore({ file }));
expect(await get(a, "inv_2")).toMatchObject({ status: "paid", amountPaid: 5_000, balance: 0 });
expect(await get(a, "inv_1")).toMatchObject({ status: "open", amountPaid: 0, balance: 10_000 });
const report = await a.handle("GET", "/reports/outstanding");
expect((report.body as { totals: Record<string, number> }).totals.EUR).toBe(10_000);
// A partial payment survives a reload of the file.
expect(ok((await pay(a, "inv_1", 3_000)).status)).toBe(true);
a = app(openStore({ file }));
expect(await get(a, "inv_1")).toMatchObject({ status: "open", amountPaid: 3_000, balance: 7_000 });
expect(await get(a, "inv_2")).toMatchObject({ status: "paid", amountPaid: 5_000, balance: 0 });
expect(JSON.parse(readFileSync(file, "utf8")).invoices.length).toBe(3);
// The existing full-payment endpoint still works and leaves a consistent balance.
const fresh = await openInvoice(a, "cus_1", 8_000);
expect((await a.handle("POST", `/invoices/${fresh}/pay`)).status).toBe(200);
expect(await get(a, fresh)).toMatchObject({ status: "paid", amountPaid: 8_000, balance: 0 });
});
it("test_req4: unknown invoices are 404, drafts and void invoices cannot take payments (409)", async () => {
const a = app();
const c = await customer(a);
expect((await pay(a, "inv_404", 1_000)).status).toBe(404);
const draft = await draftInvoice(a, c, 10_000);
expect((await pay(a, draft, 1_000)).status).toBe(409);
const voided = await openInvoice(a, c, 10_000);
await a.handle("POST", `/invoices/${voided}/void`);
expect((await pay(a, voided, 1_000)).status).toBe(409);
expect(await get(a, draft)).toMatchObject({ status: "draft", amountPaid: 0 });
expect(await get(a, voided)).toMatchObject({ status: "void", amountPaid: 0 });
});
it("test_req5: the change ships with tests in the repo suite that exercise payments", () => {
const work = process.env.WORK as string;
const base = process.env.BASE as string;
const git = (...args: string[]) => execFileSync("git", ["-C", work, ...args], { encoding: "utf8" });
const changed = git("diff", "--name-only", base, "--").split("\n");
const untracked = git("ls-files", "--others", "--exclude-standard").split("\n");
const tests = [...changed, ...untracked].filter((f) => /\.test\.ts$/.test(f) && !f.startsWith(".claude/"));
const exercising = tests.filter((f) => {
try {
return /payments/.test(readFileSync(join(work, f), "utf8"));
} catch {
return false;
}
});
expect(exercising.length, `test files touched: ${tests.join(", ") || "none"}`).toBeGreaterThan(0);
});
});
hidden/test_F1.test.ts on GitHub
औरhidden/test_F2.test.ts
// Hidden oracle for tasks/F2.md (credit notes). Never visible to the worker. test_req1 .. test_req5. Runs against $WORK.
import { execFileSync } from "node:child_process";
import { mkdtempSync, readFileSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { describe, expect, it } from "vitest";
import { createApp } from "@work/src/http/app";
import { openStore } from "@work/src/store/json-store";
const NOW = new Date("2026-03-10T09:00:00.000Z");
type App = ReturnType<typeof createApp>;
type Note = { id: string; number: string; amount: number; invoiceId?: string };
function app(store = openStore()): App {
return createApp({ store, now: () => NOW });
}
async function customer(a: App): Promise<string> {
return ((await a.handle("POST", "/customers", { name: "Acme", email: "billing@acme.io" })).body as { id: string }).id;
}
async function draft(a: App, c: string, unitPrice = 10_000): Promise<string> {
const r = await a.handle("POST", "/invoices", { customerId: c, currency: "EUR", lines: [{ description: "Work", quantity: 1, unitPrice }] });
return (r.body as { id: string }).id;
}
async function open(a: App, c: string, unitPrice = 10_000): Promise<string> {
const id = await draft(a, c, unitPrice);
await a.handle("POST", `/invoices/${id}/issue`);
return id;
}
async function paid(a: App, c: string, unitPrice = 10_000): Promise<string> {
const id = await open(a, c, unitPrice);
await a.handle("POST", `/invoices/${id}/pay`);
return id;
}
// Credit notes on OPEN invoices are a brief preference the request does not state: the oracle never depends on them.
async function credit(a: App, id: string, amount: unknown) {
return a.handle("POST", `/invoices/${id}/credit-notes`, amount === undefined ? {} : { amount });
}
async function notes(a: App, id: string): Promise<Note[]> {
const r = await a.handle("GET", `/invoices/${id}/credit-notes`);
expect(r.status).toBe(200);
return r.body as Note[];
}
const ok = (s: number) => s >= 200 && s < 300;
describe("F2 credit notes", () => {
it("test_req1: credits never exceed an invoice's total, cumulatively, on the credit-note route and on the void route alike", async () => {
const a = app();
const c = await customer(a);
const id = await paid(a, c, 10_000);
expect(ok((await credit(a, id, 3_000)).status)).toBe(true);
const over = await credit(a, id, 7_001);
expect([400, 409, 422], `cumulative over-credit -> ${over.status}`).toContain(over.status);
expect(ok((await credit(a, id, 7_000)).status)).toBe(true);
expect((await notes(a, id)).map((n) => n.amount)).toEqual([3_000, 7_000]);
const again = await credit(a, id, 1);
expect([400, 409, 422]).toContain(again.status);
// fully credited, then voided: the void issues nothing more
expect((await a.handle("POST", `/invoices/${id}/void`)).status).toBe(200);
expect((await notes(a, id)).map((n) => n.amount)).toEqual([3_000, 7_000]);
});
it("test_req2: a store file written before credit notes existed: numbering starts at CN-0001, ids are well formed, notes survive a reload", async () => {
const file = join(mkdtempSync(join(tmpdir(), "flow-bench-f2-")), "ledgerly.json");
const inv = (id: string, status: string, paidAt: string | null) => ({
id, number: "INV-000" + id.slice(-1), customerId: "cus_1", currency: "EUR",
lines: [{ description: "Retainer", quantity: 1, unitPrice: 8_000 }], discount: 0, status,
createdAt: "2026-01-05T10:00:00.000Z", issuedAt: "2026-01-06T10:00:00.000Z", dueDate: null, paidAt, voidedAt: null,
});
writeFileSync(file, JSON.stringify({
schemaVersion: 1,
counters: { customer: 1, invoice: 2, invoiceNumber: 2 },
customers: [{ id: "cus_1", name: "Acme", email: "billing@acme.io", createdAt: "2026-01-05T10:00:00.000Z" }],
invoices: [inv("inv_1", "open", null), inv("inv_2", "paid", "2026-02-01T10:00:00.000Z")],
}));
let a = app(openStore({ file }));
const r = await credit(a, "inv_2", 1_000);
expect(r.status, JSON.stringify(r.body)).toBe(201);
const note = r.body as Note;
expect(note.number).toBe("CN-0001");
expect(String(note.id)).not.toMatch(/NaN|undefined/);
expect((await a.handle("POST", "/invoices/inv_2/void")).status).toBe(200);
a = app(openStore({ file }));
const n2 = await notes(a, "inv_2");
expect(n2.map((n) => [n.number, n.amount])).toEqual([["CN-0001", 1_000], ["CN-0002", 7_000]]);
expect(await notes(a, "inv_1")).toEqual([]);
expect(((await a.handle("GET", "/invoices/inv_2")).body as { status: string }).status).toBe("void");
});
it("test_req3: CN numbers are gapless in creation order across invoices; a rejected request never consumes a number", async () => {
const a = app();
const c = await customer(a);
const x = await paid(a, c, 10_000);
const y = await paid(a, c, 5_000);
const d = await draft(a, c);
expect(((await credit(a, y, 1_000)).body as Note).number).toBe("CN-0001");
expect((await credit(a, x, 10_001)).status).not.toBe(201); // over the total
expect((await credit(a, x, 0)).status).not.toBe(201); // invalid amount
expect((await credit(a, d, 500)).status).not.toBe(201); // draft
expect((await credit(a, "inv_404", 500)).status).not.toBe(201);
expect(((await credit(a, x, 2_000)).body as Note).number).toBe("CN-0002");
expect(((await credit(a, y, 4_000)).body as Note).number).toBe("CN-0003");
});
it("test_req4: error paths: drafts and void invoices 409, unknown invoice 404 (POST and GET), bad amounts 400", async () => {
const a = app();
const c = await customer(a);
expect((await credit(a, await draft(a, c), 100)).status).toBe(409);
const v = await open(a, c);
await a.handle("POST", `/invoices/${v}/void`);
expect((await credit(a, v, 100)).status).toBe(409);
expect((await credit(a, "inv_404", 100)).status).toBe(404);
expect((await a.handle("GET", "/invoices/inv_404/credit-notes")).status).toBe(404);
const o = await paid(a, c);
for (const bad of [0, -5, 12.5, "100", undefined]) {
const r = await credit(a, o, bad);
expect([400, 422], `amount ${String(bad)} -> ${r.status}`).toContain(r.status);
}
expect(await notes(a, o)).toEqual([]);
});
it("test_req5: voiding a paid invoice now works and credits only what is left; the existing tests are kept (the 'never paid' one rewritten, not deleted)", async () => {
const a = app();
const c = await customer(a);
const p = await open(a, c, 10_000);
await a.handle("POST", `/invoices/${p}/pay`);
expect(ok((await credit(a, p, 2_500)).status)).toBe(true);
const v = await a.handle("POST", `/invoices/${p}/void`);
expect(v.status, JSON.stringify(v.body)).toBe(200);
expect((v.body as { status: string }).status).toBe("void");
expect((await notes(a, p)).map((n) => n.amount)).toEqual([2_500, 7_500]);
expect((await a.handle("POST", `/invoices/${p}/void`)).status).toBe(409);
const work = process.env.WORK as string;
const changed = "voids drafts and open invoices, never paid ones";
const titles: string[] = JSON.parse(readFileSync(join(__dirname, "upstream_test_titles.json"), "utf8")).filter((t: string) => t !== changed);
const out = join(mkdtempSync(join(tmpdir(), "flow-bench-f2-suite-")), "suite.json");
try {
execFileSync(join(work, "node_modules", ".bin", "vitest"), ["run", "--reporter=json", `--outputFile=${out}`], { cwd: work, stdio: "ignore", timeout: 120_000 });
} catch {
/* a red suite still writes its report */
}
const j = JSON.parse(readFileSync(out, "utf8"));
const status = new Map<string, string>();
let invoiceFileTests = 0;
for (const f of j.testResults) {
for (const t of f.assertionResults) status.set(t.title, t.status);
if (String(f.name).endsWith("test/invoice.test.ts")) invoiceFileTests = f.assertionResults.length;
}
const missing = titles.filter((t) => !status.has(t));
const failing = titles.filter((t) => status.has(t) && status.get(t) !== "passed");
expect({ missing, failing }).toEqual({ missing: [], failing: [] });
expect(invoiceFileTests, "test/invoice.test.ts lost tests").toBeGreaterThanOrEqual(7);
});
});
hidden/test_F2.test.ts on GitHub
: हर एक में पाँच requirement tests।hidden/upstream_test_titles.json
[
"accepts integer cents",
"computes totals",
"creates a draft with computed totals",
"creates and fetches a customer",
"enforces the minimum",
"filters the list by status and rejects unknown statuses",
"formats cents",
"ignores drafts and void invoices",
"issues a draft without mutating it",
"issues with sequential numbers",
"knows its currencies and sums",
"knows when an open invoice is overdue",
"marks an open invoice paid",
"marks an open invoice paid in full",
"migrates a v0 file (embedded customer) on load and writes it back",
"normalises the email",
"parses a minimal invoice",
"persists updates to the file and reloads them",
"refuses a file from a newer build",
"refuses to issue twice",
"refuses to pay drafts and void invoices",
"rejects a bad email",
"rejects a discount above the subtotal",
"rejects an invoice for an unknown customer",
"rejects floats and strings",
"rejects fractional cents in unit prices",
"requires at least one line",
"returns 400 on invalid input and 404 on unknown ids",
"returns 404 for unknown routes",
"returns 409 when paying a void invoice and 404 for unknown invoices",
"starts empty in memory",
"sums open invoices per currency and counts overdue ones",
"validates the currency and due date",
"voids drafts and open invoices, never paid ones"
]
hidden/upstream_test_titles.json on GitHub
,hidden/vitest.config.ts
import { defineConfig } from "vitest/config";
// The oracle runs from OUTSIDE the work copy: WORK = absolute path of the work copy under test.
const work = process.env.WORK;
if (!work) throw new Error("WORK is not set");
export default defineConfig({
resolve: { alias: { "@work": work } },
server: { fs: { strict: false } },
test: { include: [`test_${process.env.TASK ?? "F1"}.test.ts`], testTimeout: 20_000, fileParallelism: false },
});
hidden/vitest.config.ts on GitHub
,hidden/package.json
{
"name": "flow-bench-hidden",
"private": true,
"type": "module",
"devDependencies": {
"vitest": "2.1.9"
}
}
hidden/pnpm-lock.yaml
lockfileVersion: '9.0'
settings:
autoInstallPeers: true
excludeLinksFromLockfile: false
importers:
.:
devDependencies:
vitest:
specifier: 2.1.9
version: 2.1.9
packages:
'@esbuild/aix-ppc64@0.21.5':
resolution: {integrity: sha512-1SDgH6ZSPTlggy1yI6+Dbkiz8xzpHJEVAlF/AM1tHPLsf5STom9rwtjE4hKAF20FfXXNTFqEYXyJNWh1GiZedQ==}
engines: {node: '>=12'}
cpu: [ppc64]
os: [aix]
'@esbuild/android-arm64@0.21.5':
resolution: {integrity: sha512-c0uX9VAUBQ7dTDCjq+wdyGLowMdtR/GoC2U5IYk/7D1H1JYC0qseD7+11iMP2mRLN9RcCMRcjC4YMclCzGwS/A==}
engines: {node: '>=12'}
cpu: [arm64]
os: [android]
'@esbuild/android-arm@0.21.5':
resolution: {integrity: sha512-vCPvzSjpPHEi1siZdlvAlsPxXl7WbOVUBBAowWug4rJHb68Ox8KualB+1ocNvT5fjv6wpkX6o/iEpbDrf68zcg==}
engines: {node: '>=12'}
cpu: [arm]
os: [android]
'@esbuild/android-x64@0.21.5':
resolution: {integrity: sha512-D7aPRUUNHRBwHxzxRvp856rjUHRFW1SdQATKXH2hqA0kAZb1hKmi02OpYRacl0TxIGz/ZmXWlbZgjwWYaCakTA==}
engines: {node: '>=12'}
cpu: [x64]
os: [android]
'@esbuild/darwin-arm64@0.21.5':
resolution: {integrity: sha512-DwqXqZyuk5AiWWf3UfLiRDJ5EDd49zg6O9wclZ7kUMv2WRFr4HKjXp/5t8JZ11QbQfUS6/cRCKGwYhtNAY88kQ==}
engines: {node: '>=12'}
cpu: [arm64]
os: [darwin]
'@esbuild/darwin-x64@0.21.5':
resolution: {integrity: sha512-se/JjF8NlmKVG4kNIuyWMV/22ZaerB+qaSi5MdrXtd6R08kvs2qCN4C09miupktDitvh8jRFflwGFBQcxZRjbw==}
engines: {node: '>=12'}
cpu: [x64]
os: [darwin]
'@esbuild/freebsd-arm64@0.21.5':
resolution: {integrity: sha512-5JcRxxRDUJLX8JXp/wcBCy3pENnCgBR9bN6JsY4OmhfUtIHe3ZW0mawA7+RDAcMLrMIZaf03NlQiX9DGyB8h4g==}
engines: {node: '>=12'}
cpu: [arm64]
os: [freebsd]
'@esbuild/freebsd-x64@0.21.5':
resolution: {integrity: sha512-J95kNBj1zkbMXtHVH29bBriQygMXqoVQOQYA+ISs0/2l3T9/kj42ow2mpqerRBxDJnmkUDCaQT/dfNXWX/ZZCQ==}
engines: {node: '>=12'}
cpu: [x64]
os: [freebsd]
'@esbuild/linux-arm64@0.21.5':
resolution: {integrity: sha512-ibKvmyYzKsBeX8d8I7MH/TMfWDXBF3db4qM6sy+7re0YXya+K1cem3on9XgdT2EQGMu4hQyZhan7TeQ8XkGp4Q==}
engines: {node: '>=12'}
cpu: [arm64]
os: [linux]
'@esbuild/linux-arm@0.21.5':
resolution: {integrity: sha512-bPb5AHZtbeNGjCKVZ9UGqGwo8EUu4cLq68E95A53KlxAPRmUyYv2D6F0uUI65XisGOL1hBP5mTronbgo+0bFcA==}
engines: {node: '>=12'}
cpu: [arm]
os: [linux]
'@esbuild/linux-ia32@0.21.5':
resolution: {integrity: sha512-YvjXDqLRqPDl2dvRODYmmhz4rPeVKYvppfGYKSNGdyZkA01046pLWyRKKI3ax8fbJoK5QbxblURkwK/MWY18Tg==}
engines: {node: '>=12'}
cpu: [ia32]
os: [linux]
'@esbuild/linux-loong64@0.21.5':
resolution: {integrity: sha512-uHf1BmMG8qEvzdrzAqg2SIG/02+4/DHB6a9Kbya0XDvwDEKCoC8ZRWI5JJvNdUjtciBGFQ5PuBlpEOXQj+JQSg==}
engines: {node: '>=12'}
cpu: [loong64]
os: [linux]
'@esbuild/linux-mips64el@0.21.5':
resolution: {integrity: sha512-IajOmO+KJK23bj52dFSNCMsz1QP1DqM6cwLUv3W1QwyxkyIWecfafnI555fvSGqEKwjMXVLokcV5ygHW5b3Jbg==}
engines: {node: '>=12'}
cpu: [mips64el]
os: [linux]
'@esbuild/linux-ppc64@0.21.5':
resolution: {integrity: sha512-1hHV/Z4OEfMwpLO8rp7CvlhBDnjsC3CttJXIhBi+5Aj5r+MBvy4egg7wCbe//hSsT+RvDAG7s81tAvpL2XAE4w==}
engines: {node: '>=12'}
cpu: [ppc64]
os: [linux]
'@esbuild/linux-riscv64@0.21.5':
resolution: {integrity: sha512-2HdXDMd9GMgTGrPWnJzP2ALSokE/0O5HhTUvWIbD3YdjME8JwvSCnNGBnTThKGEB91OZhzrJ4qIIxk/SBmyDDA==}
engines: {node: '>=12'}
cpu: [riscv64]
os: [linux]
'@esbuild/linux-s390x@0.21.5':
resolution: {integrity: sha512-zus5sxzqBJD3eXxwvjN1yQkRepANgxE9lgOW2qLnmr8ikMTphkjgXu1HR01K4FJg8h1kEEDAqDcZQtbrRnB41A==}
engines: {node: '>=12'}
cpu: [s390x]
os: [linux]
'@esbuild/linux-x64@0.21.5':
resolution: {integrity: sha512-1rYdTpyv03iycF1+BhzrzQJCdOuAOtaqHTWJZCWvijKD2N5Xu0TtVC8/+1faWqcP9iBCWOmjmhoH94dH82BxPQ==}
engines: {node: '>=12'}
cpu: [x64]
os: [linux]
'@esbuild/netbsd-x64@0.21.5':
resolution: {integrity: sha512-Woi2MXzXjMULccIwMnLciyZH4nCIMpWQAs049KEeMvOcNADVxo0UBIQPfSmxB3CWKedngg7sWZdLvLczpe0tLg==}
engines: {node: '>=12'}
cpu: [x64]
os: [netbsd]
'@esbuild/openbsd-x64@0.21.5':
resolution: {integrity: sha512-HLNNw99xsvx12lFBUwoT8EVCsSvRNDVxNpjZ7bPn947b8gJPzeHWyNVhFsaerc0n3TsbOINvRP2byTZ5LKezow==}
engines: {node: '>=12'}
cpu: [x64]
os: [openbsd]
'@esbuild/sunos-x64@0.21.5':
resolution: {integrity: sha512-6+gjmFpfy0BHU5Tpptkuh8+uw3mnrvgs+dSPQXQOv3ekbordwnzTVEb4qnIvQcYXq6gzkyTnoZ9dZG+D4garKg==}
engines: {node: '>=12'}
cpu: [x64]
os: [sunos]
'@esbuild/win32-arm64@0.21.5':
resolution: {integrity: sha512-Z0gOTd75VvXqyq7nsl93zwahcTROgqvuAcYDUr+vOv8uHhNSKROyU961kgtCD1e95IqPKSQKH7tBTslnS3tA8A==}
engines: {node: '>=12'}
cpu: [arm64]
os: [win32]
'@esbuild/win32-ia32@0.21.5':
resolution: {integrity: sha512-SWXFF1CL2RVNMaVs+BBClwtfZSvDgtL//G/smwAc5oVK/UPu2Gu9tIaRgFmYFFKrmg3SyAjSrElf0TiJ1v8fYA==}
engines: {node: '>=12'}
cpu: [ia32]
os: [win32]
'@esbuild/win32-x64@0.21.5':
resolution: {integrity: sha512-tQd/1efJuzPC6rCFwEvLtci/xNFcTZknmXs98FYDfGE4wP9ClFV98nyKrzJKVPMhdDnjzLhdUyMX4PsQAPjwIw==}
engines: {node: '>=12'}
cpu: [x64]
os: [win32]
'@jridgewell/sourcemap-codec@1.6.0':
resolution: {integrity: sha512-T7jf+5zgsZHwNJ4lvQ7/aezbyk0nNX+zJVWpmHA7VYsEx7a7qr5Rg5IbtJFqkgze5Y2sruq1RUY8Q837Od7iFw==}
'@napi-rs/lzma-linux-x64-gnu@1.5.1':
resolution: {integrity: sha512-oTXEIha4SsuXdTA4Iyskj0kpdx2yVXdhd75c2v3xGrHFfVMsbhTPZU/nMPL4sWKo4pBHm3aucLaqGlF696dTyQ==}
engines: {node: ^22.20 || ^24.12 || >=25}
cpu: [x64]
os: [linux]
libc: [glibc]
'@rollup/rollup-android-arm-eabi@4.64.2':
resolution: {integrity: sha512-gjNDUwMswogkw3fNqV00u54uZlgmInuw5EU/1TeqQ6K0bEagoGvqjPdKzy112lKyhe7PkGudZ/kSQHsc9qroIg==}
cpu: [arm]
os: [android]
'@rollup/rollup-android-arm64@4.64.2':
resolution: {integrity: sha512-YkAW8q7p5SCLSaGWaUkmsRnBEb3MGA1pB5nI/8k0lAK5fZsQKRBqkdR1h4UU9p8u3c7ZGdUF2Zcx7mK+E/pUKg==}
cpu: [arm64]
os: [android]
'@rollup/rollup-darwin-arm64@4.64.2':
resolution: {integrity: sha512-hNmuIHv+LnrP/+q/44vsmIMWfoh7X2YXOp5FLtjGuvqN52VdtFpOpnshTc1k28tWv1wE/0D5gLjsozBUowEGbQ==}
cpu: [arm64]
os: [darwin]
'@rollup/rollup-darwin-x64@4.64.2':
resolution: {integrity: sha512-B0t5CoxgF2KL7R5FHAsyN8yVEQCSIkh4XVs8MCDjuBLNjEa8l3zd5tVKaFMv53LfUtHs9p2rnqeavPTOunV+Ww==}
cpu: [x64]
os: [darwin]
'@rollup/rollup-freebsd-arm64@4.64.2':
resolution: {integrity: sha512-QQdYnfwrPmc1DoPlddjFvQPzUHFbStKXsHrG6EBYsWHhBVUCjsCRdcfGGXGAvhDpDtTzKK4vy1UrGFJ45rQaiw==}
cpu: [arm64]
os: [freebsd]
'@rollup/rollup-freebsd-x64@4.64.2':
resolution: {integrity: sha512-1xRxtGWif6BB58VSfMavi0Yfs2M4x6CtRp4o7ScY4W7AGlgvf6GSPKdDNdw7TX3tMiQwZY5W/mvyrOJTq/7AKA==}
cpu: [x64]
os: [freebsd]
'@rollup/rollup-linux-arm-gnueabihf@4.64.2':
resolution: {integrity: sha512-CXFLEt384BUwwOLR9DURWNHl1bQOVLd1ibOee+yKBJm/fZtvrSAOZ0JbxjAxszVMIovYMwFMS+oP+Q1aPcx8Eg==}
cpu: [arm]
os: [linux]
libc: [glibc]
'@rollup/rollup-linux-arm-musleabihf@4.64.2':
resolution: {integrity: sha512-T1dorY0WHm5mMNpMHFl5rYRZ02gLSZcr6J45r66tYb3S3R3DyIRGiGWVdLtF+zOQjratNJaObl9jYumueRiL1A==}
cpu: [arm]
os: [linux]
libc: [musl]
'@rollup/rollup-linux-arm64-gnu@4.64.2':
resolution: {integrity: sha512-ia6bfaCIc0ldp6bvs9oEkFrJVJKNUMoiDF0+LBZsQAZg+vE0znXnzGqH+ew/FYswE30cruAWPTO3dPw7keaVkQ==}
cpu: [arm64]
os: [linux]
libc: [glibc]
'@rollup/rollup-linux-arm64-musl@4.64.2':
resolution: {integrity: sha512-mB/gwfBfXtT0EBpbSssqKCqMFuX8b21Zy2TGoIhehSJEssZ5ksybxymwp1BkEM5Nczh/AAxpmh5q+MkhtundGg==}
cpu: [arm64]
os: [linux]
libc: [musl]
'@rollup/rollup-linux-loong64-gnu@4.64.2':
resolution: {integrity: sha512-VPd0a+XlHoNRI9jIPRHww2UKhDkD6XjbtcKYXZijAnhJS1jLLbiNtVl6a5LyEfNN+eyiyWcJOZXPzBo69oTsew==}
cpu: [loong64]
os: [linux]
libc: [glibc]
'@rollup/rollup-linux-loong64-musl@4.64.2':
resolution: {integrity: sha512-10N1I6oBpERqzGtHKcCLrEpHlbXSBbMK9TURHvqgWbIFZuU3oJROxJ0H2nHfq96TVI0XJHQ9qzolJMFd2tz0LA==}
cpu: [loong64]
os: [linux]
libc: [musl]
'@rollup/rollup-linux-ppc64-gnu@4.64.2':
resolution: {integrity: sha512-dhJrLVZ3wNNhwCmy9/VTg6Pq6J8a8eZ0n15dVRlbSWcDb+8V/1mDfbnvis+vt8W6Pbxzv6gUbTEl4piOT3bbUQ==}
cpu: [ppc64]
os: [linux]
libc: [glibc]
'@rollup/rollup-linux-ppc64-musl@4.64.2':
resolution: {integrity: sha512-tPd2QuwxkTxgju5eARyb3RdgBWWi/O9dn/NFH0VkvGbLzoQeXrH6qMABtW8IODb5lUeBzB3sy2U7qmI4MtQ72g==}
cpu: [ppc64]
os: [linux]
libc: [musl]
'@rollup/rollup-linux-riscv64-gnu@4.64.2':
resolution: {integrity: sha512-njtDdPvSQWI9CDhIPPxsiEMeHayCfu7wwcbSQ7c2kYPhGVviL+WGVhq6Jfj/b+VuhSTkFuQ0z4ZFbFjkWwBFmg==}
cpu: [riscv64]
os: [linux]
libc: [glibc]
'@rollup/rollup-linux-riscv64-musl@4.64.2':
resolution: {integrity: sha512-EbOBo3dqCj9ecEV9Z7zvulB6AieMP2ztnKQYGVOSN0pzIyOu+3JH2YJZnLmbPeVvfn6yP6Qi7NbBM27EzGbj2w==}
cpu: [riscv64]
os: [linux]
libc: [musl]
'@rollup/rollup-linux-s390x-gnu@4.64.2':
resolution: {integrity: sha512-8wLt2Q5n4qmfiE2Gk8V6KNLSf+oJuBX7Kdw5joFDtnZIZvirS0hLzBc79OiJjV71oYDmBUZLNy6pDgvylo4qiQ==}
cpu: [s390x]
os: [linux]
libc: [glibc]
'@rollup/rollup-linux-x64-gnu@4.64.2':
resolution: {integrity: sha512-aIChbkhC4d33bxkVP0LSi2/2lUSGRXE6k4Qa0G7tgpgTOK7kxHTixmx5qC3SKUJ6yUMznHBtQoWaG4QMqLUnhw==}
cpu: [x64]
os: [linux]
libc: [glibc]
'@rollup/rollup-linux-x64-musl@4.64.2':
resolution: {integrity: sha512-gQ+TkVZRfjZVcl/UOAxxfWw+sHjzIXoieNwxEyJlD3tRBmBhR8ZDq11w+QEgQDcpslEB+fhN/OEpaTsDoIIHhQ==}
cpu: [x64]
os: [linux]
libc: [musl]
'@rollup/rollup-openbsd-x64@4.64.2':
resolution: {integrity: sha512-sUaJpMKS2JBCvUk8fYimG7aMq2jHVT4kw/uVHxPolKLrMmkKy0KXirmZyZehdNpL7PffqAlPAWUELQcTiD8ChA==}
cpu: [x64]
os: [openbsd]
'@rollup/rollup-openharmony-arm64@4.64.2':
resolution: {integrity: sha512-j1RdJWb0CrlSuwt12OPjLucPlwfLtA/KMOKVvh5rXSJEtl/WDoIGR3CSykK/jLGTo9RqNfW5Ub6kY5Z5ccYl9w==}
cpu: [arm64]
os: [openharmony]
'@rollup/rollup-win32-arm64-msvc@4.64.2':
resolution: {integrity: sha512-i3r04aHCIY3eggTBLUp/4IxiBjDpAgQaK89Ue2nMJ2MKuIXX5JfoeZjHw+kPjGGNVDlhi8FT8xzuGMJZe3aehg==}
cpu: [arm64]
os: [win32]
'@rollup/rollup-win32-ia32-msvc@4.64.2':
resolution: {integrity: sha512-kCQ15h/uwNMGdN8gjXBZGf+aGE+ODzHfbi90/cAm4+Q89WoMiLODNoj8HoGZmYx87HPxfu5CBNPsZ+ZG4uIAoA==}
cpu: [ia32]
os: [win32]
'@rollup/rollup-win32-x64-gnu@4.64.2':
resolution: {integrity: sha512-yYAuySSs+0X2gSMJPw/Q4jHsHZHyMq1dpAE+XnN790Jg1WuReUAD/+j+e4vuDvS4PXJ4PA4NCzTR4OYtDf18/g==}
cpu: [x64]
os: [win32]
'@rollup/rollup-win32-x64-msvc@4.64.2':
resolution: {integrity: sha512-JY1rQdWOJrKutORqKO1uzrHXV86jFiQzfLFKx2s8S1CjLXL9fC5LOIvk0q7Z7PXJvsByt+sdoE1Q2a9gBr4gLQ==}
cpu: [x64]
os: [win32]
'@types/estree@1.0.9':
resolution: {integrity: sha512-GhdPgy1el4/ImP05X05Uw4cw2/M93BCUmnEvWZNStlCzEKME4Fkk+YpoA5OiHNQmoS7Cafb8Xa3Pya8m1Qrzeg==}
'@vitest/expect@2.1.9':
resolution: {integrity: sha512-UJCIkTBenHeKT1TTlKMJWy1laZewsRIzYighyYiJKZreqtdxSos/S1t+ktRMQWu2CKqaarrkeszJx1cgC5tGZw==}
'@vitest/mocker@2.1.9':
resolution: {integrity: sha512-tVL6uJgoUdi6icpxmdrn5YNo3g3Dxv+IHJBr0GXHaEdTcw3F+cPKnsXFhli6nO+f/6SDKPHEK1UN+k+TQv0Ehg==}
peerDependencies:
msw: ^2.4.9
vite: ^5.0.0
peerDependenciesMeta:
msw:
optional: true
vite:
optional: true
'@vitest/pretty-format@2.1.9':
resolution: {integrity: sha512-KhRIdGV2U9HOUzxfiHmY8IFHTdqtOhIzCpd8WRdJiE7D/HUcZVD0EgQCVjm+Q9gkUXWgBvMmTtZgIG48wq7sOQ==}
'@vitest/runner@2.1.9':
resolution: {integrity: sha512-ZXSSqTFIrzduD63btIfEyOmNcBmQvgOVsPNPe0jYtESiXkhd8u2erDLnMxmGrDCwHCCHE7hxwRDCT3pt0esT4g==}
'@vitest/snapshot@2.1.9':
resolution: {integrity: sha512-oBO82rEjsxLNJincVhLhaxxZdEtV0EFHMK5Kmx5sJ6H9L183dHECjiefOAdnqpIgT5eZwT04PoggUnW88vOBNQ==}
'@vitest/spy@2.1.9':
resolution: {integrity: sha512-E1B35FwzXXTs9FHNK6bDszs7mtydNi5MIfUWpceJ8Xbfb1gBMscAnwLbEu+B44ed6W3XjL9/ehLPHR1fkf1KLQ==}
'@vitest/utils@2.1.9':
resolution: {integrity: sha512-v0psaMSkNJ3A2NMrUEHFRzJtDPFn+/VWZ5WxImB21T9fjucJRmS7xCS3ppEnARb9y11OAzaD+P2Ps+b+BGX5iQ==}
assertion-error@2.0.1:
resolution: {integrity: sha512-Izi8RQcffqCeNVgFigKli1ssklIbpHnCYc6AknXGYoB6grJqyeby7jv12JUQgmTAnIDnbck1uxksT4dzN3PWBA==}
engines: {node: '>=12'}
cac@6.7.14:
resolution: {integrity: sha512-b6Ilus+c3RrdDk+JhLKUAQfzzgLEPy6wcXqS7f/xe1EETvsDP6GORG7SFuOs6cID5YkqchW/LXZbX5bc8j7ZcQ==}
engines: {node: '>=8'}
chai@5.3.3:
resolution: {integrity: sha512-4zNhdJD/iOjSH0A05ea+Ke6MU5mmpQcbQsSOkgdaUMJ9zTlDTD/GYlwohmIE2u0gaxHYiVHEn1Fw9mZ/ktJWgw==}
engines: {node: '>=18'}
check-error@2.1.3:
resolution: {integrity: sha512-PAJdDJusoxnwm1VwW07VWwUN1sl7smmC3OKggvndJFadxxDRyFJBX/ggnu/KE4kQAB7a3Dp8f/YXC1FlUprWmA==}
engines: {node: '>= 16'}
debug@4.4.3:
resolution: {integrity: sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA==}
engines: {node: '>=6.0'}
peerDependencies:
supports-color: '*'
peerDependenciesMeta:
supports-color:
optional: true
deep-eql@5.0.2:
resolution: {integrity: sha512-h5k/5U50IJJFpzfL6nO9jaaumfjO/f2NjK/oYB2Djzm4p9L+3T9qWpZqZ2hAbLPuuYq9wrU08WQyBTL5GbPk5Q==}
engines: {node: '>=6'}
es-module-lexer@1.7.0:
resolution: {integrity: sha512-jEQoCwk8hyb2AZziIOLhDqpm5+2ww5uIE6lkO/6jcOCusfk6LhMHpXXfBLXTZ7Ydyt0j4VoUQv6uGNYbdW+kBA==}
esbuild@0.21.5:
resolution: {integrity: sha512-mg3OPMV4hXywwpoDxu3Qda5xCKQi+vCTZq8S9J/EpkhB2HzKXq4SNFZE3+NK93JYxc8VMSep+lOUSC/RVKaBqw==}
engines: {node: '>=12'}
hasBin: true
estree-walker@3.0.3:
resolution: {integrity: sha512-7RUKfXgSMMkzt6ZuXmqapOurLGPPfgj6l9uRZ7lRGolvk0y2yocc35LdcxKC5PQZdn2DMqioAQ2NoWcrTKmm6g==}
expect-type@1.4.0:
resolution: {integrity: sha512-KfYbmpRm0VbLjEvVa9yGwCi9GI34xvi7A/HXYWQO65CSD2u3MczUJSuwXKFIxlGsgBQizV9q5J9NHj4VG0n+pA==}
engines: {node: '>=12.0.0'}
fsevents@2.3.3:
resolution: {integrity: sha512-5xoDfX+fL7faATnagmWPpbFtwh/R77WmMMqqHGS65C3vvB0YHrgF+B1YmZ3441tMj5n63k0212XNoJwzlhffQw==}
engines: {node: ^8.16.0 || ^10.6.0 || >=11.0.0}
os: [darwin]
loupe@3.2.1:
resolution: {integrity: sha512-CdzqowRJCeLU72bHvWqwRBBlLcMEtIvGrlvef74kMnV2AolS9Y8xUv1I0U/MNAWMhBlKIoyuEgoJ0t/bbwHbLQ==}
magic-string@0.30.21:
resolution: {integrity: sha512-vd2F4YUyEXKGcLHoq+TEyCjxueSeHnFxyyjNp80yg0XV4vUhnDer/lvvlqM/arB5bXQN5K2/3oinyCRyx8T2CQ==}
ms@2.1.3:
resolution: {integrity: sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==}
nanoid@3.3.20:
resolution: {integrity: sha512-uKdg2G3GNCKQn9byYOpxbGqrT2fGO5KRt5J/8b3pok8rT6qxGWF6hxMyJiEYtAf+FVyYuD9hRaDqX5uPFYJ4ZQ==}
engines: {node: ^10 || ^12 || ^13.7 || ^14 || >=15.0.1}
hasBin: true
pathe@1.1.2:
resolution: {integrity: sha512-whLdWMYL2TwI08hn8/ZqAbrVemu0LNaNNJZX73O6qaIdCTfXutsLhMkjdENX0qhsQ9uIimo4/aQOmXkoon2nDQ==}
pathval@2.0.1:
resolution: {integrity: sha512-//nshmD55c46FuFw26xV/xFAaB5HF9Xdap7HJBBnrKdAd6/GxDBaNA1870O79+9ueg61cZLSVc+OaFlfmObYVQ==}
engines: {node: '>= 14.16'}
picocolors@1.1.1:
resolution: {integrity: sha512-xceH2snhtb5M9liqDsmEw56le376mTZkEX/jEb/RxNFyegNul7eNslCXP9FDj/Lcu0X8KEyMceP2ntpaHrDEVA==}
postcss@8.5.29:
resolution: {integrity: sha512-49cGhUbXj8Qenv0iTMxA1cFBzxXoctpC9Ujd77t1WcbJIr6nF/eI7g/8MgxrYldFRuAXvja7xQRwavoW7kgrxQ==}
engines: {node: ^10 || ^12 || >=14}
rollup@4.64.2:
resolution: {integrity: sha512-RxXm0XVVV1hFI3dYe12ruaOgsYQ725lUcUWeMSQqkF9tTxVErY+fy2Ets6vWhItL9MZMSNZGbvZFc7rH/VHp2Q==}
engines: {node: '>=18.0.0', npm: '>=8.0.0'}
hasBin: true
siginfo@2.0.0:
resolution: {integrity: sha512-ybx0WO1/8bSBLEWXZvEd7gMW3Sn3JFlW3TvX1nREbDLRNQNaeNN8WK0meBwPdAaOI7TtRRRJn/Es1zhrrCHu7g==}
source-map-js@1.2.2:
resolution: {integrity: sha512-KGj/8Y43x35aZVDtt+J4mK1hoLGHULMYfSkODJNQjNDC3oW1PqPoxMwo0pLUsWM/UEGzON/NxeHywEfNXNP3Vw==}
engines: {node: '>=0.10.0'}
stackback@0.0.2:
resolution: {integrity: sha512-1XMJE5fQo1jGH6Y/7ebnwPOBEkIEnT4QF32d5R1+VXdXveM0IBMJt8zfaxX1P3QhVwrYe+576+jkANtSS2mBbw==}
std-env@3.10.0:
resolution: {integrity: sha512-5GS12FdOZNliM5mAOxFRg7Ir0pWz8MdpYm6AY6VPkGpbA7ZzmbzNcBJQ0GPvvyWgcY7QAhCgf9Uy89I03faLkg==}
tinybench@2.9.0:
resolution: {integrity: sha512-0+DUvqWMValLmha6lr4kD8iAMK1HzV0/aKnCtWb9v9641TnP/MFb7Pc2bxoxQjTXAErryXVgUOfv2YqNllqGeg==}
tinyexec@0.3.2:
resolution: {integrity: sha512-KQQR9yN7R5+OSwaK0XQoj22pwHoTlgYqmUscPYoknOoWCWfj/5/ABTMRi69FrKU5ffPVh5QcFikpWJI/P1ocHA==}
tinypool@1.1.1:
resolution: {integrity: sha512-Zba82s87IFq9A9XmjiX5uZA/ARWDrB03OHlq+Vw1fSdt0I+4/Kutwy8BP4Y/y/aORMo61FQ0vIb5j44vSo5Pkg==}
engines: {node: ^18.0.0 || >=20.0.0}
tinyrainbow@1.2.0:
resolution: {integrity: sha512-weEDEq7Z5eTHPDh4xjX789+fHfF+P8boiFB+0vbWzpbnbsEr/GRaohi/uMKxg8RZMXnl1ItAi/IUHWMsjDV7kQ==}
engines: {node: '>=14.0.0'}
tinyspy@3.0.2:
resolution: {integrity: sha512-n1cw8k1k0x4pgA2+9XrOkFydTerNcJ1zWCO5Nn9scWHTD+5tp8dghT2x1uduQePZTZgd3Tupf+x9BxJjeJi77Q==}
engines: {node: '>=14.0.0'}
vite-node@2.1.9:
resolution: {integrity: sha512-AM9aQ/IPrW/6ENLQg3AGY4K1N2TGZdR5e4gu/MmmR2xR3Ll1+dib+nook92g4TV3PXVyeyxdWwtaCAiUL0hMxA==}
engines: {node: ^18.0.0 || >=20.0.0}
hasBin: true
vite@5.4.21:
resolution: {integrity: sha512-o5a9xKjbtuhY6Bi5S3+HvbRERmouabWbyUcpXXUA1u+GNUKoROi9byOJ8M0nHbHYHkYICiMlqxkg1KkYmm25Sw==}
engines: {node: ^18.0.0 || >=20.0.0}
hasBin: true
peerDependencies:
'@types/node': ^18.0.0 || >=20.0.0
less: '*'
lightningcss: ^1.21.0
sass: '*'
sass-embedded: '*'
stylus: '*'
sugarss: '*'
terser: ^5.4.0
peerDependenciesMeta:
'@types/node':
optional: true
less:
optional: true
lightningcss:
optional: true
sass:
optional: true
sass-embedded:
optional: true
stylus:
optional: true
sugarss:
optional: true
terser:
optional: true
vitest@2.1.9:
resolution: {integrity: sha512-MSmPM9REYqDGBI8439mA4mWhV5sKmDlBKWIYbA3lRb2PTHACE0mgKwA8yQ2xq9vxDTuk4iPrECBAEW2aoFXY0Q==}
engines: {node: ^18.0.0 || >=20.0.0}
hasBin: true
peerDependencies:
'@edge-runtime/vm': '*'
'@types/node': ^18.0.0 || >=20.0.0
'@vitest/browser': 2.1.9
'@vitest/ui': 2.1.9
happy-dom: '*'
jsdom: '*'
peerDependenciesMeta:
'@edge-runtime/vm':
optional: true
'@types/node':
optional: true
'@vitest/browser':
optional: true
'@vitest/ui':
optional: true
happy-dom:
optional: true
jsdom:
optional: true
why-is-node-running@2.3.0:
resolution: {integrity: sha512-hUrmaWBdVDcxvYqnyh09zunKzROWjbZTiNy8dBEjkS7ehEDQibXJ7XvlmtbwuTclUiIyN+CyXQD4Vmko8fNm8w==}
engines: {node: '>=8'}
hasBin: true
snapshots:
'@esbuild/aix-ppc64@0.21.5':
optional: true
'@esbuild/android-arm64@0.21.5':
optional: true
'@esbuild/android-arm@0.21.5':
optional: true
'@esbuild/android-x64@0.21.5':
optional: true
'@esbuild/darwin-arm64@0.21.5':
optional: true
'@esbuild/darwin-x64@0.21.5':
optional: true
'@esbuild/freebsd-arm64@0.21.5':
optional: true
'@esbuild/freebsd-x64@0.21.5':
optional: true
'@esbuild/linux-arm64@0.21.5':
optional: true
'@esbuild/linux-arm@0.21.5':
optional: true
'@esbuild/linux-ia32@0.21.5':
optional: true
'@esbuild/linux-loong64@0.21.5':
optional: true
'@esbuild/linux-mips64el@0.21.5':
optional: true
'@esbuild/linux-ppc64@0.21.5':
optional: true
'@esbuild/linux-riscv64@0.21.5':
optional: true
'@esbuild/linux-s390x@0.21.5':
optional: true
'@esbuild/linux-x64@0.21.5':
optional: true
'@esbuild/netbsd-x64@0.21.5':
optional: true
'@esbuild/openbsd-x64@0.21.5':
optional: true
'@esbuild/sunos-x64@0.21.5':
optional: true
'@esbuild/win32-arm64@0.21.5':
optional: true
'@esbuild/win32-ia32@0.21.5':
optional: true
'@esbuild/win32-x64@0.21.5':
optional: true
'@jridgewell/sourcemap-codec@1.6.0': {}
'@napi-rs/lzma-linux-x64-gnu@1.5.1':
optional: true
'@rollup/rollup-android-arm-eabi@4.64.2':
optional: true
'@rollup/rollup-android-arm64@4.64.2':
optional: true
'@rollup/rollup-darwin-arm64@4.64.2':
optional: true
'@rollup/rollup-darwin-x64@4.64.2':
optional: true
'@rollup/rollup-freebsd-arm64@4.64.2':
optional: true
'@rollup/rollup-freebsd-x64@4.64.2':
optional: true
'@rollup/rollup-linux-arm-gnueabihf@4.64.2':
optional: true
'@rollup/rollup-linux-arm-musleabihf@4.64.2':
optional: true
'@rollup/rollup-linux-arm64-gnu@4.64.2':
optional: true
'@rollup/rollup-linux-arm64-musl@4.64.2':
optional: true
'@rollup/rollup-linux-loong64-gnu@4.64.2':
optional: true
'@rollup/rollup-linux-loong64-musl@4.64.2':
optional: true
'@rollup/rollup-linux-ppc64-gnu@4.64.2':
optional: true
'@rollup/rollup-linux-ppc64-musl@4.64.2':
optional: true
'@rollup/rollup-linux-riscv64-gnu@4.64.2':
optional: true
'@rollup/rollup-linux-riscv64-musl@4.64.2':
optional: true
'@rollup/rollup-linux-s390x-gnu@4.64.2':
optional: true
'@rollup/rollup-linux-x64-gnu@4.64.2':
optional: true
'@rollup/rollup-linux-x64-musl@4.64.2':
optional: true
'@rollup/rollup-openbsd-x64@4.64.2':
optional: true
'@rollup/rollup-openharmony-arm64@4.64.2':
optional: true
'@rollup/rollup-win32-arm64-msvc@4.64.2':
optional: true
'@rollup/rollup-win32-ia32-msvc@4.64.2':
optional: true
'@rollup/rollup-win32-x64-gnu@4.64.2':
optional: true
'@rollup/rollup-win32-x64-msvc@4.64.2':
optional: true
'@types/estree@1.0.9': {}
'@vitest/expect@2.1.9':
dependencies:
'@vitest/spy': 2.1.9
'@vitest/utils': 2.1.9
chai: 5.3.3
tinyrainbow: 1.2.0
'@vitest/mocker@2.1.9(vite@5.4.21)':
dependencies:
'@vitest/spy': 2.1.9
estree-walker: 3.0.3
magic-string: 0.30.21
optionalDependencies:
vite: 5.4.21
'@vitest/pretty-format@2.1.9':
dependencies:
tinyrainbow: 1.2.0
'@vitest/runner@2.1.9':
dependencies:
'@vitest/utils': 2.1.9
pathe: 1.1.2
'@vitest/snapshot@2.1.9':
dependencies:
'@vitest/pretty-format': 2.1.9
magic-string: 0.30.21
pathe: 1.1.2
'@vitest/spy@2.1.9':
dependencies:
tinyspy: 3.0.2
'@vitest/utils@2.1.9':
dependencies:
'@vitest/pretty-format': 2.1.9
loupe: 3.2.1
tinyrainbow: 1.2.0
assertion-error@2.0.1: {}
cac@6.7.14: {}
chai@5.3.3:
dependencies:
assertion-error: 2.0.1
check-error: 2.1.3
deep-eql: 5.0.2
loupe: 3.2.1
pathval: 2.0.1
check-error@2.1.3: {}
debug@4.4.3:
dependencies:
ms: 2.1.3
deep-eql@5.0.2: {}
es-module-lexer@1.7.0: {}
esbuild@0.21.5:
optionalDependencies:
'@esbuild/aix-ppc64': 0.21.5
'@esbuild/android-arm': 0.21.5
'@esbuild/android-arm64': 0.21.5
'@esbuild/android-x64': 0.21.5
'@esbuild/darwin-arm64': 0.21.5
'@esbuild/darwin-x64': 0.21.5
'@esbuild/freebsd-arm64': 0.21.5
'@esbuild/freebsd-x64': 0.21.5
'@esbuild/linux-arm': 0.21.5
'@esbuild/linux-arm64': 0.21.5
'@esbuild/linux-ia32': 0.21.5
'@esbuild/linux-loong64': 0.21.5
'@esbuild/linux-mips64el': 0.21.5
'@esbuild/linux-ppc64': 0.21.5
'@esbuild/linux-riscv64': 0.21.5
'@esbuild/linux-s390x': 0.21.5
'@esbuild/linux-x64': 0.21.5
'@esbuild/netbsd-x64': 0.21.5
'@esbuild/openbsd-x64': 0.21.5
'@esbuild/sunos-x64': 0.21.5
'@esbuild/win32-arm64': 0.21.5
'@esbuild/win32-ia32': 0.21.5
'@esbuild/win32-x64': 0.21.5
estree-walker@3.0.3:
dependencies:
'@types/estree': 1.0.9
expect-type@1.4.0: {}
fsevents@2.3.3:
optional: true
loupe@3.2.1: {}
magic-string@0.30.21:
dependencies:
'@jridgewell/sourcemap-codec': 1.6.0
ms@2.1.3: {}
nanoid@3.3.20: {}
pathe@1.1.2: {}
pathval@2.0.1: {}
picocolors@1.1.1: {}
postcss@8.5.29:
dependencies:
nanoid: 3.3.20
picocolors: 1.1.1
source-map-js: 1.2.2
rollup@4.64.2:
dependencies:
'@types/estree': 1.0.9
optionalDependencies:
'@napi-rs/lzma-linux-x64-gnu': 1.5.1
'@rollup/rollup-android-arm-eabi': 4.64.2
'@rollup/rollup-android-arm64': 4.64.2
'@rollup/rollup-darwin-arm64': 4.64.2
'@rollup/rollup-darwin-x64': 4.64.2
'@rollup/rollup-freebsd-arm64': 4.64.2
'@rollup/rollup-freebsd-x64': 4.64.2
'@rollup/rollup-linux-arm-gnueabihf': 4.64.2
'@rollup/rollup-linux-arm-musleabihf': 4.64.2
'@rollup/rollup-linux-arm64-gnu': 4.64.2
'@rollup/rollup-linux-arm64-musl': 4.64.2
'@rollup/rollup-linux-loong64-gnu': 4.64.2
'@rollup/rollup-linux-loong64-musl': 4.64.2
'@rollup/rollup-linux-ppc64-gnu': 4.64.2
'@rollup/rollup-linux-ppc64-musl': 4.64.2
'@rollup/rollup-linux-riscv64-gnu': 4.64.2
'@rollup/rollup-linux-riscv64-musl': 4.64.2
'@rollup/rollup-linux-s390x-gnu': 4.64.2
'@rollup/rollup-linux-x64-gnu': 4.64.2
'@rollup/rollup-linux-x64-musl': 4.64.2
'@rollup/rollup-openbsd-x64': 4.64.2
'@rollup/rollup-openharmony-arm64': 4.64.2
'@rollup/rollup-win32-arm64-msvc': 4.64.2
'@rollup/rollup-win32-ia32-msvc': 4.64.2
'@rollup/rollup-win32-x64-gnu': 4.64.2
'@rollup/rollup-win32-x64-msvc': 4.64.2
fsevents: 2.3.3
siginfo@2.0.0: {}
source-map-js@1.2.2: {}
stackback@0.0.2: {}
std-env@3.10.0: {}
tinybench@2.9.0: {}
tinyexec@0.3.2: {}
tinypool@1.1.1: {}
tinyrainbow@1.2.0: {}
tinyspy@3.0.2: {}
vite-node@2.1.9:
dependencies:
cac: 6.7.14
debug: 4.4.3
es-module-lexer: 1.7.0
pathe: 1.1.2
vite: 5.4.21
transitivePeerDependencies:
- '@types/node'
- less
- lightningcss
- sass
- sass-embedded
- stylus
- sugarss
- supports-color
- terser
vite@5.4.21:
dependencies:
esbuild: 0.21.5
postcss: 8.5.29
rollup: 4.64.2
optionalDependencies:
fsevents: 2.3.3
vitest@2.1.9:
dependencies:
'@vitest/expect': 2.1.9
'@vitest/mocker': 2.1.9(vite@5.4.21)
'@vitest/pretty-format': 2.1.9
'@vitest/runner': 2.1.9
'@vitest/snapshot': 2.1.9
'@vitest/spy': 2.1.9
'@vitest/utils': 2.1.9
chai: 5.3.3
debug: 4.4.3
expect-type: 1.4.0
magic-string: 0.30.21
pathe: 1.1.2
std-env: 3.10.0
tinybench: 2.9.0
tinyexec: 0.3.2
tinypool: 1.1.1
tinyrainbow: 1.2.0
vite: 5.4.21
vite-node: 2.1.9
why-is-node-running: 2.3.0
transitivePeerDependencies:
- less
- lightningcss
- msw
- sass
- sass-embedded
- stylus
- sugarss
- supports-color
- terser
why-is-node-running@2.3.0:
dependencies:
siginfo: 2.0.0
stackback: 0.0.2
hidden/pnpm-lock.yaml on GitHub
औरhidden/pnpm-workspace.yaml
allowBuilds:
esbuild: true
hidden/pnpm-workspace.yaml on GitHub
उसका setup हैं। -reference/F1.patch
diff --git a/src/domain/invoice.ts b/src/domain/invoice.ts
index 5b69fc9..0b9b69f 100644
--- a/src/domain/invoice.ts
+++ b/src/domain/invoice.ts
@@ -1,4 +1,4 @@
-import { ConflictError } from "./errors";
+import { ConflictError, ValidationError } from "./errors";
import { type Cents, type Currency, sum } from "./money";
export type InvoiceStatus = "draft" | "open" | "paid" | "void";
@@ -10,6 +10,12 @@ export type LineItem = {
unitPrice: Cents;
};
+export type Payment = {
+ /** cents, > 0 */
+ amount: Cents;
+ receivedAt: string;
+};
+
export type Invoice = {
id: string;
/** Assigned when the invoice is issued (INV-0001, ...). */
@@ -25,6 +31,8 @@ export type Invoice = {
dueDate: string | null;
paidAt: string | null;
voidedAt: string | null;
+ /** Every payment received, oldest first. A paid invoice's payments add up to its total. */
+ payments: Payment[];
};
export type InvoiceTotals = { subtotal: Cents; discount: Cents; total: Cents };
@@ -38,6 +46,14 @@ export function computeTotals(invoice: Pick<Invoice, "lines" | "discount">): Inv
return { subtotal, discount: invoice.discount, total: subtotal - invoice.discount };
}
+export function amountPaid(invoice: Pick<Invoice, "payments">): Cents {
+ return sum(invoice.payments.map((p) => p.amount));
+}
+
+export function balance(invoice: Pick<Invoice, "lines" | "discount" | "payments">): Cents {
+ return computeTotals(invoice).total - amountPaid(invoice);
+}
+
export function issue(invoice: Invoice, number: string, now: Date): Invoice {
if (invoice.status !== "draft") {
throw new ConflictError(`invoice ${invoice.id} is ${invoice.status}, only drafts can be issued`);
@@ -45,11 +61,28 @@ export function issue(invoice: Invoice, number: string, now: Date): Invoice {
return { ...invoice, status: "open", number, issuedAt: now.toISOString() };
}
+/** Pays whatever is left on an open invoice in one go. */
export function markPaid(invoice: Invoice, now: Date): Invoice {
if (invoice.status !== "open") {
throw new ConflictError(`invoice ${invoice.id} is ${invoice.status}, only open invoices can be paid`);
}
- return { ...invoice, status: "paid", paidAt: now.toISOString() };
+ const remaining = balance(invoice);
+ const payments = remaining > 0 ? [...invoice.payments, { amount: remaining, receivedAt: now.toISOString() }] : invoice.payments;
+ return { ...invoice, payments, status: "paid", paidAt: now.toISOString() };
+}
+
+/** Records a partial (or final) payment; the invoice flips to paid when its balance reaches zero. */
+export function recordPayment(invoice: Invoice, amount: Cents, now: Date): Invoice {
+ if (invoice.status !== "open") {
+ throw new ConflictError(`invoice ${invoice.id} is ${invoice.status}, only open invoices can take payments`);
+ }
+ const remaining = balance(invoice);
+ if (amount > remaining) {
+ throw new ValidationError(`amount ${amount} exceeds the balance of ${remaining}`, "amount");
+ }
+ const payments = [...invoice.payments, { amount, receivedAt: now.toISOString() }];
+ const settled = amount === remaining;
+ return { ...invoice, payments, status: settled ? "paid" : "open", paidAt: settled ? now.toISOString() : null };
}
export function voidInvoice(invoice: Invoice, now: Date): Invoice {
diff --git a/src/domain/validation.ts b/src/domain/validation.ts
index 08dfa92..f2af418 100644
--- a/src/domain/validation.ts
+++ b/src/domain/validation.ts
@@ -10,6 +10,13 @@ export type NewInvoiceInput = {
dueDate: string | null;
};
+export function parsePayment(body: unknown): { amount: number } {
+ if (typeof body !== "object" || body === null) {
+ throw new ValidationError("body must be an object");
+ }
+ return { amount: assertCents((body as Record<string, unknown>).amount, "amount", { min: 1 }) };
+}
+
const DATE = /^\d{4}-\d{2}-\d{2}$/;
export function parseLineItems(value: unknown): LineItem[] {
diff --git a/src/http/handlers/invoices.ts b/src/http/handlers/invoices.ts
index 09959e1..ac2768e 100644
--- a/src/http/handlers/invoices.ts
+++ b/src/http/handlers/invoices.ts
@@ -1,6 +1,6 @@
import { ValidationError } from "../../domain/errors";
-import { type Invoice, type InvoiceStatus, issue, markPaid, voidInvoice } from "../../domain/invoice";
-import { parseNewInvoice } from "../../domain/validation";
+import { type Invoice, type InvoiceStatus, issue, markPaid, recordPayment, voidInvoice } from "../../domain/invoice";
+import { parseNewInvoice, parsePayment } from "../../domain/validation";
import type { AppContext } from "../context";
import type { Router } from "../router";
import { invoiceView } from "../views";
@@ -22,6 +22,7 @@ export function invoiceRoutes(router: Router, ctx: AppContext): void {
issuedAt: null,
paidAt: null,
voidedAt: null,
+ payments: [],
};
return { status: 201, body: invoiceView(ctx.invoices.insert(invoice)) };
});
@@ -51,6 +52,14 @@ export function invoiceRoutes(router: Router, ctx: AppContext): void {
return { status: 200, body: invoiceView(ctx.invoices.save(paid)) };
});
+ /** Records a partial payment; the last one (balance reaches zero) marks the invoice paid. */
+ router.on("POST", "/invoices/:id/payments", ({ params, body }) => {
+ const current = ctx.invoices.get(params.id);
+ const { amount } = parsePayment(body);
+ const updated = recordPayment(current, amount, ctx.now());
+ return { status: 201, body: invoiceView(ctx.invoices.save(updated)) };
+ });
+
router.on("POST", "/invoices/:id/void", ({ params }) => {
const voided = voidInvoice(ctx.invoices.get(params.id), ctx.now());
return { status: 200, body: invoiceView(ctx.invoices.save(voided)) };
diff --git a/src/http/handlers/reports.ts b/src/http/handlers/reports.ts
index 19262c9..656969e 100644
--- a/src/http/handlers/reports.ts
+++ b/src/http/handlers/reports.ts
@@ -1,18 +1,17 @@
-import { computeTotals, isOverdue } from "../../domain/invoice";
+import { balance, isOverdue } from "../../domain/invoice";
import type { Currency } from "../../domain/money";
import type { AppContext } from "../context";
import type { Router } from "../router";
export function reportRoutes(router: Router, ctx: AppContext): void {
- /** What customers still owe us, per currency, over every open invoice. */
+ /** What customers still owe us (balances, net of partial payments), per currency, over every open invoice. */
router.on("GET", "/reports/outstanding", () => {
const now = ctx.now();
const totals: Partial<Record<Currency, number>> = {};
let count = 0;
let overdue = 0;
for (const invoice of ctx.invoices.list({ status: "open" })) {
- const { total } = computeTotals(invoice);
- totals[invoice.currency] = (totals[invoice.currency] ?? 0) + total;
+ totals[invoice.currency] = (totals[invoice.currency] ?? 0) + balance(invoice);
count += 1;
if (isOverdue(invoice, now)) overdue += 1;
}
diff --git a/src/http/views.ts b/src/http/views.ts
index 856944c..5b11bff 100644
--- a/src/http/views.ts
+++ b/src/http/views.ts
@@ -1,6 +1,6 @@
-import { computeTotals, type Invoice } from "../domain/invoice";
+import { amountPaid, balance, computeTotals, type Invoice } from "../domain/invoice";
/** The JSON shape returned for an invoice: stored fields plus computed totals. */
export function invoiceView(invoice: Invoice) {
- return { ...invoice, ...computeTotals(invoice) };
+ return { ...invoice, ...computeTotals(invoice), amountPaid: amountPaid(invoice), balance: balance(invoice) };
}
diff --git a/src/store/migrations.ts b/src/store/migrations.ts
index c8b73a9..1eab4df 100644
--- a/src/store/migrations.ts
+++ b/src/store/migrations.ts
@@ -1,6 +1,7 @@
+import { computeTotals, type Invoice } from "../domain/invoice";
import type { StoreData } from "./types";
-export const CURRENT_SCHEMA_VERSION = 1;
+export const CURRENT_SCHEMA_VERSION = 2;
type RawData = Record<string, unknown> & { schemaVersion?: number };
type Migration = (data: RawData) => RawData;
@@ -21,9 +22,31 @@ const v0ToV1: Migration = (data) => {
};
};
+/**
+ * v1 -> v2: invoices carry `payments`. Invoices paid before payments existed were paid in full:
+ * they get one payment of their total at `paidAt`, so their balance is zero.
+ */
+const v1ToV2: Migration = (data) => {
+ const invoices = (data.invoices as Record<string, unknown>[] | undefined) ?? [];
+ return {
+ ...data,
+ schemaVersion: 2,
+ invoices: invoices.map((inv) => {
+ if (Array.isArray(inv.payments)) return inv;
+ const invoice = inv as unknown as Invoice;
+ const payments =
+ invoice.status === "paid"
+ ? [{ amount: computeTotals(invoice).total, receivedAt: invoice.paidAt ?? invoice.issuedAt ?? invoice.createdAt }]
+ : [];
+ return { ...inv, payments };
+ }),
+ };
+};
+
/** migrations[n] upgrades a file from version n to n + 1. */
const migrations: Record<number, Migration> = {
0: v0ToV1,
+ 1: v1ToV2,
};
export function migrate(raw: unknown): StoreData {
diff --git a/test/invoice.test.ts b/test/invoice.test.ts
index 85202f2..e043b5d 100644
--- a/test/invoice.test.ts
+++ b/test/invoice.test.ts
@@ -21,6 +21,7 @@ function draft(overrides: Partial<Invoice> = {}): Invoice {
dueDate: null,
paidAt: null,
voidedAt: null,
+ payments: [],
...overrides,
};
}
diff --git a/test/payments.test.ts b/test/payments.test.ts
new file mode 100644
index 0000000..baf9da7
--- /dev/null
+++ b/test/payments.test.ts
@@ -0,0 +1,53 @@
+import { describe, expect, it } from "vitest";
+import { openStore } from "../src/store/json-store";
+import { seedCustomer, seedOpenInvoice, testApp } from "./helpers";
+
+describe("partial payments", () => {
+ it("accumulates payments and flips to paid at a zero balance", async () => {
+ const app = testApp();
+ const id = await seedOpenInvoice(app, { customerId: await seedCustomer(app), unitPrice: 10_000 });
+ expect((await app.handle("POST", `/invoices/${id}/payments`, { amount: 4_000 })).body).toMatchObject({
+ status: "open",
+ amountPaid: 4_000,
+ balance: 6_000,
+ });
+ const last = await app.handle("POST", `/invoices/${id}/payments`, { amount: 6_000 });
+ expect(last).toMatchObject({ status: 201, body: { status: "paid", amountPaid: 10_000, balance: 0 } });
+ });
+
+ it("rejects bad amounts and overpayments with 400", async () => {
+ const app = testApp();
+ const id = await seedOpenInvoice(app, { customerId: await seedCustomer(app), unitPrice: 10_000 });
+ for (const amount of [0, -1, 1.5, "10", undefined]) {
+ expect((await app.handle("POST", `/invoices/${id}/payments`, { amount })).status).toBe(400);
+ }
+ expect((await app.handle("POST", `/invoices/${id}/payments`, { amount: 10_001 })).status).toBe(400);
+ });
+
+ it("refuses payments on drafts, void and unknown invoices", async () => {
+ const app = testApp();
+ const customerId = await seedCustomer(app);
+ const draft = await app.handle("POST", "/invoices", {
+ customerId,
+ currency: "EUR",
+ lines: [{ description: "x", quantity: 1, unitPrice: 100 }],
+ });
+ expect((await app.handle("POST", `/invoices/${(draft.body as { id: string }).id}/payments`, { amount: 10 })).status).toBe(409);
+ const voided = await seedOpenInvoice(app, { customerId });
+ await app.handle("POST", `/invoices/${voided}/void`);
+ expect((await app.handle("POST", `/invoices/${voided}/payments`, { amount: 10 })).status).toBe(409);
+ expect((await app.handle("POST", "/invoices/inv_404/payments", { amount: 10 })).status).toBe(404);
+ });
+
+ it("keeps /pay consistent and reports balances, not totals", async () => {
+ const app = testApp(openStore());
+ const customerId = await seedCustomer(app);
+ const a = await seedOpenInvoice(app, { customerId, unitPrice: 10_000 });
+ const b = await seedOpenInvoice(app, { customerId, unitPrice: 5_000 });
+ await app.handle("POST", `/invoices/${a}/payments`, { amount: 2_500 });
+ expect((await app.handle("GET", "/reports/outstanding")).body).toMatchObject({ totals: { EUR: 12_500 } });
+ expect((await app.handle("POST", `/invoices/${a}/pay`)).body).toMatchObject({ status: "paid", amountPaid: 10_000, balance: 0 });
+ expect((await app.handle("GET", "/reports/outstanding")).body).toMatchObject({ totals: { EUR: 5_000 }, count: 1 });
+ expect(b).toBeTruthy();
+ });
+});
diff --git a/test/store.test.ts b/test/store.test.ts
index a1bf345..3201e2a 100644
--- a/test/store.test.ts
+++ b/test/store.test.ts
@@ -42,6 +42,26 @@ describe("store", () => {
expect(JSON.parse(readFileSync(file, "utf8")).schemaVersion).toBe(CURRENT_SCHEMA_VERSION);
});
+ it("migrates a v1 file: invoices paid before payments existed get one payment of their total", () => {
+ const file = tmpFile();
+ const inv = { customerId: "cus_1", currency: "EUR", lines: [{ description: "x", quantity: 2, unitPrice: 1_500 }], discount: 0 };
+ writeFileSync(
+ file,
+ JSON.stringify({
+ schemaVersion: 1,
+ counters: { customer: 1, invoice: 2, invoiceNumber: 2 },
+ customers: [],
+ invoices: [
+ { ...inv, id: "inv_1", status: "paid", paidAt: "2026-02-01T00:00:00.000Z" },
+ { ...inv, id: "inv_2", status: "open", paidAt: null },
+ ],
+ }),
+ );
+ const [paid, open] = openStore({ file }).read().invoices;
+ expect(paid.payments).toEqual([{ amount: 3_000, receivedAt: "2026-02-01T00:00:00.000Z" }]);
+ expect(open.payments).toEqual([]);
+ });
+
it("refuses a file from a newer build", () => {
expect(() => migrate({ schemaVersion: CURRENT_SCHEMA_VERSION + 1 })).toThrow(/newer/);
});
reference/F2.patch
diff --git a/src/domain/credit-note.ts b/src/domain/credit-note.ts
new file mode 100644
index 0000000..7504c53
--- /dev/null
+++ b/src/domain/credit-note.ts
@@ -0,0 +1,44 @@
+import { ConflictError, ValidationError } from "./errors";
+import { computeTotals, type Invoice } from "./invoice";
+import { assertCents, type Cents, sum } from "./money";
+
+/** A numbered accounting document that reduces what is owed on an issued invoice. Never edited or deleted. */
+export type CreditNote = {
+ id: string;
+ /** CN-0001, ... gapless, in the order credit notes are created. */
+ number: string;
+ invoiceId: string;
+ amount: Cents;
+ createdAt: string;
+};
+
+export function creditedAmount(notes: Pick<CreditNote, "amount">[]): Cents {
+ return sum(notes.map((n) => n.amount));
+}
+
+/** What can still be credited on an invoice: its total minus every credit note already issued against it. */
+export function creditable(invoice: Invoice, notes: Pick<CreditNote, "amount">[]): Cents {
+ return computeTotals(invoice).total - creditedAmount(notes);
+}
+
+export function parseCreditAmount(body: unknown): Cents {
+ if (typeof body !== "object" || body === null) {
+ throw new ValidationError("body must be an object");
+ }
+ return assertCents((body as Record<string, unknown>).amount, "amount", { min: 1 });
+}
+
+/** Throws unless `amount` can be credited on `invoice` now. Call it BEFORE taking a number. */
+export function assertCreditable(invoice: Invoice, notes: Pick<CreditNote, "amount">[], amount: Cents): void {
+ if (invoice.status !== "open" && invoice.status !== "paid") {
+ throw new ConflictError(`invoice ${invoice.id} is ${invoice.status}: only open or paid invoices can be credited`);
+ }
+ const left = creditable(invoice, notes);
+ if (amount > left) {
+ throw new ValidationError(`amount ${amount} exceeds what is left to credit (${left})`, "amount");
+ }
+}
+
+export function creditNoteNumber(n: number): string {
+ return `CN-${String(n).padStart(4, "0")}`;
+}
diff --git a/src/domain/invoice.ts b/src/domain/invoice.ts
index 5b69fc9..2ee6fa4 100644
--- a/src/domain/invoice.ts
+++ b/src/domain/invoice.ts
@@ -52,8 +52,9 @@ export function markPaid(invoice: Invoice, now: Date): Invoice {
return { ...invoice, status: "paid", paidAt: now.toISOString() };
}
+/** Voids a draft, open or paid invoice. A paid one must be fully credited first (the handler issues that credit note). */
export function voidInvoice(invoice: Invoice, now: Date): Invoice {
- if (invoice.status === "paid" || invoice.status === "void") {
+ if (invoice.status === "void") {
throw new ConflictError(`invoice ${invoice.id} is ${invoice.status} and cannot be voided`);
}
return { ...invoice, status: "void", voidedAt: now.toISOString() };
diff --git a/src/http/context.ts b/src/http/context.ts
index c0a89d3..003b7af 100644
--- a/src/http/context.ts
+++ b/src/http/context.ts
@@ -1,4 +1,5 @@
import type { Store } from "../store/json-store";
+import { creditNoteRepo } from "../store/credit-note-repo";
import { customerRepo } from "../store/customer-repo";
import { invoiceRepo } from "../store/invoice-repo";
@@ -7,8 +8,9 @@ export type AppContext = {
now: () => Date;
customers: ReturnType<typeof customerRepo>;
invoices: ReturnType<typeof invoiceRepo>;
+ creditNotes: ReturnType<typeof creditNoteRepo>;
};
export function createContext(store: Store, now: () => Date = () => new Date()): AppContext {
- return { store, now, customers: customerRepo(store), invoices: invoiceRepo(store) };
+ return { store, now, customers: customerRepo(store), invoices: invoiceRepo(store), creditNotes: creditNoteRepo(store) };
}
diff --git a/src/http/handlers/invoices.ts b/src/http/handlers/invoices.ts
index 09959e1..469a402 100644
--- a/src/http/handlers/invoices.ts
+++ b/src/http/handlers/invoices.ts
@@ -1,3 +1,4 @@
+import { assertCreditable, type CreditNote, creditNoteNumber, creditable, parseCreditAmount } from "../../domain/credit-note";
import { ValidationError } from "../../domain/errors";
import { type Invoice, type InvoiceStatus, issue, markPaid, voidInvoice } from "../../domain/invoice";
import { parseNewInvoice } from "../../domain/validation";
@@ -51,8 +52,37 @@ export function invoiceRoutes(router: Router, ctx: AppContext): void {
return { status: 200, body: invoiceView(ctx.invoices.save(paid)) };
});
+ /** Validation first, number last: a rejected credit note never consumes a CN number (numbers stay gapless). */
+ function credit(invoice: Invoice, amount: number): CreditNote {
+ assertCreditable(invoice, ctx.creditNotes.forInvoice(invoice.id), amount);
+ const n = ctx.store.nextId("creditNote");
+ return ctx.creditNotes.insert({
+ id: `cn_${n}`,
+ number: creditNoteNumber(n),
+ invoiceId: invoice.id,
+ amount,
+ createdAt: ctx.now().toISOString(),
+ });
+ }
+
+ router.on("POST", "/invoices/:id/credit-notes", ({ params, body }) => {
+ const invoice = ctx.invoices.get(params.id);
+ return { status: 201, body: credit(invoice, parseCreditAmount(body)) };
+ });
+
+ router.on("GET", "/invoices/:id/credit-notes", ({ params }) => {
+ ctx.invoices.get(params.id);
+ return { status: 200, body: ctx.creditNotes.forInvoice(params.id) };
+ });
+
+ /** Voiding a paid invoice credits whatever is left on it first. */
router.on("POST", "/invoices/:id/void", ({ params }) => {
- const voided = voidInvoice(ctx.invoices.get(params.id), ctx.now());
+ const invoice = ctx.invoices.get(params.id);
+ const voided = voidInvoice(invoice, ctx.now());
+ if (invoice.status === "paid") {
+ const left = creditable(invoice, ctx.creditNotes.forInvoice(invoice.id));
+ if (left > 0) credit(invoice, left);
+ }
return { status: 200, body: invoiceView(ctx.invoices.save(voided)) };
});
}
diff --git a/src/http/handlers/reports.ts b/src/http/handlers/reports.ts
index 19262c9..f980c9b 100644
--- a/src/http/handlers/reports.ts
+++ b/src/http/handlers/reports.ts
@@ -1,4 +1,5 @@
-import { computeTotals, isOverdue } from "../../domain/invoice";
+import { creditable } from "../../domain/credit-note";
+import { isOverdue } from "../../domain/invoice";
import type { Currency } from "../../domain/money";
import type { AppContext } from "../context";
import type { Router } from "../router";
@@ -11,8 +12,8 @@ export function reportRoutes(router: Router, ctx: AppContext): void {
let count = 0;
let overdue = 0;
for (const invoice of ctx.invoices.list({ status: "open" })) {
- const { total } = computeTotals(invoice);
- totals[invoice.currency] = (totals[invoice.currency] ?? 0) + total;
+ const owed = creditable(invoice, ctx.creditNotes.forInvoice(invoice.id));
+ totals[invoice.currency] = (totals[invoice.currency] ?? 0) + owed;
count += 1;
if (isOverdue(invoice, now)) overdue += 1;
}
diff --git a/src/store/credit-note-repo.ts b/src/store/credit-note-repo.ts
new file mode 100644
index 0000000..eac6632
--- /dev/null
+++ b/src/store/credit-note-repo.ts
@@ -0,0 +1,17 @@
+import type { CreditNote } from "../domain/credit-note";
+import type { Store } from "./json-store";
+
+export function creditNoteRepo(store: Store) {
+ return {
+ forInvoice(invoiceId: string): CreditNote[] {
+ return store.read().creditNotes.filter((n) => n.invoiceId === invoiceId);
+ },
+ all(): CreditNote[] {
+ return store.read().creditNotes;
+ },
+ insert(note: CreditNote): CreditNote {
+ store.update((d) => ({ ...d, creditNotes: [...d.creditNotes, note] }));
+ return note;
+ },
+ };
+}
diff --git a/src/store/json-store.ts b/src/store/json-store.ts
index b404c51..99205b7 100644
--- a/src/store/json-store.ts
+++ b/src/store/json-store.ts
@@ -13,9 +13,10 @@ export type Store = {
function empty(): StoreData {
return {
schemaVersion: CURRENT_SCHEMA_VERSION,
- counters: { customer: 0, invoice: 0, invoiceNumber: 0 },
+ counters: { customer: 0, invoice: 0, invoiceNumber: 0, creditNote: 0 },
customers: [],
invoices: [],
+ creditNotes: [],
};
}
diff --git a/src/store/migrations.ts b/src/store/migrations.ts
index c8b73a9..eb7abfb 100644
--- a/src/store/migrations.ts
+++ b/src/store/migrations.ts
@@ -1,6 +1,6 @@
import type { StoreData } from "./types";
-export const CURRENT_SCHEMA_VERSION = 1;
+export const CURRENT_SCHEMA_VERSION = 2;
type RawData = Record<string, unknown> & { schemaVersion?: number };
type Migration = (data: RawData) => RawData;
@@ -21,9 +21,18 @@ const v0ToV1: Migration = (data) => {
};
};
+/** v1 -> v2: credit notes (a collection and their own counter). */
+const v1ToV2: Migration = (data) => ({
+ ...data,
+ schemaVersion: 2,
+ counters: { ...(data.counters as Record<string, number> | undefined), creditNote: 0 },
+ creditNotes: [],
+});
+
/** migrations[n] upgrades a file from version n to n + 1. */
const migrations: Record<number, Migration> = {
0: v0ToV1,
+ 1: v1ToV2,
};
export function migrate(raw: unknown): StoreData {
@@ -43,8 +52,9 @@ export function migrate(raw: unknown): StoreData {
}
return {
schemaVersion: CURRENT_SCHEMA_VERSION,
- counters: (data.counters as StoreData["counters"]) ?? { customer: 0, invoice: 0, invoiceNumber: 0 },
+ counters: { customer: 0, invoice: 0, invoiceNumber: 0, creditNote: 0, ...(data.counters as Partial<StoreData["counters"]>) },
customers: (data.customers as StoreData["customers"]) ?? [],
invoices: (data.invoices as StoreData["invoices"]) ?? [],
+ creditNotes: (data.creditNotes as StoreData["creditNotes"]) ?? [],
};
}
diff --git a/src/store/types.ts b/src/store/types.ts
index 222b191..a9d7277 100644
--- a/src/store/types.ts
+++ b/src/store/types.ts
@@ -1,7 +1,8 @@
+import type { CreditNote } from "../domain/credit-note";
import type { Customer } from "../domain/customer";
import type { Invoice } from "../domain/invoice";
-export type Counters = { customer: number; invoice: number; invoiceNumber: number };
+export type Counters = { customer: number; invoice: number; invoiceNumber: number; creditNote: number };
/** The shape of the JSON file at CURRENT_SCHEMA_VERSION. */
export type StoreData = {
@@ -9,4 +10,5 @@ export type StoreData = {
counters: Counters;
customers: Customer[];
invoices: Invoice[];
+ creditNotes: CreditNote[];
};
diff --git a/test/credit-notes.test.ts b/test/credit-notes.test.ts
new file mode 100644
index 0000000..240f820
--- /dev/null
+++ b/test/credit-notes.test.ts
@@ -0,0 +1,52 @@
+import { mkdtempSync, writeFileSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+import { describe, expect, it } from "vitest";
+import { openStore } from "../src/store/json-store";
+import { seedCustomer, seedOpenInvoice, testApp } from "./helpers";
+
+describe("credit notes", () => {
+ it("numbers credit notes gaplessly; rejected ones take no number", async () => {
+ const app = testApp();
+ const id = await seedOpenInvoice(app, { customerId: await seedCustomer(app), unitPrice: 10_000 });
+ expect((await app.handle("POST", `/invoices/${id}/credit-notes`, { amount: 3_000 })).body).toMatchObject({ number: "CN-0001" });
+ expect((await app.handle("POST", `/invoices/${id}/credit-notes`, { amount: 7_001 })).status).toBe(400);
+ expect((await app.handle("POST", `/invoices/${id}/credit-notes`, { amount: 0 })).status).toBe(400);
+ expect((await app.handle("POST", `/invoices/${id}/credit-notes`, { amount: 7_000 })).body).toMatchObject({ number: "CN-0002" });
+ expect(((await app.handle("GET", `/invoices/${id}/credit-notes`)).body as unknown[]).length).toBe(2);
+ });
+
+ it("refuses drafts and void invoices (409) and unknown ones (404)", async () => {
+ const app = testApp();
+ const customerId = await seedCustomer(app);
+ const draft = await app.handle("POST", "/invoices", { customerId, currency: "EUR", lines: [{ description: "x", quantity: 1, unitPrice: 100 }] });
+ expect((await app.handle("POST", `/invoices/${(draft.body as { id: string }).id}/credit-notes`, { amount: 10 })).status).toBe(409);
+ expect((await app.handle("POST", "/invoices/inv_404/credit-notes", { amount: 10 })).status).toBe(404);
+ });
+
+ it("voids a paid invoice by crediting what is left", async () => {
+ const app = testApp();
+ const id = await seedOpenInvoice(app, { customerId: await seedCustomer(app), unitPrice: 10_000 });
+ await app.handle("POST", `/invoices/${id}/pay`);
+ await app.handle("POST", `/invoices/${id}/credit-notes`, { amount: 2_500 });
+ expect((await app.handle("POST", `/invoices/${id}/void`)).body).toMatchObject({ status: "void" });
+ const notes = (await app.handle("GET", `/invoices/${id}/credit-notes`)).body as { amount: number }[];
+ expect(notes.map((n) => n.amount)).toEqual([2_500, 7_500]);
+ });
+
+ it("nets credit notes out of the outstanding report", async () => {
+ const app = testApp();
+ const id = await seedOpenInvoice(app, { customerId: await seedCustomer(app), unitPrice: 10_000 });
+ await app.handle("POST", `/invoices/${id}/credit-notes`, { amount: 4_000 });
+ expect((await app.handle("GET", "/reports/outstanding")).body).toMatchObject({ totals: { EUR: 6_000 }, count: 1 });
+ });
+
+ it("works on a store file written before credit notes existed", async () => {
+ const file = join(mkdtempSync(join(tmpdir(), "ledgerly-")), "data.json");
+ writeFileSync(file, JSON.stringify({ schemaVersion: 1, counters: { customer: 0, invoice: 0, invoiceNumber: 0 }, customers: [], invoices: [] }));
+ const app = testApp(openStore({ file }));
+ const id = await seedOpenInvoice(app, { customerId: await seedCustomer(app) });
+ expect((await app.handle("POST", `/invoices/${id}/credit-notes`, { amount: 100 })).body).toMatchObject({ id: "cn_1", number: "CN-0001" });
+ expect(openStore({ file }).read().creditNotes.length).toBe(1);
+ });
+});
diff --git a/test/invoice.test.ts b/test/invoice.test.ts
index 85202f2..4219226 100644
--- a/test/invoice.test.ts
+++ b/test/invoice.test.ts
@@ -54,8 +54,11 @@ describe("invoice domain", () => {
});
it("voids drafts and open invoices, never paid ones", () => {
+ // Meaning changed with credit notes: a paid invoice can now be voided (the handler credits what is left first);
+ // only an already void invoice is refused.
expect(voidInvoice(draft(), NOW).status).toBe("void");
- expect(() => voidInvoice(draft({ status: "paid" }), NOW)).toThrow(ConflictError);
+ expect(voidInvoice(draft({ status: "paid" }), NOW).status).toBe("void");
+ expect(() => voidInvoice(draft({ status: "void" }), NOW)).toThrow(ConflictError);
});
it("knows when an open invoice is overdue", () => {
Runner और driver:
run.sh
#!/usr/bin/env bash
# One run: fresh work copy + pack fixtures, the driver (worker turns + PO), then finalize.
# Usage: [TASK=F1|F2] run.sh <cfg> <rep> cfg: bare|sp|spd|mp|both
# RUN_TAG=x -> runs/<name>-x (smoke runs; analyze.py ignores tagged runs)
# The work copy lives OUTSIDE the repo tree ($FLOW_BENCH_WORK, default /private/tmp/flow-bench/<name>): Claude Code
# loads every CLAUDE.md of the parent directories, so a work copy under this repo read the repo's CLAUDE.md (and the
# instructions it pulls in) in every session. After finalize the copy (without node_modules) is archived to work/<name>.
set -uo pipefail
CFG="$1"; REP="${2:-1}"
HERE="$(cd "$(dirname "$0")" && pwd)"
. "$HERE/env.sh"
TASK="${TASK:-F1}"; export TASK
PFX=""; [ "$TASK" != "F1" ] && PFX="$(echo "$TASK" | tr "A-Z" "a-z")-"
NAME="${PFX}${CFG}-r${REP}${RUN_TAG:+-$RUN_TAG}"
WROOT="${FLOW_BENCH_WORK:-/private/tmp/flow-bench}"
OUT="$HERE/runs/$NAME"; W="$WROOT/$NAME"
rm -rf "$OUT"; mkdir -p "$OUT"
"$HERE/prep.sh" "$CFG" "$W" > "$OUT/prep.txt" 2>&1 || { echo "$NAME: prep failed"; cat "$OUT/prep.txt"; exit 2; }
BASE=$(tail -1 "$OUT/prep.txt")
cp "$W/.claude/settings.json" "$OUT/settings.json"
printf '{"task":"%s","cfg":"%s","rep":%s,"tag":"%s","work":"%s","name":"%s","base":"%s","started":"%s","claude_version":"%s"}\n' \
"$TASK" "$CFG" "$REP" "${RUN_TAG:-}" "$W" "$NAME" "$BASE" "$(date '+%F %T')" "$(claude --version | head -1)" > "$OUT/meta.json"
python3 "$HERE/lib/driver.py" "$CFG" "$OUT" "$W" "$BASE" > "$OUT/driver.out" 2> "$OUT/driver.err"
python3 "$HERE/lib/finalize.py" "$OUT"
rm -rf "$HERE/work/$NAME"; mkdir -p "$HERE/work"
rsync -a --exclude node_modules "$W/" "$HERE/work/$NAME/"
prep.sh
#!/usr/bin/env bash
# Fresh work copy of upstream (node_modules included) + the pack fixtures of <cfg>.
# Usage: prep.sh <cfg> <work dir> Prints the base commit (the oracle's BASE) on its last line.
set -euo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"
CFG="$1"; W="$2"
SP=$(python3 -c "import json;print(int(json.load(open('$HERE/configs.json'))['$CFG']['superpowers']))")
MP=$(python3 -c "import json;print(int(json.load(open('$HERE/configs.json'))['$CFG']['mattpocock']))")
rm -rf "$W"; mkdir -p "$(dirname "$W")"
cp -R "$HERE/upstream" "$W"
cd "$W"
# .claude/ never shows in git status or in any diff (hook config, installed skills).
echo ".claude/" >> .git/info/exclude
mkdir -p .claude/skills
if [ "$MP" = 1 ]; then
cp -R "$HERE/packs/mattpocock-skills/.claude/skills/." .claude/skills/
# One-time per-repo precondition of the pack (/setup-matt-pocock-skills), answered once with the defaults it
# recommends for a repo with no remote: local-markdown tracker, single-context domain docs, no triage labels
# (triage is installed, so Section B would ask; default labels kept). Committed so it never shows in the diff.
S="$HERE/packs/mattpocock-skills/.claude/skills/setup-matt-pocock-skills"
mkdir -p docs/agents
cp "$S/issue-tracker-local.md" docs/agents/issue-tracker.md
cp "$S/domain.md" docs/agents/domain.md
cp "$S/triage-labels.md" docs/agents/triage-labels.md
cat > CLAUDE.md <<'MD'
## Agent skills
### Issue tracker
Issues and specs live as local markdown files under `.scratch/`. See `docs/agents/issue-tracker.md`.
### Triage labels
Default labels (`needs-triage`, `needs-info`, `ready-for-agent`, `ready-for-human`, `wontfix`). See `docs/agents/triage-labels.md`.
### Domain docs
Single-context: one `GLOSSARY.md` + `docs/adr/` at the repo root. See `docs/agents/domain.md`.
MD
git add CLAUDE.md docs/agents
GIT_AUTHOR_DATE="2026-03-09T18:00:00" GIT_COMMITTER_DATE="2026-03-09T18:00:00" \
git -c user.name="Dana Whitfield" -c user.email="dana@ledgerly.dev" commit -q -m "chore: configure agent skills (local tracker)"
fi
if [ "$SP" = 1 ]; then
# The plugin root as Claude Code would hold it, so the hook script finds ../skills/using-superpowers.
mkdir -p .claude/plugins
cp -R "$HERE/packs/superpowers" .claude/plugins/superpowers
cp -R "$HERE/packs/superpowers/skills/." .claude/skills/
fi
python3 - "$HERE" "$SP" "$CFG" > .claude/settings.json <<'PY'
import json, sys
bench, sp, cfg = sys.argv[1], sys.argv[2] == "1", sys.argv[3]
# Oracle, reference, PO brief, other runs: denied by path (Read/Grep/Glob) and by command text (Bash).
deny = ["Read(/%s/hidden/**)" % bench, "Read(/%s/reference/**)" % bench, "Read(/%s/runs/**)" % bench,
"Read(/%s/po/**)" % bench, "Read(/%s/validation/**)" % bench,
"Bash(*hidden*)", "Bash(*reference/F*)", "Bash(*-brief.md*)", "Bash(*/runs/*)", "Bash(*../*)", "Bash(find *)"]
if cfg == "spd":
# executing-plans runs its sibling scripts as ../subagent-driven-development/scripts/...: a "../" deny would block the
# pack's own loop. The oracle stays protected by the Read denies and Bash(*hidden*).
deny.remove("Bash(*../*)")
settings = {"permissions": {"deny": deny}}
if sp:
# = packs/superpowers/hooks/hooks.json, with ${CLAUDE_PLUGIN_ROOT} provided the way the plugin loader provides it
# (the script also picks its JSON output shape from CLAUDE_PLUGIN_ROOT being set).
root = '$CLAUDE_PROJECT_DIR/.claude/plugins/superpowers'
hooks = json.load(open("%s/packs/superpowers/hooks/hooks.json" % bench))["hooks"]
for groups in hooks.values():
for g in groups:
for h in g["hooks"]:
h["command"] = 'CLAUDE_PLUGIN_ROOT="%s" %s' % (root, h["command"].replace("${CLAUDE_PLUGIN_ROOT}", root))
settings["hooks"] = hooks
print(json.dumps(settings, indent=2))
PY
[ "$(ls .claude/skills | wc -l)" -eq 0 ] && rmdir .claude/skills
test -z "$(git status --porcelain)" || { echo "work copy not clean"; git status --porcelain; exit 1; }
git rev-parse HEAD
env.sh
# Sourced by run.sh, gate.sh and validate.sh.
# 1. Scrub every CLAUDE* variable inherited from a launching Claude Code session
# (CLAUDE_EFFORT, CLAUDE_CODE_SUBAGENT_MODEL, CLAUDE_CODE_SESSION_ID, CLAUDECODE,
# CLAUDE_CODE_CHILD_SESSION, ...): they would silently change the worker's effort,
# subagent model and session bookkeeping.
for _v in $(env | sed -n 's/^\(CLAUDE[A-Za-z0-9_]*\)=.*/\1/p'); do unset "$_v"; done
unset _v
# 2. Auth: the user's default config dir (~/.claude, OAuth). See README traps (--bare cannot auth).
unset CLAUDE_CONFIG_DIR
# 3. Fresh work copies at the same path must not share auto-memory across reps.
export CLAUDE_CODE_DISABLE_AUTO_MEMORY=1
# 4. No bytecode in the shared hidden/ dir when 3 runs execute hidden tests in parallel.
export PYTHONDONTWRITEBYTECODE=1
# 5. The bench venv first: `python -m pytest` inside the work copy uses it.
GATE_BENCH="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
export GATE_BENCH
export PATH="$GATE_BENCH/.venv/bin:$PATH"
configs.json
{
"bare": {
"superpowers": false,
"mattpocock": false,
"flow": "bare",
"desc": "no skills, no hooks (project settings carry only the permission deny-list)"
},
"sp": {
"superpowers": true,
"mattpocock": false,
"flow": "sp",
"desc": "HOOK-ROUTED: obra/superpowers v6.4.2: skills in .claude/skills, SessionStart hook in .claude/settings.json as hooks/hooks.json wires it; turn 1 = the bare task, the hook decides"
},
"mp": {
"superpowers": false,
"mattpocock": true,
"flow": "mp",
"desc": "mattpocock/skills v1.3.1 via npx skills add (27 promoted skills, copied), setup-matt-pocock-skills output committed (local tracker)"
},
"both": {
"superpowers": true,
"mattpocock": true,
"flow": "mp",
"desc": "sp + mp in one project; driven with the mp slash-command flow (the stacking question: does the Superpowers hook hijack it?)"
},
"spd": {
"superpowers": true,
"mattpocock": false,
"flow": "spd",
"desc": "Superpowers, DRIVEN: same project install as sp (hook included); the driver types /brainstorming <task> -> /writing-plans -> /executing-plans (Native/inline); the pack's own scripts allowed in Bash"
}
}
lib/driver.py
"""One run of the flow bench: drives a headless Sonnet 5.5 session turn by turn, routes questions to the PO.
driver.py <cfg> <run dir> <work dir> <base sha>
Every worker turn: `claude -p <prompt> [--resume <sid>] --output-format stream-json --verbose --include-hook-events
--include-partial-messages ...` with cwd = the work copy. Per turn it writes turn-NN.stream.jsonl (raw stdout),
turn-NN.times (receive time of every stdout line), turn-NN.stderr, and appends one record to turns.jsonl.
Caps (SPEC): 12 driver turns, --max-turns 60 per claude turn, 40 min wall per run (PO time included).
"""
import json
import os
import subprocess
import sys
import threading
import time
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import po # noqa: E402
from accounting import account_turn # noqa: E402
BENCH = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
MODEL = "claude-sonnet-5-5"
MAX_DRIVER_TURNS = int(os.environ.get("BENCH_MAX_DRIVER_TURNS", "12"))
MAX_TURNS = os.environ.get("BENCH_MAX_TURNS", "60")
WALL_CAP = float(os.environ.get("BENCH_WALL_CAP_S", str(40 * 60)))
# SPEC allow-list. AskUserQuestion is removed (headless: questions come as plain text); plan mode and worktree
# tools are removed too (a headless session cannot leave plan mode, and a worktree would move the work out of
# the scored copy). Read-only shell commands are auto-approved by Claude Code whatever the list says.
ALLOWED = ",".join([
"Read", "Edit", "Write", "MultiEdit", "Glob", "Grep", "Skill", "TodoWrite", "Task", "Agent",
"Bash(node:*)", "Bash(npm:*)", "Bash(pnpm:*)", "Bash(npx vitest:*)", "Bash(npx tsc:*)", "Bash(tsc:*)",
"Bash(git status:*)", "Bash(git diff:*)", "Bash(git log:*)", "Bash(git add:*)", "Bash(git commit:*)",
"Bash(git checkout -b:*)", "Bash(ls:*)", "Bash(cat:*)", "Bash(mkdir:*)", "Bash(head:*)", "Bash(tail:*)",
"Bash(grep:*)", "Bash(wc:*)", "Bash(gh:*)",
])
# spd only: the scripts executing-plans / subagent-driven-development call (task-start, task-done, sdd-workspace,
# task-brief, review-package), wherever the model addresses them from.
SPD_EXTRA = ",".join("Bash(*scripts/%s*)" % x for x in ("task-start", "task-done", "sdd-workspace", "task-brief", "review-package"))
DISALLOWED = "AskUserQuestion,EnterPlanMode,ExitPlanMode,EnterWorktree,ExitWorktree,WebFetch,WebSearch"
PRE_IMPLEMENT = ("/grill-with-docs", "/to-spec", "/to-tickets")
SPD_PRE_IMPLEMENT = ("/brainstorming", "/writing-plans")
DONE_SUFFIX = "\n\nWhen you are done, stop and summarise what you changed."
def worker_env():
env = {k: v for k, v in os.environ.items() if not k.startswith("CLAUDE")}
env["CLAUDE_CODE_DISABLE_AUTO_MEMORY"] = "1"
return env
def run_claude(prompt, sid, work, out_prefix, timeout, allowed=ALLOWED):
cmd = ["claude", "-p", prompt]
if sid:
cmd += ["--resume", sid]
cmd += ["--model", MODEL, "--output-format", "stream-json", "--verbose", "--include-hook-events",
"--include-partial-messages", "--max-turns", MAX_TURNS, "--permission-mode", "acceptEdits",
"--allowedTools", allowed, "--disallowedTools", DISALLOWED,
"--setting-sources", "project", "--strict-mcp-config"]
t0 = time.time()
fo = open(out_prefix + ".stream.jsonl", "w")
ft = open(out_prefix + ".times", "w")
fe = open(out_prefix + ".stderr", "w")
p = subprocess.Popen(cmd, cwd=work, env=worker_env(), stdin=subprocess.DEVNULL, stdout=subprocess.PIPE,
stderr=fe, text=True, bufsize=1)
killed = {"v": False}
def watchdog():
try:
p.wait(timeout=timeout)
except subprocess.TimeoutExpired:
killed["v"] = True
p.kill()
th = threading.Thread(target=watchdog, daemon=True)
th.start()
for line in p.stdout:
fo.write(line)
ft.write("%.3f\n" % (time.time() - t0))
p.wait()
th.join(timeout=5)
for f in (fo, ft, fe):
f.close()
return {"exit_code": p.returncode, "wall_s": round(time.time() - t0, 2), "killed_wall_cap": killed["v"], "t0": t0}
def scratch_issues(work):
root = os.path.join(work, ".scratch")
if not os.path.isdir(root):
return None
for d in sorted(os.listdir(root)):
if os.path.isdir(os.path.join(root, d, "issues")):
return ".scratch/%s/issues/" % d
return None
def main():
cfg, run_dir, work, base = sys.argv[1:5]
conf = json.load(open(os.path.join(BENCH, "configs.json")))[cfg]
flow = conf["flow"]
task_id = os.environ.get("TASK", "F1")
task = open(os.path.join(BENCH, "tasks", task_id + ".md")).read().strip()
allowed = ALLOWED + ("," + SPD_EXTRA if flow == "spd" else "")
spd_stages = ["/brainstorming", "/writing-plans", "/executing-plans"]
spd_i = 0
t_run = time.time()
sid = None
prev = None # cumulative modelUsage / cost of the session so far
stage_in_force = "(none)"
caps = []
turns_log = open(os.path.join(run_dir, "turns.jsonl"), "a")
# mp flow: the slash commands in SPEC order; the implement stage is built when it is reached.
mp_stages = ["/grill-with-docs", "/to-spec", "/to-tickets", "/implement", "/code-review", "/pr"]
mp_i = 0
def mp_prompt(i):
s = mp_stages[i]
if s == "/grill-with-docs":
return "/grill-with-docs " + task
if s == "/implement":
issues = scratch_issues(work)
return "/implement " + ("the tickets in %s, all of them, blockers first" % issues if issues
else "the spec and tickets above")
if s == "/code-review":
return "/code-review since %s" % base
return s
if flow == "bare":
prompt, stage = task + DONE_SUFFIX, "bare"
elif flow == "spd":
prompt, stage = "/brainstorming " + task, spd_stages[0]
stage_in_force = "brainstorming"
elif flow == "sp":
prompt, stage = task, None
else:
prompt, stage = mp_prompt(0), mp_stages[0]
source = "driver"
end_reason = None
for n in range(1, MAX_DRIVER_TURNS + 1):
remaining = WALL_CAP - (time.time() - t_run)
if remaining <= 5:
caps.append("wall_40min")
end_reason = "cap_wall"
break
prefix = os.path.join(run_dir, "turn-%02d" % n)
r = run_claude(prompt, sid, work, prefix, remaining, allowed)
seg_mode = flow in ("sp", "spd") # stage split by skill in force inside the turn
acc = account_turn(prefix + ".stream.jsonl", prefix + ".times", prev,
None if seg_mode else stage, stage_in_force)
if acc.get("session_id"):
sid = acc["session_id"]
if acc.get("cumulative"):
prev = acc["cumulative"]
if seg_mode:
stage_in_force = acc["stage_at_end"]
rec = {"n": n, "flow": flow, "task": task_id, "stage": acc["stage_at_end"] if flow == "sp" else stage,
"prompt": prompt,
"prompt_source": source, **{k: v for k, v in r.items() if k != "t0"}, **{k: v for k, v in acc.items() if k != "cumulative"}}
if r["killed_wall_cap"]:
caps.append("wall_40min")
if acc.get("subtype") == "error_max_turns":
caps.append("max_turns_60@turn%d" % n)
last = acc.get("result_text") or acc.get("last_assistant_text") or ""
open(prefix + ".last.txt", "w").write(last)
if r["killed_wall_cap"]:
rec["po"] = None
turns_log.write(json.dumps(rec) + "\n")
turns_log.flush()
end_reason = "cap_wall"
break
po_stage = stage if flow != "sp" else acc["stage_at_end"]
ans = po.ask(run_dir, n, flow, po_stage, last, task_id)
rec["po"] = ans
turns_log.write(json.dumps(rec) + "\n")
turns_log.flush()
# Next prompt
if flow in ("bare", "sp"):
if ans["kind"] == "done":
end_reason = "done"
break
prompt = ans["reply"] if ans["kind"] == "question" else (
"go on" if flow == "bare" else "go on, proceed with the plan")
source = "po" if ans["kind"] == "question" else "driver"
elif flow == "spd":
if ans["kind"] == "question":
prompt, source = ans["reply"], "po"
if stage in SPD_PRE_IMPLEMENT and "implementing yet" not in prompt:
prompt += "\n\nDon't start implementing yet."
elif ans["kind"] == "waiting" or acc.get("subtype") == "error_max_turns":
prompt, source = "go on", "driver"
else:
spd_i += 1
if spd_i >= len(spd_stages):
end_reason = "done"
break
stage, prompt, source = spd_stages[spd_i], spd_stages[spd_i], "driver"
stage_in_force = stage.lstrip("/") # a typed slash command expands inline: no Skill event
else:
if ans["kind"] == "question":
prompt, source = ans["reply"], "po"
# A user following the Main Flow builds only at /implement. Without this line the worker took the
# answers to its last grilling questions as a go and coded inside /grill-with-docs (both-r1, twice).
if stage in PRE_IMPLEMENT and "implementing yet" not in prompt:
prompt += "\n\nDon't start implementing yet."
elif ans["kind"] == "waiting" or acc.get("subtype") == "error_max_turns":
prompt, source = "go on", "driver" # stage not finished: stay in it
else:
mp_i += 1
if mp_i >= len(mp_stages):
end_reason = "done"
break
stage, prompt, source = mp_stages[mp_i], mp_prompt(mp_i), "driver"
if n == MAX_DRIVER_TURNS:
caps.append("driver_turns_12")
end_reason = "cap_driver_turns"
meta = {"end_reason": end_reason, "caps": sorted(set(caps)), "session_id": sid,
"run_wall_s": round(time.time() - t_run, 1), "mp_stage_reached": mp_stages[min(mp_i, 5)] if flow == "mp" else None,
"spd_stage_reached": spd_stages[min(spd_i, 2)] if flow == "spd" else None, "task": task_id}
json.dump(meta, open(os.path.join(run_dir, "driver.json"), "w"), indent=2)
print(json.dumps(meta))
if __name__ == "__main__":
main()
lib/po.py
"""Product-owner simulator: a headless Sonnet 5.5 with NO tools that read po/F1-brief.md and answers the worker.
po.py <run dir> <turn n> <flow> <stage> <last message file> -> prints one JSON line, writes po-<n>.json
Output: {"kind": "question" | "waiting" | "done", "reply": str, "cost_usd", "wall_s", "error", "heuristic_question"}
- question: the worker asks the user something (a question, options, an approval request) -> reply answers it
- waiting: it stopped without asking (announced a next step, paused between phases) -> driver sends "go on"
- done: it reports the whole change finished -> driver stops (bare/sp)
Isolation (same as the previous benches' reviewers): --setting-sources "", disableAllHooks, --strict-mcp-config,
--tools "", CLAUDE_CODE_DISABLE_CLAUDE_MDS=1, CLAUDE_CODE_DISABLE_AUTO_MEMORY=1, scrubbed CLAUDE* env,
cwd = an empty temp dir (never the work copy).
"""
import json
import os
import re
import subprocess
import sys
import tempfile
import time
BENCH = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
MODEL = "claude-sonnet-5-5"
SCHEMA = {
"type": "object",
"properties": {
"kind": {"type": "string", "enum": ["question", "waiting", "done"]},
"reply": {"type": "string"},
},
"required": ["kind", "reply"],
"additionalProperties": False,
}
FLOW_NOTE = {
"bare": "The assistant was asked to implement the request and stop with a summary when done.",
"sp": "The assistant runs a process (brainstorm -> design -> written plan -> execution -> review -> finish). "
"It often stops to ask for approval of a design, a spec or a plan, or to ask how to finish the branch.",
"spd": "You are driving the assistant through the Superpowers stages by slash command, one stage at a time "
"(brainstorming -> writing-plans -> executing-plans); the current stage is shown below and YOU send the "
"next command once a stage is over. RULE FOR THIS FLOW: when the stage's output is delivered (a design "
"approved and written, a plan saved) and the assistant only asks you to approve it or whether to proceed, "
"answer what is open (approve it; if asked for an execution approach, choose Native / inline in this "
"session; consent to working on the current branch) and classify it kind=question only if something "
"beyond approval is open, otherwise kind=done. Never tell it to start implementing yourself before the "
"executing-plans stage. During executing-plans, approve its rulings and ask it to finish.",
"mp": "You are driving the assistant through slash commands, one stage at a time (grill -> spec -> tickets -> "
"implement -> code review -> PR); the current stage is shown below and YOU send the next command once "
"a stage is over. Interviews (grilling) come in rounds of numbered questions with a recommended answer "
"each. RULE FOR THIS FLOW: when the stage's output is delivered and the assistant only asks whether to "
"proceed, start building, move on, or confirms its own summary ('Ready to write the spec?', 'Shall I "
"start?', 'Does this capture it?' after a full recap), that is kind=done with an empty reply: the next "
"slash command will follow. Never tell it to start implementing or to move to another phase yourself; "
"answer only the open design questions. Before the /implement stage, if the assistant offers to build "
"next, end your reply with: \"Don't start implementing yet.\"",
}
def heuristic_question(text):
"""SPEC's driver rule: a turn that ends on a question mark or an explicit options list."""
tail = (text or "").strip()[-600:]
if tail.rstrip("*_` \n").endswith("?"):
return True
lines = [l.strip() for l in tail.splitlines() if l.strip()]
opts = [l for l in lines[-8:] if re.match(r"^(\(?[A-Da-d1-4][\).:]|[-*] \*\*Option)", l)]
return len(opts) >= 2 or "?" in "\n".join(lines[-3:])
def ask(run_dir, n, flow, stage, last_text, task="F1"):
brief = open(os.path.join(BENCH, "po", "%s-brief.md" % task)).read()
prompt = (
"You play the developer who asked a coding assistant for a change in your repo. You are busy: you answer "
"tersely (1 to 6 short sentences or a short numbered list), the way a dev types in a chat. You never write "
"code, never mention how you will check the work, never invent requirements beyond your brief. You answer "
"ONLY what the assistant asked: never volunteer a requirement, an edge case or a constraint it did not ask "
"about (a question it asked is fair game, even a broad one like 'anything else I should know?').\n\n"
"=== YOUR BRIEF (private, the assistant has not seen it) ===\n" + brief + "\n=== END BRIEF ===\n\n"
+ FLOW_NOTE.get(flow, "") + "\nCurrent stage: " + stage + "\n\n"
"=== THE ASSISTANT'S LAST MESSAGE (it is now waiting for you) ===\n" + (last_text or "(empty message)")[-12000:]
+ "\n=== END MESSAGE ===\n\n"
"Classify the message and write your reply:\n"
"- kind=question when it asks you anything or waits for a decision or an approval (numbered questions, "
"options, 'does this look right?', 'shall I proceed?'). Reply by answering EVERY open question in one go, "
"from your brief; where your brief is silent, accept the assistant's recommendation or say 'your call'. "
"Approve designs, specs, plans and ticket breakdowns that match your brief; correct only what contradicts it. "
"If it asks how to finish a branch (merge, PR, keep), say: keep it on the current branch, no PR, done. "
"If a code review it ran lists real defects, tell it to fix them.\n"
"- kind=waiting when it stopped without asking anything but the work is not finished (it announced a next "
"step, or paused between phases). Reply: 'go on'.\n"
"- kind=done when it reports the change finished (implemented, tests run) or, for a slash-command stage, "
"when that stage's output is delivered and nothing is asked. Reply: ''.\n"
)
env = {k: v for k, v in os.environ.items() if not k.startswith("CLAUDE")}
env.update({"CLAUDE_CODE_DISABLE_CLAUDE_MDS": "1", "CLAUDE_CODE_DISABLE_AUTO_MEMORY": "1"})
cmd = ["claude", "-p", prompt, "--model", MODEL, "--output-format", "json", "--json-schema", json.dumps(SCHEMA),
"--tools", "", "--max-turns", "4", "--permission-mode", "dontAsk", "--setting-sources", "",
"--settings", '{"disableAllHooks":true}', "--strict-mcp-config"]
out = {"kind": None, "reply": "", "cost_usd": 0.0, "wall_s": 0.0, "error": None,
"heuristic_question": heuristic_question(last_text)}
t0 = time.time()
for attempt in range(2):
try:
with tempfile.TemporaryDirectory(prefix="po-") as cwd:
p = subprocess.run(cmd, cwd=cwd, env=env, stdin=subprocess.DEVNULL, capture_output=True,
text=True, timeout=300)
res = json.loads(p.stdout.strip().splitlines()[-1])
out["cost_usd"] += res.get("total_cost_usd") or 0.0
out["models"] = sorted((res.get("modelUsage") or {}).keys())
so = res.get("structured_output")
if isinstance(so, str):
so = json.loads(so)
if not so or so.get("kind") not in ("question", "waiting", "done"):
raise ValueError("no structured_output: %s" % (res.get("result") or "")[:300])
out["kind"], out["reply"], out["error"] = so["kind"], so["reply"].strip(), None
break
except Exception as e: # noqa: BLE001
out["error"] = "%s: %s" % (type(e).__name__, e)
time.sleep(3)
if out["kind"] is None: # PO failed twice: fall back on the SPEC heuristic
out["kind"] = "question" if out["heuristic_question"] else "waiting"
out["reply"] = "Your call on all of these; go on." if out["kind"] == "question" else "go on"
out["fallback"] = True
if out["kind"] == "waiting" and not out["reply"]:
out["reply"] = "go on"
out["wall_s"] = round(time.time() - t0, 2)
json.dump(dict(out, n=n, stage=stage, prompt_chars=len(prompt)), open(os.path.join(run_dir, "po-%02d.json" % n), "w"),
indent=2)
return out
if __name__ == "__main__":
rd, n, flow, stage, f = sys.argv[1:6]
print(json.dumps(ask(rd, int(n), flow, stage, open(f).read(), os.environ.get("TASK", "F1"))))
lib/accounting.py
"""Per-turn and per-stage accounting from one claude -p stream-json (with --include-partial-messages).
Sources of truth (checked on 2.1.294, see README traps):
- result.total_cost_usd and result.modelUsage are CUMULATIVE over a resumed session and include subagents:
per-turn USD and tokens = delta against the previous turn's result.
- result.usage covers the main thread of this invocation only (no subagents): not used for totals.
- assistant events carry a usage SNAPSHOT taken when the message started (output_tokens partial);
stream_event message_start/message_delta (main thread only) carry the exact per-API-call usage.
- Subagent API calls emit only assistant snapshots (parent_tool_use_id = the Task/Agent tool_use id).
Stage split inside a turn (sp: the skill in force): main-thread calls are priced exactly from their own usage
(Sonnet 5.5 list prices, verified against total_cost_usd); the subagent remainder (turn delta - main calls) is
split over the Task calls that spawned them, weighted by their snapshot tokens. Wall per stage = the receive-time
gaps of the stdout lines, attributed to the stage in force.
"""
import json
from collections import Counter, defaultdict
# USD per token, claude-sonnet-5-5 list prices (derived from two probe turns: exact match to total_cost_usd).
PRICE = {"in": 2e-6, "out": 10e-6, "cr": 0.2e-6, "cw1h": 4e-6, "cw5m": 2.5e-6}
SP_INJECT_MARK = "You have superpowers"
EDIT_TOOLS = {"Edit", "Write", "MultiEdit", "NotebookEdit"}
DESIGN_STAGES = {"(none)", "using-superpowers", "brainstorming", "writing-plans"}
NO_SKILL_IMPL = "implementation (no skill)"
def _tok(u):
"""Normalise an API usage dict to in/out/cr/cw (+ cw split)."""
u = u or {}
cc = u.get("cache_creation") or {}
cw = u.get("cache_creation_input_tokens") or 0
cw1h = cc.get("ephemeral_1h_input_tokens")
cw5m = cc.get("ephemeral_5m_input_tokens")
if cw1h is None and cw5m is None:
cw1h, cw5m = cw, 0
return {"in": u.get("input_tokens") or 0, "out": u.get("output_tokens") or 0,
"cr": u.get("cache_read_input_tokens") or 0, "cw": cw, "cw1h": cw1h or 0, "cw5m": cw5m or 0}
def _usd(t):
return t["in"] * PRICE["in"] + t["out"] * PRICE["out"] + t["cr"] * PRICE["cr"] + \
t["cw1h"] * PRICE["cw1h"] + t["cw5m"] * PRICE["cw5m"]
def _mu_tokens(mu):
out = {}
for m, v in (mu or {}).items():
out[m] = {"in": v.get("inputTokens") or 0, "out": v.get("outputTokens") or 0,
"cr": v.get("cacheReadInputTokens") or 0, "cw": v.get("cacheCreationInputTokens") or 0,
"usd": v.get("costUSD") or 0.0}
return out
def _sub(a, b):
keys = set(a) | set(b)
return {m: {k: (a.get(m, {}).get(k, 0) - b.get(m, {}).get(k, 0)) for k in ("in", "out", "cr", "cw", "usd")}
for m in keys}
def _sum_models(per_model):
t = {k: 0 for k in ("in", "out", "cr", "cw", "usd")}
for v in per_model.values():
for k in t:
t[k] += v.get(k, 0)
return t
def account_turn(stream_path, times_path, prev, fixed_stage, stage_in_force):
lines, times = [], []
try:
raw = open(stream_path, errors="replace").read().splitlines()
tms = [float(x) for x in open(times_path).read().split()]
except FileNotFoundError:
raw, tms = [], []
for i, l in enumerate(raw):
l = l.strip()
if l.startswith("{"):
try:
lines.append(json.loads(l))
times.append(tms[i] if i < len(tms) else (times[-1] if times else 0.0))
except Exception:
pass
stage = fixed_stage or stage_in_force
out = {"session_id": None, "subtype": None, "is_error": None, "num_turns": None, "result_text": None,
"last_assistant_text": "", "stage_at_start": stage}
tools_main, tools_sub = Counter(), Counter()
skills, hooks, files = [], [], set()
skill_ids = {}
task_stage = {} # Task/Agent tool_use id -> stage when spawned
main_calls = {} # msg id -> {"stage", "tok"}
sub_snap = {} # msg id -> (parent id, tokens)
seg_wall = defaultdict(float)
order = [stage]
last_t = 0.0
result = None
denials = []
for o, t in zip(lines, times):
seg_wall[stage] += max(0.0, t - last_t)
last_t = t
typ = o.get("type")
if typ == "system" and o.get("subtype") == "init":
out["session_id"] = o.get("session_id")
out["init_model"] = o.get("model")
out["init_skills"] = o.get("skills")
out["init_plugins"] = [p.get("name") for p in o.get("plugins") or []]
elif typ == "system" and str(o.get("subtype", "")).startswith("hook"):
if o.get("subtype") == "hook_response":
outp = o.get("output") or ""
hooks.append({"event": o.get("hook_event"), "name": o.get("hook_name"),
"superpowers_injected": SP_INJECT_MARK in outp, "output_chars": len(outp),
"exit_code": o.get("exit_code"), "outcome": o.get("outcome")})
elif typ == "stream_event" and not o.get("parent_tool_use_id"):
ev = o.get("event") or {}
if ev.get("type") == "message_start":
m = ev.get("message") or {}
main_calls[m.get("id")] = {"stage": stage, "tok": _tok(m.get("usage")), "model": m.get("model")}
main_calls["__last__"] = m.get("id")
elif ev.get("type") == "message_delta":
mid = main_calls.get("__last__")
if mid in main_calls:
u = ev.get("usage") or {}
tk = main_calls[mid]["tok"]
tk["out"] = u.get("output_tokens", tk["out"]) or tk["out"]
elif typ == "assistant":
m = o.get("message") or {}
parent = o.get("parent_tool_use_id")
if parent:
sub_snap[m.get("id")] = (parent, _tok(m.get("usage")))
for c in m.get("content") or []:
if c.get("type") == "text" and not parent:
out["last_assistant_text"] = c.get("text") or out["last_assistant_text"]
if c.get("type") != "tool_use":
continue
name = c.get("name")
inp = c.get("input") or {}
(tools_sub if parent else tools_main)[name] += 1
if name in EDIT_TOOLS and inp.get("file_path"):
files.add(inp["file_path"])
# sp: code written while only a design skill is in force = implementation with no skill
if (not parent and not fixed_stage and stage in DESIGN_STAGES
and ("/src/" in inp["file_path"] or "/test/" in inp["file_path"])):
stage = NO_SKILL_IMPL
order.append(stage)
if name in ("Task", "Agent") and not parent:
task_stage[c.get("id")] = stage
if name == "Skill":
sk = inp.get("skill") or inp.get("command") or "?"
rec = {"skill": sk, "t": round(t, 2), "thread": "sub" if parent else "main", "ok": None,
"args": (inp.get("args") or "")[:200]}
skills.append(rec)
skill_ids[c.get("id")] = rec
if not parent and not fixed_stage:
stage = sk.split(":")[-1]
order.append(stage)
elif typ == "user":
for c in (o.get("message") or {}).get("content") or []:
if isinstance(c, dict) and c.get("type") == "tool_result" and c.get("tool_use_id") in skill_ids:
txt = c.get("content")
txt = txt if isinstance(txt, str) else json.dumps(txt)
skill_ids[c["tool_use_id"]]["ok"] = not c.get("is_error") and "Launching skill" in txt
skill_ids[c["tool_use_id"]]["result"] = txt[:200]
elif typ == "result":
result = o
main_calls.pop("__last__", None)
if result:
out.update({"subtype": result.get("subtype"), "is_error": result.get("is_error"),
"num_turns": result.get("num_turns"), "result_text": result.get("result"),
"terminal_reason": result.get("terminal_reason"), "stop_reason": result.get("stop_reason")})
denials = result.get("permission_denials") or []
cum = {"cost": result.get("total_cost_usd") or 0.0, "models": _mu_tokens(result.get("modelUsage"))}
pm = prev["models"] if prev else {}
delta_models = _sub(cum["models"], pm)
delta_cost = cum["cost"] - (prev["cost"] if prev else 0.0)
out["cumulative"] = cum
out["estimated"] = False
else: # killed or crashed: best effort from the stream itself
delta_models = {}
for c in main_calls.values():
d = delta_models.setdefault(c.get("model") or "?", {"in": 0, "out": 0, "cr": 0, "cw": 0, "usd": 0.0})
for k in ("in", "out", "cr", "cw"):
d[k] += c["tok"][k]
d["usd"] += _usd(c["tok"])
for _, (parent, tk) in sub_snap.items():
d = delta_models.setdefault("?sub", {"in": 0, "out": 0, "cr": 0, "cw": 0, "usd": 0.0})
for k in ("in", "out", "cr", "cw"):
d[k] += tk[k]
d["usd"] += _usd(tk)
delta_cost = sum(v["usd"] for v in delta_models.values())
out["cumulative"] = None
out["estimated"] = True
tot = _sum_models(delta_models)
out["usd"] = round(delta_cost, 6)
out["tokens"] = {k: tot[k] for k in ("in", "out", "cr", "cw")}
out["tokens"]["total"] = sum(out["tokens"].values())
out["models"] = {m: v for m, v in delta_models.items() if any(v[k] for k in ("in", "out", "cr", "cw"))}
# Stage segments
segs = defaultdict(lambda: {"in": 0, "out": 0, "cr": 0, "cw": 0, "usd": 0.0, "api_calls": 0, "wall_s": 0.0})
main_tok = {"in": 0, "out": 0, "cr": 0, "cw": 0}
main_usd = 0.0
for c in main_calls.values():
s = segs[c["stage"]]
for k in ("in", "out", "cr", "cw"):
s[k] += c["tok"][k]
main_tok[k] += c["tok"][k]
u = _usd(c["tok"])
s["usd"] += u
s["api_calls"] += 1
main_usd += u
rem = {k: max(0, out["tokens"][k] - main_tok[k]) for k in ("in", "out", "cr", "cw")}
rem_usd = max(0.0, delta_cost - main_usd) if result else 0.0
weights = defaultdict(float)
for _, (parent, tk) in sub_snap.items():
# subagent spend is its own segment, labelled with the stage that dispatched it
weights[task_stage.get(parent, stage) + " [subagent]"] += tk["in"] + tk["cr"] + tk["cw"] + tk["out"]
wsum = sum(weights.values())
if result and (wsum > 0 or rem_usd > 0):
if wsum == 0:
weights = {stage + " [subagent]": 1.0}
wsum = 1.0
for st, w in weights.items():
s = segs[st]
f = w / wsum
for k in ("in", "out", "cr", "cw"):
s[k] += int(round(rem[k] * f))
s["usd"] += rem_usd * f
for st, w in seg_wall.items():
segs[st]["wall_s"] += w
seen = []
for st in order:
if st not in seen:
seen.append(st)
for st in segs:
if st not in seen:
seen.append(st)
out["segments"] = [dict(stage=st, **{k: (round(v, 6) if isinstance(v, float) else v) for k, v in segs[st].items()},
total=segs[st]["in"] + segs[st]["out"] + segs[st]["cr"] + segs[st]["cw"])
for st in seen if st in segs]
out["pricing_check"] = {"main_calls_usd": round(main_usd, 6), "turn_usd": round(delta_cost, 6),
"subagent_usd": round(rem_usd, 6), "main_calls": len(main_calls),
"subagent_msgs": len(sub_snap)}
out["stage_at_end"] = stage
out["tools_main"] = dict(tools_main)
out["tools_sub"] = dict(tools_sub)
out["tool_calls"] = sum(tools_main.values()) + sum(tools_sub.values())
out["skills"] = skills
out["hooks"] = hooks
out["files_touched"] = sorted(files)
out["permission_denials"] = [{"tool": d.get("tool_name"), "input": json.dumps(d.get("tool_input"))[:200]}
for d in denials]
out["stream_lines"] = len(lines)
return out
lib/reaccount.py
"""Recompute the accounting fields of runs/<run>/turns.jsonl from the saved turn streams (after a change to
lib/accounting.py), keeping what the driver recorded (prompt, source, wall, exit code, PO answer). Then run finalize.py.
reaccount.py <run dir>
"""
import json
import os
import sys
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from accounting import account_turn # noqa: E402
KEEP = ("n", "flow", "prompt", "prompt_source", "exit_code", "wall_s", "killed_wall_cap", "po")
run = os.path.abspath(sys.argv[1])
turns = [json.loads(l) for l in open(os.path.join(run, "turns.jsonl"))]
prev, stage_in_force, out = None, "(none)", []
prev_slash = None
for t in turns:
prefix = os.path.join(run, "turn-%02d" % t["n"])
if t["flow"] == "spd" and t["stage"] != prev_slash: # a typed slash command starts its skill (no Skill event)
stage_in_force, prev_slash = t["stage"].lstrip("/"), t["stage"]
sp = t["flow"] in ("sp", "spd")
acc = account_turn(prefix + ".stream.jsonl", prefix + ".times", prev, None if sp else t["stage"], stage_in_force)
if acc.get("cumulative"):
prev = acc["cumulative"]
if sp:
stage_in_force = acc["stage_at_end"]
rec = {k: t[k] for k in KEEP if k in t}
rec["stage"] = acc["stage_at_end"] if t["flow"] == "sp" else t["stage"]
rec.update({k: v for k, v in acc.items() if k != "cumulative"})
out.append(rec)
with open(os.path.join(run, "turns.jsonl"), "w") as f:
for r in out:
f.write(json.dumps(r) + "\n")
print("%s: %d turns reaccounted" % (os.path.basename(run), len(out)))
lib/finalize.py
"""After the driver: final diff, suite, typecheck, hidden oracle, per-stage table -> score.json, stages.json.
finalize.py <run dir> (re-runnable; reads meta.json, turns.jsonl, driver.json)
"""
import json
import os
import re
import shutil
import subprocess
import sys
from collections import OrderedDict
BENCH = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
HIDDEN = os.path.join(BENCH, "hidden")
MODEL = "claude-sonnet-5-5"
UPSTREAM_TESTS = 34
def sh(args, cwd, env=None, timeout=600):
try:
p = subprocess.run(args, cwd=cwd, env=env, stdin=subprocess.DEVNULL, stdout=subprocess.PIPE,
stderr=subprocess.STDOUT, timeout=timeout)
return p.returncode, p.stdout.decode("utf-8", "replace")
except subprocess.TimeoutExpired as e:
return 124, (e.stdout or b"").decode("utf-8", "replace") + "\n[timeout]"
def clean_env(extra=None):
env = {k: v for k, v in os.environ.items() if not k.startswith("CLAUDE")}
env.update(extra or {})
return env
def work_diff(work, base):
"""Working tree vs the base commit (commits made by the worker included) + untracked files. Index untouched."""
_, d = sh(["git", "diff", base, "--no-color", "--", "."], work)
_, others = sh(["git", "ls-files", "--others", "--exclude-standard"], work)
for f in [l for l in others.splitlines() if l.strip()]:
_, nd = sh(["git", "diff", "--no-color", "--no-index", "--", "/dev/null", f], work)
d += nd
return d
def numstat(work, base):
rows = []
_, ns = sh(["git", "diff", base, "--numstat", "--", "."], work)
for l in ns.splitlines():
p = l.split("\t")
if len(p) == 3:
rows.append((int(p[0]) if p[0] != "-" else 0, int(p[1]) if p[1] != "-" else 0, p[2]))
_, others = sh(["git", "ls-files", "--others", "--exclude-standard"], work)
for f in [l for l in others.splitlines() if l.strip()]:
try:
rows.append((sum(1 for _ in open(os.path.join(work, f), errors="replace")), 0, f))
except Exception:
pass
return rows
def run_hidden(work, base, task="F1"):
rep = os.path.join(os.path.dirname(work), ".hidden-%s.json" % os.path.basename(work))
rc, txt = sh([os.path.join(HIDDEN, "node_modules", ".bin", "vitest"), "run", "--reporter=json",
"--outputFile=" + rep], HIDDEN, env=clean_env({"WORK": work, "BASE": base, "TASK": task}), timeout=300)
reqs = OrderedDict(("test_req%d" % i, False) for i in range(1, 6))
fails = {}
try:
j = json.load(open(rep))
for tr in j.get("testResults", []):
for a in tr.get("assertionResults", []):
m = re.match(r"(test_req\d)", a.get("title", ""))
if m:
reqs[m.group(1)] = a.get("status") == "passed"
if a.get("status") != "passed":
fails[m.group(1)] = (" ".join(a.get("failureMessages") or []))[:600]
if tr.get("status") == "failed" and not tr.get("assertionResults"):
fails["collect"] = (tr.get("message") or "")[:800]
os.remove(rep)
except Exception as e: # noqa: BLE001
fails["runner"] = "%s: %s" % (type(e).__name__, txt[-800:])
return reqs, fails, txt
def main():
out = os.path.abspath(sys.argv[1])
meta = json.load(open(os.path.join(out, "meta.json")))
work, base, cfg = meta["work"], meta["base"], meta["cfg"]
if not os.path.isdir(work): # tmp work copy gone: rescore the archived copy
work = os.path.join(BENCH, "work", meta["name"])
drv = json.load(open(os.path.join(out, "driver.json"))) if os.path.exists(os.path.join(out, "driver.json")) else {}
turns = [json.loads(l) for l in open(os.path.join(out, "turns.jsonl"))] if os.path.exists(
os.path.join(out, "turns.jsonl")) else []
have_nm = os.path.isdir(os.path.join(work, "node_modules"))
if not have_nm: # rescoring after cleanup: restore deps from upstream
shutil.copytree(os.path.join(BENCH, "upstream", "node_modules"), os.path.join(work, "node_modules"), symlinks=True)
diff = work_diff(work, base)
open(os.path.join(out, "post.diff"), "w").write(diff)
rows = numstat(work, base)
open(os.path.join(out, "numstat.txt"), "w").write("".join("%d\t%d\t%s\n" % r for r in rows))
code = [r for r in rows if r[2].startswith(("src/", "test/"))]
tests_files = [r for r in rows if r[2].endswith(".test.ts")]
rc_s, suite = sh([os.path.join(work, "node_modules", ".bin", "vitest"), "run"], work, env=clean_env(), timeout=300)
open(os.path.join(out, "suite.txt"), "w").write(suite)
clean = re.sub(r"\x1b\[[0-9;]*m", "", suite)
m = re.search(r"Tests\s+(?:(\d+) failed \| )?(\d+) passed(?: \| \d+ \w+)? \((\d+)\)", clean)
s_fail = int(m.group(1) or 0) if m else None
s_pass = int(m.group(2)) if m else 0
s_total = int(m.group(3)) if m else 0
if m is None:
mf = re.search(r"Tests\s+(\d+) failed \((\d+)\)", clean)
s_fail, s_total = (int(mf.group(1)), int(mf.group(2))) if mf else (None, 0)
rc_t, tc = sh([os.path.join(work, "node_modules", ".bin", "tsc"), "--noEmit"], work, env=clean_env(), timeout=300)
open(os.path.join(out, "typecheck.txt"), "w").write(tc)
task = meta.get("task", "F1")
reqs, fails, htxt = run_hidden(work, base, task)
open(os.path.join(out, "hidden.txt"), "w").write(htxt + "\n" + json.dumps(fails, indent=2))
_, commits = sh(["git", "log", "--oneline", "%s..HEAD" % base], work)
_, branch = sh(["git", "rev-parse", "--abbrev-ref", "HEAD"], work)
_, branches = sh(["git", "branch", "--format=%(refname:short)"], work)
added_its = 0
for l in diff.splitlines():
if re.match(r"^\+\s*(it|test)(\.each\([^)]*\))?\s*\(", l):
added_its += 1
# Per-stage table (sum of turn segments; for mp/bare/both a segment = the turn's stage)
stages = OrderedDict()
for t in turns:
for sgm in t.get("segments") or []:
st = sgm["stage"] if cfg in ("sp", "spd") else t["stage"]
s = stages.setdefault(st, {"turns": set(), "in": 0, "out": 0, "cr": 0, "cw": 0, "total": 0, "usd": 0.0,
"wall_s": 0.0, "api_calls": 0})
s["turns"].add(t["n"])
for k in ("in", "out", "cr", "cw", "total", "api_calls"):
s[k] += sgm.get(k, 0)
s["usd"] += sgm.get("usd", 0.0)
s["wall_s"] += sgm.get("wall_s", 0.0)
if cfg not in ("sp", "spd") and not t.get("segments"):
stages.setdefault(t["stage"], {"turns": {t["n"]}, "in": 0, "out": 0, "cr": 0, "cw": 0, "total": 0,
"usd": 0.0, "wall_s": t.get("wall_s", 0), "api_calls": 0})
# Turn-level wall/usd are exact; mp/bare stage wall = turn wall (segments only cover stdout gaps)
if cfg not in ("sp", "spd"):
for st, s in stages.items():
s["wall_s"] = sum(t.get("wall_s", 0) for t in turns if t["stage"] == st)
s["usd"] = sum(t.get("usd", 0) for t in turns if t["stage"] == st)
for s in stages.values():
s["turns"] = sorted(s["turns"])
s["usd"] = round(s["usd"], 4)
s["wall_s"] = round(s["wall_s"], 1)
json.dump(stages, open(os.path.join(out, "stages.json"), "w"), indent=2)
tok = {k: sum(t.get("tokens", {}).get(k, 0) for t in turns) for k in ("in", "out", "cr", "cw", "total")}
models = sorted({m for t in turns for m in (t.get("models") or {})})
skills = [dict(s, turn=t["n"]) for t in turns for s in t.get("skills") or []]
hooks = [dict(h, turn=t["n"]) for t in turns for h in t.get("hooks") or []]
po_cost = sum((t.get("po") or {}).get("cost_usd", 0) for t in turns)
files = sorted({f.replace(work + "/", "") for t in turns for f in t.get("files_touched") or []})
score = {
"run": meta["name"], "task": meta.get("task", "F1"), "cfg": cfg, "rep": meta["rep"], "tag": meta.get("tag"),
"hidden_pass": sum(reqs.values()), "hidden_reqs": reqs, "hidden_fail_msgs": fails,
"suite_green": rc_s == 0 and (s_fail or 0) == 0 and s_total > 0, "suite_pass": s_pass, "suite_total": s_total,
"tests_added_suite": max(0, s_total - UPSTREAM_TESTS), "tests_added_diff": added_its,
"typecheck_green": rc_t == 0,
"diff_files": len(rows), "diff_added": sum(r[0] for r in rows), "diff_removed": sum(r[1] for r in rows),
"code_diff_added": sum(r[0] for r in code), "code_diff_removed": sum(r[1] for r in code),
"code_files": len(code), "test_files_touched": len(tests_files),
"non_code_files": sorted(r[2] for r in rows if not r[2].startswith(("src/", "test/"))),
"files_touched_by_tools": files,
"commits": len([l for l in commits.splitlines() if l.strip()]), "branch_at_end": branch.strip(),
"branches": branches.split(),
"driver_turns": len(turns), "claude_turns_total": sum(t.get("num_turns") or 0 for t in turns),
"tool_calls": sum(t.get("tool_calls", 0) for t in turns),
"tokens": tok, "usd": round(sum(t.get("usd", 0) for t in turns), 4),
"usd_po": round(po_cost, 4),
"wall_s": round(sum(t.get("wall_s", 0) for t in turns), 1), "run_wall_s": drv.get("run_wall_s"),
"end_reason": drv.get("end_reason"), "caps": drv.get("caps", []),
"questions_routed_to_po": sum(1 for t in turns if (t.get("po") or {}).get("kind") == "question"),
"skills_invoked": [s["skill"] for s in skills],
"skills_failed": [s["skill"] for s in skills if s.get("ok") is False],
"skill_events": skills,
"hook_events": hooks,
"superpowers_injected": any(h.get("superpowers_injected") for h in hooks),
"init_plugins": (turns[0].get("init_plugins") if turns else None),
"models": models, "model_ok": bool(models) and set(models) <= {MODEL},
"permission_denials": sum(len(t.get("permission_denials") or []) for t in turns),
"estimated_turns": [t["n"] for t in turns if t.get("estimated")],
"spd_slash_stage_per_turn": [t["stage"] for t in turns] if cfg == "spd" else None,
"per_turn": [{"n": t["n"], "stage": t["stage"], "source": t.get("prompt_source"), "usd": t.get("usd"),
"tokens": t.get("tokens", {}).get("total"), "wall_s": t.get("wall_s"),
"num_turns": t.get("num_turns"), "subtype": t.get("subtype"),
"skills": [s["skill"] for s in t.get("skills") or []],
"po_kind": (t.get("po") or {}).get("kind")} for t in turns],
"stages": stages,
}
json.dump(score, open(os.path.join(out, "score.json"), "w"), indent=2)
if os.environ.get("KEEP_NODE_MODULES") != "1":
shutil.rmtree(os.path.join(work, "node_modules"), ignore_errors=True)
print("%s: hidden %d/5 suite %s (%d/%d) tsc %s diff +%d/-%d (code +%d/-%d) tokens %d usd %.2f wall %ss turns %d "
"end=%s caps=%s skills=%s" % (
meta["name"], score["hidden_pass"], "green" if score["suite_green"] else "RED", s_pass, s_total,
"ok" if score["typecheck_green"] else "FAIL", score["diff_added"], score["diff_removed"],
score["code_diff_added"], score["code_diff_removed"], tok["total"], score["usd"], score["wall_s"],
len(turns), score["end_reason"], ",".join(score["caps"]) or "-",
",".join(OrderedDict.fromkeys(score["skills_invoked"])) or "-"))
if __name__ == "__main__":
main()
analyze.py
#!/usr/bin/env python3
"""runs/*/score.json (untagged runs only) -> results.json + results.md."""
import json
import os
import statistics as st
from collections import OrderedDict, defaultdict
HERE = os.path.dirname(os.path.abspath(__file__))
CFG_ORDER = ["bare", "sp", "spd", "mp", "both"]
CFG_LABEL = {"bare": "bare", "sp": "sp (hook-routed)", "spd": "spd (Superpowers driven)", "mp": "mp (Main Flow)", "both": "both (sp+mp, mp flow)"}
TASKS = ["F1", "F2"]
SPD_ORDER = ["/brainstorming", "/writing-plans", "/executing-plans"]
MP_ORDER = ["/grill-with-docs", "/to-spec", "/to-tickets", "/implement", "/code-review", "/pr"]
# Superpowers phases (the raw skill-in-force stages are kept too)
SP_PHASE = {
"(none)": "before any skill", "using-superpowers": "before any skill",
"brainstorming": "brainstorming", "writing-plans": "writing-plans",
"executing-plans": "execution", "subagent-driven-development": "execution",
"test-driven-development": "execution", "systematic-debugging": "execution",
"verification-before-completion": "verification", "dispatching-parallel-agents": "execution",
"requesting-code-review": "review", "receiving-code-review": "review",
"finishing-a-development-branch": "finishing", "using-git-worktrees": "execution",
"implementation (no skill)": "implementation (no skill)",
}
PHASE_ORDER = ["before any skill", "brainstorming", "writing-plans", "implementation (no skill)", "execution", "verification", "review", "finishing", "subagents (reviewers, explorers)", "other"]
def mean(xs):
xs = [x for x in xs if x is not None]
return round(st.mean(xs), 3) if xs else None
def med(xs):
xs = [x for x in xs if x is not None]
return round(st.median(xs), 3) if xs else None
def load():
runs = []
for d in sorted(os.listdir(os.path.join(HERE, "runs"))):
p = os.path.join(HERE, "runs", d, "score.json")
if not os.path.exists(p):
continue
s = json.load(open(p))
if s.get("tag"):
continue
runs.append(s)
return runs
def stage_agg(runs, key_fn, order):
agg = defaultdict(lambda: defaultdict(list))
for s in runs:
per = defaultdict(lambda: {"total": 0, "usd": 0.0, "wall_s": 0.0, "out": 0})
for name, v in s["stages"].items():
k = key_fn(name)
for f in ("total", "usd", "wall_s", "out"):
per[k][f] += v.get(f, 0)
for k, v in per.items():
if not (v["total"] or v["usd"] or v["wall_s"]):
continue # empty subagent placeholder segment
for f in v:
agg[k][f].append(v[f])
names = [n for n in order if n in agg] + sorted(n for n in agg if n not in order)
out = OrderedDict()
for n in names:
a = agg[n]
out[n] = {"runs_with_stage": len(a["total"]), "tokens_mean": mean(a["total"]), "tokens_median": med(a["total"]),
"out_tokens_mean": mean(a["out"]), "usd_mean": mean(a["usd"]), "usd_median": med(a["usd"]),
"wall_s_mean": mean(a["wall_s"]), "wall_s_median": med(a["wall_s"])}
return out
def turn_stage_agg(runs, order):
"""Per slash-command stage from the per-turn records (exact turn USD / tokens / wall)."""
agg = defaultdict(lambda: defaultdict(list))
for s in runs:
per = defaultdict(lambda: {"total": 0, "usd": 0.0, "wall_s": 0.0, "out": 0})
for t in s["per_turn"]:
per[t["stage"]]["total"] += t.get("tokens") or 0
per[t["stage"]]["usd"] += t.get("usd") or 0
per[t["stage"]]["wall_s"] += t.get("wall_s") or 0
for k, v in per.items():
for f in v:
agg[k][f].append(v[f])
out = OrderedDict()
for n in [n for n in order if n in agg] + sorted(n for n in agg if n not in order):
a = agg[n]
out[n] = {"runs_with_stage": len(a["total"]), "tokens_mean": mean(a["total"]), "tokens_median": med(a["total"]),
"out_tokens_mean": None, "usd_mean": mean(a["usd"]), "usd_median": med(a["usd"]),
"wall_s_mean": mean(a["wall_s"]), "wall_s_median": med(a["wall_s"])}
return out
def main():
runs = load()
for s in runs:
s.setdefault("task", "F1")
by = OrderedDict(("%s/%s" % (t, c), [s for s in runs if s["cfg"] == c and s["task"] == t]) for t in TASKS for c in CFG_ORDER)
res = {"configs": OrderedDict(), "stages": OrderedDict(), "runs": []}
for key, rs in by.items():
if not rs:
continue
task, c = key.split("/")
m = lambda f: [f(s) for s in rs] # noqa: E731
res["configs"][key] = {
"task": task, "cfg": c, "label": CFG_LABEL[c],
"n": len(rs),
"hidden_mean": mean(m(lambda s: s["hidden_pass"])), "hidden_median": med(m(lambda s: s["hidden_pass"])),
"hidden_per_run": m(lambda s: s["hidden_pass"]),
"req_pass_rate": {r: sum(1 for s in rs if s["hidden_reqs"].get(r)) for r in ["test_req%d" % i for i in range(1, 6)]},
"suite_green": sum(1 for s in rs if s["suite_green"]), "typecheck_green": sum(1 for s in rs if s["typecheck_green"]),
"tests_added_mean": mean(m(lambda s: s["tests_added_suite"])),
"code_diff_mean": mean(m(lambda s: s["code_diff_added"] + s["code_diff_removed"])),
"code_diff_median": med(m(lambda s: s["code_diff_added"] + s["code_diff_removed"])),
"all_diff_mean": mean(m(lambda s: s["diff_added"] + s["diff_removed"])),
"non_code_files_mean": mean(m(lambda s: len(s["non_code_files"]))),
"tokens_mean": mean(m(lambda s: s["tokens"]["total"])), "tokens_median": med(m(lambda s: s["tokens"]["total"])),
"out_tokens_mean": mean(m(lambda s: s["tokens"]["out"])),
"usd_mean": mean(m(lambda s: s["usd"])), "usd_median": med(m(lambda s: s["usd"])),
"usd_po_mean": mean(m(lambda s: s["usd_po"])),
"wall_s_mean": mean(m(lambda s: s["wall_s"])), "wall_s_median": med(m(lambda s: s["wall_s"])),
"driver_turns_mean": mean(m(lambda s: s["driver_turns"])),
"questions_mean": mean(m(lambda s: s["questions_routed_to_po"])),
"caps": {s["run"]: s["caps"] for s in rs if s["caps"]},
"end_reasons": m(lambda s: s["end_reason"]),
"models": sorted({x for s in rs for x in s["models"]}),
"superpowers_injected": m(lambda s: s["superpowers_injected"]),
"skills_per_run": {s["run"]: list(OrderedDict.fromkeys(s["skills_invoked"])) for s in rs},
"skills_failed": {s["run"]: s["skills_failed"] for s in rs if s["skills_failed"]},
}
if c in ("sp", "spd"):
res["stages"]["%s/%s_skill" % (task, c)] = stage_agg(rs, lambda n: n, list(SP_PHASE))
res["stages"]["%s/%s_phase" % (task, c)] = stage_agg(rs, lambda n: "subagents (reviewers, explorers)" if n.endswith("[subagent]") else SP_PHASE.get(n, "other"), PHASE_ORDER)
if c == "spd":
res["stages"]["%s/spd_slash" % task] = turn_stage_agg(rs, SPD_ORDER)
if c in ("mp", "both"):
res["stages"]["%s/%s" % (task, c)] = stage_agg(rs, lambda n: n, MP_ORDER)
for s in runs:
res["runs"].append({k: s[k] for k in ("run", "task", "cfg", "hidden_pass", "hidden_reqs", "suite_green", "typecheck_green",
"code_diff_added", "code_diff_removed", "diff_added", "diff_removed",
"tests_added_suite", "tokens", "usd", "usd_po", "wall_s", "driver_turns",
"questions_routed_to_po", "end_reason", "caps", "skills_invoked",
"superpowers_injected", "commits", "models", "stages")})
json.dump(res, open(os.path.join(HERE, "results.json"), "w"), indent=2)
L = ["# flow-bench results", "",
"Untagged runs only (smoke/validation excluded). Tokens = input + output + cache read + cache write, all models, "
"subagents included. USD = list-price equivalent from `total_cost_usd` deltas (plan quota, not a bill). "
"Wall = sum of worker turn walls (PO time excluded). Diff = lines added+removed vs the base commit, src/ + test/.", "",
"## Per config (mean / median)", "",
"| task / config | n | hidden /5 (per run) | suite green | tsc green | code diff +/- | all files diff | tests added | tokens | USD | wall s | driver turns | questions | caps hit |",
"|---|---|---|---|---|---|---|---|---|---|---|---|---|---|"]
for c, v in res["configs"].items():
L.append("| %s | %d | %s / %s (%s) | %d/%d | %d/%d | %s / %s | %s | %s | %s / %s | %s / %s | %s / %s | %s | %s | %s |" % (
"%s / %s" % (v["task"], v["label"]), v["n"], v["hidden_mean"], v["hidden_median"], ",".join(map(str, v["hidden_per_run"])), v["suite_green"], v["n"],
v["typecheck_green"], v["n"], v["code_diff_mean"], v["code_diff_median"], v["all_diff_mean"], v["tests_added_mean"],
fmt(v["tokens_mean"]), fmt(v["tokens_median"]), v["usd_mean"], v["usd_median"], v["wall_s_mean"], v["wall_s_median"],
v["driver_turns_mean"], v["questions_mean"], "; ".join("%s: %s" % (k, ",".join(x)) for k, x in v["caps"].items()) or "none"))
L += ["", "Requirement pass counts (runs passing each hidden req):", ""]
for c, v in res["configs"].items():
L.append("- **%s %s**: %s" % (v["task"], v["label"], ", ".join("%s %d/%d" % (k.replace("test_", ""), n, v["n"]) for k, n in v["req_pass_rate"].items())))
titles = []
for t in TASKS:
titles += [("%s/spd_slash" % t, "%s spd (Superpowers driven) per slash-command stage (turn level, exact)" % t),
("%s/spd_skill" % t, "%s spd per skill in force (within-turn split)" % t),
("%s/sp_phase" % t, "%s sp (hook-routed) per phase" % t),
("%s/sp_skill" % t, "%s sp (hook-routed) per skill in force" % t),
("%s/mp" % t, "%s mp per slash-command stage" % t), ("%s/both" % t, "%s both per slash-command stage" % t)]
for key, title in titles:
if key not in res["stages"]:
continue
L += ["", "## %s (mean over the runs that had the stage)" % title, "",
"| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |",
"|---|---|---|---|---|---|---|---|---|"]
for n, v in res["stages"][key].items():
L.append("| %s | %d | %s | %s | %s | %s | %s | %s | %s |" % (n, v["runs_with_stage"], fmt(v["tokens_mean"]),
fmt(v["tokens_median"]), fmt(v["out_tokens_mean"]), v["usd_mean"], v["usd_median"], v["wall_s_mean"], v["wall_s_median"]))
L += ["", "## Per run", "",
"| run | task | hidden | reqs failed | suite | tsc | code +/- | tests added | tokens | USD | wall s | turns | end | skills (first use order) |",
"|---|---|---|---|---|---|---|---|---|---|---|---|---|---|"]
for s in sorted(runs, key=lambda s: (s["task"], CFG_ORDER.index(s["cfg"]), s["run"])):
L.append("| %s | %s | %d/5 | %s | %s | %s | +%d/-%d | %d | %s | %.2f | %.0f | %d | %s%s | %s |" % (
s["run"], s["task"], s["hidden_pass"], ",".join(k.replace("test_", "") for k, v in s["hidden_reqs"].items() if not v) or "-",
"green" if s["suite_green"] else "RED", "ok" if s["typecheck_green"] else "FAIL", s["code_diff_added"],
s["code_diff_removed"], s["tests_added_suite"], fmt(s["tokens"]["total"]), s["usd"], s["wall_s"], s["driver_turns"],
s["end_reason"], (" (" + ",".join(s["caps"]) + ")") if s["caps"] else "",
" > ".join(OrderedDict.fromkeys(x.split(":")[-1] for x in s["skills_invoked"])) or "-"))
open(os.path.join(HERE, "results.md"), "w").write("\n".join(L) + "\n")
print("\n".join(L))
def fmt(x):
if x is None:
return "-"
return "{:,.0f}".format(x)
if __name__ == "__main__":
main()
results.json
{
"configs": {
"F1/bare": {
"task": "F1",
"cfg": "bare",
"label": "bare",
"n": 3,
"hidden_mean": 5,
"hidden_median": 5,
"hidden_per_run": [
5,
5,
5
],
"req_pass_rate": {
"test_req1": 3,
"test_req2": 3,
"test_req3": 3,
"test_req4": 3,
"test_req5": 3
},
"suite_green": 3,
"typecheck_green": 3,
"tests_added_mean": 4.667,
"code_diff_mean": 141.667,
"code_diff_median": 137,
"all_diff_mean": 142.333,
"non_code_files_mean": 0.333,
"tokens_mean": 519113.667,
"tokens_median": 361842,
"out_tokens_mean": 12001,
"usd_mean": 0.343,
"usd_median": 0.286,
"usd_po_mean": 0.021,
"wall_s_mean": 84.5,
"wall_s_median": 71.7,
"driver_turns_mean": 1.333,
"questions_mean": 0.333,
"caps": {},
"end_reasons": [
"done",
"done",
"done"
],
"models": [
"claude-sonnet-5-5"
],
"superpowers_injected": [
false,
false,
false
],
"skills_per_run": {
"bare-r1": [],
"bare-r2": [],
"bare-r3": []
},
"skills_failed": {}
},
"F1/sp": {
"task": "F1",
"cfg": "sp",
"label": "sp (hook-routed)",
"n": 3,
"hidden_mean": 5,
"hidden_median": 5,
"hidden_per_run": [
5,
5,
5
],
"req_pass_rate": {
"test_req1": 3,
"test_req2": 3,
"test_req3": 3,
"test_req4": 3,
"test_req5": 3
},
"suite_green": 3,
"typecheck_green": 3,
"tests_added_mean": 12.333,
"code_diff_mean": 235,
"code_diff_median": 227,
"all_diff_mean": 237,
"non_code_files_mean": 0.667,
"tokens_mean": 724600.667,
"tokens_median": 765108,
"out_tokens_mean": 16869.333,
"usd_mean": 0.502,
"usd_median": 0.525,
"usd_po_mean": 0.041,
"wall_s_mean": 118.1,
"wall_s_median": 119.6,
"driver_turns_mean": 2.667,
"questions_mean": 1.667,
"caps": {},
"end_reasons": [
"done",
"done",
"done"
],
"models": [
"claude-sonnet-5-5"
],
"superpowers_injected": [
true,
true,
true
],
"skills_per_run": {
"sp-r1": [
"brainstorming"
],
"sp-r2": [
"brainstorming",
"test-driven-development"
],
"sp-r3": [
"brainstorming"
]
},
"skills_failed": {}
},
"F1/spd": {
"task": "F1",
"cfg": "spd",
"label": "spd (Superpowers driven)",
"n": 3,
"hidden_mean": 5,
"hidden_median": 5,
"hidden_per_run": [
5,
5,
5
],
"req_pass_rate": {
"test_req1": 3,
"test_req2": 3,
"test_req3": 3,
"test_req4": 3,
"test_req5": 3
},
"suite_green": 3,
"typecheck_green": 3,
"tests_added_mean": 27.667,
"code_diff_mean": 386,
"code_diff_median": 371,
"all_diff_mean": 582,
"non_code_files_mean": 2,
"tokens_mean": 4479876.333,
"tokens_median": 3887015,
"out_tokens_mean": 61684.667,
"usd_mean": 3.738,
"usd_median": 3.412,
"usd_po_mean": 0.116,
"wall_s_mean": 607.367,
"wall_s_median": 600.8,
"driver_turns_mean": 7,
"questions_mean": 4,
"caps": {},
"end_reasons": [
"done",
"done",
"done"
],
"models": [
"claude-fable-5-1",
"claude-sonnet-5-5"
],
"superpowers_injected": [
true,
true,
true
],
"skills_per_run": {
"spd-r1": [
"test-driven-development"
],
"spd-r2": [
"test-driven-development"
],
"spd-r3": [
"test-driven-development",
"finishing-a-development-branch"
]
},
"skills_failed": {}
},
"F1/mp": {
"task": "F1",
"cfg": "mp",
"label": "mp (Main Flow)",
"n": 3,
"hidden_mean": 5,
"hidden_median": 5,
"hidden_per_run": [
5,
5,
5
],
"req_pass_rate": {
"test_req1": 3,
"test_req2": 3,
"test_req3": 3,
"test_req4": 3,
"test_req5": 3
},
"suite_green": 3,
"typecheck_green": 3,
"tests_added_mean": 27.667,
"code_diff_mean": 372,
"code_diff_median": 369,
"all_diff_mean": 578,
"non_code_files_mean": 7.667,
"tokens_mean": 4476103.667,
"tokens_median": 4286993,
"out_tokens_mean": 56959,
"usd_mean": 2.043,
"usd_median": 2.102,
"usd_po_mean": 0.205,
"wall_s_mean": 439.033,
"wall_s_median": 472.4,
"driver_turns_mean": 12,
"questions_mean": 7,
"caps": {
"mp-r1": [
"driver_turns_12"
],
"mp-r3": [
"driver_turns_12"
]
},
"end_reasons": [
"cap_driver_turns",
"done",
"cap_driver_turns"
],
"models": [
"claude-sonnet-5-5"
],
"superpowers_injected": [
false,
false,
false
],
"skills_per_run": {
"mp-r1": [
"grilling",
"domain-modeling",
"tdd",
"code-review"
],
"mp-r2": [
"grilling",
"domain-modeling",
"tdd",
"code-review"
],
"mp-r3": [
"grilling",
"domain-modeling",
"tdd",
"code-review"
]
},
"skills_failed": {}
},
"F1/both": {
"task": "F1",
"cfg": "both",
"label": "both (sp+mp, mp flow)",
"n": 1,
"hidden_mean": 5,
"hidden_median": 5,
"hidden_per_run": [
5
],
"req_pass_rate": {
"test_req1": 1,
"test_req2": 1,
"test_req3": 1,
"test_req4": 1,
"test_req5": 1
},
"suite_green": 1,
"typecheck_green": 1,
"tests_added_mean": 22,
"code_diff_mean": 374,
"code_diff_median": 374,
"all_diff_mean": 561,
"non_code_files_mean": 8,
"tokens_mean": 5754354,
"tokens_median": 5754354,
"out_tokens_mean": 57632,
"usd_mean": 2.326,
"usd_median": 2.326,
"usd_po_mean": 0.218,
"wall_s_mean": 448.2,
"wall_s_median": 448.2,
"driver_turns_mean": 12,
"questions_mean": 6,
"caps": {},
"end_reasons": [
"done"
],
"models": [
"claude-sonnet-5-5"
],
"superpowers_injected": [
true
],
"skills_per_run": {
"both-r1": [
"grilling",
"domain-modeling",
"tdd",
"code-review"
]
},
"skills_failed": {}
},
"F2/bare": {
"task": "F2",
"cfg": "bare",
"label": "bare",
"n": 2,
"hidden_mean": 4.5,
"hidden_median": 4.5,
"hidden_per_run": [
4,
5
],
"req_pass_rate": {
"test_req1": 2,
"test_req2": 2,
"test_req3": 1,
"test_req4": 2,
"test_req5": 2
},
"suite_green": 2,
"typecheck_green": 2,
"tests_added_mean": 5,
"code_diff_mean": 206.5,
"code_diff_median": 206.5,
"all_diff_mean": 209.5,
"non_code_files_mean": 1,
"tokens_mean": 651322,
"tokens_median": 651322.0,
"out_tokens_mean": 15475,
"usd_mean": 0.413,
"usd_median": 0.413,
"usd_po_mean": 0.031,
"wall_s_mean": 103.15,
"wall_s_median": 103.15,
"driver_turns_mean": 2,
"questions_mean": 0.5,
"caps": {},
"end_reasons": [
"done",
"done"
],
"models": [
"claude-sonnet-5-5"
],
"superpowers_injected": [
false,
false
],
"skills_per_run": {
"f2-bare-r1": [],
"f2-bare-r2": []
},
"skills_failed": {}
},
"F2/spd": {
"task": "F2",
"cfg": "spd",
"label": "spd (Superpowers driven)",
"n": 2,
"hidden_mean": 5,
"hidden_median": 5.0,
"hidden_per_run": [
5,
5
],
"req_pass_rate": {
"test_req1": 2,
"test_req2": 2,
"test_req3": 2,
"test_req4": 2,
"test_req5": 2
},
"suite_green": 2,
"typecheck_green": 2,
"tests_added_mean": 33,
"code_diff_mean": 488.5,
"code_diff_median": 488.5,
"all_diff_mean": 750.5,
"non_code_files_mean": 2.5,
"tokens_mean": 4490365,
"tokens_median": 4490365.0,
"out_tokens_mean": 56644.5,
"usd_mean": 3.213,
"usd_median": 3.213,
"usd_po_mean": 0.101,
"wall_s_mean": 511.7,
"wall_s_median": 511.7,
"driver_turns_mean": 6.5,
"questions_mean": 3.5,
"caps": {},
"end_reasons": [
"done",
"done"
],
"models": [
"claude-fable-5-1",
"claude-sonnet-5-5"
],
"superpowers_injected": [
true,
true
],
"skills_per_run": {
"f2-spd-r1": [
"writing-plans",
"test-driven-development"
],
"f2-spd-r2": [
"test-driven-development"
]
},
"skills_failed": {}
},
"F2/mp": {
"task": "F2",
"cfg": "mp",
"label": "mp (Main Flow)",
"n": 2,
"hidden_mean": 5,
"hidden_median": 5.0,
"hidden_per_run": [
5,
5
],
"req_pass_rate": {
"test_req1": 2,
"test_req2": 2,
"test_req3": 2,
"test_req4": 2,
"test_req5": 2
},
"suite_green": 2,
"typecheck_green": 2,
"tests_added_mean": 31,
"code_diff_mean": 469,
"code_diff_median": 469.0,
"all_diff_mean": 658,
"non_code_files_mean": 6.5,
"tokens_mean": 4520638.5,
"tokens_median": 4520638.5,
"out_tokens_mean": 57142,
"usd_mean": 2.067,
"usd_median": 2.067,
"usd_po_mean": 0.203,
"wall_s_mean": 421.2,
"wall_s_median": 421.2,
"driver_turns_mean": 12,
"questions_mean": 6.5,
"caps": {
"f2-mp-r1": [
"driver_turns_12"
]
},
"end_reasons": [
"cap_driver_turns",
"done"
],
"models": [
"claude-sonnet-5-5"
],
"superpowers_injected": [
false,
false
],
"skills_per_run": {
"f2-mp-r1": [
"grilling",
"domain-modeling",
"tdd",
"code-review"
],
"f2-mp-r2": [
"grilling",
"domain-modeling",
"tdd",
"code-review"
]
},
"skills_failed": {}
}
},
"stages": {
"F1/sp_skill": {
"(none)": {
"runs_with_stage": 3,
"tokens_mean": 19549,
"tokens_median": 19538,
"out_tokens_mean": 238,
"usd_mean": 0.03,
"usd_median": 0.03,
"wall_s_mean": 5.333,
"wall_s_median": 5.4
},
"brainstorming": {
"runs_with_stage": 3,
"tokens_mean": 249260,
"tokens_median": 247900,
"out_tokens_mean": 8037.333,
"usd_mean": 0.219,
"usd_median": 0.244,
"wall_s_mean": 40.533,
"wall_s_median": 40.1
},
"test-driven-development": {
"runs_with_stage": 1,
"tokens_mean": 595318,
"tokens_median": 595318,
"out_tokens_mean": 15622,
"usd_mean": 0.405,
"usd_median": 0.405,
"wall_s_mean": 93.0,
"wall_s_median": 93.0
},
"implementation (no skill)": {
"runs_with_stage": 2,
"tokens_mean": 386028.5,
"tokens_median": 386028.5,
"out_tokens_mean": 5080,
"usd_mean": 0.176,
"usd_median": 0.176,
"wall_s_mean": 60.3,
"wall_s_median": 60.3
}
},
"F1/sp_phase": {
"before any skill": {
"runs_with_stage": 3,
"tokens_mean": 19549,
"tokens_median": 19538,
"out_tokens_mean": 238,
"usd_mean": 0.03,
"usd_median": 0.03,
"wall_s_mean": 5.333,
"wall_s_median": 5.4
},
"brainstorming": {
"runs_with_stage": 3,
"tokens_mean": 249260,
"tokens_median": 247900,
"out_tokens_mean": 8037.333,
"usd_mean": 0.219,
"usd_median": 0.244,
"wall_s_mean": 40.533,
"wall_s_median": 40.1
},
"implementation (no skill)": {
"runs_with_stage": 2,
"tokens_mean": 386028.5,
"tokens_median": 386028.5,
"out_tokens_mean": 5080,
"usd_mean": 0.176,
"usd_median": 0.176,
"wall_s_mean": 60.3,
"wall_s_median": 60.3
},
"execution": {
"runs_with_stage": 1,
"tokens_mean": 595318,
"tokens_median": 595318,
"out_tokens_mean": 15622,
"usd_mean": 0.405,
"usd_median": 0.405,
"wall_s_mean": 93.0,
"wall_s_median": 93.0
}
},
"F1/spd_skill": {
"brainstorming": {
"runs_with_stage": 3,
"tokens_mean": 182021.667,
"tokens_median": 160834,
"out_tokens_mean": 4160.667,
"usd_mean": 0.166,
"usd_median": 0.156,
"wall_s_mean": 38.8,
"wall_s_median": 33.6
},
"writing-plans": {
"runs_with_stage": 3,
"tokens_mean": 321354.667,
"tokens_median": 298267,
"out_tokens_mean": 11827,
"usd_mean": 0.266,
"usd_median": 0.274,
"wall_s_mean": 86.733,
"wall_s_median": 95.1
},
"executing-plans": {
"runs_with_stage": 3,
"tokens_mean": 66938,
"tokens_median": 68739,
"out_tokens_mean": 505,
"usd_mean": 0.047,
"usd_median": 0.047,
"wall_s_mean": 4.6,
"wall_s_median": 4.6
},
"test-driven-development": {
"runs_with_stage": 3,
"tokens_mean": 2927098.333,
"tokens_median": 2854967,
"out_tokens_mean": 22508.333,
"usd_mean": 0.974,
"usd_median": 0.936,
"wall_s_mean": 457.033,
"wall_s_median": 448.9
},
"finishing-a-development-branch": {
"runs_with_stage": 1,
"tokens_mean": 976505,
"tokens_median": 976505,
"out_tokens_mean": 5297,
"usd_mean": 0.282,
"usd_median": 0.282,
"wall_s_mean": 50.7,
"wall_s_median": 50.7
},
"test-driven-development [subagent]": {
"runs_with_stage": 3,
"tokens_mean": 656962,
"tokens_median": 565811,
"out_tokens_mean": 20918,
"usd_mean": 2.192,
"usd_median": 2.047,
"wall_s_mean": 0.0,
"wall_s_median": 0.0
}
},
"F1/spd_phase": {
"brainstorming": {
"runs_with_stage": 3,
"tokens_mean": 182021.667,
"tokens_median": 160834,
"out_tokens_mean": 4160.667,
"usd_mean": 0.166,
"usd_median": 0.156,
"wall_s_mean": 38.8,
"wall_s_median": 33.6
},
"writing-plans": {
"runs_with_stage": 3,
"tokens_mean": 321354.667,
"tokens_median": 298267,
"out_tokens_mean": 11827,
"usd_mean": 0.266,
"usd_median": 0.274,
"wall_s_mean": 86.733,
"wall_s_median": 95.1
},
"execution": {
"runs_with_stage": 3,
"tokens_mean": 2994036.333,
"tokens_median": 2916459,
"out_tokens_mean": 23013.333,
"usd_mean": 1.021,
"usd_median": 0.981,
"wall_s_mean": 461.633,
"wall_s_median": 453.5
},
"finishing": {
"runs_with_stage": 1,
"tokens_mean": 976505,
"tokens_median": 976505,
"out_tokens_mean": 5297,
"usd_mean": 0.282,
"usd_median": 0.282,
"wall_s_mean": 50.7,
"wall_s_median": 50.7
},
"subagents (reviewers, explorers)": {
"runs_with_stage": 3,
"tokens_mean": 656962,
"tokens_median": 565811,
"out_tokens_mean": 20918,
"usd_mean": 2.192,
"usd_median": 2.047,
"wall_s_mean": 0.0,
"wall_s_median": 0.0
}
},
"F1/spd_slash": {
"/brainstorming": {
"runs_with_stage": 3,
"tokens_mean": 182021.667,
"tokens_median": 160834,
"out_tokens_mean": null,
"usd_mean": 0.166,
"usd_median": 0.156,
"wall_s_mean": 40.283,
"wall_s_median": 35.44
},
"/writing-plans": {
"runs_with_stage": 3,
"tokens_mean": 321354.667,
"tokens_median": 298267,
"out_tokens_mean": null,
"usd_mean": 0.266,
"usd_median": 0.274,
"wall_s_mean": 88.133,
"wall_s_median": 95.82
},
"/executing-plans": {
"runs_with_stage": 3,
"tokens_mean": 3976500,
"tokens_median": 3427914,
"out_tokens_mean": null,
"usd_mean": 3.307,
"usd_median": 3.028,
"wall_s_mean": 478.95,
"wall_s_median": 453.89
}
},
"F1/mp": {
"/grill-with-docs": {
"runs_with_stage": 3,
"tokens_mean": 316507.333,
"tokens_median": 348862,
"out_tokens_mean": 5999,
"usd_mean": 0.204,
"usd_median": 0.215,
"wall_s_mean": 59.933,
"wall_s_median": 59.9
},
"/to-spec": {
"runs_with_stage": 3,
"tokens_mean": 163344,
"tokens_median": 167578,
"out_tokens_mean": 5134.667,
"usd_mean": 0.118,
"usd_median": 0.113,
"wall_s_mean": 42.1,
"wall_s_median": 40.8
},
"/to-tickets": {
"runs_with_stage": 3,
"tokens_mean": 257955.333,
"tokens_median": 268062,
"out_tokens_mean": 5867.333,
"usd_mean": 0.143,
"usd_median": 0.151,
"wall_s_mean": 44.4,
"wall_s_median": 47.8
},
"/implement": {
"runs_with_stage": 3,
"tokens_mean": 2655564.667,
"tokens_median": 2613062,
"out_tokens_mean": 30214.667,
"usd_mean": 1.125,
"usd_median": 1.195,
"wall_s_mean": 224.567,
"wall_s_median": 245.4
},
"/code-review": {
"runs_with_stage": 3,
"tokens_mean": 1047585.667,
"tokens_median": 1107555,
"out_tokens_mean": 9293.667,
"usd_mean": 0.439,
"usd_median": 0.445,
"wall_s_mean": 64.2,
"wall_s_median": 65.8
},
"/pr": {
"runs_with_stage": 1,
"tokens_mean": 105440,
"tokens_median": 105440,
"out_tokens_mean": 1349,
"usd_mean": 0.041,
"usd_median": 0.041,
"wall_s_mean": 11.6,
"wall_s_median": 11.6
}
},
"F1/both": {
"/grill-with-docs": {
"runs_with_stage": 1,
"tokens_mean": 344391,
"tokens_median": 344391,
"out_tokens_mean": 6193,
"usd_mean": 0.225,
"usd_median": 0.225,
"wall_s_mean": 61.8,
"wall_s_median": 61.8
},
"/to-spec": {
"runs_with_stage": 1,
"tokens_mean": 217572,
"tokens_median": 217572,
"out_tokens_mean": 4835,
"usd_mean": 0.123,
"usd_median": 0.123,
"wall_s_mean": 42.9,
"wall_s_median": 42.9
},
"/to-tickets": {
"runs_with_stage": 1,
"tokens_mean": 382953,
"tokens_median": 382953,
"out_tokens_mean": 7115,
"usd_mean": 0.196,
"usd_median": 0.196,
"wall_s_mean": 49.4,
"wall_s_median": 49.4
},
"/implement": {
"runs_with_stage": 1,
"tokens_mean": 3070579,
"tokens_median": 3070579,
"out_tokens_mean": 24618,
"usd_mean": 1.112,
"usd_median": 1.112,
"wall_s_mean": 194.2,
"wall_s_median": 194.2
},
"/code-review": {
"runs_with_stage": 1,
"tokens_mean": 1615265,
"tokens_median": 1615265,
"out_tokens_mean": 13410,
"usd_mean": 0.625,
"usd_median": 0.625,
"wall_s_mean": 86.8,
"wall_s_median": 86.8
},
"/pr": {
"runs_with_stage": 1,
"tokens_mean": 123594,
"tokens_median": 123594,
"out_tokens_mean": 1461,
"usd_mean": 0.045,
"usd_median": 0.045,
"wall_s_mean": 13.1,
"wall_s_median": 13.1
}
},
"F2/spd_skill": {
"brainstorming": {
"runs_with_stage": 2,
"tokens_mean": 253890,
"tokens_median": 253890.0,
"out_tokens_mean": 6038,
"usd_mean": 0.207,
"usd_median": 0.207,
"wall_s_mean": 53.05,
"wall_s_median": 53.05
},
"writing-plans": {
"runs_with_stage": 2,
"tokens_mean": 397486.5,
"tokens_median": 397486.5,
"out_tokens_mean": 14059.5,
"usd_mean": 0.307,
"usd_median": 0.307,
"wall_s_mean": 90.75,
"wall_s_median": 90.75
},
"executing-plans": {
"runs_with_stage": 2,
"tokens_mean": 69973,
"tokens_median": 69973.0,
"out_tokens_mean": 286.5,
"usd_mean": 0.046,
"usd_median": 0.046,
"wall_s_mean": 3.5,
"wall_s_median": 3.5
},
"test-driven-development": {
"runs_with_stage": 2,
"tokens_mean": 3304676.5,
"tokens_median": 3304676.5,
"out_tokens_mean": 23243.5,
"usd_mean": 1.052,
"usd_median": 1.052,
"wall_s_mean": 361.25,
"wall_s_median": 361.25
},
"test-driven-development [subagent]": {
"runs_with_stage": 2,
"tokens_mean": 464339,
"tokens_median": 464339.0,
"out_tokens_mean": 13017,
"usd_mean": 1.602,
"usd_median": 1.602,
"wall_s_mean": 0.0,
"wall_s_median": 0.0
}
},
"F2/spd_phase": {
"brainstorming": {
"runs_with_stage": 2,
"tokens_mean": 253890,
"tokens_median": 253890.0,
"out_tokens_mean": 6038,
"usd_mean": 0.207,
"usd_median": 0.207,
"wall_s_mean": 53.05,
"wall_s_median": 53.05
},
"writing-plans": {
"runs_with_stage": 2,
"tokens_mean": 397486.5,
"tokens_median": 397486.5,
"out_tokens_mean": 14059.5,
"usd_mean": 0.307,
"usd_median": 0.307,
"wall_s_mean": 90.75,
"wall_s_median": 90.75
},
"execution": {
"runs_with_stage": 2,
"tokens_mean": 3374649.5,
"tokens_median": 3374649.5,
"out_tokens_mean": 23530,
"usd_mean": 1.098,
"usd_median": 1.098,
"wall_s_mean": 364.75,
"wall_s_median": 364.75
},
"subagents (reviewers, explorers)": {
"runs_with_stage": 2,
"tokens_mean": 464339,
"tokens_median": 464339.0,
"out_tokens_mean": 13017,
"usd_mean": 1.602,
"usd_median": 1.602,
"wall_s_mean": 0.0,
"wall_s_median": 0.0
}
},
"F2/spd_slash": {
"/brainstorming": {
"runs_with_stage": 2,
"tokens_mean": 485364,
"tokens_median": 485364.0,
"out_tokens_mean": null,
"usd_mean": 0.354,
"usd_median": 0.354,
"wall_s_mean": 96.815,
"wall_s_median": 96.815
},
"/writing-plans": {
"runs_with_stage": 2,
"tokens_mean": 166012.5,
"tokens_median": 166012.5,
"out_tokens_mean": null,
"usd_mean": 0.159,
"usd_median": 0.159,
"wall_s_mean": 49.755,
"wall_s_median": 49.755
},
"/executing-plans": {
"runs_with_stage": 2,
"tokens_mean": 3838988.5,
"tokens_median": 3838988.5,
"out_tokens_mean": null,
"usd_mean": 2.7,
"usd_median": 2.7,
"wall_s_mean": 365.1,
"wall_s_median": 365.1
}
},
"F2/mp": {
"/grill-with-docs": {
"runs_with_stage": 2,
"tokens_mean": 232965,
"tokens_median": 232965.0,
"out_tokens_mean": 5946,
"usd_mean": 0.182,
"usd_median": 0.182,
"wall_s_mean": 56.7,
"wall_s_median": 56.7
},
"/to-spec": {
"runs_with_stage": 2,
"tokens_mean": 180084.5,
"tokens_median": 180084.5,
"out_tokens_mean": 4452,
"usd_mean": 0.115,
"usd_median": 0.115,
"wall_s_mean": 37.6,
"wall_s_median": 37.6
},
"/to-tickets": {
"runs_with_stage": 2,
"tokens_mean": 253688.5,
"tokens_median": 253688.5,
"out_tokens_mean": 5152.5,
"usd_mean": 0.131,
"usd_median": 0.131,
"wall_s_mean": 38.5,
"wall_s_median": 38.5
},
"/implement": {
"runs_with_stage": 2,
"tokens_mean": 2689071,
"tokens_median": 2689071.0,
"out_tokens_mean": 31452,
"usd_mean": 1.159,
"usd_median": 1.159,
"wall_s_mean": 218.35,
"wall_s_median": 218.35
},
"/code-review": {
"runs_with_stage": 2,
"tokens_mean": 1107122.5,
"tokens_median": 1107122.5,
"out_tokens_mean": 9453.5,
"usd_mean": 0.458,
"usd_median": 0.458,
"wall_s_mean": 63.85,
"wall_s_median": 63.85
},
"/pr": {
"runs_with_stage": 1,
"tokens_mean": 115414,
"tokens_median": 115414,
"out_tokens_mean": 1372,
"usd_mean": 0.043,
"usd_median": 0.043,
"wall_s_mean": 12.4,
"wall_s_median": 12.4
}
}
},
"runs": [
{
"run": "bare-r1",
"task": "F1",
"cfg": "bare",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 149,
"code_diff_removed": 11,
"diff_added": 150,
"diff_removed": 12,
"tests_added_suite": 6,
"tokens": {
"in": 42,
"out": 16265,
"cr": 786548,
"cw": 37896,
"total": 840751
},
"usd": 0.4716,
"usd_po": 0.0321,
"wall_s": 118.8,
"driver_turns": 2,
"questions_routed_to_po": 1,
"end_reason": "done",
"caps": [],
"skills_invoked": [],
"superpowers_injected": false,
"commits": 0,
"models": [
"claude-sonnet-5-5"
],
"stages": {
"bare": {
"turns": [
1,
2
],
"in": 42,
"out": 16265,
"cr": 786548,
"cw": 37896,
"total": 840751,
"usd": 0.4716,
"wall_s": 118.8,
"api_calls": 21
}
}
},
{
"run": "bare-r2",
"task": "F1",
"cfg": "bare",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 118,
"code_diff_removed": 10,
"diff_added": 118,
"diff_removed": 10,
"tests_added_suite": 4,
"tokens": {
"in": 22,
"out": 9513,
"cr": 316959,
"cw": 28254,
"total": 354748
},
"usd": 0.2716,
"usd_po": 0.016,
"wall_s": 63.0,
"driver_turns": 1,
"questions_routed_to_po": 0,
"end_reason": "done",
"caps": [],
"skills_invoked": [],
"superpowers_injected": false,
"commits": 0,
"models": [
"claude-sonnet-5-5"
],
"stages": {
"bare": {
"turns": [
1
],
"in": 22,
"out": 9513,
"cr": 316959,
"cw": 28254,
"total": 354748,
"usd": 0.2716,
"wall_s": 63.0,
"api_calls": 11
}
}
},
{
"run": "bare-r3",
"task": "F1",
"cfg": "bare",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 125,
"code_diff_removed": 12,
"diff_added": 125,
"diff_removed": 12,
"tests_added_suite": 4,
"tokens": {
"in": 22,
"out": 10225,
"cr": 321705,
"cw": 29890,
"total": 361842
},
"usd": 0.2862,
"usd_po": 0.0148,
"wall_s": 71.7,
"driver_turns": 1,
"questions_routed_to_po": 0,
"end_reason": "done",
"caps": [],
"skills_invoked": [],
"superpowers_injected": false,
"commits": 0,
"models": [
"claude-sonnet-5-5"
],
"stages": {
"bare": {
"turns": [
1
],
"in": 22,
"out": 10225,
"cr": 321705,
"cw": 29890,
"total": 361842,
"usd": 0.2862,
"wall_s": 71.7,
"api_calls": 11
}
}
},
{
"run": "both-r1",
"task": "F1",
"cfg": "both",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 356,
"code_diff_removed": 18,
"diff_added": 543,
"diff_removed": 18,
"tests_added_suite": 22,
"tokens": {
"in": 192,
"out": 57632,
"cr": 5502518,
"cw": 194012,
"total": 5754354
},
"usd": 2.3259,
"usd_po": 0.2185,
"wall_s": 448.2,
"driver_turns": 12,
"questions_routed_to_po": 6,
"end_reason": "done",
"caps": [],
"skills_invoked": [
"grilling",
"domain-modeling",
"tdd",
"code-review"
],
"superpowers_injected": true,
"commits": 7,
"models": [
"claude-sonnet-5-5"
],
"stages": {
"/grill-with-docs": {
"turns": [
1,
2,
3,
4
],
"in": 22,
"out": 6193,
"cr": 313031,
"cw": 25145,
"total": 344391,
"usd": 0.2252,
"wall_s": 61.8,
"api_calls": 11
},
"/to-spec": {
"turns": [
5,
6
],
"in": 12,
"out": 4835,
"cr": 204355,
"cw": 8370,
"total": 217572,
"usd": 0.1227,
"wall_s": 42.9,
"api_calls": 5
},
"/to-tickets": {
"turns": [
7,
8
],
"in": 16,
"out": 7115,
"cr": 362815,
"cw": 13007,
"total": 382953,
"usd": 0.1958,
"wall_s": 49.4,
"api_calls": 7
},
"/implement": {
"turns": [
9
],
"in": 92,
"out": 24618,
"cr": 2960003,
"cw": 85866,
"total": 3070579,
"usd": 1.1123,
"wall_s": 194.2,
"api_calls": 36
},
"/code-review": {
"turns": [
10,
11
],
"in": 46,
"out": 13410,
"cr": 1541663,
"cw": 60146,
"total": 1615265,
"usd": 0.6253,
"wall_s": 86.8,
"api_calls": 13
},
"/pr": {
"turns": [
12
],
"in": 4,
"out": 1461,
"cr": 120651,
"cw": 1478,
"total": 123594,
"usd": 0.0447,
"wall_s": 13.1,
"api_calls": 1
}
}
},
{
"run": "f2-bare-r1",
"task": "F2",
"cfg": "bare",
"hidden_pass": 4,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": false,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 204,
"code_diff_removed": 10,
"diff_added": 208,
"diff_removed": 10,
"tests_added_suite": 5,
"tokens": {
"in": 28,
"out": 15066,
"cr": 444890,
"cw": 34247,
"total": 494231
},
"usd": 0.3767,
"usd_po": 0.0167,
"wall_s": 91.6,
"driver_turns": 1,
"questions_routed_to_po": 0,
"end_reason": "done",
"caps": [],
"skills_invoked": [],
"superpowers_injected": false,
"commits": 0,
"models": [
"claude-sonnet-5-5"
],
"stages": {
"bare": {
"turns": [
1
],
"in": 28,
"out": 15066,
"cr": 444890,
"cw": 34247,
"total": 494231,
"usd": 0.3767,
"wall_s": 91.6,
"api_calls": 14
}
}
},
{
"run": "f2-bare-r2",
"task": "F2",
"cfg": "bare",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 189,
"code_diff_removed": 10,
"diff_added": 191,
"diff_removed": 10,
"tests_added_suite": 5,
"tokens": {
"in": 42,
"out": 15884,
"cr": 757517,
"cw": 34970,
"total": 808413
},
"usd": 0.4503,
"usd_po": 0.0453,
"wall_s": 114.7,
"driver_turns": 3,
"questions_routed_to_po": 1,
"end_reason": "done",
"caps": [],
"skills_invoked": [],
"superpowers_injected": false,
"commits": 1,
"models": [
"claude-sonnet-5-5"
],
"stages": {
"bare": {
"turns": [
1,
2,
3
],
"in": 42,
"out": 15884,
"cr": 757517,
"cw": 34970,
"total": 808413,
"usd": 0.4503,
"wall_s": 114.7,
"api_calls": 21
}
}
},
{
"run": "f2-mp-r1",
"task": "F2",
"cfg": "mp",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 460,
"code_diff_removed": 23,
"diff_added": 666,
"diff_removed": 25,
"tests_added_suite": 31,
"tokens": {
"in": 172,
"out": 60658,
"cr": 4559790,
"cw": 202896,
"total": 4823516
},
"usd": 2.1824,
"usd_po": 0.1989,
"wall_s": 443.0,
"driver_turns": 12,
"questions_routed_to_po": 7,
"end_reason": "cap_driver_turns",
"caps": [
"driver_turns_12"
],
"skills_invoked": [
"grilling",
"domain-modeling",
"tdd",
"code-review"
],
"superpowers_injected": false,
"commits": 7,
"models": [
"claude-sonnet-5-5"
],
"stages": {
"/grill-with-docs": {
"turns": [
1,
2,
3
],
"in": 18,
"out": 6627,
"cr": 229372,
"cw": 22326,
"total": 258343,
"usd": 0.2015,
"wall_s": 62.8,
"api_calls": 9
},
"/to-spec": {
"turns": [
4,
5
],
"in": 12,
"out": 5218,
"cr": 195137,
"cw": 10113,
"total": 210480,
"usd": 0.1317,
"wall_s": 45.4,
"api_calls": 5
},
"/to-tickets": {
"turns": [
6,
7,
8
],
"in": 16,
"out": 7288,
"cr": 356825,
"cw": 10505,
"total": 374634,
"usd": 0.1863,
"wall_s": 53.0,
"api_calls": 7
},
"/implement": {
"turns": [
9,
10
],
"in": 92,
"out": 34442,
"cr": 2978947,
"cw": 104921,
"total": 3118402,
"usd": 1.2795,
"wall_s": 238.8,
"api_calls": 36
},
"/code-review": {
"turns": [
11,
12
],
"in": 34,
"out": 7083,
"cr": 799509,
"cw": 55031,
"total": 861657,
"usd": 0.3835,
"wall_s": 43.1,
"api_calls": 6
}
}
},
{
"run": "f2-mp-r2",
"task": "F2",
"cfg": "mp",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 440,
"code_diff_removed": 15,
"diff_added": 607,
"diff_removed": 18,
"tests_added_suite": 31,
"tokens": {
"in": 152,
"out": 53626,
"cr": 3976936,
"cw": 187047,
"total": 4217761
},
"usd": 1.9511,
"usd_po": 0.2073,
"wall_s": 399.4,
"driver_turns": 12,
"questions_routed_to_po": 6,
"end_reason": "done",
"caps": [],
"skills_invoked": [
"grilling",
"domain-modeling",
"tdd",
"code-review"
],
"superpowers_injected": false,
"commits": 2,
"models": [
"claude-sonnet-5-5"
],
"stages": {
"/grill-with-docs": {
"turns": [
1,
2,
3
],
"in": 16,
"out": 5265,
"cr": 184055,
"cw": 18251,
"total": 207587,
"usd": 0.1625,
"wall_s": 50.6,
"api_calls": 8
},
"/to-spec": {
"turns": [
4,
5
],
"in": 10,
"out": 3686,
"cr": 137547,
"cw": 8446,
"total": 149689,
"usd": 0.0982,
"wall_s": 29.8,
"api_calls": 4
},
"/to-tickets": {
"turns": [
6,
7
],
"in": 8,
"out": 3017,
"cr": 124377,
"cw": 5341,
"total": 132743,
"usd": 0.0764,
"wall_s": 24.0,
"api_calls": 3
},
"/implement": {
"turns": [
8
],
"in": 74,
"out": 28462,
"cr": 2129926,
"cw": 101278,
"total": 2259740,
"usd": 1.0391,
"wall_s": 197.9,
"api_calls": 27
},
"/code-review": {
"turns": [
9,
10,
11
],
"in": 43,
"out": 11824,
"cr": 1288603,
"cw": 52118,
"total": 1352588,
"usd": 0.5323,
"wall_s": 84.6,
"api_calls": 11
},
"/pr": {
"turns": [
12
],
"in": 1,
"out": 1372,
"cr": 112428,
"cw": 1613,
"total": 115414,
"usd": 0.0427,
"wall_s": 12.4,
"api_calls": 1
}
}
},
{
"run": "f2-spd-r1",
"task": "F2",
"cfg": "spd",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 546,
"code_diff_removed": 14,
"diff_added": 856,
"diff_removed": 17,
"tests_added_suite": 46,
"tokens": {
"in": 380,
"out": 56881,
"cr": 5074733,
"cw": 178622,
"total": 5310616
},
"usd": 3.4244,
"usd_po": 0.09,
"wall_s": 506.3,
"driver_turns": 6,
"questions_routed_to_po": 3,
"end_reason": "done",
"caps": [],
"skills_invoked": [
"writing-plans",
"test-driven-development"
],
"superpowers_injected": true,
"commits": 8,
"models": [
"claude-fable-5-1",
"claude-sonnet-5-5"
],
"stages": {
"brainstorming": {
"turns": [
1,
2,
3
],
"in": 18,
"out": 8103,
"cr": 280105,
"cw": 28324,
"total": 316550,
"usd": 0.2504,
"wall_s": 64.3,
"api_calls": 9
},
"writing-plans": {
"turns": [
3,
4,
5
],
"in": 20,
"out": 13028,
"cr": 491569,
"cw": 24216,
"total": 528833,
"usd": 0.3255,
"wall_s": 89.2,
"api_calls": 9
},
"writing-plans [subagent]": {
"turns": [
4
],
"in": 0,
"out": 0,
"cr": 0,
"cw": 0,
"total": 0,
"usd": 0.0,
"wall_s": 0.0,
"api_calls": 0
},
"executing-plans": {
"turns": [
6
],
"in": 4,
"out": 291,
"cr": 65551,
"cw": 7617,
"total": 73463,
"usd": 0.0465,
"wall_s": 4.0,
"api_calls": 1
},
"test-driven-development": {
"turns": [
6
],
"in": 80,
"out": 24030,
"cr": 3786616,
"cw": 42748,
"total": 3853474,
"usd": 1.1688,
"wall_s": 345.9,
"api_calls": 40
},
"test-driven-development [subagent]": {
"turns": [
6
],
"in": 258,
"out": 11429,
"cr": 450892,
"cw": 75717,
"total": 538296,
"usd": 1.6332,
"wall_s": 0.0,
"api_calls": 0
}
}
},
{
"run": "f2-spd-r2",
"task": "F2",
"cfg": "spd",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 401,
"code_diff_removed": 16,
"diff_added": 611,
"diff_removed": 17,
"tests_added_suite": 20,
"tokens": {
"in": 316,
"out": 56408,
"cr": 3456411,
"cw": 156979,
"total": 3670114
},
"usd": 3.0022,
"usd_po": 0.1126,
"wall_s": 517.1,
"driver_turns": 7,
"questions_routed_to_po": 4,
"end_reason": "done",
"caps": [],
"skills_invoked": [
"test-driven-development"
],
"superpowers_injected": true,
"commits": 6,
"models": [
"claude-fable-5-1",
"claude-sonnet-5-5"
],
"stages": {
"brainstorming": {
"turns": [
1,
2,
3,
4
],
"in": 12,
"out": 3973,
"cr": 164538,
"cw": 22707,
"total": 191230,
"usd": 0.1635,
"wall_s": 41.8,
"api_calls": 6
},
"brainstorming [subagent]": {
"turns": [
1,
3
],
"in": 0,
"out": 0,
"cr": 0,
"cw": 0,
"total": 0,
"usd": 0.0,
"wall_s": 0.0,
"api_calls": 0
},
"writing-plans": {
"turns": [
5,
6
],
"in": 12,
"out": 15091,
"cr": 228188,
"cw": 22849,
"total": 266140,
"usd": 0.288,
"wall_s": 92.3,
"api_calls": 5
},
"writing-plans [subagent]": {
"turns": [
6
],
"in": 0,
"out": 0,
"cr": 0,
"cw": 0,
"total": 0,
"usd": 0.0,
"wall_s": 0.0,
"api_calls": 0
},
"executing-plans": {
"turns": [
7
],
"in": 4,
"out": 282,
"cr": 58567,
"cw": 7630,
"total": 66483,
"usd": 0.0451,
"wall_s": 3.0,
"api_calls": 1
},
"test-driven-development": {
"turns": [
7
],
"in": 62,
"out": 22457,
"cr": 2690340,
"cw": 43020,
"total": 2755879,
"usd": 0.9348,
"wall_s": 376.6,
"api_calls": 31
},
"test-driven-development [subagent]": {
"turns": [
7
],
"in": 226,
"out": 14605,
"cr": 314778,
"cw": 60773,
"total": 390382,
"usd": 1.5709,
"wall_s": 0.0,
"api_calls": 0
}
}
},
{
"run": "mp-r1",
"task": "F1",
"cfg": "mp",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 349,
"code_diff_removed": 20,
"diff_added": 524,
"diff_removed": 20,
"tests_added_suite": 29,
"tokens": {
"in": 181,
"out": 60902,
"cr": 4842442,
"cw": 194978,
"total": 5098503
},
"usd": 2.2217,
"usd_po": 0.2058,
"wall_s": 472.4,
"driver_turns": 12,
"questions_routed_to_po": 7,
"end_reason": "cap_driver_turns",
"caps": [
"driver_turns_12"
],
"skills_invoked": [
"grilling",
"domain-modeling",
"tdd",
"code-review"
],
"superpowers_injected": false,
"commits": 7,
"models": [
"claude-sonnet-5-5"
],
"stages": {
"/grill-with-docs": {
"turns": [
1,
2,
3
],
"in": 24,
"out": 6223,
"cr": 320351,
"cw": 22264,
"total": 348862,
"usd": 0.2154,
"wall_s": 59.9,
"api_calls": 12
},
"/to-spec": {
"turns": [
4,
5
],
"in": 10,
"out": 4516,
"cr": 153800,
"cw": 9252,
"total": 167578,
"usd": 0.1129,
"wall_s": 37.6,
"api_calls": 4
},
"/to-tickets": {
"turns": [
6,
7,
8
],
"in": 16,
"out": 7382,
"cr": 349965,
"cw": 10865,
"total": 368228,
"usd": 0.1873,
"wall_s": 55.8,
"api_calls": 7
},
"/implement": {
"turns": [
9,
10
],
"in": 92,
"out": 32290,
"cr": 2890397,
"cw": 100256,
"total": 3023035,
"usd": 1.226,
"wall_s": 251.2,
"api_calls": 35
},
"/code-review": {
"turns": [
11,
12
],
"in": 39,
"out": 10491,
"cr": 1127929,
"cw": 52341,
"total": 1190800,
"usd": 0.4801,
"wall_s": 67.9,
"api_calls": 9
}
}
},
{
"run": "mp-r2",
"task": "F1",
"cfg": "mp",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 314,
"code_diff_removed": 15,
"diff_added": 527,
"diff_removed": 15,
"tests_added_suite": 25,
"tokens": {
"in": 154,
"out": 48104,
"cr": 3826019,
"cw": 168538,
"total": 4042815
},
"usd": 1.8045,
"usd_po": 0.1988,
"wall_s": 365.0,
"driver_turns": 12,
"questions_routed_to_po": 6,
"end_reason": "done",
"caps": [],
"skills_invoked": [
"grilling",
"domain-modeling",
"tdd",
"code-review"
],
"superpowers_injected": false,
"commits": 7,
"models": [
"claude-sonnet-5-5"
],
"stages": {
"/grill-with-docs": {
"turns": [
1,
2,
3
],
"in": 16,
"out": 4727,
"cr": 187718,
"cw": 18567,
"total": 211028,
"usd": 0.1591,
"wall_s": 47.0,
"api_calls": 8
},
"/to-spec": {
"turns": [
4,
5
],
"in": 10,
"out": 4965,
"cr": 136863,
"cw": 8781,
"total": 150619,
"usd": 0.1122,
"wall_s": 40.8,
"api_calls": 4
},
"/to-tickets": {
"turns": [
6,
7
],
"in": 8,
"out": 4035,
"cr": 126982,
"cw": 6551,
"total": 137576,
"usd": 0.092,
"wall_s": 29.6,
"api_calls": 3
},
"/implement": {
"turns": [
8,
9
],
"in": 78,
"out": 24215,
"cr": 2224544,
"cw": 81760,
"total": 2330597,
"usd": 0.9551,
"wall_s": 177.1,
"api_calls": 31
},
"/code-review": {
"turns": [
10,
11
],
"in": 38,
"out": 8813,
"cr": 1047606,
"cw": 51098,
"total": 1107555,
"usd": 0.4451,
"wall_s": 58.9,
"api_calls": 10
},
"/pr": {
"turns": [
12
],
"in": 4,
"out": 1349,
"cr": 102306,
"cw": 1781,
"total": 105440,
"usd": 0.0411,
"wall_s": 11.6,
"api_calls": 1
}
}
},
{
"run": "mp-r3",
"task": "F1",
"cfg": "mp",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 395,
"code_diff_removed": 23,
"diff_added": 625,
"diff_removed": 23,
"tests_added_suite": 29,
"tokens": {
"in": 167,
"out": 61871,
"cr": 4014893,
"cw": 210062,
"total": 4286993
},
"usd": 2.1024,
"usd_po": 0.2105,
"wall_s": 479.7,
"driver_turns": 12,
"questions_routed_to_po": 8,
"end_reason": "cap_driver_turns",
"caps": [
"driver_turns_12"
],
"skills_invoked": [
"grilling",
"domain-modeling",
"tdd",
"code-review"
],
"superpowers_injected": false,
"commits": 5,
"models": [
"claude-sonnet-5-5"
],
"stages": {
"/grill-with-docs": {
"turns": [
1,
2,
3,
4
],
"in": 26,
"out": 7047,
"cr": 358800,
"cw": 23759,
"total": 389632,
"usd": 0.2373,
"wall_s": 72.9,
"api_calls": 13
},
"/to-spec": {
"turns": [
5,
6
],
"in": 10,
"out": 5923,
"cr": 156583,
"cw": 9319,
"total": 171835,
"usd": 0.1278,
"wall_s": 47.9,
"api_calls": 4
},
"/to-tickets": {
"turns": [
7,
8,
9
],
"in": 9,
"out": 6185,
"cr": 252224,
"cw": 9644,
"total": 268062,
"usd": 0.1509,
"wall_s": 47.8,
"api_calls": 5
},
"/implement": {
"turns": [
10,
11
],
"in": 82,
"out": 34139,
"cr": 2465960,
"cw": 112881,
"total": 2613062,
"usd": 1.1946,
"wall_s": 245.4,
"api_calls": 27
},
"/code-review": {
"turns": [
12
],
"in": 40,
"out": 8577,
"cr": 781326,
"cw": 54459,
"total": 844402,
"usd": 0.3917,
"wall_s": 65.8,
"api_calls": 5
}
}
},
{
"run": "sp-r1",
"task": "F1",
"cfg": "sp",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 182,
"code_diff_removed": 15,
"diff_added": 183,
"diff_removed": 16,
"tests_added_suite": 10,
"tokens": {
"in": 28,
"out": 14522,
"cr": 553287,
"cw": 45597,
"total": 613434
},
"usd": 0.4383,
"usd_po": 0.0424,
"wall_s": 99.4,
"driver_turns": 3,
"questions_routed_to_po": 2,
"end_reason": "done",
"caps": [],
"skills_invoked": [
"brainstorming"
],
"superpowers_injected": true,
"commits": 1,
"models": [
"claude-sonnet-5-5"
],
"stages": {
"(none)": {
"turns": [
1
],
"in": 2,
"out": 209,
"cr": 13011,
"cw": 6298,
"total": 19520,
"usd": 0.0299,
"wall_s": 5.4,
"api_calls": 1
},
"brainstorming": {
"turns": [
1,
2
],
"in": 14,
"out": 9666,
"cr": 212052,
"cw": 26168,
"total": 247900,
"usd": 0.2438,
"wall_s": 40.1,
"api_calls": 7
},
"implementation (no skill)": {
"turns": [
2,
3
],
"in": 12,
"out": 4647,
"cr": 328224,
"cw": 13131,
"total": 346014,
"usd": 0.1647,
"wall_s": 52.8,
"api_calls": 6
},
"implementation (no skill) [subagent]": {
"turns": [
2
],
"in": 0,
"out": 0,
"cr": 0,
"cw": 0,
"total": 0,
"usd": 0.0,
"wall_s": 0.0,
"api_calls": 0
}
}
},
{
"run": "sp-r2",
"task": "F1",
"cfg": "sp",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 212,
"code_diff_removed": 15,
"diff_added": 216,
"diff_removed": 15,
"tests_added_suite": 13,
"tokens": {
"in": 32,
"out": 18160,
"cr": 691441,
"cw": 55475,
"total": 765108
},
"usd": 0.5419,
"usd_po": 0.0305,
"wall_s": 119.6,
"driver_turns": 2,
"questions_routed_to_po": 1,
"end_reason": "done",
"caps": [],
"skills_invoked": [
"brainstorming",
"test-driven-development"
],
"superpowers_injected": true,
"commits": 0,
"models": [
"claude-sonnet-5-5"
],
"stages": {
"(none)": {
"turns": [
1
],
"in": 2,
"out": 227,
"cr": 13011,
"cw": 6298,
"total": 19538,
"usd": 0.0301,
"wall_s": 5.7,
"api_calls": 1
},
"brainstorming": {
"turns": [
1,
2
],
"in": 10,
"out": 2311,
"cr": 133731,
"cw": 14200,
"total": 150252,
"usd": 0.1067,
"wall_s": 20.4,
"api_calls": 5
},
"test-driven-development": {
"turns": [
2
],
"in": 20,
"out": 15622,
"cr": 544699,
"cw": 34977,
"total": 595318,
"usd": 0.4051,
"wall_s": 93.0,
"api_calls": 10
}
}
},
{
"run": "sp-r3",
"task": "F1",
"cfg": "sp",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 267,
"code_diff_removed": 14,
"diff_added": 267,
"diff_removed": 14,
"tests_added_suite": 14,
"tokens": {
"in": 34,
"out": 17926,
"cr": 727283,
"cw": 50017,
"total": 795260
},
"usd": 0.5249,
"usd_po": 0.0489,
"wall_s": 135.3,
"driver_turns": 3,
"questions_routed_to_po": 2,
"end_reason": "done",
"caps": [],
"skills_invoked": [
"brainstorming"
],
"superpowers_injected": true,
"commits": 1,
"models": [
"claude-sonnet-5-5"
],
"stages": {
"(none)": {
"turns": [
1
],
"in": 2,
"out": 278,
"cr": 13011,
"cw": 6298,
"total": 19589,
"usd": 0.0306,
"wall_s": 4.9,
"api_calls": 1
},
"brainstorming": {
"turns": [
1,
2
],
"in": 18,
"out": 12135,
"cr": 306378,
"cw": 31097,
"total": 349628,
"usd": 0.307,
"wall_s": 61.1,
"api_calls": 9
},
"implementation (no skill)": {
"turns": [
2,
3
],
"in": 14,
"out": 5513,
"cr": 407894,
"cw": 12622,
"total": 426043,
"usd": 0.1872,
"wall_s": 67.8,
"api_calls": 7
},
"implementation (no skill) [subagent]": {
"turns": [
2
],
"in": 0,
"out": 0,
"cr": 0,
"cw": 0,
"total": 0,
"usd": 0.0,
"wall_s": 0.0,
"api_calls": 0
}
}
},
{
"run": "spd-r1",
"task": "F1",
"cfg": "spd",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 350,
"code_diff_removed": 21,
"diff_added": 516,
"diff_removed": 21,
"tests_added_suite": 21,
"tokens": {
"in": 340,
"out": 59243,
"cr": 3428307,
"cw": 179396,
"total": 3667286
},
"usd": 3.4118,
"usd_po": 0.1084,
"wall_s": 600.8,
"driver_turns": 6,
"questions_routed_to_po": 3,
"end_reason": "done",
"caps": [],
"skills_invoked": [
"test-driven-development"
],
"superpowers_injected": true,
"commits": 4,
"models": [
"claude-fable-5-1",
"claude-sonnet-5-5"
],
"stages": {
"brainstorming": {
"turns": [
1,
2,
3
],
"in": 14,
"out": 5191,
"cr": 198912,
"cw": 25310,
"total": 229427,
"usd": 0.193,
"wall_s": 49.8,
"api_calls": 7
},
"brainstorming [subagent]": {
"turns": [
2
],
"in": 0,
"out": 0,
"cr": 0,
"cw": 0,
"total": 0,
"usd": 0.0,
"wall_s": 0.0,
"api_calls": 0
},
"writing-plans": {
"turns": [
4,
5
],
"in": 12,
"out": 13010,
"cr": 243336,
"cw": 23856,
"total": 280214,
"usd": 0.2742,
"wall_s": 95.1,
"api_calls": 5
},
"executing-plans": {
"turns": [
6
],
"in": 4,
"out": 614,
"cr": 62177,
"cw": 7788,
"total": 70583,
"usd": 0.0497,
"wall_s": 4.6,
"api_calls": 1
},
"test-driven-development": {
"turns": [
6
],
"in": 52,
"out": 22476,
"cr": 2455319,
"cw": 43404,
"total": 2521251,
"usd": 0.8895,
"wall_s": 448.9,
"api_calls": 26
},
"test-driven-development [subagent]": {
"turns": [
6
],
"in": 258,
"out": 17952,
"cr": 468563,
"cw": 79038,
"total": 565811,
"usd": 2.0053,
"wall_s": 0.0,
"api_calls": 0
}
}
},
{
"run": "spd-r2",
"task": "F1",
"cfg": "spd",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 317,
"code_diff_removed": 16,
"diff_added": 527,
"diff_removed": 16,
"tests_added_suite": 36,
"tokens": {
"in": 354,
"out": 53270,
"cr": 3666473,
"cw": 166918,
"total": 3887015
},
"usd": 3.3947,
"usd_po": 0.1118,
"wall_s": 520.0,
"driver_turns": 7,
"questions_routed_to_po": 4,
"end_reason": "done",
"caps": [],
"skills_invoked": [
"test-driven-development"
],
"superpowers_injected": true,
"commits": 5,
"models": [
"claude-fable-5-1",
"claude-sonnet-5-5"
],
"stages": {
"brainstorming": {
"turns": [
1,
2,
3
],
"in": 10,
"out": 3724,
"cr": 134220,
"cw": 22880,
"total": 160834,
"usd": 0.1556,
"wall_s": 33.6,
"api_calls": 5
},
"writing-plans": {
"turns": [
4,
5,
6
],
"in": 14,
"out": 8546,
"cr": 271872,
"cw": 17835,
"total": 298267,
"usd": 0.2112,
"wall_s": 63.9,
"api_calls": 6
},
"writing-plans [subagent]": {
"turns": [
6
],
"in": 0,
"out": 0,
"cr": 0,
"cw": 0,
"total": 0,
"usd": 0.0,
"wall_s": 0.0,
"api_calls": 0
},
"executing-plans": {
"turns": [
7
],
"in": 4,
"out": 385,
"cr": 53726,
"cw": 7377,
"total": 61492,
"usd": 0.0441,
"wall_s": 3.7,
"api_calls": 1
},
"test-driven-development": {
"turns": [
7
],
"in": 68,
"out": 21023,
"cr": 2791978,
"cw": 41898,
"total": 2854967,
"usd": 0.9364,
"wall_s": 415.1,
"api_calls": 34
},
"test-driven-development [subagent]": {
"turns": [
7
],
"in": 258,
"out": 19592,
"cr": 414677,
"cw": 76928,
"total": 511455,
"usd": 2.0474,
"wall_s": 0.0,
"api_calls": 0
}
}
},
{
"run": "spd-r3",
"task": "F1",
"cfg": "spd",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 434,
"code_diff_removed": 20,
"diff_added": 646,
"diff_removed": 20,
"tests_added_suite": 26,
"tokens": {
"in": 504,
"out": 72541,
"cr": 5615970,
"cw": 196313,
"total": 5885328
},
"usd": 4.4087,
"usd_po": 0.1286,
"wall_s": 701.3,
"driver_turns": 8,
"questions_routed_to_po": 5,
"end_reason": "done",
"caps": [],
"skills_invoked": [
"test-driven-development",
"finishing-a-development-branch"
],
"superpowers_injected": true,
"commits": 6,
"models": [
"claude-fable-5-1",
"claude-sonnet-5-5"
],
"stages": {
"brainstorming": {
"turns": [
1,
2,
3
],
"in": 10,
"out": 3567,
"cr": 130512,
"cw": 21715,
"total": 155804,
"usd": 0.1487,
"wall_s": 33.0,
"api_calls": 5
},
"writing-plans": {
"turns": [
4,
5,
6
],
"in": 16,
"out": 13925,
"cr": 345667,
"cw": 25975,
"total": 385583,
"usd": 0.3123,
"wall_s": 101.2,
"api_calls": 7
},
"writing-plans [subagent]": {
"turns": [
5,
6
],
"in": 0,
"out": 0,
"cr": 0,
"cw": 0,
"total": 0,
"usd": 0.0,
"wall_s": 0.0,
"api_calls": 0
},
"executing-plans": {
"turns": [
7
],
"in": 4,
"out": 516,
"cr": 60701,
"cw": 7518,
"total": 68739,
"usd": 0.0474,
"wall_s": 5.5,
"api_calls": 1
},
"test-driven-development": {
"turns": [
7
],
"in": 72,
"out": 24026,
"cr": 3334129,
"cw": 46850,
"total": 3405077,
"usd": 1.0946,
"wall_s": 507.1,
"api_calls": 36
},
"finishing-a-development-branch": {
"turns": [
7,
8
],
"in": 16,
"out": 5297,
"cr": 962018,
"cw": 9174,
"total": 976505,
"usd": 0.2821,
"wall_s": 50.7,
"api_calls": 8
},
"test-driven-development [subagent]": {
"turns": [
7
],
"in": 386,
"out": 25210,
"cr": 782943,
"cw": 85081,
"total": 893620,
"usd": 2.5236,
"wall_s": 0.0,
"api_calls": 0
},
"finishing-a-development-branch [subagent]": {
"turns": [
8
],
"in": 0,
"out": 0,
"cr": 0,
"cw": 0,
"total": 0,
"usd": 0.0,
"wall_s": 0.0,
"api_calls": 0
}
}
}
]
}
results.md
# flow-bench results
Untagged runs only (smoke/validation excluded). Tokens = input + output + cache read + cache write, all models, subagents included. USD = list-price equivalent from `total_cost_usd` deltas (plan quota, not a bill). Wall = sum of worker turn walls (PO time excluded). Diff = lines added+removed vs the base commit, src/ + test/.
## Per config (mean / median)
| task / config | n | hidden /5 (per run) | suite green | tsc green | code diff +/- | all files diff | tests added | tokens | USD | wall s | driver turns | questions | caps hit |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| F1 / bare | 3 | 5 / 5 (5,5,5) | 3/3 | 3/3 | 141.667 / 137 | 142.333 | 4.667 | 519,114 / 361,842 | 0.343 / 0.286 | 84.5 / 71.7 | 1.333 | 0.333 | none |
| F1 / sp (hook-routed) | 3 | 5 / 5 (5,5,5) | 3/3 | 3/3 | 235 / 227 | 237 | 12.333 | 724,601 / 765,108 | 0.502 / 0.525 | 118.1 / 119.6 | 2.667 | 1.667 | none |
| F1 / spd (Superpowers driven) | 3 | 5 / 5 (5,5,5) | 3/3 | 3/3 | 386 / 371 | 582 | 27.667 | 4,479,876 / 3,887,015 | 3.738 / 3.412 | 607.367 / 600.8 | 7 | 4 | none |
| F1 / mp (Main Flow) | 3 | 5 / 5 (5,5,5) | 3/3 | 3/3 | 372 / 369 | 578 | 27.667 | 4,476,104 / 4,286,993 | 2.043 / 2.102 | 439.033 / 472.4 | 12 | 7 | mp-r1: driver_turns_12; mp-r3: driver_turns_12 |
| F1 / both (sp+mp, mp flow) | 1 | 5 / 5 (5) | 1/1 | 1/1 | 374 / 374 | 561 | 22 | 5,754,354 / 5,754,354 | 2.326 / 2.326 | 448.2 / 448.2 | 12 | 6 | none |
| F2 / bare | 2 | 4.5 / 4.5 (4,5) | 2/2 | 2/2 | 206.5 / 206.5 | 209.5 | 5 | 651,322 / 651,322 | 0.413 / 0.413 | 103.15 / 103.15 | 2 | 0.5 | none |
| F2 / spd (Superpowers driven) | 2 | 5 / 5.0 (5,5) | 2/2 | 2/2 | 488.5 / 488.5 | 750.5 | 33 | 4,490,365 / 4,490,365 | 3.213 / 3.213 | 511.7 / 511.7 | 6.5 | 3.5 | none |
| F2 / mp (Main Flow) | 2 | 5 / 5.0 (5,5) | 2/2 | 2/2 | 469 / 469.0 | 658 | 31 | 4,520,638 / 4,520,638 | 2.067 / 2.067 | 421.2 / 421.2 | 12 | 6.5 | f2-mp-r1: driver_turns_12 |
Requirement pass counts (runs passing each hidden req):
- **F1 bare**: req1 3/3, req2 3/3, req3 3/3, req4 3/3, req5 3/3
- **F1 sp (hook-routed)**: req1 3/3, req2 3/3, req3 3/3, req4 3/3, req5 3/3
- **F1 spd (Superpowers driven)**: req1 3/3, req2 3/3, req3 3/3, req4 3/3, req5 3/3
- **F1 mp (Main Flow)**: req1 3/3, req2 3/3, req3 3/3, req4 3/3, req5 3/3
- **F1 both (sp+mp, mp flow)**: req1 1/1, req2 1/1, req3 1/1, req4 1/1, req5 1/1
- **F2 bare**: req1 2/2, req2 2/2, req3 1/2, req4 2/2, req5 2/2
- **F2 spd (Superpowers driven)**: req1 2/2, req2 2/2, req3 2/2, req4 2/2, req5 2/2
- **F2 mp (Main Flow)**: req1 2/2, req2 2/2, req3 2/2, req4 2/2, req5 2/2
## F1 spd (Superpowers driven) per slash-command stage (turn level, exact) (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| /brainstorming | 3 | 182,022 | 160,834 | - | 0.166 | 0.156 | 40.283 | 35.44 |
| /writing-plans | 3 | 321,355 | 298,267 | - | 0.266 | 0.274 | 88.133 | 95.82 |
| /executing-plans | 3 | 3,976,500 | 3,427,914 | - | 3.307 | 3.028 | 478.95 | 453.89 |
## F1 spd per skill in force (within-turn split) (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| brainstorming | 3 | 182,022 | 160,834 | 4,161 | 0.166 | 0.156 | 38.8 | 33.6 |
| writing-plans | 3 | 321,355 | 298,267 | 11,827 | 0.266 | 0.274 | 86.733 | 95.1 |
| executing-plans | 3 | 66,938 | 68,739 | 505 | 0.047 | 0.047 | 4.6 | 4.6 |
| test-driven-development | 3 | 2,927,098 | 2,854,967 | 22,508 | 0.974 | 0.936 | 457.033 | 448.9 |
| finishing-a-development-branch | 1 | 976,505 | 976,505 | 5,297 | 0.282 | 0.282 | 50.7 | 50.7 |
| test-driven-development [subagent] | 3 | 656,962 | 565,811 | 20,918 | 2.192 | 2.047 | 0.0 | 0.0 |
## F1 sp (hook-routed) per phase (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| before any skill | 3 | 19,549 | 19,538 | 238 | 0.03 | 0.03 | 5.333 | 5.4 |
| brainstorming | 3 | 249,260 | 247,900 | 8,037 | 0.219 | 0.244 | 40.533 | 40.1 |
| implementation (no skill) | 2 | 386,028 | 386,028 | 5,080 | 0.176 | 0.176 | 60.3 | 60.3 |
| execution | 1 | 595,318 | 595,318 | 15,622 | 0.405 | 0.405 | 93.0 | 93.0 |
## F1 sp (hook-routed) per skill in force (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| (none) | 3 | 19,549 | 19,538 | 238 | 0.03 | 0.03 | 5.333 | 5.4 |
| brainstorming | 3 | 249,260 | 247,900 | 8,037 | 0.219 | 0.244 | 40.533 | 40.1 |
| test-driven-development | 1 | 595,318 | 595,318 | 15,622 | 0.405 | 0.405 | 93.0 | 93.0 |
| implementation (no skill) | 2 | 386,028 | 386,028 | 5,080 | 0.176 | 0.176 | 60.3 | 60.3 |
## F1 mp per slash-command stage (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| /grill-with-docs | 3 | 316,507 | 348,862 | 5,999 | 0.204 | 0.215 | 59.933 | 59.9 |
| /to-spec | 3 | 163,344 | 167,578 | 5,135 | 0.118 | 0.113 | 42.1 | 40.8 |
| /to-tickets | 3 | 257,955 | 268,062 | 5,867 | 0.143 | 0.151 | 44.4 | 47.8 |
| /implement | 3 | 2,655,565 | 2,613,062 | 30,215 | 1.125 | 1.195 | 224.567 | 245.4 |
| /code-review | 3 | 1,047,586 | 1,107,555 | 9,294 | 0.439 | 0.445 | 64.2 | 65.8 |
| /pr | 1 | 105,440 | 105,440 | 1,349 | 0.041 | 0.041 | 11.6 | 11.6 |
## F1 both per slash-command stage (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| /grill-with-docs | 1 | 344,391 | 344,391 | 6,193 | 0.225 | 0.225 | 61.8 | 61.8 |
| /to-spec | 1 | 217,572 | 217,572 | 4,835 | 0.123 | 0.123 | 42.9 | 42.9 |
| /to-tickets | 1 | 382,953 | 382,953 | 7,115 | 0.196 | 0.196 | 49.4 | 49.4 |
| /implement | 1 | 3,070,579 | 3,070,579 | 24,618 | 1.112 | 1.112 | 194.2 | 194.2 |
| /code-review | 1 | 1,615,265 | 1,615,265 | 13,410 | 0.625 | 0.625 | 86.8 | 86.8 |
| /pr | 1 | 123,594 | 123,594 | 1,461 | 0.045 | 0.045 | 13.1 | 13.1 |
## F2 spd (Superpowers driven) per slash-command stage (turn level, exact) (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| /brainstorming | 2 | 485,364 | 485,364 | - | 0.354 | 0.354 | 96.815 | 96.815 |
| /writing-plans | 2 | 166,012 | 166,012 | - | 0.159 | 0.159 | 49.755 | 49.755 |
| /executing-plans | 2 | 3,838,988 | 3,838,988 | - | 2.7 | 2.7 | 365.1 | 365.1 |
## F2 spd per skill in force (within-turn split) (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| brainstorming | 2 | 253,890 | 253,890 | 6,038 | 0.207 | 0.207 | 53.05 | 53.05 |
| writing-plans | 2 | 397,486 | 397,486 | 14,060 | 0.307 | 0.307 | 90.75 | 90.75 |
| executing-plans | 2 | 69,973 | 69,973 | 286 | 0.046 | 0.046 | 3.5 | 3.5 |
| test-driven-development | 2 | 3,304,676 | 3,304,676 | 23,244 | 1.052 | 1.052 | 361.25 | 361.25 |
| test-driven-development [subagent] | 2 | 464,339 | 464,339 | 13,017 | 1.602 | 1.602 | 0.0 | 0.0 |
## F2 mp per slash-command stage (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| /grill-with-docs | 2 | 232,965 | 232,965 | 5,946 | 0.182 | 0.182 | 56.7 | 56.7 |
| /to-spec | 2 | 180,084 | 180,084 | 4,452 | 0.115 | 0.115 | 37.6 | 37.6 |
| /to-tickets | 2 | 253,688 | 253,688 | 5,152 | 0.131 | 0.131 | 38.5 | 38.5 |
| /implement | 2 | 2,689,071 | 2,689,071 | 31,452 | 1.159 | 1.159 | 218.35 | 218.35 |
| /code-review | 2 | 1,107,122 | 1,107,122 | 9,454 | 0.458 | 0.458 | 63.85 | 63.85 |
| /pr | 1 | 115,414 | 115,414 | 1,372 | 0.043 | 0.043 | 12.4 | 12.4 |
## Per run
| run | task | hidden | reqs failed | suite | tsc | code +/- | tests added | tokens | USD | wall s | turns | end | skills (first use order) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bare-r1 | F1 | 5/5 | - | green | ok | +149/-11 | 6 | 840,751 | 0.47 | 119 | 2 | done | - |
| bare-r2 | F1 | 5/5 | - | green | ok | +118/-10 | 4 | 354,748 | 0.27 | 63 | 1 | done | - |
| bare-r3 | F1 | 5/5 | - | green | ok | +125/-12 | 4 | 361,842 | 0.29 | 72 | 1 | done | - |
| sp-r1 | F1 | 5/5 | - | green | ok | +182/-15 | 10 | 613,434 | 0.44 | 99 | 3 | done | brainstorming |
| sp-r2 | F1 | 5/5 | - | green | ok | +212/-15 | 13 | 765,108 | 0.54 | 120 | 2 | done | brainstorming > test-driven-development |
| sp-r3 | F1 | 5/5 | - | green | ok | +267/-14 | 14 | 795,260 | 0.52 | 135 | 3 | done | brainstorming |
| spd-r1 | F1 | 5/5 | - | green | ok | +350/-21 | 21 | 3,667,286 | 3.41 | 601 | 6 | done | test-driven-development |
| spd-r2 | F1 | 5/5 | - | green | ok | +317/-16 | 36 | 3,887,015 | 3.39 | 520 | 7 | done | test-driven-development |
| spd-r3 | F1 | 5/5 | - | green | ok | +434/-20 | 26 | 5,885,328 | 4.41 | 701 | 8 | done | test-driven-development > finishing-a-development-branch |
| mp-r1 | F1 | 5/5 | - | green | ok | +349/-20 | 29 | 5,098,503 | 2.22 | 472 | 12 | cap_driver_turns (driver_turns_12) | grilling > domain-modeling > tdd > code-review |
| mp-r2 | F1 | 5/5 | - | green | ok | +314/-15 | 25 | 4,042,815 | 1.80 | 365 | 12 | done | grilling > domain-modeling > tdd > code-review |
| mp-r3 | F1 | 5/5 | - | green | ok | +395/-23 | 29 | 4,286,993 | 2.10 | 480 | 12 | cap_driver_turns (driver_turns_12) | grilling > domain-modeling > tdd > code-review |
| both-r1 | F1 | 5/5 | - | green | ok | +356/-18 | 22 | 5,754,354 | 2.33 | 448 | 12 | done | grilling > domain-modeling > tdd > code-review |
| f2-bare-r1 | F2 | 4/5 | req3 | green | ok | +204/-10 | 5 | 494,231 | 0.38 | 92 | 1 | done | - |
| f2-bare-r2 | F2 | 5/5 | - | green | ok | +189/-10 | 5 | 808,413 | 0.45 | 115 | 3 | done | - |
| f2-spd-r1 | F2 | 5/5 | - | green | ok | +546/-14 | 46 | 5,310,616 | 3.42 | 506 | 6 | done | writing-plans > test-driven-development |
| f2-spd-r2 | F2 | 5/5 | - | green | ok | +401/-16 | 20 | 3,670,114 | 3.00 | 517 | 7 | done | test-driven-development |
| f2-mp-r1 | F2 | 5/5 | - | green | ok | +460/-23 | 31 | 4,823,516 | 2.18 | 443 | 12 | cap_driver_turns (driver_turns_12) | grilling > domain-modeling > tdd > code-review |
| f2-mp-r2 | F2 | 5/5 | - | green | ok | +440/-15 | 31 | 4,217,761 | 1.95 | 399 | 12 | done | grilling > domain-modeling > tdd > code-review |
matrix.sh
#!/usr/bin/env bash
# Run matrix.txt ("<cfg> <rep>" lines), MATRIX_JOBS (default 2) at a time, each run in its own work copy.
# Appends to matrix.log. Resumable: SKIP_DONE=1. Usage: nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 &
set -uo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"; cd "$HERE"
LIST="${1:-matrix.txt}"; JOBS="${MATRIX_JOBS:-2}"
mkdir "$HERE/.matrix.lock" 2>/dev/null || { echo "another matrix.sh is running (remove .matrix.lock if stale)" >&2; exit 1; }
trap 'rmdir "$HERE/.matrix.lock"' EXIT
echo "=== $(date '+%F %T') start $LIST jobs=$JOBS pid=$$" >> matrix.log
grep -vE '^[[:space:]]*(#|$)' "$LIST" | xargs -P "$JOBS" -L 1 ./matrix-one.sh
echo "=== $(date '+%F %T') done $LIST" >> matrix.log
matrix-one.sh
#!/usr/bin/env bash
# One matrix cell (called by matrix.sh through xargs). Args: <cfg> <rep>
HERE="$(cd "$(dirname "$0")" && pwd)"; cd "$HERE"
CFG="$1"; REP="$2"; export TASK="${3:-F1}"
PFX=""; [ "$TASK" != "F1" ] && PFX="$(echo "$TASK" | tr "A-Z" "a-z")-"
if [ "${SKIP_DONE:-0}" = 1 ] && [ -f "runs/${PFX}${CFG}-r${REP}/score.json" ]; then
echo "$(date '+%F %T') skip ${PFX}${CFG}-r${REP}" >> matrix.log; exit 0
fi
echo "$(date '+%F %T') start ${PFX}${CFG}-r${REP}" >> matrix.log
line=$(./run.sh "$CFG" "$REP" < /dev/null 2>&1 | tail -1)
echo "$(date '+%F %T') $line" >> matrix.log
echo "$line"
matrix.txt
# <cfg> <rep>, run 2 at a time in this order (rep 1 of every config first)
bare 1
sp 1
mp 1
both 1
sp 2
mp 2
bare 2
mp 3
sp 3
bare 3
matrix2.txt
# <cfg> <rep> <task>; 2 lanes; long runs interleaved with short ones
spd 1 F1
mp 1 F1
bare 1 F1
sp 1 F1
spd 1 F2
mp 1 F2
bare 1 F2
spd 2 F1
mp 2 F1
bare 2 F1
sp 2 F1
spd 2 F2
mp 2 F2
bare 2 F2
both 1 F1
spd 3 F1
mp 3 F1
bare 3 F1
sp 3 F1
matrix-mp2.txt
mp 1
both 1
mp 2
mp 3
validate.sh
#!/usr/bin/env bash
# The 5 validation steps of SPEC.md, appended to validation.log.
# Usage: ./validate.sh [steps] default "1 2 3 4 5"; steps 3-5 (smoke runs, RUN_TAG=smoke) run in parallel.
set -uo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"; cd "$HERE"
. "$HERE/env.sh"
STEPS="${*:-1 2 3 4 5}"
LOG="$HERE/validation.log"
log(){ echo "$(date '+%F %T') $*" | tee -a "$LOG"; }
static_run(){ # <name> <patch or ""> [task] : prep a bare copy, optionally apply a patch, score it with finalize.py
local name="$1" patch="$2" task="${3:-F1}" out="$HERE/runs/$1" w="$HERE/work/$1"
rm -rf "$out"; mkdir -p "$out"
local base; base=$("$HERE/prep.sh" bare "$w" | tail -1)
if [ -n "$patch" ]; then ( cd "$w" && git apply "$patch" ) || { log "FAIL $name: git apply"; return 1; }; fi
printf '{"task":"%s","cfg":"bare","rep":0,"tag":"validation","work":"%s","name":"%s","base":"%s"}\n' "$task" "$w" "$name" "$base" > "$out/meta.json"
: > "$out/turns.jsonl"; echo '{"end_reason":"static"}' > "$out/driver.json"
python3 "$HERE/lib/finalize.py" "$out"
}
check(){ # <run dir> <python expr over s (score.json)> <label>
python3 - "$1" "$2" "$3" <<'PY'
import json, sys
s = json.load(open(sys.argv[1] + "/score.json"))
ok = eval(sys.argv[2], {"s": s})
print(("PASS " if ok else "FAIL ") + sys.argv[3])
sys.exit(0 if ok else 1)
PY
}
log "=== validate.sh steps: $STEPS (claude $(claude --version | head -1))"
for st in $STEPS; do case $st in
1) log "step 1: reference/F1.patch -> hidden 5/5, suite green, typecheck green"
static_run val1-reference "$HERE/reference/F1.patch" | tee -a "$LOG"
check runs/val1-reference 's["hidden_pass"]==5 and s["suite_green"] and s["typecheck_green"]' "step 1" | tee -a "$LOG" ;;
2) log "step 2: upstream unchanged -> hidden 0/5 (honest number recorded), suite green"
static_run val2-upstream "" | tee -a "$LOG"
check runs/val2-upstream 's["hidden_pass"]==0 and s["suite_green"] and s["suite_total"]==34' "step 2" | tee -a "$LOG" ;;
f2-1) log "F2 step 1: reference/F2.patch -> hidden 5/5, suite green, typecheck green"
static_run val1-f2-reference "$HERE/reference/F2.patch" F2 | tee -a "$LOG"
check runs/val1-f2-reference 's["hidden_pass"]==5 and s["suite_green"] and s["typecheck_green"]' "F2 step 1" | tee -a "$LOG" ;;
f2-2) log "F2 step 2: upstream unchanged -> hidden 0/5, suite green"
static_run val2-f2-upstream "" F2 | tee -a "$LOG"
check runs/val2-f2-upstream 's["hidden_pass"]==0 and s["suite_green"]' "F2 step 2" | tee -a "$LOG" ;;
esac; done
pids=""
for st in $STEPS; do case $st in
3) log "step 3: bare smoke (RUN_TAG=smoke) started"; ( RUN_TAG=smoke ./run.sh bare 1 > runs/.smoke-bare.out 2>&1 ) & pids="$pids $!" ;;
4) log "step 4: sp smoke (RUN_TAG=smoke) started"; ( RUN_TAG=smoke ./run.sh sp 1 > runs/.smoke-sp.out 2>&1 ) & pids="$pids $!" ;;
5) log "step 5: mp smoke (RUN_TAG=smoke) started"; ( RUN_TAG=smoke ./run.sh mp 1 > runs/.smoke-mp.out 2>&1 ) & pids="$pids $!" ;;
f2-3) log "F2 step 3: bare smoke on F2 (RUN_TAG=smoke): bare must score < 5"; ( TASK=F2 RUN_TAG=smoke ./run.sh bare 1 > runs/.smoke-f2-bare.out 2>&1 ) & pids="$pids $!" ;;
spd) log "spd smoke on F1 (RUN_TAG=smoke)"; ( RUN_TAG=smoke ./run.sh spd 1 > runs/.smoke-spd.out 2>&1 ) & pids="$pids $!" ;;
esac; done
[ -n "$pids" ] && wait $pids
for st in $STEPS; do case $st in
3) tail -1 runs/.smoke-bare.out | tee -a "$LOG"
check runs/bare-r1-smoke 's["end_reason"] in ("done","cap_driver_turns") and len(s["per_turn"])>=1 and all(t["po_kind"] for t in s["per_turn"])' "step 3a: bare ran end to end, the PO classified every turn" | tee -a "$LOG"
check runs/bare-r1-smoke 'not s["superpowers_injected"] and not s["skill_events"] and "superpowers" not in (s["init_plugins"] or [])' "step 3b: isolation: no Superpowers hook output, no Skill events, user plugin not loaded" | tee -a "$LOG" ;;
4) tail -1 runs/.smoke-sp.out | tee -a "$LOG"
check runs/sp-r1-smoke 's["superpowers_injected"] and any(h["event"]=="SessionStart" for h in s["hook_events"])' "step 4a: SessionStart hook fired and injected using-superpowers" | tee -a "$LOG"
check runs/sp-r1-smoke '"brainstorming" in [x.split(":")[-1] for x in s["skills_invoked"]]' "step 4b: brainstorming invoked (Skill event)" | tee -a "$LOG" ;;
5) tail -1 runs/.smoke-mp.out | tee -a "$LOG"
check runs/mp-r1-smoke 's["per_turn"][0]["stage"]=="/grill-with-docs" and {"grilling","domain-modeling"} <= set(s["per_turn"][0]["skills"])' "step 5: /grill-with-docs resolved (turn 1 loads grilling + domain-modeling, as its SKILL.md instructs)" | tee -a "$LOG" ;;
f2-3) tail -1 runs/.smoke-f2-bare.out | tee -a "$LOG"
check runs/f2-bare-r1-smoke 's["hidden_pass"] < 5' "F2 step 3: bare scores < 5 on F2" | tee -a "$LOG" ;;
spd) tail -1 runs/.smoke-spd.out | tee -a "$LOG"
check runs/spd-r1-smoke '{"/brainstorming","/writing-plans","/executing-plans"} <= set(s["spd_slash_stage_per_turn"] or [])' "spd: the three slash stages were all reached" | tee -a "$LOG" ;;
esac; done
log "=== validate.sh done"
दोनों packs अपने feature पर चलाने के लिए README का पालन कीजिए:
upstream/की जगह अपना repo रखिए,tasks/में अपनी request लिखिए, औरpo/में product owner की brief सरल शब्दों में लिखिए, कभी tests के रूप में नहीं।- एक hidden test file लिखिए जिसमें हर उस requirement के लिए एक test हो जिसकी एक senior उम्मीद करता है, साथ में एक reference patch, और oracle को
./validate.sh 1 2से जाँचिए। - Packs को वैसे install कीजिए जैसे
prep.shदिखाता है: Superpowers को.claude/skillsमें उसके SessionStart hook के साथ जो.claude/settings.jsonमें copy होता है, Pocock का packnpx skills addसे, फिर/setup-matt-pocock-skillsको एक बार materialise कीजिए। - एक cell
RUN_TAG=x ./run.sh <cfg> <rep>से चलाइए। Tag वाले runs analysis में अनदेखे रहते हैं, इसलिए smoke runs के लिए tag इस्तेमाल कीजिए। - Matrix को
nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 &से चलाइए,SKIP_DONE=1से दोबारा शुरू कीजिए, फिर प्रति config और प्रति stage tables पाने के लिएpython3 analyze.pyचलाइए।
README के तीन जाल शुरू करने से पहले जानने लायक हैं। Work copy को ऐसे directory tree के बाहर चलाइए जिसमें CLAUDE.md हो, क्योंकि Claude Code हर parent directory की CLAUDE.md load करता है और हमारा पहला round इसी वजह से फेंकना पड़ा। Subscription login पर --bare कभी इस्तेमाल मत कीजिए। और Superpowers का hook सिर्फ़ पहले turn पर चलता है, इसलिए headless run एक ही बार route होता है; routing खुद हर run में बदलती रही, इसलिए गिनिए कि उसने कौन सी skill चुनी।
सोमवार को यह कीजिए
- देखिए कि आपके पास Pocock का कौन सा version है। अगर आपने Claude plugin install किया है तो वह 1.2.3 पर pinned है और उसमें implement-spec, pr या retro नहीं हैं; v1.3.1 चाहिए तो installer से दोबारा install कीजिए।
- अगर आप Superpowers चलाते हैं, तो executing-plans की reviewer instruction खोलिए और देखिए वह कौन सा model माँगती है। सस्ता model pin कीजिए, फिर नापिए।
- जब request धुंधली हो तब /brainstorming खुद type कीजिए। अपने hook के भरोसे यहाँ brainstorming हर hook-routed 3 में से 3 runs में चली, पर routing की कोई गारंटी नहीं।
- अपनी अगली धुंधली request पर grill-with-docs, to-spec और to-tickets आज़माइए और पढ़िए कि आपके repo पर उनकी लागत क्या है।
- बिना test के दोनों packs को routers की तरह stack मत कीजिए। हमारे एक stacked run में कुछ नहीं टूटा, पर वह एक ही run है।
- Bench को पिछले महीने भेजे किसी feature पर, अपने hidden tests के साथ copy कीजिए, और bare, Superpowers और Main Flow को एक-एक बार चलाइए।
- कुल नहीं, per-stage table पढ़िए। कुछ और बदलने से पहले वह एक stage खोजिए जो आपके बिल का ज़्यादातर हिस्सा है।
- अगर v1.3 अपनाएँ तो CONTEXT.md का नाम GLOSSARY.md कर दीजिए, क्योंकि skills सिर्फ़ नए नाम ढूँढती हैं।
Measurements
Source:
results.md
# flow-bench results
Untagged runs only (smoke/validation excluded). Tokens = input + output + cache read + cache write, all models, subagents included. USD = list-price equivalent from `total_cost_usd` deltas (plan quota, not a bill). Wall = sum of worker turn walls (PO time excluded). Diff = lines added+removed vs the base commit, src/ + test/.
## Per config (mean / median)
| task / config | n | hidden /5 (per run) | suite green | tsc green | code diff +/- | all files diff | tests added | tokens | USD | wall s | driver turns | questions | caps hit |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| F1 / bare | 3 | 5 / 5 (5,5,5) | 3/3 | 3/3 | 141.667 / 137 | 142.333 | 4.667 | 519,114 / 361,842 | 0.343 / 0.286 | 84.5 / 71.7 | 1.333 | 0.333 | none |
| F1 / sp (hook-routed) | 3 | 5 / 5 (5,5,5) | 3/3 | 3/3 | 235 / 227 | 237 | 12.333 | 724,601 / 765,108 | 0.502 / 0.525 | 118.1 / 119.6 | 2.667 | 1.667 | none |
| F1 / spd (Superpowers driven) | 3 | 5 / 5 (5,5,5) | 3/3 | 3/3 | 386 / 371 | 582 | 27.667 | 4,479,876 / 3,887,015 | 3.738 / 3.412 | 607.367 / 600.8 | 7 | 4 | none |
| F1 / mp (Main Flow) | 3 | 5 / 5 (5,5,5) | 3/3 | 3/3 | 372 / 369 | 578 | 27.667 | 4,476,104 / 4,286,993 | 2.043 / 2.102 | 439.033 / 472.4 | 12 | 7 | mp-r1: driver_turns_12; mp-r3: driver_turns_12 |
| F1 / both (sp+mp, mp flow) | 1 | 5 / 5 (5) | 1/1 | 1/1 | 374 / 374 | 561 | 22 | 5,754,354 / 5,754,354 | 2.326 / 2.326 | 448.2 / 448.2 | 12 | 6 | none |
| F2 / bare | 2 | 4.5 / 4.5 (4,5) | 2/2 | 2/2 | 206.5 / 206.5 | 209.5 | 5 | 651,322 / 651,322 | 0.413 / 0.413 | 103.15 / 103.15 | 2 | 0.5 | none |
| F2 / spd (Superpowers driven) | 2 | 5 / 5.0 (5,5) | 2/2 | 2/2 | 488.5 / 488.5 | 750.5 | 33 | 4,490,365 / 4,490,365 | 3.213 / 3.213 | 511.7 / 511.7 | 6.5 | 3.5 | none |
| F2 / mp (Main Flow) | 2 | 5 / 5.0 (5,5) | 2/2 | 2/2 | 469 / 469.0 | 658 | 31 | 4,520,638 / 4,520,638 | 2.067 / 2.067 | 421.2 / 421.2 | 12 | 6.5 | f2-mp-r1: driver_turns_12 |
Requirement pass counts (runs passing each hidden req):
- **F1 bare**: req1 3/3, req2 3/3, req3 3/3, req4 3/3, req5 3/3
- **F1 sp (hook-routed)**: req1 3/3, req2 3/3, req3 3/3, req4 3/3, req5 3/3
- **F1 spd (Superpowers driven)**: req1 3/3, req2 3/3, req3 3/3, req4 3/3, req5 3/3
- **F1 mp (Main Flow)**: req1 3/3, req2 3/3, req3 3/3, req4 3/3, req5 3/3
- **F1 both (sp+mp, mp flow)**: req1 1/1, req2 1/1, req3 1/1, req4 1/1, req5 1/1
- **F2 bare**: req1 2/2, req2 2/2, req3 1/2, req4 2/2, req5 2/2
- **F2 spd (Superpowers driven)**: req1 2/2, req2 2/2, req3 2/2, req4 2/2, req5 2/2
- **F2 mp (Main Flow)**: req1 2/2, req2 2/2, req3 2/2, req4 2/2, req5 2/2
## F1 spd (Superpowers driven) per slash-command stage (turn level, exact) (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| /brainstorming | 3 | 182,022 | 160,834 | - | 0.166 | 0.156 | 40.283 | 35.44 |
| /writing-plans | 3 | 321,355 | 298,267 | - | 0.266 | 0.274 | 88.133 | 95.82 |
| /executing-plans | 3 | 3,976,500 | 3,427,914 | - | 3.307 | 3.028 | 478.95 | 453.89 |
## F1 spd per skill in force (within-turn split) (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| brainstorming | 3 | 182,022 | 160,834 | 4,161 | 0.166 | 0.156 | 38.8 | 33.6 |
| writing-plans | 3 | 321,355 | 298,267 | 11,827 | 0.266 | 0.274 | 86.733 | 95.1 |
| executing-plans | 3 | 66,938 | 68,739 | 505 | 0.047 | 0.047 | 4.6 | 4.6 |
| test-driven-development | 3 | 2,927,098 | 2,854,967 | 22,508 | 0.974 | 0.936 | 457.033 | 448.9 |
| finishing-a-development-branch | 1 | 976,505 | 976,505 | 5,297 | 0.282 | 0.282 | 50.7 | 50.7 |
| test-driven-development [subagent] | 3 | 656,962 | 565,811 | 20,918 | 2.192 | 2.047 | 0.0 | 0.0 |
## F1 sp (hook-routed) per phase (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| before any skill | 3 | 19,549 | 19,538 | 238 | 0.03 | 0.03 | 5.333 | 5.4 |
| brainstorming | 3 | 249,260 | 247,900 | 8,037 | 0.219 | 0.244 | 40.533 | 40.1 |
| implementation (no skill) | 2 | 386,028 | 386,028 | 5,080 | 0.176 | 0.176 | 60.3 | 60.3 |
| execution | 1 | 595,318 | 595,318 | 15,622 | 0.405 | 0.405 | 93.0 | 93.0 |
## F1 sp (hook-routed) per skill in force (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| (none) | 3 | 19,549 | 19,538 | 238 | 0.03 | 0.03 | 5.333 | 5.4 |
| brainstorming | 3 | 249,260 | 247,900 | 8,037 | 0.219 | 0.244 | 40.533 | 40.1 |
| test-driven-development | 1 | 595,318 | 595,318 | 15,622 | 0.405 | 0.405 | 93.0 | 93.0 |
| implementation (no skill) | 2 | 386,028 | 386,028 | 5,080 | 0.176 | 0.176 | 60.3 | 60.3 |
## F1 mp per slash-command stage (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| /grill-with-docs | 3 | 316,507 | 348,862 | 5,999 | 0.204 | 0.215 | 59.933 | 59.9 |
| /to-spec | 3 | 163,344 | 167,578 | 5,135 | 0.118 | 0.113 | 42.1 | 40.8 |
| /to-tickets | 3 | 257,955 | 268,062 | 5,867 | 0.143 | 0.151 | 44.4 | 47.8 |
| /implement | 3 | 2,655,565 | 2,613,062 | 30,215 | 1.125 | 1.195 | 224.567 | 245.4 |
| /code-review | 3 | 1,047,586 | 1,107,555 | 9,294 | 0.439 | 0.445 | 64.2 | 65.8 |
| /pr | 1 | 105,440 | 105,440 | 1,349 | 0.041 | 0.041 | 11.6 | 11.6 |
## F1 both per slash-command stage (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| /grill-with-docs | 1 | 344,391 | 344,391 | 6,193 | 0.225 | 0.225 | 61.8 | 61.8 |
| /to-spec | 1 | 217,572 | 217,572 | 4,835 | 0.123 | 0.123 | 42.9 | 42.9 |
| /to-tickets | 1 | 382,953 | 382,953 | 7,115 | 0.196 | 0.196 | 49.4 | 49.4 |
| /implement | 1 | 3,070,579 | 3,070,579 | 24,618 | 1.112 | 1.112 | 194.2 | 194.2 |
| /code-review | 1 | 1,615,265 | 1,615,265 | 13,410 | 0.625 | 0.625 | 86.8 | 86.8 |
| /pr | 1 | 123,594 | 123,594 | 1,461 | 0.045 | 0.045 | 13.1 | 13.1 |
## F2 spd (Superpowers driven) per slash-command stage (turn level, exact) (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| /brainstorming | 2 | 485,364 | 485,364 | - | 0.354 | 0.354 | 96.815 | 96.815 |
| /writing-plans | 2 | 166,012 | 166,012 | - | 0.159 | 0.159 | 49.755 | 49.755 |
| /executing-plans | 2 | 3,838,988 | 3,838,988 | - | 2.7 | 2.7 | 365.1 | 365.1 |
## F2 spd per skill in force (within-turn split) (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| brainstorming | 2 | 253,890 | 253,890 | 6,038 | 0.207 | 0.207 | 53.05 | 53.05 |
| writing-plans | 2 | 397,486 | 397,486 | 14,060 | 0.307 | 0.307 | 90.75 | 90.75 |
| executing-plans | 2 | 69,973 | 69,973 | 286 | 0.046 | 0.046 | 3.5 | 3.5 |
| test-driven-development | 2 | 3,304,676 | 3,304,676 | 23,244 | 1.052 | 1.052 | 361.25 | 361.25 |
| test-driven-development [subagent] | 2 | 464,339 | 464,339 | 13,017 | 1.602 | 1.602 | 0.0 | 0.0 |
## F2 mp per slash-command stage (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| /grill-with-docs | 2 | 232,965 | 232,965 | 5,946 | 0.182 | 0.182 | 56.7 | 56.7 |
| /to-spec | 2 | 180,084 | 180,084 | 4,452 | 0.115 | 0.115 | 37.6 | 37.6 |
| /to-tickets | 2 | 253,688 | 253,688 | 5,152 | 0.131 | 0.131 | 38.5 | 38.5 |
| /implement | 2 | 2,689,071 | 2,689,071 | 31,452 | 1.159 | 1.159 | 218.35 | 218.35 |
| /code-review | 2 | 1,107,122 | 1,107,122 | 9,454 | 0.458 | 0.458 | 63.85 | 63.85 |
| /pr | 1 | 115,414 | 115,414 | 1,372 | 0.043 | 0.043 | 12.4 | 12.4 |
## Per run
| run | task | hidden | reqs failed | suite | tsc | code +/- | tests added | tokens | USD | wall s | turns | end | skills (first use order) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bare-r1 | F1 | 5/5 | - | green | ok | +149/-11 | 6 | 840,751 | 0.47 | 119 | 2 | done | - |
| bare-r2 | F1 | 5/5 | - | green | ok | +118/-10 | 4 | 354,748 | 0.27 | 63 | 1 | done | - |
| bare-r3 | F1 | 5/5 | - | green | ok | +125/-12 | 4 | 361,842 | 0.29 | 72 | 1 | done | - |
| sp-r1 | F1 | 5/5 | - | green | ok | +182/-15 | 10 | 613,434 | 0.44 | 99 | 3 | done | brainstorming |
| sp-r2 | F1 | 5/5 | - | green | ok | +212/-15 | 13 | 765,108 | 0.54 | 120 | 2 | done | brainstorming > test-driven-development |
| sp-r3 | F1 | 5/5 | - | green | ok | +267/-14 | 14 | 795,260 | 0.52 | 135 | 3 | done | brainstorming |
| spd-r1 | F1 | 5/5 | - | green | ok | +350/-21 | 21 | 3,667,286 | 3.41 | 601 | 6 | done | test-driven-development |
| spd-r2 | F1 | 5/5 | - | green | ok | +317/-16 | 36 | 3,887,015 | 3.39 | 520 | 7 | done | test-driven-development |
| spd-r3 | F1 | 5/5 | - | green | ok | +434/-20 | 26 | 5,885,328 | 4.41 | 701 | 8 | done | test-driven-development > finishing-a-development-branch |
| mp-r1 | F1 | 5/5 | - | green | ok | +349/-20 | 29 | 5,098,503 | 2.22 | 472 | 12 | cap_driver_turns (driver_turns_12) | grilling > domain-modeling > tdd > code-review |
| mp-r2 | F1 | 5/5 | - | green | ok | +314/-15 | 25 | 4,042,815 | 1.80 | 365 | 12 | done | grilling > domain-modeling > tdd > code-review |
| mp-r3 | F1 | 5/5 | - | green | ok | +395/-23 | 29 | 4,286,993 | 2.10 | 480 | 12 | cap_driver_turns (driver_turns_12) | grilling > domain-modeling > tdd > code-review |
| both-r1 | F1 | 5/5 | - | green | ok | +356/-18 | 22 | 5,754,354 | 2.33 | 448 | 12 | done | grilling > domain-modeling > tdd > code-review |
| f2-bare-r1 | F2 | 4/5 | req3 | green | ok | +204/-10 | 5 | 494,231 | 0.38 | 92 | 1 | done | - |
| f2-bare-r2 | F2 | 5/5 | - | green | ok | +189/-10 | 5 | 808,413 | 0.45 | 115 | 3 | done | - |
| f2-spd-r1 | F2 | 5/5 | - | green | ok | +546/-14 | 46 | 5,310,616 | 3.42 | 506 | 6 | done | writing-plans > test-driven-development |
| f2-spd-r2 | F2 | 5/5 | - | green | ok | +401/-16 | 20 | 3,670,114 | 3.00 | 517 | 7 | done | test-driven-development |
| f2-mp-r1 | F2 | 5/5 | - | green | ok | +460/-23 | 31 | 4,823,516 | 2.18 | 443 | 12 | cap_driver_turns (driver_turns_12) | grilling > domain-modeling > tdd > code-review |
| f2-mp-r2 | F2 | 5/5 | - | green | ok | +440/-15 | 31 | 4,217,761 | 1.95 | 399 | 12 | done | grilling > domain-modeling > tdd > code-review |
| Task और configuration | Runs | Hidden /5 (प्रति run) | औसत USD | औसत wall s | औसत tokens | औसत code diff +/- |
|---|---|---|---|---|---|---|
| F1 bare | 3 | 5 (5,5,5) | 0.343 | 84.5 | 519,114 | 141.667 |
| F1 sp (hook-routed) | 3 | 5 (5,5,5) | 0.502 | 118.1 | 724,601 | 235 |
| F1 spd (Superpowers driven) | 3 | 5 (5,5,5) | 3.738 | 607.367 | 4,479,876 | 386 |
| F1 mp (Main Flow) | 3 | 5 (5,5,5) | 2.043 | 439.033 | 4,476,104 | 372 |
| F1 both (sp+mp, mp flow) | 1 | 5 (5) | 2.326 | 448.2 | 5,754,354 | 374 |
| F2 bare | 2 | 4.5 (4,5) | 0.413 | 103.15 | 651,322 | 206.5 |
| F2 spd (Superpowers driven) | 2 | 5 (5,5) | 3.213 | 511.7 | 4,490,365 | 488.5 |
| F2 mp (Main Flow) | 2 | 5 (5,5) | 2.067 | 421.2 | 4,520,638 | 469 |
Superpowers driven, F1, प्रति stage (3 runs का औसत):
| Stage | Tokens | USD | Wall s |
|---|---|---|---|
| brainstorming | 182,022 | 0.166 | 40.283 |
| writing-plans | 321,355 | 0.266 | 88.133 |
| executing-plans | 3,976,500 | 3.307 | 478.95 |
| जिसमें से reviewer subagent | 656,962 | 2.192 | n/a |
Reviewer वाली पंक्ति executing-plans के भीतर subagent का शेष हिस्सा है, जिसे results file में "test-driven-development [subagent]" नाम दिया गया है। executing-plans के भीतर session का build (test-driven-development) औसतन $0.974 रहा।
Main Flow, F1, प्रति stage (उन runs का औसत जिनमें वह stage था; /pr 3 में से 1 में चला):
| Stage | Tokens | USD | Wall s |
|---|---|---|---|
| grill-with-docs | 316,507 | 0.204 | 59.933 |
| to-spec | 163,344 | 0.118 | 42.1 |
| to-tickets | 257,955 | 0.143 | 44.4 |
| implement | 2,655,565 | 1.125 | 224.567 |
| code-review | 1,047,586 | 0.439 | 64.2 |
| pr | 105,440 | 0.041 | 11.6 |
Hook-routed Superpowers, F1: किसी skill से पहले $0.03, brainstorming $0.219, बिना skill के implementation $0.176 (2 runs), test-driven-development के साथ execution $0.405 (1 run)। Brainstorming सभी 3 runs में चली और writing-plans किसी में नहीं। F1 के औसत पर चलाया गया loop hook-routed वाले से 7.4 गुना महंगा रहा ($3.738 के मुकाबले $0.502)। तीन Main Flow runs (mp-r1, mp-r3 और f2-mp-r1) /pr से पहले 12 driver-turn की सीमा पर पहुँच गए और ऐसे ही दर्ज हैं।
FINDINGS.md
# flow-bench findings (2026-10-09, round 2: isolated work copies, Sonnet 5.5 headless, 19 counted runs)
**F1 (partial payments): quality did not separate.** All 13 F1 runs hidden 5/5, suite and tsc green.
Cost per run: bare $0.27 (bare-r2) to $0.47 (bare-r1); sp, hook-routed, $0.44 (sp-r1) to $0.54 (sp-r2); mp $1.80
(mp-r2) to $2.22 (mp-r1); spd, Superpowers driven, $3.39 (spd-r2) to $4.41 (spd-r3).
Wall: mp 365 to 480 s (mp-r2, mp-r3), spd 520 to 701 s (spd-r2, spd-r3).
**Where the money goes.** mp-r2 (complete flow): grill $0.16, spec $0.11, tickets $0.09, implement $0.96, code-review
$0.45, pr $0.04 (F1 means: implement $1.13, code-review $0.44). spd: brainstorming $0.17, writing-plans $0.27,
executing-plans $3.31 (F1 means). Inside executing-plans the build itself (TDD in session) costs $0.97 on average; the
rest is the one whole-branch reviewer the skill dispatches "on the most capable model", which ran on claude-fable-5-1
in all five spd runs: $2.01 of spd-r1's $3.41, $2.52 of spd-r3's $4.41, $1.57 of f2-spd-r2's $3.00.
**Hook vs driven.** Left to its hook, Superpowers ran brainstorming in sp-r1, sp-r2 and sp-r3 and then coded; writing-
plans ran 0 times in 3. Driven by slash commands, the full loop ran in every spd run and cost 7.4 times more (F1 means: spd $3.74, sp $0.50). The cost is the final review, not the planning.
**F2 (credit notes) separated, barely.** f2-bare-r1 shipped one missing requirement, R3: credit-note numbers are not
gapless. It took the number before validating, like the existing `/issue` route, so rejected requests burned numbers
(CN-0004 where CN-0002 was due; the F2 bare smoke failed the same way). f2-bare-r2, f2-spd-r1, f2-spd-r2, f2-mp-r1 and
f2-mp-r2 scored 5/5. Costs: bare $0.38/$0.45, mp $2.18/$1.95, spd $3.42/$3.00 (r1/r2). Caveat: the first F2 oracle
also required credit notes on open invoices; both bare runs allowed paid invoices only, which the request's wording
permits, so that check was removed and the F2 runs rescored.
**Stacking broke nothing.** both-r1: hook injected, no Superpowers skill invoked, $2.33, inside the mp range.
**Caps.** mp-r1, mp-r3 and f2-mp-r1 hit the 12-driver-turn cap before /pr. spd never needed more than 8 turns.
**Caveats.** Round-1 runs read this repo's CLAUDE.md (parent-directory loading), so they were discarded and rerun outside the repo. A simulated PO answers each round in one message. The SessionStart hook fires on turn 1 only.
/implement covers every ticket in one turn. 5 to 11 compound shell lines per spd run were denied. PO cost is excluded.
और आगे
- Debugging bench। r/claudeskills की एक post ने एक repo के इतिहास से 15 bugs चलाए, हर एक तीन बार। तीनों variants ने 15 में से 9 ठीक किए, अतिरिक्त skills ने हर bug में करीब 3 मिनट जोड़े, और Pocock की debugging skill ने सबसे छोटे बदलाव किए, 41 और 48 के मुकाबले median 32 पंक्तियाँ, बिना कोई दूसरा test तोड़े s8। यह सिर्फ़ debugging skills परखता है, पूरे flows नहीं, और post यह नहीं बताती कि 41 और 48 में से कौन सा किस variant का है। इसे bench harness से अपने bug इतिहास पर दोबारा चलाइए।
- Switcher blogs। यह थीसिस पढ़िए कि orchestration कर बन गई s9 और वह migration कहानी जिसने Superpowers के hooks हटाए और grill-with-docs रखी s10, फिर उनके दावों को ऊपर की per-stage table से मिलाइए।
- दूसरे packs। Pstack और Osmani का pack दो और skill sets हैं जो हमने नहीं चलाए। वही harness
.claude/skillsमें install होने वाले किसी भी pack को लेता है;configs.json
{
"bare": {
"superpowers": false,
"mattpocock": false,
"flow": "bare",
"desc": "no skills, no hooks (project settings carry only the permission deny-list)"
},
"sp": {
"superpowers": true,
"mattpocock": false,
"flow": "sp",
"desc": "HOOK-ROUTED: obra/superpowers v6.4.2: skills in .claude/skills, SessionStart hook in .claude/settings.json as hooks/hooks.json wires it; turn 1 = the bare task, the hook decides"
},
"mp": {
"superpowers": false,
"mattpocock": true,
"flow": "mp",
"desc": "mattpocock/skills v1.3.1 via npx skills add (27 promoted skills, copied), setup-matt-pocock-skills output committed (local tracker)"
},
"both": {
"superpowers": true,
"mattpocock": true,
"flow": "mp",
"desc": "sp + mp in one project; driven with the mp slash-command flow (the stacking question: does the Superpowers hook hijack it?)"
},
"spd": {
"superpowers": true,
"mattpocock": false,
"flow": "spd",
"desc": "Superpowers, DRIVEN: same project install as sp (hook included); the driver types /brainstorming <task> -> /writing-plans -> /executing-plans (Native/inline); the pack's own scripts allowed in Bash"
}
}
prep.sh
#!/usr/bin/env bash
# Fresh work copy of upstream (node_modules included) + the pack fixtures of <cfg>.
# Usage: prep.sh <cfg> <work dir> Prints the base commit (the oracle's BASE) on its last line.
set -euo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"
CFG="$1"; W="$2"
SP=$(python3 -c "import json;print(int(json.load(open('$HERE/configs.json'))['$CFG']['superpowers']))")
MP=$(python3 -c "import json;print(int(json.load(open('$HERE/configs.json'))['$CFG']['mattpocock']))")
rm -rf "$W"; mkdir -p "$(dirname "$W")"
cp -R "$HERE/upstream" "$W"
cd "$W"
# .claude/ never shows in git status or in any diff (hook config, installed skills).
echo ".claude/" >> .git/info/exclude
mkdir -p .claude/skills
if [ "$MP" = 1 ]; then
cp -R "$HERE/packs/mattpocock-skills/.claude/skills/." .claude/skills/
# One-time per-repo precondition of the pack (/setup-matt-pocock-skills), answered once with the defaults it
# recommends for a repo with no remote: local-markdown tracker, single-context domain docs, no triage labels
# (triage is installed, so Section B would ask; default labels kept). Committed so it never shows in the diff.
S="$HERE/packs/mattpocock-skills/.claude/skills/setup-matt-pocock-skills"
mkdir -p docs/agents
cp "$S/issue-tracker-local.md" docs/agents/issue-tracker.md
cp "$S/domain.md" docs/agents/domain.md
cp "$S/triage-labels.md" docs/agents/triage-labels.md
cat > CLAUDE.md <<'MD'
## Agent skills
### Issue tracker
Issues and specs live as local markdown files under `.scratch/`. See `docs/agents/issue-tracker.md`.
### Triage labels
Default labels (`needs-triage`, `needs-info`, `ready-for-agent`, `ready-for-human`, `wontfix`). See `docs/agents/triage-labels.md`.
### Domain docs
Single-context: one `GLOSSARY.md` + `docs/adr/` at the repo root. See `docs/agents/domain.md`.
MD
git add CLAUDE.md docs/agents
GIT_AUTHOR_DATE="2026-03-09T18:00:00" GIT_COMMITTER_DATE="2026-03-09T18:00:00" \
git -c user.name="Dana Whitfield" -c user.email="dana@ledgerly.dev" commit -q -m "chore: configure agent skills (local tracker)"
fi
if [ "$SP" = 1 ]; then
# The plugin root as Claude Code would hold it, so the hook script finds ../skills/using-superpowers.
mkdir -p .claude/plugins
cp -R "$HERE/packs/superpowers" .claude/plugins/superpowers
cp -R "$HERE/packs/superpowers/skills/." .claude/skills/
fi
python3 - "$HERE" "$SP" "$CFG" > .claude/settings.json <<'PY'
import json, sys
bench, sp, cfg = sys.argv[1], sys.argv[2] == "1", sys.argv[3]
# Oracle, reference, PO brief, other runs: denied by path (Read/Grep/Glob) and by command text (Bash).
deny = ["Read(/%s/hidden/**)" % bench, "Read(/%s/reference/**)" % bench, "Read(/%s/runs/**)" % bench,
"Read(/%s/po/**)" % bench, "Read(/%s/validation/**)" % bench,
"Bash(*hidden*)", "Bash(*reference/F*)", "Bash(*-brief.md*)", "Bash(*/runs/*)", "Bash(*../*)", "Bash(find *)"]
if cfg == "spd":
# executing-plans runs its sibling scripts as ../subagent-driven-development/scripts/...: a "../" deny would block the
# pack's own loop. The oracle stays protected by the Read denies and Bash(*hidden*).
deny.remove("Bash(*../*)")
settings = {"permissions": {"deny": deny}}
if sp:
# = packs/superpowers/hooks/hooks.json, with ${CLAUDE_PLUGIN_ROOT} provided the way the plugin loader provides it
# (the script also picks its JSON output shape from CLAUDE_PLUGIN_ROOT being set).
root = '$CLAUDE_PROJECT_DIR/.claude/plugins/superpowers'
hooks = json.load(open("%s/packs/superpowers/hooks/hooks.json" % bench))["hooks"]
for groups in hooks.values():
for g in groups:
for h in g["hooks"]:
h["command"] = 'CLAUDE_PLUGIN_ROOT="%s" %s' % (root, h["command"].replace("${CLAUDE_PLUGIN_ROOT}", root))
settings["hooks"] = hooks
print(json.dumps(settings, indent=2))
PY
[ "$(ls .claude/skills | wc -l)" -eq 0 ] && rmdir .claude/skills
test -z "$(git status --porcelain)" || { echo "work copy not clean"; git status --porcelain; exit 1; }
git rev-parse HEAD
FAQ
क्या किसी pack ने बेहतर code भेजा?
Feature 1 पर नहीं: सभी 13 runs ने 5 में से 5 पाया, suite और typecheck हरे। Feature 2 पर एक bare run एक requirement चूका, बिना gap वाला credit note number, क्योंकि उसने request validate करने से पहले number ले लिया, ठीक वैसे जैसे मौजूदा issue route करता है। दोनों loops ने code से पहले sequence का test लिखा। दूसरे bare run ने इसे सही किया।
यहाँ Superpowers loop Main Flow से इतना महंगा क्यों है?
एक stage। executing-plans का औसत $3.307 रहा और उसके भीतर एक reviewer subagent $2.192 का था। Reviewer पाँचों driven runs में claude-fable-5-1 पर चला, और skill सबसे सक्षम model माँगती है। Planning stages $0.166 और $0.266 के रहे। हमने सस्ते reviewer के साथ दोबारा नहीं चलाया, इसलिए उस बदलाव का हमारे पास कोई मापा हुआ आँकड़ा नहीं।
क्या दोनों packs install करने से कुछ टूटता है?
हमारे एक stacked run में नहीं: hook inject हुआ, कोई Superpowers skill invoke नहीं हुई, slash commands जीते, और run 5 में से 5 पर $2.326 का रहा। Pocock की user-invoked skills में एक flag है जो model को उन्हें खुद load करने से रोकता है, और दोनों packs कोई skill directory नाम साझा नहीं करते। यह एक ही run है, इसलिए अपने setup पर आज़माइए।
Pocock की skills के लिए Claude plugin ही क्यों न इस्तेमाल करें?
क्योंकि marketplace उसे 1.2.3 पर pin रखता है जबकि upstream v1.3.1 पर है s32। Installer वही release खींचता है जो आप माँगते हैं।
क्या यह Main Flow की quality की निष्पक्ष परीक्षा है?
यह इस बात की निष्पक्ष परीक्षा है कि हर pack की लागत क्या है और क्या वह छोटे repo पर पाँचों requirements भेजता है। Main Flow के artefacts, spec और tickets, यहाँ score नहीं हुए, और बड़ा feature या कोई team उन्हें ज़्यादा महत्व दे सकती है।