AIDiveTL;DR i werdykt
- Ta sama funkcja zbudowana na jednym wymyślonym repo przez czystą sesję Claude Code, przez Superpowers i przez Main Flow Matta Pococka, headless Sonnet 5.5, 19 zliczonych przebiegów. Tokeny, czas i koszt według cennika są odczytane na każdym etapie, a nie na przebieg.
- Mapa etapów: zostaw Superpowers kierowany hookiem. Pożycz przednią połowę Pococka, grill-with-docs, to-spec, to-tickets, gdy zlecenie jest mgliste. Zmień linię modelu w recenzencie executing-plans, bo ten jeden etap to prawie cała różnica między paczkami.
- Funkcja 1 (płatności częściowe), średnie z 3 przebiegów na konfigurację: bare $0.343 i 84.5 s, Superpowers zostawiony swojemu hookowi $0.502 i 118.1 s, Superpowers prowadzony przez pełną pętlę $3.738 i 607.367 s, Main Flow $2.043 i 439.033 s. Jeden przebieg z obiema paczkami naraz: $2.326. Każdy z tych przebiegów przeszedł 5 z 5 ukrytych testów.
- W prowadzonej pętli Superpowers executing-plans kosztował $3.307 z całych $3.738, a $2.192 z tego to jeden subagent-recenzent, którego skill wysyła do najmocniejszego modelu. Brainstorming kosztował $0.166, writing-plans $0.266.
- Funkcja 2 (noty kredytowe), trudniejsza: bare $0.413 z jednym potknięciem w dwóch przebiegach, Superpowers prowadzony $3.213, Main Flow $2.067, oba ostatnie na 5 z 5. Jedyne niespełnione wymaganie w 19 przebiegach to ciągły numer noty kredytowej, wypuszczony przez przebieg bare.
- Ograniczenia: jedno wymyślone repo, jeden model workera, dwie funkcje, skryptowany product owner, małe n w komórce. Jakość nie rozdzieliła tu paczek, rozdzielił je koszt.
Co mówią źródła
Wersje w momencie przebiegu: Superpowers jest w v6.4.2, wydanym 2026-09-25 s20, a mattpocock/skills w v1.3.1, wydanym 2026-10-04 s26. Superpowers dostarcza 15 skilli, repo Pococka wymienia 27 dostarczanych skilli s1 s3. Starsze zestawienie 4 wspólnych, 10 tylko Superpowers i 12 tylko Pococka wykonano na Superpowers 6.0.3 wobec Pococka 1.0.1, przy 14 wobec 17 skilli, więc jest nieaktualne s6.
Obie paczki to przeciwne zakłady. Superpowers okablowuje hook SessionStart, który wstrzykuje skill using-superpowers, owinięty w tag EXTREMELY_IMPORTANT, przy startup, clear i compact s23. Jego tekst: "If you think there is even a 1% chance a skill might apply to what you are doing, you ABSOLUTELY MUST invoke the skill." s22 Mówi też, by wywoływać odpowiednie skille przed jakąkolwiek odpowiedzią lub działaniem, łącznie z pytaniami doprecyzowującymi s22. Paczka Pococka nie ma hooka sesji: "There is no session hook; nothing gets injected at startup." s6 Drzewo repo też nie zawiera pliku hooków s3.
Pliki skilli są krótkie. grill-with-docs ma 35 słów i robi tylko dwa wywołania narzędzia Skill, dla grilling i domain-modeling s27. implement ma 103 słowa, niesie disable-model-invocation i kończy się wywołaniem code-review s28. Liczba 63%, którą ogłosił Matt, to redukcja kosztu tokenów opisów skilli, a nie przebiegu s34 s35.
Main Flow, tak jak rozpisuje go skill routera, przechodzi przez krok pull requesta i kończy retro, które domyka pętlę s29. Strona przeglądowa wymienia 6 skilli w Main Flow i trzyma /pr wśród skilli referencyjnych s1. We własnych notatkach do v1.3 Matt pisze: "The agent loop is worse than a deterministic loop. But it is a good way to start with AFK work" s2. Wymienia też szorstkie krawędzie: "worktrees do not remove collisions, they move them to merge time" s2. v1.3 zmienia nazwę CONTEXT.md na GLOSSARY.md i CONTEXT-MAP.md na GLOSSARY-MAP.md s2 s26.
Ścieżki instalacji. README podaje komendę pluginu i komendę installera, claude plugin install mattpocock-skills@claude-plugins-official oraz npx skills@latest add mattpocock/skills -a <agent>, i mówi, by wybrać jedną na agenta, bo instalacja obu daje każdy skill dwa razy s3. Kopia z marketplace się spóźnia: oficjalny marketplace pluginów Claude przypina mattpocock-skills na 1.2.3, a prośba o podbicie jest nadal otwarta s32. W naszym sprawdzeniu upstream main był 71 commitów przed tym pinem, a przypięta kopia nie ma implement-spec, pr ani retro s31 s32. Zgłoszenie użytkownika o tym samym spóźnieniu jest otwarte w repo Pococka s31, a 2026-10-08 Matt poprosił na X o podbicie wersji z cache: "All my users are stuck on v1.2." s33 W tym samym sprawdzeniu marketplace przypinał Superpowers na 6.4.1, o jeden patch za v6.4.2 s20.
Po stronie Superpowers: 6.4.1 przebudowało executing-plans z natywnym trybem inline, "the cheapest way to run a plan" s21, a 6.4.2 odchudziło writing-plans s20. Otwarte issues, które ludzie przytaczają: spowolnienie po instalacji skilla s24 oraz przebieg Codex z subagentami, który zjadł cały 5-godzinny budżet, zgłoszony na Superpowers 5.0.7 i Codex, a nie na wydaniu 6.4 s25. Issue Pococka o koszcie tokenów /implement to dyskusja użytkowników, zamknięta przez Matta jako dyskusja, a nie awaria skilla s30.
Co ludzie mówią bez mierzenia. Jedna osoba po przesiadce stwierdziła, że Q&A Pococka zużywało mniej tokenów niż brainstorming Superpowers "with no worse results" s10. Inna pisze, że obowiązkowa orkiestracja przestaje być siatką bezpieczeństwa i zaczyna być podatkiem s9. Wątek na Hacker News prosi o benchmarki wobec niestosowania Superpowers w ogóle, obok obrony, że jest budowany z evalami s13. Wątek o grillingu opisuje "5 hours wayfinding and spec reading" przed krótką implementacją, a jeden komentujący uważa, że grilling z nowszymi modelami "feels like a hinderance" s7. Post o łączeniu dwóch paczek jako routerów twierdzi, że ich komendy slash i hooki sesji kolidują s11. Żaden z nich nie podaje liczb tokenów na etap, a tę lukę wypełnia ten bench.
Bench
Bench to mały wymyślony API do fakturowania w TypeScript o nazwie ledgerly: warstwa domeny, plikowy magazyn JSON z migracjami schematu, warstwa HTTP, 34 przechodzące testy, 4 commity. Każde zlecenie funkcji ma trzy linie, napisane tak, jak pisze deweloper. Senior oczekiwałby pięciu rzeczy, których zlecenie nie wymienia, i te pięć to ukryte testy, których worker nigdy nie widzi. Worker to headless Sonnet 5.5. Drugi headless model gra product ownera, nie ma narzędzi, czyta prywatny brief i odpowiada zwięźle na pytania workera. Każda tura jest logowana z tokenami, kosztem według cennika, czasem, wywołaniami narzędzi i wywołaniami skilli. Koszt to total_cost_usd z CLI, ekwiwalent cennikowy, a nie rachunek. Nic z tego nie zostało uruchomione na niczyim prawdziwym kodzie, więc traktuj to jako szablon dla własnego repo.
Zacznij od
README.md
# flow-bench: Superpowers loop vs Matt Pocock's Main Flow, one feature, one repo
Built 2026-10-09 from `SPEC.md` (the fixed decisions are implemented as written; deviations and their reasons are
listed under "Decisions taken where SPEC left a choice"). Worker = headless `claude-sonnet-5-5` (`claude -p`, OAuth
from `~/.claude`), Claude Code 2.1.294.
## Commands (from this dir)
```bash
./validate.sh # SPEC validation steps 1-5 -> validation.log (3-5 are smoke runs, RUN_TAG=smoke)
./validate.sh 1 2 # only the static steps (reference patch, untouched upstream)
RUN_TAG=x ./run.sh <cfg> <rep> # one run -> runs/<cfg>-r<rep>-x/, work copy work/<same>/ ; tagged runs are ignored by analyze.py
nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 & # the 10-run matrix, 2 lanes (MATRIX_JOBS=2)
SKIP_DONE=1 nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 & # resume
python3 analyze.py # results.json + results.md
python3 lib/finalize.py runs/<run> # rescore one run (restores node_modules into the work copy if it was removed)
python3 lib/reaccount.py runs/<run> # recompute turns.jsonl accounting from the saved streams (then finalize)
```
## Layout
| path | what |
|---|---|
| `upstream/` | **ledgerly**, the invented repo: Node 20 + TS + vitest + pnpm, 18 source files (domain / store / http layers), 34 tests in 7 files, 4 commits (HEAD 980d542), `test` + `typecheck` scripts, `node_modules` installed (pnpm 11) |
| `tasks/F1.md` | the task, verbatim (3 lines, the way a dev types it) |
| `po/F1-brief.md` | the product owner's full intent: the 5 requirements in plain words, preferences, constraints (never the tests) |
| `hidden/` | `test_F1.test.ts` (the oracle: `test_req1..5`), its own `vitest.config.ts` (alias `@work` -> `$WORK`) and `node_modules` (vitest 2.1.9). Runs from OUTSIDE the work copy |
| `reference/F1.patch` | reference solution: hidden 5/5, suite 39/39, tsc green |
| `configs.json` | `bare`, `sp`, `mp`, `both` |
| `packs/` | `superpowers/` (plugin root of obra/superpowers v6.4.2), `mattpocock-skills/.claude/skills/` (27 skills of mattpocock/skills v1.3.1 installed by `npx skills add`), `VERSIONS.md`, `mattpocock-install.log`; `*-src/` = the clones (gitignored) |
| `env.sh` | copied from an earlier bench of the same kind: scrubs inherited `CLAUDE*` vars, `CLAUDE_CODE_DISABLE_AUTO_MEMORY=1` |
| `prep.sh <cfg> <work>` | fresh work copy + the pack fixtures of the config + `.claude/settings.json`; prints the base commit |
| `run.sh <cfg> <rep>` | prep -> `lib/driver.py` -> `lib/finalize.py` |
| `lib/driver.py` | the turn loop (flows below), caps, `turns.jsonl` |
| `lib/accounting.py` | per-turn and per-stage tokens / USD / wall / tools / Skill events / hook events from one turn's stream |
| `lib/po.py` | the product-owner simulator |
| `lib/finalize.py` | diff vs base, suite, typecheck, hidden oracle, stages -> `score.json`, `stages.json` |
| `analyze.py` | `results.json`, `results.md` |
| `validate.sh`, `validation.log`, `validation/routing-probes/` | pre-matrix checks (+ 6 one-turn probes: plugin wiring vs project wiring) |
| `matrix.txt`, `matrix.sh`, `matrix-one.sh`, `matrix.log` | 10 runs, 2 at a time |
### The feature and the oracle
`tasks/F1.md`: partial payments, `POST /invoices/:id/payments` with an amount in cents, `amountPaid`/`balance` on
invoices, flip to `paid` at zero balance, keep `/pay` working, "we have prod data in the JSON store".
The 5 hidden requirements (one `it` each) and the codebase precedent that makes each one a senior's expectation:
| req | checks | precedent in the repo |
|---|---|---|
| R1 edge cases | payments accumulate; status stays non-`paid` until the exact remainder; `paidAt` set; `/reports/outstanding` sums balances, not totals | the report sums `total` of open invoices |
| R2 validation | 0, negative, 12.5, "100", null, missing -> 400/422; overpayment (of the total, then of the balance) -> 400/409/422; state unchanged | `assertCents`, "discount cannot exceed the subtotal" |
| R3 migration of existing data | a schemaVersion-1 file: legacy `paid` invoice shows amountPaid=total, balance 0; open shows 0/total; payments persist across a reload; `/pay` leaves balance 0 | `schemaVersion` + `src/store/migrations.ts` (v0->v1), README house rule |
| R4 error path | unknown id 404; draft 409; void 409; nothing recorded | `markPaid` throws `ConflictError` on draft/void |
| R5 a test | a `*.test.ts` changed or added since the base commit mentions `payments` | 34 existing tests |
Lenient where the brief would be a guess (status codes for overpayment accept 400/409/422, payment response 2xx,
response body never inspected: state is read back with GET).
## Configs and how each pack is wired (`prep.sh`)
- Every config: `cp -R upstream`, `.claude/` in `.git/info/exclude`, `.claude/settings.json` with a `permissions.deny`
list (Read of `hidden/ reference/ runs/ po/ validation/` of this bench; Bash commands containing `hidden`,
`reference/F1`, `F1-brief`, `/runs/`, `../`, and `find`). Bare has nothing else: no skills, no hooks.
- `sp`: the plugin root is copied to `.claude/plugins/superpowers/` (the hook script reads
`../skills/using-superpowers/SKILL.md` relative to itself), the 15 skills to `.claude/skills/<name>/`, and
`hooks/hooks.json` is reproduced in `.claude/settings.json` verbatim (`SessionStart`, matcher
`startup|clear|compact`, `"shell": "bash"`, `"async": false`) with the command prefixed by
`CLAUDE_PLUGIN_ROOT="$CLAUDE_PROJECT_DIR/.claude/plugins/superpowers"`: Claude Code sets that variable for plugin hooks,
and `hooks/session-start` uses it to pick the `hookSpecificOutput.additionalContext` output shape. Superpowers ships no
other hook (`hooks-cursor.json` is Cursor's).
- `mp`: the 27 skills to `.claude/skills/`. Pocock ships **no hook** (verified: no settings or hooks file in the install;
`git-guardrails-claude-code`, a skill that writes hooks, is outside the promoted set and was trimmed). The pack's
per-repo precondition `/setup-matt-pocock-skills` is materialised once: `docs/agents/issue-tracker.md` (local
markdown under `.scratch/`), `docs/agents/domain.md`, `docs/agents/triage-labels.md` from the skill's own seed
templates, and a `CLAUDE.md` "## Agent skills" block, committed as "chore: configure agent skills (local tracker)"
so the diff only shows the feature. Base commit for mp/both = that commit.
- `both`: sp + mp in the same `.claude/` (no skill-name collision between the two packs), driven with the mp flow.
## Flows (`lib/driver.py`)
Every worker turn: `claude -p <prompt> [--resume <sid>] --model claude-sonnet-5-5 --output-format stream-json --verbose
--include-hook-events --include-partial-messages --max-turns 60 --permission-mode acceptEdits --allowedTools <SPEC list>
--disallowedTools AskUserQuestion,EnterPlanMode,ExitPlanMode,EnterWorktree,ExitWorktree,WebFetch,WebSearch
--setting-sources project --strict-mcp-config`, cwd = work copy, stdin /dev/null, scrubbed env.
Allow-list: `Read,Edit,Write,MultiEdit,Glob,Grep,Skill,TodoWrite,Task,Agent`, Bash `node npm pnpm npx vitest npx tsc tsc
git status|diff|log|add|commit|checkout -b ls cat mkdir head tail grep wc gh`.
- `bare`: turn 1 = task + "When you are done, stop and summarise what you changed."
- `sp`: turn 1 = the task alone (the SessionStart hook is supposed to route it).
- `mp` / `both`: `/grill-with-docs <task>` -> `/to-spec` -> `/to-tickets` -> `/implement the tickets in
.scratch/<slug>/issues/, all of them, blockers first` -> `/code-review since <base sha>` -> `/pr`.
- After every turn the PO reads the last assistant message and classifies it: `question` (its reply is the next
prompt), `waiting` (driver sends "go on" / "go on, proceed with the plan"; mp: stays in the stage), `done` (bare/sp:
stop; mp: next slash command; after `/pr`: stop). If the PO call fails twice the SPEC heuristic (ends on "?" or an
options list) decides.
- Caps: 12 driver turns, 60 `--max-turns` per claude turn (a turn that hits it gets "go on"), 40 min wall per run
(the claude process is killed at the deadline). Caps hit are recorded in `driver.json` / `score.json`, never discarded.
### PO simulator (`lib/po.py`)
Headless `claude-sonnet-5-5`, `--tools ""`, `--setting-sources ""`, `--settings '{"disableAllHooks":true}'`,
`--strict-mcp-config`, `CLAUDE_CODE_DISABLE_CLAUDE_MDS=1`, `CLAUDE_CODE_DISABLE_AUTO_MEMORY=1`, cwd = an empty temp dir,
`--json-schema {kind, reply}`. Prompt = role + `po/F1-brief.md` + flow note + stage + the last message. Rules: terse,
answer every open question at once, approve what matches the brief, correct only contradictions, "your call" where
the brief is silent, **answer only what was asked** (first smoke: the PO volunteered the migration requirement nobody
had asked about; the rule was added before any counted run), how to finish a branch: "keep it on the current branch,
no PR". mp rule: a stage whose output is delivered and that only asks "shall I proceed / start?" is `done` (first mp
smoke: the PO's "yes, go" after the grilling recap made the worker start implementing inside `/grill-with-docs`; that
smoke was killed and rerun). PO cost is recorded apart (`usd_po`), never in the worker's USD.
## Accounting (the per-turn / per-stage numbers)
Checked on 2.1.294 with probes before writing `lib/accounting.py`:
- `result.total_cost_usd` and `result.modelUsage` are **cumulative over a resumed session** and **include subagents**:
per-turn USD and tokens = delta against the previous turn's result line.
- `result.usage` = main thread of the invocation only (a Task subagent's tokens are missing from it): not used.
- `assistant` events carry the usage snapshot of the message START (`output_tokens` partial, e.g. 3 for a 783-token
message): useless for output. `--include-partial-messages` adds `stream_event` `message_start` / `message_delta`
with the exact per-API-call usage, main thread only.
- Sonnet 5.5 list prices derived from two probe turns and matched to the cent: $2/M input, $10/M output, $0.20/M cache
read, $4/M cache write (1 h TTL, what Claude Code uses here); 5-min writes priced $2.5/M (unseen). In every turn without
subagents, the per-call priced sum equals the `total_cost_usd` delta exactly (`pricing_check` in `turns.jsonl`).
- Stage = slash command of the turn (mp/both, exact) or the skill in force (sp: switches at each main-thread `Skill`
tool_use; turn 1 starts at `(none)`). Inside a turn, main-thread calls are attributed exactly; the subagent remainder
(turn delta minus main calls) goes to the stage that spawned the Task, weighted by the subagents' snapshot tokens.
Wall per stage = receive-time gaps of the stdout lines (`turn-NN.times`) attributed to the stage in force.
- `analyze.py` also groups sp skills into phases (brainstorming / writing-plans / execution incl. TDD /
verification / review / finishing).
- Tokens reported = input + output + cache read + cache write. Cache reads dominate (re-reading the context each call).
### Per run (`runs/<cfg>-r<rep>/`)
`meta.json`, `prep.txt`, `settings.json` (copy of the work copy's), per turn `turn-NN.stream.jsonl` (raw stdout, every
line), `turn-NN.times`, `turn-NN.stderr`, `turn-NN.last.txt`, `po-NN.json`; `turns.jsonl` (one accounting record per
turn: prompt, source, wall, subtype, USD, tokens per model, segments, tools, Skill events, hook events, files, denials,
PO answer), `driver.json` (end reason, caps), `post.diff`, `numstat.txt`, `suite.txt`, `typecheck.txt`, `hidden.txt`,
`stages.json`, `score.json`. Work copies stay in `work/` without `node_modules` (49 MB each; finalize removes it,
`KEEP_NODE_MODULES=1` keeps it).
## Decisions taken where SPEC left a choice
- **Implement stage = `/implement`, not `/implement-spec`.** v1.3.1 ships both. `/implement-spec` requires implementer
subagents "each in its own worktree on its own branch" plus merger subagents: `git worktree`, `git merge`, `git branch`
are outside the SPEC Bash allow-list, so it could not run as designed. `/implement` is sent once with the ticket
directory ("all of them, blockers first"), since one session cannot `/clear` between tickets as `ask-matt` suggests.
- `/code-review` gets `since <base sha>` (the skill reviews "the diff since a fixed point"); Pocock's `code-review`
replaces Claude Code's bundled `code-review` skill of the same name in the session (init lists it once, the Skill
event loads `.claude/skills/code-review`).
- `/retro` not run (optional in SPEC).
- `both` uses the mp flow: the stacking question is whether the Superpowers SessionStart routing hijacks the slash
commands.
- Reps / order: `matrix.txt` (rep 1 of every config first), 2 lanes.
## Traps met
- **Superpowers routing is stochastic headless.** On turn 1 the hook always fires, but the model picks
`test-driven-development` instead of `brainstorming` part of the time. Six one-turn probes on the F1 prompt
(`validation/routing-probes/`): real plugin wiring (`--plugin-dir packs/superpowers`, namespaced
`superpowers:*`) 1/3 brainstorming, 2/3 TDD; project wiring (this bench) 2/3 brainstorming, 1/3 TDD. So the project
wiring is not the cause; the first sp smoke went TDD-only (one turn, hidden 5/5). A second sp smoke (`smoke2`) was run
to see the brainstorming path end to end.
- **The SessionStart hook fires only on turn 1.** Its matcher is `startup|clear|compact`; every later turn is
`--resume` (source `resume`), so the injection is not repeated; it stays in the transcript.
- **User plugins do not leak** with `--setting-sources project`: the init event lists only Claude Code's built-in
plugins (`cc-plugin-agents-md`, `cc-plugin-telemetry`, `cc-plugin-plugin-authoring`), never the user's Superpowers
6.4.1; bare smoke: no hook event, no Skill event. Built-in skills (`verify`, `debug`, `simplify`, `code-review`, ...)
are still listed in every config.
- **`--bare` is never used** (cannot authenticate on this OAuth account; see the previous bench).
- **Read-only shell commands bypass the allow-list.** Claude Code auto-approves read-only commands (`git ls-files`,
`sed -n`, pipes of read-only commands) whatever `--allowedTools` says; `python3` edits are denied (the bare smoke had 4
`permission_denials`, all `python3` heredoc edits; the worker fell back to Edit). `cat > /tmp/x <<EOF` matches
`Bash(cat:*)`: the allow-list is a prefix match, not a sandbox.
- **pnpm 11** refuses to install without approving esbuild's build script: `pnpm-workspace.yaml` with
`allowBuilds: {esbuild: true}` (pnpm 11 no longer reads the `pnpm` field of package.json). No `packages:` key, so it
is not a monorepo signal for Pocock's setup.
- **`npx skills add` ignores `--skill a,b` and `--skill "a b"`** ("No matching skills found"); repeated `--skill x`
flags installed every skill of the repo (37). Trimmed to the plugin's 27 (`packs/VERSIONS.md`). Pinning works with
the `https://github.com/mattpocock/skills/tree/v1.3.1` source (main is past the release).
- **mattpocock skills are mostly user-invoked** (`disable-model-invocation: true` on grill-with-docs, to-spec,
to-tickets, implement, implement-spec, ...): reachable as slash commands only; `/grill-with-docs` itself is one line
("Call the Skill tool twice, for grilling and domain-modeling"), so the proof that it resolved is those two Skill
events on turn 1.
- **The model mistypes the bench path.** Sonnet 5.5 regularly writes absolute paths with
`superpowers-vs-matt-pocock-skins-...` (or `skips`) instead of `skills`: 0 to 10 tool calls per run (0 to 7% of calls;
e.g. mp-r3-po1 10/145, mp-r2 0/124, sp-r2 3/48, bare-r1 0/28), in every config. Edits on the wrong path fail and are
retried (a small, roughly proportional token tax); **Writes succeed outside the work copy** because `Write` in
`--allowedTools` is not path-limited: the runs left stray files under a mistyped copy of the bench path (a spec
copy, an `x`) and three 1-byte placeholder files, deleted after the matrix
(nothing else in the repo was touched; `git status` checked). A future bench should put work copies on a short
path without the word `skills` and deny `Write(//**)` outside the work copy.
- **Shared plan meter.** Workers and PO bill the user's plan quota; USD = the CLI's list-price equivalent.
- **rtk** rewrites interactive `diff`/`grep` in the launching session only; scripts call binaries directly.
## Validation (2026-10-09, `validation.log`)
1. `reference/F1.patch` -> hidden 5/5, suite 39/39 green, tsc green: PASS.
2. upstream unchanged -> hidden 0/5, suite 34/34 green: PASS.
3. `bare-r1-smoke` -> 2 driver turns (the PO answered one closing question), hidden 5/5, $0.52, 125 s; isolation: no
hook event, no Skill event, only built-in plugins in init: PASS.
4. `sp-r1-smoke` -> hook fired and injected `using-superpowers` (PASS) but the model went straight to
`test-driven-development` (brainstorming FAIL in that run). Retry `sp-r1-smoke2`: hook fired, `brainstorming` on
turn 1, PO approved the design, then `test-driven-development`: PASS. Routing probes: see Traps.
5. `mp-r1-smoke` -> turn 1 loads `grilling` + `domain-modeling` (what `/grill-with-docs` says to do): PASS. The run then
used all 12 driver turns (grill 4, to-spec 2, to-tickets 2, implement 2, code-review 2) and never reached `/pr`.
Fixes made during validation, before any counted run: the PO volunteering unasked requirements; the PO telling the
worker "yes, go" at the end of the grilling (the worker started implementing inside `/grill-with-docs`; that first mp
smoke was killed at turn 4 and rerun).
## Matrix launch
`nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 &` at 2026-10-09 10:41:33, 2 lanes, order in `matrix.txt`.
Done 11:06:29 (10 runs, 25 min).
### Reruns of the mp-flow runs (harness fixes, `matrix.log`)
- `both-r1` (first): the PO answered the grilling recap's branch question with an approval and the worker coded inside
`/grill-with-docs` (turn 4, 12 src/test files). Kept as `runs/both-r1-pofault` (tag `pofault`, ignored by analyze).
PO rule added (mp flow: before `/implement`, end with "Don't start implementing yet." when the worker offers to build).
- `both-r1` (second): the PO did not apply that rule and the worker coded inside `/grill-with-docs` again (turn 3, after
"say so and I'll start with failing tests"). The rule was made deterministic in `lib/driver.py`: in `/grill-with-docs`,
`/to-spec`, `/to-tickets`, the driver appends "Don't start implementing yet." to every PO answer. Since that changes
the conditions of all mp-flow runs, mp-r1..r3 and both-r1 were moved to `runs/*-po1` (tag `po1`; mp-r1..r3-po1 never
coded early, hidden 5/5 each, $2.22 to $2.57) and rerun with `matrix-mp2.txt` (11:18 to 11:40). The counted mp/both
runs are the reruns.
- sp stage labels: brainstorming's SKILL.md ends at the approved design, but in sp-r1 and sp-r3 the model then coded with
no new Skill call, so "skill in force" would bill the whole build to brainstorming. `lib/accounting.py` now switches the
stage to `implementation (no skill)` at the first main-thread Edit/Write under `src/` or `test/` while only a design
skill (`(none)`, `using-superpowers`, `brainstorming`, `writing-plans`) is in force; the sp runs were re-accounted from
their saved streams with `lib/reaccount.py` (tokens, USD and wall totals unchanged, only the split).
## Results
`results.md` / `results.json` (`python3 analyze.py`), reading in `FINDINGS.md`.
---
# Round 2 (2026-10-09 afternoon): `spd`, feature F2, isolation fix, full rerun
## Isolation leak found and fixed (all round-1 runs superseded)
The first `spd` smoke said "the most capable model your CLAUDE.md permits for subagents": the workers were reading
CLAUDE.md files. `--setting-sources project` does keep `~/.claude/CLAUDE.md` out (probe in an empty dir outside the
repo: "NONE"), but Claude Code also loads **every CLAUDE.md of the parent directories**, and the work copies lived under
the host repository, so every round-1 session carried that repository's CLAUDE.md and what it imports
(a probe in a round-1 work copy showed the host rules present; the round-1 mp code
reviews even cited them). Fix: `run.sh` builds the work copy in
`/private/tmp/flow-bench/<run>` (outside any CLAUDE.md tree) and archives it to `work/<run>` (without node_modules)
after finalize; probe in an mp copy there: only the pack's own "## Agent skills" CLAUDE.md block appears.
Every round-1 run was renamed `runs/*-leak` (tag `leak`, ignored by analyze) and the whole matrix rerun
(`matrix2.txt`). The short path also removes the "pocock-skins" typo surface (see Traps).
## `spd`: Superpowers, driven
Same project install as `sp` (skills in `.claude/skills/`, SessionStart hook). Installed in the project, the skills
resolve **without the plugin namespace**: init `slash_commands` lists `brainstorming`, `writing-plans`,
`executing-plans` (with `--plugin-dir` they are `superpowers:brainstorming`, ...; see `validation/routing-probes/`),
so the driver types `/brainstorming <task>` -> `/writing-plans` -> `/executing-plans`. writing-plans ends by asking
"Subagent-driven or Native?": the PO answers Native (added to both briefs: "If asked to choose how to execute a written
plan: Native (inline, in this session) ... working on main is fine: you consent"). Before `/executing-plans` the driver
appends "Don't start implementing yet." to PO answers (same rule as mp). A stage whose output is delivered and that
only asks for approval/go is `done` -> next command; after `/executing-plans` the run ends when the PO says done
(executing-plans itself runs the final whole-branch review and finishing-a-development-branch).
- Allow-list: the SPEC list + `Bash(*scripts/task-start*)`, `task-done`, `sdd-workspace`, `task-brief`,
`review-package` (executing-plans drives its loop with these pack scripts), and the `Bash(*../*)` deny is dropped for
spd (the skill calls `../subagent-driven-development/scripts/...`); the oracle stays behind the Read denies and
`Bash(*hidden*)`.
- Stages: per slash command (turn level, exact) AND per skill in force inside the turn (a typed slash command starts
its skill; later Skill events switch). Subagent spend is now its own segment `<dispatching stage> [subagent]`
(round 1 folded it into the dispatching stage); executing-plans' final reviewer is dispatched "on the most capable
available model": it ran on `claude-fable-5-1` in the spd smoke ($2.19 of $3.88).
- `sp` is unchanged and labelled **hook-routed** in results.
## F2: credit notes (`tasks/F2.md`, `po/F2-brief.md`, `hidden/test_F2.test.ts`, `reference/F2.patch`)
Request (2 lines): credit notes `POST/GET /invoices/:id/credit-notes`, numbered `CN-0001`..., and "voiding a paid
invoice should then work, by crediting whatever is left on it". Implicit requirements (hidden, one `it` each):
| req | category | checks | why a careless build misses it |
|---|---|---|---|
| R1 | cross-cutting invariant, 2 layers | credits never exceed the total cumulatively (domain) and `/reports/outstanding` owes total minus credits (report) | the report is not in the request |
| R2 | stored-data compat | a schemaVersion-1 file (no `creditNotes`, no credit-note counter): first note `CN-0001`, id not `NaN`, notes survive a reload; a legacy paid invoice can be voided | `migrate()` rebuilds only known keys and casts `counters`: a new counter is `undefined + 1` |
| R3 | ordering rule | one gapless CN sequence across invoices; rejected requests (over-credit, bad amount, draft, unknown) consume no number | the existing `/issue` route takes `nextId()` before validating: copying it burns numbers |
| R4 | error path, specific status | draft/void 409, unknown 404 on POST and GET, bad amounts 400, nothing recorded | |
| R5 | existing test changes meaning | void of a paid invoice: 200, credits only the remainder, void again 409; the 33 other upstream tests present and green and `test/invoice.test.ts` keeps >= 7 tests ("voids drafts and open invoices, never paid ones" must be rewritten, not deleted) | |
Validation (`validation.log`): reference 5/5, suite 39/39, tsc green; upstream 0/5, suite 34/34; bare smoke (isolated)
**4/5** (R3 failed: `CN-0004` where `CN-0002` was due, rejected requests burned numbers). A first F2 (payment terms,
`validation/f2-v1/`) was dropped: its bare smoke scored 5/5.
`hidden/upstream_test_titles.json` = the 34 upstream test titles (for R5). `hidden/vitest.config.ts` picks
`test_${TASK}.test.ts`. Run names: F1 `<cfg>-r<rep>`, F2 `f2-<cfg>-r<rep>`; `TASK=F2 ./run.sh <cfg> <rep>`.
## Matrix round 2 (`matrix2.txt`, 2 lanes, started 12:09:53)
F1: bare x3, sp x3, spd x3, mp x3, both x1. F2: bare x2, spd x2, mp x2. Same caps (12 driver turns, 60 max-turns,
40 min).
Done 13:0x (19 runs). Then `analyze.py` (results.md / results.json carry F1 and F2, five configs).
### After the matrix
- **F2 oracle v3.** Both F2 bare runs allowed credit notes on `paid` invoices only (409 on open ones), a reading the
request's framing ("paid invoices can't be voided, so finance needs credit notes") permits; oracle v2 ran R1-R3 on
open invoices, so that one choice failed three requirements. R1-R4 now run on paid invoices; credit notes on open
invoices and the report netting are out of the oracle (the brief still prefers them). v2 kept in
`validation/test_F2.v2-open-invoices.test.ts`; reference 5/5 and upstream 0/5 re-checked; the 7 F2 runs rescored
with `lib/finalize.py` (worker runs untouched). With v3: f2-bare-r1 4/5 (R3, numbers burned), every other F2 run 5/5.
- **spd permission denials:** 5 to 11 per run, mostly compound shell lines (`W=.superpowers/sdd/...; mkdir ...; cat ...`)
that match no allow rule; the pack scripts themselves ran (task-start/task-done in every spd run).
- **"pocock-skins" path typos** (Traps) concern round 1 only: round-2 work copies live at `/private/tmp/flow-bench/<run>`,
a path without the word `skills`; no stray file was found in the repo after round 2.
- Zero-cost `[subagent]` placeholder segments are skipped by analyze.py.
SPEC.md
# flow-bench - Superpowers loop vs Matt Pocock's Main Flow, same feature, one repo
Question the video must answer: on one small invented repo, for one bounded brownfield feature,
what does each pack cost PER STAGE (tokens, wall-clock, USD) and what does it deliver (hidden
acceptance tests passed, repo suite green, diff size, bugs shipped), against a bare Claude Code run?
Secondary question: does stacking both packs in one project break (hook / slash conflicts)?
## Fixed decisions (2026-10-09)
- Worker model: `claude-sonnet-5-5`, headless (`claude -p`), OAuth from `~/.claude` (see traps below).
- The repo `upstream/`: INVENTED, built here, a small TypeScript project (about 12-20 source files,
vitest, pnpm or npm, a green suite of ~25-40 tests): a plausible brownfield service, e.g. an
invoicing API or a booking CLI with a store layer, a domain layer and an HTTP or CLI layer. It is a
git repo with a clean history (3-6 commits), a short README and a `package.json` with
`test` and `typecheck` scripts. No framework exotica; plain Node 20 + TS + vitest.
- ONE feature task `tasks/F1.md`: a terse developer request (headline ask + one or two constraints,
2-4 lines, the way a dev types it), whose honest completion needs 5 requirements that a senior
would expect (edge cases, a validation, a migration of existing data, an error path, a test).
`hidden/test_F1.test.ts` is the oracle (5 requirement tests, never visible to the worker, run
from OUTSIDE the work copy against it). `reference/F1.patch` passes all 5 and the repo suite.
- Configs (`configs.json`):
- `bare`: no skills, no hooks.
- `sp`: obra/superpowers installed in the PROJECT (not the user's plugin): clone the repo at the
latest release tag (record it), expose its skills under `<work>/.claude/skills/` and its
SessionStart hook + any hooks it ships under `<work>/.claude/settings.json` exactly the way the
plugin wires them (read its `hooks/` and `plugin.json`; reproduce, don't improvise). Verify in a
smoke run that the `using-superpowers` text actually appears in the session (stream-json shows
the hook output) and that `brainstorming` is invoked.
- `mp`: mattpocock/skills installed in the project via the official installer
(`npx skills add mattpocock/skills` or the README's exact command, latest release, record the
version); verify the Main Flow slash commands resolve (`/grill-with-docs`, `/to-spec`,
`/to-tickets`, `/implement` or `/implement-spec`, `/code-review`, `/pr`, `/retro`; use the names
the installed version really has).
- `both`: `sp` + `mp` together (one or two runs only: the stacking question).
- Flows (the driver `run.sh <cfg> <rep>` / `lib/driver.py`):
- `bare`: one session, the task prompt, "when done, stop and summarise".
- `sp`: one session; turn 1 = the task prompt (the hook is supposed to route into brainstorming).
Then the Superpowers loop as it unfolds (brainstorming → writing-plans → executing-plans →
review); the driver only answers questions and says "go on / proceed with the plan" when the
assistant ends a turn waiting. Record which skills were invoked and when (Skill tool events).
- `mp`: one session, one stage per driver turn, in this order: `/grill-with-docs <task>` →
`/to-spec` → `/to-tickets` → the implement stage (`/implement` or `/implement-spec`, whichever
v1.3 ships for the loop over tickets) → `/code-review` → `/pr`. (`/retro` optional, one run.)
The driver answers questions between turns.
- Every turn is `claude -p … --resume <session_id> --output-format stream-json --verbose
--include-hook-events`; the first turn creates the session. Per TURN: tokens in/out/cache
read/cache write, USD (`result.total_cost_usd` deltas), wall-clock, tool calls, Skill
invocations, files touched. Per STAGE: the sum of its turns. A "stage" for `sp` is the skill in
force (from the Skill events), for `mp` the slash command of the turn.
- **Product-owner simulator** (`lib/po.py`): both packs interrogate the user. A headless
`claude-sonnet-5-5` with NO tools, `--setting-sources ""`, `CLAUDE_CODE_DISABLE_CLAUDE_MDS=1`,
reads `po/F1-brief.md` (the full intent, the 5 requirements in plain words, preferences and
constraints; never the tests) and answers the worker's questions tersely, as a busy dev would.
When the worker ends a turn without a question, the PO replies "go on" (or the next slash
command for `mp`). Hard caps: 12 driver turns per run, 60 `--max-turns` per claude turn, 40 min
wall per run. A run that hits a cap is recorded as such, not discarded.
- Reps: 3 per config for `bare`, `sp`, `mp`; 1 for `both`. 10 runs. Parallel 2.
- Scoring (`lib/finalize.py`): hidden 0-5, repo suite pass/fail, typecheck pass/fail, `git diff
--stat` lines added/removed, files touched, tests added, plus the per-stage table. `analyze.py`
→ `results.json` + `results.md` (per config: mean/median per metric; per stage: tokens, USD, wall).
- Validation BEFORE the matrix (`validate.sh`, logged): (1) `reference/F1.patch` → hidden 5/5 and
suite green; (2) upstream unchanged → hidden 0/5 (or the honest number, recorded) and suite green;
(3) one `bare` smoke run end to end with the PO loop exercised; (4) one `sp` smoke proving the
hook fired and brainstorming ran; (5) one `mp` smoke proving `/grill-with-docs` resolved. A
smoke run is tagged (`RUN_TAG=smoke`) and ignored by analyze.py.
## Traps already known (from a previous bench of the same kind)
- `claude -p --bare` cannot authenticate on an OAuth (Max plan) account: never use `--bare`. Use
`--setting-sources project` for workers so the user's own plugins (Superpowers 6.4.1 is installed
at user scope on this machine) never leak into a `bare` or `mp` run. VERIFY this isolation in the
smoke: a `bare` run must show no Superpowers hook output and no Skill events.
- Scrub inherited `CLAUDE*` env vars (copy `env.sh` from the previous bench), set
`CLAUDE_CODE_DISABLE_AUTO_MEMORY=1`.
- Fresh work copy per run at a unique path; `.claude/` excluded via `.git/info/exclude` so diffs
stay about the feature.
- A worker may try to read the hidden tests: the hidden dir lives outside the work copy and the
allow-list has no `find`/`ls` outside cwd.
- Allowed tools: Read,Edit,Write,MultiEdit,Glob,Grep,Skill,TodoWrite,Task,AskUserQuestion is NOT
available headless (questions come as plain text; the driver detects a turn that ends on a
question mark or an explicit "?"/options list and routes it to the PO); Bash limited to
node/npm/pnpm/npx vitest/tsc/git status|diff|log|add|commit|checkout -b/ls/cat/mkdir/head/tail/
grep/wc. Pocock's `/pr` wants `gh`: allow `Bash(gh:*)` but with no remote the PR step records
its output text as the artefact (no network push).
## Deliverables
`README.md` (how to run, layout, every trap met), `SPEC.md` (this), `upstream/`, `tasks/F1.md`,
`po/F1-brief.md`, `hidden/`, `reference/`, `configs.json`, `env.sh`, `prep.sh`, `run.sh`,
`validate.sh`, `matrix.sh`, `lib/{driver.py,po.py,finalize.py}`, `analyze.py`, `validation.log`,
`runs/<cfg>-r<rep>/` (stream.jsonl per turn, stages.json, score.json), `results.json`,
`results.md`, and `FINDINGS.md`: the honest reading in 300 words, every number with its run name.
FINDINGS.md
# flow-bench findings (2026-10-09, round 2: isolated work copies, Sonnet 5.5 headless, 19 counted runs)
**F1 (partial payments): quality did not separate.** All 13 F1 runs hidden 5/5, suite and tsc green.
Cost per run: bare $0.27 (bare-r2) to $0.47 (bare-r1); sp, hook-routed, $0.44 (sp-r1) to $0.54 (sp-r2); mp $1.80
(mp-r2) to $2.22 (mp-r1); spd, Superpowers driven, $3.39 (spd-r2) to $4.41 (spd-r3).
Wall: mp 365 to 480 s (mp-r2, mp-r3), spd 520 to 701 s (spd-r2, spd-r3).
**Where the money goes.** mp-r2 (complete flow): grill $0.16, spec $0.11, tickets $0.09, implement $0.96, code-review
$0.45, pr $0.04 (F1 means: implement $1.13, code-review $0.44). spd: brainstorming $0.17, writing-plans $0.27,
executing-plans $3.31 (F1 means). Inside executing-plans the build itself (TDD in session) costs $0.97 on average; the
rest is the one whole-branch reviewer the skill dispatches "on the most capable model", which ran on claude-fable-5-1
in all five spd runs: $2.01 of spd-r1's $3.41, $2.52 of spd-r3's $4.41, $1.57 of f2-spd-r2's $3.00.
**Hook vs driven.** Left to its hook, Superpowers ran brainstorming in sp-r1, sp-r2 and sp-r3 and then coded; writing-
plans ran 0 times in 3. Driven by slash commands, the full loop ran in every spd run and cost 7.4 times more (F1 means: spd $3.74, sp $0.50). The cost is the final review, not the planning.
**F2 (credit notes) separated, barely.** f2-bare-r1 shipped one missing requirement, R3: credit-note numbers are not
gapless. It took the number before validating, like the existing `/issue` route, so rejected requests burned numbers
(CN-0004 where CN-0002 was due; the F2 bare smoke failed the same way). f2-bare-r2, f2-spd-r1, f2-spd-r2, f2-mp-r1 and
f2-mp-r2 scored 5/5. Costs: bare $0.38/$0.45, mp $2.18/$1.95, spd $3.42/$3.00 (r1/r2). Caveat: the first F2 oracle
also required credit notes on open invoices; both bare runs allowed paid invoices only, which the request's wording
permits, so that check was removed and the F2 runs rescored.
**Stacking broke nothing.** both-r1: hook injected, no Superpowers skill invoked, $2.33, inside the mp range.
**Caps.** mp-r1, mp-r3 and f2-mp-r1 hit the 12-driver-turn cap before /pr. spd never needed more than 8 turns.
**Caveats.** Round-1 runs read this repo's CLAUDE.md (parent-directory loading), so they were discarded and rerun outside the repo. A simulated PO answers each round in one message. The SessionStart hook fires on turn 1 only.
/implement covers every ticket in one turn. 5 to 11 compound shell lines per spd run were denied. PO cost is excluded.
Repo i zadania:
upstream/package.json
{
"name": "ledgerly",
"version": "0.4.0",
"private": true,
"description": "Small invoicing API: customers, invoices, reports. JSON-file store.",
"type": "module",
"engines": {
"node": ">=20"
},
"scripts": {
"test": "vitest run",
"typecheck": "tsc --noEmit"
},
"devDependencies": {
"@types/node": "^20.14.0",
"typescript": "^5.6.0",
"vitest": "^2.1.0"
}
}
upstream/package.json on GitHub
upstream/pnpm-lock.yaml
lockfileVersion: '9.0'
settings:
autoInstallPeers: true
excludeLinksFromLockfile: false
importers:
.:
devDependencies:
'@types/node':
specifier: ^20.14.0
version: 20.19.43
typescript:
specifier: ^5.6.0
version: 5.9.3
vitest:
specifier: ^2.1.0
version: 2.1.9(@types/node@20.19.43)
packages:
'@esbuild/aix-ppc64@0.21.5':
resolution: {integrity: sha512-1SDgH6ZSPTlggy1yI6+Dbkiz8xzpHJEVAlF/AM1tHPLsf5STom9rwtjE4hKAF20FfXXNTFqEYXyJNWh1GiZedQ==}
engines: {node: '>=12'}
cpu: [ppc64]
os: [aix]
'@esbuild/android-arm64@0.21.5':
resolution: {integrity: sha512-c0uX9VAUBQ7dTDCjq+wdyGLowMdtR/GoC2U5IYk/7D1H1JYC0qseD7+11iMP2mRLN9RcCMRcjC4YMclCzGwS/A==}
engines: {node: '>=12'}
cpu: [arm64]
os: [android]
'@esbuild/android-arm@0.21.5':
resolution: {integrity: sha512-vCPvzSjpPHEi1siZdlvAlsPxXl7WbOVUBBAowWug4rJHb68Ox8KualB+1ocNvT5fjv6wpkX6o/iEpbDrf68zcg==}
engines: {node: '>=12'}
cpu: [arm]
os: [android]
'@esbuild/android-x64@0.21.5':
resolution: {integrity: sha512-D7aPRUUNHRBwHxzxRvp856rjUHRFW1SdQATKXH2hqA0kAZb1hKmi02OpYRacl0TxIGz/ZmXWlbZgjwWYaCakTA==}
engines: {node: '>=12'}
cpu: [x64]
os: [android]
'@esbuild/darwin-arm64@0.21.5':
resolution: {integrity: sha512-DwqXqZyuk5AiWWf3UfLiRDJ5EDd49zg6O9wclZ7kUMv2WRFr4HKjXp/5t8JZ11QbQfUS6/cRCKGwYhtNAY88kQ==}
engines: {node: '>=12'}
cpu: [arm64]
os: [darwin]
'@esbuild/darwin-x64@0.21.5':
resolution: {integrity: sha512-se/JjF8NlmKVG4kNIuyWMV/22ZaerB+qaSi5MdrXtd6R08kvs2qCN4C09miupktDitvh8jRFflwGFBQcxZRjbw==}
engines: {node: '>=12'}
cpu: [x64]
os: [darwin]
'@esbuild/freebsd-arm64@0.21.5':
resolution: {integrity: sha512-5JcRxxRDUJLX8JXp/wcBCy3pENnCgBR9bN6JsY4OmhfUtIHe3ZW0mawA7+RDAcMLrMIZaf03NlQiX9DGyB8h4g==}
engines: {node: '>=12'}
cpu: [arm64]
os: [freebsd]
'@esbuild/freebsd-x64@0.21.5':
resolution: {integrity: sha512-J95kNBj1zkbMXtHVH29bBriQygMXqoVQOQYA+ISs0/2l3T9/kj42ow2mpqerRBxDJnmkUDCaQT/dfNXWX/ZZCQ==}
engines: {node: '>=12'}
cpu: [x64]
os: [freebsd]
'@esbuild/linux-arm64@0.21.5':
resolution: {integrity: sha512-ibKvmyYzKsBeX8d8I7MH/TMfWDXBF3db4qM6sy+7re0YXya+K1cem3on9XgdT2EQGMu4hQyZhan7TeQ8XkGp4Q==}
engines: {node: '>=12'}
cpu: [arm64]
os: [linux]
'@esbuild/linux-arm@0.21.5':
resolution: {integrity: sha512-bPb5AHZtbeNGjCKVZ9UGqGwo8EUu4cLq68E95A53KlxAPRmUyYv2D6F0uUI65XisGOL1hBP5mTronbgo+0bFcA==}
engines: {node: '>=12'}
cpu: [arm]
os: [linux]
'@esbuild/linux-ia32@0.21.5':
resolution: {integrity: sha512-YvjXDqLRqPDl2dvRODYmmhz4rPeVKYvppfGYKSNGdyZkA01046pLWyRKKI3ax8fbJoK5QbxblURkwK/MWY18Tg==}
engines: {node: '>=12'}
cpu: [ia32]
os: [linux]
'@esbuild/linux-loong64@0.21.5':
resolution: {integrity: sha512-uHf1BmMG8qEvzdrzAqg2SIG/02+4/DHB6a9Kbya0XDvwDEKCoC8ZRWI5JJvNdUjtciBGFQ5PuBlpEOXQj+JQSg==}
engines: {node: '>=12'}
cpu: [loong64]
os: [linux]
'@esbuild/linux-mips64el@0.21.5':
resolution: {integrity: sha512-IajOmO+KJK23bj52dFSNCMsz1QP1DqM6cwLUv3W1QwyxkyIWecfafnI555fvSGqEKwjMXVLokcV5ygHW5b3Jbg==}
engines: {node: '>=12'}
cpu: [mips64el]
os: [linux]
'@esbuild/linux-ppc64@0.21.5':
resolution: {integrity: sha512-1hHV/Z4OEfMwpLO8rp7CvlhBDnjsC3CttJXIhBi+5Aj5r+MBvy4egg7wCbe//hSsT+RvDAG7s81tAvpL2XAE4w==}
engines: {node: '>=12'}
cpu: [ppc64]
os: [linux]
'@esbuild/linux-riscv64@0.21.5':
resolution: {integrity: sha512-2HdXDMd9GMgTGrPWnJzP2ALSokE/0O5HhTUvWIbD3YdjME8JwvSCnNGBnTThKGEB91OZhzrJ4qIIxk/SBmyDDA==}
engines: {node: '>=12'}
cpu: [riscv64]
os: [linux]
'@esbuild/linux-s390x@0.21.5':
resolution: {integrity: sha512-zus5sxzqBJD3eXxwvjN1yQkRepANgxE9lgOW2qLnmr8ikMTphkjgXu1HR01K4FJg8h1kEEDAqDcZQtbrRnB41A==}
engines: {node: '>=12'}
cpu: [s390x]
os: [linux]
'@esbuild/linux-x64@0.21.5':
resolution: {integrity: sha512-1rYdTpyv03iycF1+BhzrzQJCdOuAOtaqHTWJZCWvijKD2N5Xu0TtVC8/+1faWqcP9iBCWOmjmhoH94dH82BxPQ==}
engines: {node: '>=12'}
cpu: [x64]
os: [linux]
'@esbuild/netbsd-x64@0.21.5':
resolution: {integrity: sha512-Woi2MXzXjMULccIwMnLciyZH4nCIMpWQAs049KEeMvOcNADVxo0UBIQPfSmxB3CWKedngg7sWZdLvLczpe0tLg==}
engines: {node: '>=12'}
cpu: [x64]
os: [netbsd]
'@esbuild/openbsd-x64@0.21.5':
resolution: {integrity: sha512-HLNNw99xsvx12lFBUwoT8EVCsSvRNDVxNpjZ7bPn947b8gJPzeHWyNVhFsaerc0n3TsbOINvRP2byTZ5LKezow==}
engines: {node: '>=12'}
cpu: [x64]
os: [openbsd]
'@esbuild/sunos-x64@0.21.5':
resolution: {integrity: sha512-6+gjmFpfy0BHU5Tpptkuh8+uw3mnrvgs+dSPQXQOv3ekbordwnzTVEb4qnIvQcYXq6gzkyTnoZ9dZG+D4garKg==}
engines: {node: '>=12'}
cpu: [x64]
os: [sunos]
'@esbuild/win32-arm64@0.21.5':
resolution: {integrity: sha512-Z0gOTd75VvXqyq7nsl93zwahcTROgqvuAcYDUr+vOv8uHhNSKROyU961kgtCD1e95IqPKSQKH7tBTslnS3tA8A==}
engines: {node: '>=12'}
cpu: [arm64]
os: [win32]
'@esbuild/win32-ia32@0.21.5':
resolution: {integrity: sha512-SWXFF1CL2RVNMaVs+BBClwtfZSvDgtL//G/smwAc5oVK/UPu2Gu9tIaRgFmYFFKrmg3SyAjSrElf0TiJ1v8fYA==}
engines: {node: '>=12'}
cpu: [ia32]
os: [win32]
'@esbuild/win32-x64@0.21.5':
resolution: {integrity: sha512-tQd/1efJuzPC6rCFwEvLtci/xNFcTZknmXs98FYDfGE4wP9ClFV98nyKrzJKVPMhdDnjzLhdUyMX4PsQAPjwIw==}
engines: {node: '>=12'}
cpu: [x64]
os: [win32]
'@jridgewell/sourcemap-codec@1.6.0':
resolution: {integrity: sha512-T7jf+5zgsZHwNJ4lvQ7/aezbyk0nNX+zJVWpmHA7VYsEx7a7qr5Rg5IbtJFqkgze5Y2sruq1RUY8Q837Od7iFw==}
'@napi-rs/lzma-linux-x64-gnu@1.5.1':
resolution: {integrity: sha512-oTXEIha4SsuXdTA4Iyskj0kpdx2yVXdhd75c2v3xGrHFfVMsbhTPZU/nMPL4sWKo4pBHm3aucLaqGlF696dTyQ==}
engines: {node: ^22.20 || ^24.12 || >=25}
cpu: [x64]
os: [linux]
libc: [glibc]
'@rollup/rollup-android-arm-eabi@4.64.2':
resolution: {integrity: sha512-gjNDUwMswogkw3fNqV00u54uZlgmInuw5EU/1TeqQ6K0bEagoGvqjPdKzy112lKyhe7PkGudZ/kSQHsc9qroIg==}
cpu: [arm]
os: [android]
'@rollup/rollup-android-arm64@4.64.2':
resolution: {integrity: sha512-YkAW8q7p5SCLSaGWaUkmsRnBEb3MGA1pB5nI/8k0lAK5fZsQKRBqkdR1h4UU9p8u3c7ZGdUF2Zcx7mK+E/pUKg==}
cpu: [arm64]
os: [android]
'@rollup/rollup-darwin-arm64@4.64.2':
resolution: {integrity: sha512-hNmuIHv+LnrP/+q/44vsmIMWfoh7X2YXOp5FLtjGuvqN52VdtFpOpnshTc1k28tWv1wE/0D5gLjsozBUowEGbQ==}
cpu: [arm64]
os: [darwin]
'@rollup/rollup-darwin-x64@4.64.2':
resolution: {integrity: sha512-B0t5CoxgF2KL7R5FHAsyN8yVEQCSIkh4XVs8MCDjuBLNjEa8l3zd5tVKaFMv53LfUtHs9p2rnqeavPTOunV+Ww==}
cpu: [x64]
os: [darwin]
'@rollup/rollup-freebsd-arm64@4.64.2':
resolution: {integrity: sha512-QQdYnfwrPmc1DoPlddjFvQPzUHFbStKXsHrG6EBYsWHhBVUCjsCRdcfGGXGAvhDpDtTzKK4vy1UrGFJ45rQaiw==}
cpu: [arm64]
os: [freebsd]
'@rollup/rollup-freebsd-x64@4.64.2':
resolution: {integrity: sha512-1xRxtGWif6BB58VSfMavi0Yfs2M4x6CtRp4o7ScY4W7AGlgvf6GSPKdDNdw7TX3tMiQwZY5W/mvyrOJTq/7AKA==}
cpu: [x64]
os: [freebsd]
'@rollup/rollup-linux-arm-gnueabihf@4.64.2':
resolution: {integrity: sha512-CXFLEt384BUwwOLR9DURWNHl1bQOVLd1ibOee+yKBJm/fZtvrSAOZ0JbxjAxszVMIovYMwFMS+oP+Q1aPcx8Eg==}
cpu: [arm]
os: [linux]
libc: [glibc]
'@rollup/rollup-linux-arm-musleabihf@4.64.2':
resolution: {integrity: sha512-T1dorY0WHm5mMNpMHFl5rYRZ02gLSZcr6J45r66tYb3S3R3DyIRGiGWVdLtF+zOQjratNJaObl9jYumueRiL1A==}
cpu: [arm]
os: [linux]
libc: [musl]
'@rollup/rollup-linux-arm64-gnu@4.64.2':
resolution: {integrity: sha512-ia6bfaCIc0ldp6bvs9oEkFrJVJKNUMoiDF0+LBZsQAZg+vE0znXnzGqH+ew/FYswE30cruAWPTO3dPw7keaVkQ==}
cpu: [arm64]
os: [linux]
libc: [glibc]
'@rollup/rollup-linux-arm64-musl@4.64.2':
resolution: {integrity: sha512-mB/gwfBfXtT0EBpbSssqKCqMFuX8b21Zy2TGoIhehSJEssZ5ksybxymwp1BkEM5Nczh/AAxpmh5q+MkhtundGg==}
cpu: [arm64]
os: [linux]
libc: [musl]
'@rollup/rollup-linux-loong64-gnu@4.64.2':
resolution: {integrity: sha512-VPd0a+XlHoNRI9jIPRHww2UKhDkD6XjbtcKYXZijAnhJS1jLLbiNtVl6a5LyEfNN+eyiyWcJOZXPzBo69oTsew==}
cpu: [loong64]
os: [linux]
libc: [glibc]
'@rollup/rollup-linux-loong64-musl@4.64.2':
resolution: {integrity: sha512-10N1I6oBpERqzGtHKcCLrEpHlbXSBbMK9TURHvqgWbIFZuU3oJROxJ0H2nHfq96TVI0XJHQ9qzolJMFd2tz0LA==}
cpu: [loong64]
os: [linux]
libc: [musl]
'@rollup/rollup-linux-ppc64-gnu@4.64.2':
resolution: {integrity: sha512-dhJrLVZ3wNNhwCmy9/VTg6Pq6J8a8eZ0n15dVRlbSWcDb+8V/1mDfbnvis+vt8W6Pbxzv6gUbTEl4piOT3bbUQ==}
cpu: [ppc64]
os: [linux]
libc: [glibc]
'@rollup/rollup-linux-ppc64-musl@4.64.2':
resolution: {integrity: sha512-tPd2QuwxkTxgju5eARyb3RdgBWWi/O9dn/NFH0VkvGbLzoQeXrH6qMABtW8IODb5lUeBzB3sy2U7qmI4MtQ72g==}
cpu: [ppc64]
os: [linux]
libc: [musl]
'@rollup/rollup-linux-riscv64-gnu@4.64.2':
resolution: {integrity: sha512-njtDdPvSQWI9CDhIPPxsiEMeHayCfu7wwcbSQ7c2kYPhGVviL+WGVhq6Jfj/b+VuhSTkFuQ0z4ZFbFjkWwBFmg==}
cpu: [riscv64]
os: [linux]
libc: [glibc]
'@rollup/rollup-linux-riscv64-musl@4.64.2':
resolution: {integrity: sha512-EbOBo3dqCj9ecEV9Z7zvulB6AieMP2ztnKQYGVOSN0pzIyOu+3JH2YJZnLmbPeVvfn6yP6Qi7NbBM27EzGbj2w==}
cpu: [riscv64]
os: [linux]
libc: [musl]
'@rollup/rollup-linux-s390x-gnu@4.64.2':
resolution: {integrity: sha512-8wLt2Q5n4qmfiE2Gk8V6KNLSf+oJuBX7Kdw5joFDtnZIZvirS0hLzBc79OiJjV71oYDmBUZLNy6pDgvylo4qiQ==}
cpu: [s390x]
os: [linux]
libc: [glibc]
'@rollup/rollup-linux-x64-gnu@4.64.2':
resolution: {integrity: sha512-aIChbkhC4d33bxkVP0LSi2/2lUSGRXE6k4Qa0G7tgpgTOK7kxHTixmx5qC3SKUJ6yUMznHBtQoWaG4QMqLUnhw==}
cpu: [x64]
os: [linux]
libc: [glibc]
'@rollup/rollup-linux-x64-musl@4.64.2':
resolution: {integrity: sha512-gQ+TkVZRfjZVcl/UOAxxfWw+sHjzIXoieNwxEyJlD3tRBmBhR8ZDq11w+QEgQDcpslEB+fhN/OEpaTsDoIIHhQ==}
cpu: [x64]
os: [linux]
libc: [musl]
'@rollup/rollup-openbsd-x64@4.64.2':
resolution: {integrity: sha512-sUaJpMKS2JBCvUk8fYimG7aMq2jHVT4kw/uVHxPolKLrMmkKy0KXirmZyZehdNpL7PffqAlPAWUELQcTiD8ChA==}
cpu: [x64]
os: [openbsd]
'@rollup/rollup-openharmony-arm64@4.64.2':
resolution: {integrity: sha512-j1RdJWb0CrlSuwt12OPjLucPlwfLtA/KMOKVvh5rXSJEtl/WDoIGR3CSykK/jLGTo9RqNfW5Ub6kY5Z5ccYl9w==}
cpu: [arm64]
os: [openharmony]
'@rollup/rollup-win32-arm64-msvc@4.64.2':
resolution: {integrity: sha512-i3r04aHCIY3eggTBLUp/4IxiBjDpAgQaK89Ue2nMJ2MKuIXX5JfoeZjHw+kPjGGNVDlhi8FT8xzuGMJZe3aehg==}
cpu: [arm64]
os: [win32]
'@rollup/rollup-win32-ia32-msvc@4.64.2':
resolution: {integrity: sha512-kCQ15h/uwNMGdN8gjXBZGf+aGE+ODzHfbi90/cAm4+Q89WoMiLODNoj8HoGZmYx87HPxfu5CBNPsZ+ZG4uIAoA==}
cpu: [ia32]
os: [win32]
'@rollup/rollup-win32-x64-gnu@4.64.2':
resolution: {integrity: sha512-yYAuySSs+0X2gSMJPw/Q4jHsHZHyMq1dpAE+XnN790Jg1WuReUAD/+j+e4vuDvS4PXJ4PA4NCzTR4OYtDf18/g==}
cpu: [x64]
os: [win32]
'@rollup/rollup-win32-x64-msvc@4.64.2':
resolution: {integrity: sha512-JY1rQdWOJrKutORqKO1uzrHXV86jFiQzfLFKx2s8S1CjLXL9fC5LOIvk0q7Z7PXJvsByt+sdoE1Q2a9gBr4gLQ==}
cpu: [x64]
os: [win32]
'@types/estree@1.0.9':
resolution: {integrity: sha512-GhdPgy1el4/ImP05X05Uw4cw2/M93BCUmnEvWZNStlCzEKME4Fkk+YpoA5OiHNQmoS7Cafb8Xa3Pya8m1Qrzeg==}
'@types/node@20.19.43':
resolution: {integrity: sha512-6oYBAi5ikg4Pl+kGsoYtawUMBT2zZMCvPNF7pVLnHZfd1zf38DRiWn/gT01RYCdUqkv7Fhr+C9ot4/tb+2sVvA==}
'@vitest/expect@2.1.9':
resolution: {integrity: sha512-UJCIkTBenHeKT1TTlKMJWy1laZewsRIzYighyYiJKZreqtdxSos/S1t+ktRMQWu2CKqaarrkeszJx1cgC5tGZw==}
'@vitest/mocker@2.1.9':
resolution: {integrity: sha512-tVL6uJgoUdi6icpxmdrn5YNo3g3Dxv+IHJBr0GXHaEdTcw3F+cPKnsXFhli6nO+f/6SDKPHEK1UN+k+TQv0Ehg==}
peerDependencies:
msw: ^2.4.9
vite: ^5.0.0
peerDependenciesMeta:
msw:
optional: true
vite:
optional: true
'@vitest/pretty-format@2.1.9':
resolution: {integrity: sha512-KhRIdGV2U9HOUzxfiHmY8IFHTdqtOhIzCpd8WRdJiE7D/HUcZVD0EgQCVjm+Q9gkUXWgBvMmTtZgIG48wq7sOQ==}
'@vitest/runner@2.1.9':
resolution: {integrity: sha512-ZXSSqTFIrzduD63btIfEyOmNcBmQvgOVsPNPe0jYtESiXkhd8u2erDLnMxmGrDCwHCCHE7hxwRDCT3pt0esT4g==}
'@vitest/snapshot@2.1.9':
resolution: {integrity: sha512-oBO82rEjsxLNJincVhLhaxxZdEtV0EFHMK5Kmx5sJ6H9L183dHECjiefOAdnqpIgT5eZwT04PoggUnW88vOBNQ==}
'@vitest/spy@2.1.9':
resolution: {integrity: sha512-E1B35FwzXXTs9FHNK6bDszs7mtydNi5MIfUWpceJ8Xbfb1gBMscAnwLbEu+B44ed6W3XjL9/ehLPHR1fkf1KLQ==}
'@vitest/utils@2.1.9':
resolution: {integrity: sha512-v0psaMSkNJ3A2NMrUEHFRzJtDPFn+/VWZ5WxImB21T9fjucJRmS7xCS3ppEnARb9y11OAzaD+P2Ps+b+BGX5iQ==}
assertion-error@2.0.1:
resolution: {integrity: sha512-Izi8RQcffqCeNVgFigKli1ssklIbpHnCYc6AknXGYoB6grJqyeby7jv12JUQgmTAnIDnbck1uxksT4dzN3PWBA==}
engines: {node: '>=12'}
cac@6.7.14:
resolution: {integrity: sha512-b6Ilus+c3RrdDk+JhLKUAQfzzgLEPy6wcXqS7f/xe1EETvsDP6GORG7SFuOs6cID5YkqchW/LXZbX5bc8j7ZcQ==}
engines: {node: '>=8'}
chai@5.3.3:
resolution: {integrity: sha512-4zNhdJD/iOjSH0A05ea+Ke6MU5mmpQcbQsSOkgdaUMJ9zTlDTD/GYlwohmIE2u0gaxHYiVHEn1Fw9mZ/ktJWgw==}
engines: {node: '>=18'}
check-error@2.1.3:
resolution: {integrity: sha512-PAJdDJusoxnwm1VwW07VWwUN1sl7smmC3OKggvndJFadxxDRyFJBX/ggnu/KE4kQAB7a3Dp8f/YXC1FlUprWmA==}
engines: {node: '>= 16'}
debug@4.4.3:
resolution: {integrity: sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA==}
engines: {node: '>=6.0'}
peerDependencies:
supports-color: '*'
peerDependenciesMeta:
supports-color:
optional: true
deep-eql@5.0.2:
resolution: {integrity: sha512-h5k/5U50IJJFpzfL6nO9jaaumfjO/f2NjK/oYB2Djzm4p9L+3T9qWpZqZ2hAbLPuuYq9wrU08WQyBTL5GbPk5Q==}
engines: {node: '>=6'}
es-module-lexer@1.7.0:
resolution: {integrity: sha512-jEQoCwk8hyb2AZziIOLhDqpm5+2ww5uIE6lkO/6jcOCusfk6LhMHpXXfBLXTZ7Ydyt0j4VoUQv6uGNYbdW+kBA==}
esbuild@0.21.5:
resolution: {integrity: sha512-mg3OPMV4hXywwpoDxu3Qda5xCKQi+vCTZq8S9J/EpkhB2HzKXq4SNFZE3+NK93JYxc8VMSep+lOUSC/RVKaBqw==}
engines: {node: '>=12'}
hasBin: true
estree-walker@3.0.3:
resolution: {integrity: sha512-7RUKfXgSMMkzt6ZuXmqapOurLGPPfgj6l9uRZ7lRGolvk0y2yocc35LdcxKC5PQZdn2DMqioAQ2NoWcrTKmm6g==}
expect-type@1.4.0:
resolution: {integrity: sha512-KfYbmpRm0VbLjEvVa9yGwCi9GI34xvi7A/HXYWQO65CSD2u3MczUJSuwXKFIxlGsgBQizV9q5J9NHj4VG0n+pA==}
engines: {node: '>=12.0.0'}
fsevents@2.3.3:
resolution: {integrity: sha512-5xoDfX+fL7faATnagmWPpbFtwh/R77WmMMqqHGS65C3vvB0YHrgF+B1YmZ3441tMj5n63k0212XNoJwzlhffQw==}
engines: {node: ^8.16.0 || ^10.6.0 || >=11.0.0}
os: [darwin]
loupe@3.2.1:
resolution: {integrity: sha512-CdzqowRJCeLU72bHvWqwRBBlLcMEtIvGrlvef74kMnV2AolS9Y8xUv1I0U/MNAWMhBlKIoyuEgoJ0t/bbwHbLQ==}
magic-string@0.30.21:
resolution: {integrity: sha512-vd2F4YUyEXKGcLHoq+TEyCjxueSeHnFxyyjNp80yg0XV4vUhnDer/lvvlqM/arB5bXQN5K2/3oinyCRyx8T2CQ==}
ms@2.1.3:
resolution: {integrity: sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==}
nanoid@3.3.20:
resolution: {integrity: sha512-uKdg2G3GNCKQn9byYOpxbGqrT2fGO5KRt5J/8b3pok8rT6qxGWF6hxMyJiEYtAf+FVyYuD9hRaDqX5uPFYJ4ZQ==}
engines: {node: ^10 || ^12 || ^13.7 || ^14 || >=15.0.1}
hasBin: true
pathe@1.1.2:
resolution: {integrity: sha512-whLdWMYL2TwI08hn8/ZqAbrVemu0LNaNNJZX73O6qaIdCTfXutsLhMkjdENX0qhsQ9uIimo4/aQOmXkoon2nDQ==}
pathval@2.0.1:
resolution: {integrity: sha512-//nshmD55c46FuFw26xV/xFAaB5HF9Xdap7HJBBnrKdAd6/GxDBaNA1870O79+9ueg61cZLSVc+OaFlfmObYVQ==}
engines: {node: '>= 14.16'}
picocolors@1.1.1:
resolution: {integrity: sha512-xceH2snhtb5M9liqDsmEw56le376mTZkEX/jEb/RxNFyegNul7eNslCXP9FDj/Lcu0X8KEyMceP2ntpaHrDEVA==}
postcss@8.5.29:
resolution: {integrity: sha512-49cGhUbXj8Qenv0iTMxA1cFBzxXoctpC9Ujd77t1WcbJIr6nF/eI7g/8MgxrYldFRuAXvja7xQRwavoW7kgrxQ==}
engines: {node: ^10 || ^12 || >=14}
rollup@4.64.2:
resolution: {integrity: sha512-RxXm0XVVV1hFI3dYe12ruaOgsYQ725lUcUWeMSQqkF9tTxVErY+fy2Ets6vWhItL9MZMSNZGbvZFc7rH/VHp2Q==}
engines: {node: '>=18.0.0', npm: '>=8.0.0'}
hasBin: true
siginfo@2.0.0:
resolution: {integrity: sha512-ybx0WO1/8bSBLEWXZvEd7gMW3Sn3JFlW3TvX1nREbDLRNQNaeNN8WK0meBwPdAaOI7TtRRRJn/Es1zhrrCHu7g==}
source-map-js@1.2.2:
resolution: {integrity: sha512-KGj/8Y43x35aZVDtt+J4mK1hoLGHULMYfSkODJNQjNDC3oW1PqPoxMwo0pLUsWM/UEGzON/NxeHywEfNXNP3Vw==}
engines: {node: '>=0.10.0'}
stackback@0.0.2:
resolution: {integrity: sha512-1XMJE5fQo1jGH6Y/7ebnwPOBEkIEnT4QF32d5R1+VXdXveM0IBMJt8zfaxX1P3QhVwrYe+576+jkANtSS2mBbw==}
std-env@3.10.0:
resolution: {integrity: sha512-5GS12FdOZNliM5mAOxFRg7Ir0pWz8MdpYm6AY6VPkGpbA7ZzmbzNcBJQ0GPvvyWgcY7QAhCgf9Uy89I03faLkg==}
tinybench@2.9.0:
resolution: {integrity: sha512-0+DUvqWMValLmha6lr4kD8iAMK1HzV0/aKnCtWb9v9641TnP/MFb7Pc2bxoxQjTXAErryXVgUOfv2YqNllqGeg==}
tinyexec@0.3.2:
resolution: {integrity: sha512-KQQR9yN7R5+OSwaK0XQoj22pwHoTlgYqmUscPYoknOoWCWfj/5/ABTMRi69FrKU5ffPVh5QcFikpWJI/P1ocHA==}
tinypool@1.1.1:
resolution: {integrity: sha512-Zba82s87IFq9A9XmjiX5uZA/ARWDrB03OHlq+Vw1fSdt0I+4/Kutwy8BP4Y/y/aORMo61FQ0vIb5j44vSo5Pkg==}
engines: {node: ^18.0.0 || >=20.0.0}
tinyrainbow@1.2.0:
resolution: {integrity: sha512-weEDEq7Z5eTHPDh4xjX789+fHfF+P8boiFB+0vbWzpbnbsEr/GRaohi/uMKxg8RZMXnl1ItAi/IUHWMsjDV7kQ==}
engines: {node: '>=14.0.0'}
tinyspy@3.0.2:
resolution: {integrity: sha512-n1cw8k1k0x4pgA2+9XrOkFydTerNcJ1zWCO5Nn9scWHTD+5tp8dghT2x1uduQePZTZgd3Tupf+x9BxJjeJi77Q==}
engines: {node: '>=14.0.0'}
typescript@5.9.3:
resolution: {integrity: sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw==}
engines: {node: '>=14.17'}
hasBin: true
undici-types@6.21.0:
resolution: {integrity: sha512-iwDZqg0QAGrg9Rav5H4n0M64c3mkR59cJ6wQp+7C4nI0gsmExaedaYLNO44eT4AtBBwjbTiGPMlt2Md0T9H9JQ==}
vite-node@2.1.9:
resolution: {integrity: sha512-AM9aQ/IPrW/6ENLQg3AGY4K1N2TGZdR5e4gu/MmmR2xR3Ll1+dib+nook92g4TV3PXVyeyxdWwtaCAiUL0hMxA==}
engines: {node: ^18.0.0 || >=20.0.0}
hasBin: true
vite@5.4.21:
resolution: {integrity: sha512-o5a9xKjbtuhY6Bi5S3+HvbRERmouabWbyUcpXXUA1u+GNUKoROi9byOJ8M0nHbHYHkYICiMlqxkg1KkYmm25Sw==}
engines: {node: ^18.0.0 || >=20.0.0}
hasBin: true
peerDependencies:
'@types/node': ^18.0.0 || >=20.0.0
less: '*'
lightningcss: ^1.21.0
sass: '*'
sass-embedded: '*'
stylus: '*'
sugarss: '*'
terser: ^5.4.0
peerDependenciesMeta:
'@types/node':
optional: true
less:
optional: true
lightningcss:
optional: true
sass:
optional: true
sass-embedded:
optional: true
stylus:
optional: true
sugarss:
optional: true
terser:
optional: true
vitest@2.1.9:
resolution: {integrity: sha512-MSmPM9REYqDGBI8439mA4mWhV5sKmDlBKWIYbA3lRb2PTHACE0mgKwA8yQ2xq9vxDTuk4iPrECBAEW2aoFXY0Q==}
engines: {node: ^18.0.0 || >=20.0.0}
hasBin: true
peerDependencies:
'@edge-runtime/vm': '*'
'@types/node': ^18.0.0 || >=20.0.0
'@vitest/browser': 2.1.9
'@vitest/ui': 2.1.9
happy-dom: '*'
jsdom: '*'
peerDependenciesMeta:
'@edge-runtime/vm':
optional: true
'@types/node':
optional: true
'@vitest/browser':
optional: true
'@vitest/ui':
optional: true
happy-dom:
optional: true
jsdom:
optional: true
why-is-node-running@2.3.0:
resolution: {integrity: sha512-hUrmaWBdVDcxvYqnyh09zunKzROWjbZTiNy8dBEjkS7ehEDQibXJ7XvlmtbwuTclUiIyN+CyXQD4Vmko8fNm8w==}
engines: {node: '>=8'}
hasBin: true
snapshots:
'@esbuild/aix-ppc64@0.21.5':
optional: true
'@esbuild/android-arm64@0.21.5':
optional: true
'@esbuild/android-arm@0.21.5':
optional: true
'@esbuild/android-x64@0.21.5':
optional: true
'@esbuild/darwin-arm64@0.21.5':
optional: true
'@esbuild/darwin-x64@0.21.5':
optional: true
'@esbuild/freebsd-arm64@0.21.5':
optional: true
'@esbuild/freebsd-x64@0.21.5':
optional: true
'@esbuild/linux-arm64@0.21.5':
optional: true
'@esbuild/linux-arm@0.21.5':
optional: true
'@esbuild/linux-ia32@0.21.5':
optional: true
'@esbuild/linux-loong64@0.21.5':
optional: true
'@esbuild/linux-mips64el@0.21.5':
optional: true
'@esbuild/linux-ppc64@0.21.5':
optional: true
'@esbuild/linux-riscv64@0.21.5':
optional: true
'@esbuild/linux-s390x@0.21.5':
optional: true
'@esbuild/linux-x64@0.21.5':
optional: true
'@esbuild/netbsd-x64@0.21.5':
optional: true
'@esbuild/openbsd-x64@0.21.5':
optional: true
'@esbuild/sunos-x64@0.21.5':
optional: true
'@esbuild/win32-arm64@0.21.5':
optional: true
'@esbuild/win32-ia32@0.21.5':
optional: true
'@esbuild/win32-x64@0.21.5':
optional: true
'@jridgewell/sourcemap-codec@1.6.0': {}
'@napi-rs/lzma-linux-x64-gnu@1.5.1':
optional: true
'@rollup/rollup-android-arm-eabi@4.64.2':
optional: true
'@rollup/rollup-android-arm64@4.64.2':
optional: true
'@rollup/rollup-darwin-arm64@4.64.2':
optional: true
'@rollup/rollup-darwin-x64@4.64.2':
optional: true
'@rollup/rollup-freebsd-arm64@4.64.2':
optional: true
'@rollup/rollup-freebsd-x64@4.64.2':
optional: true
'@rollup/rollup-linux-arm-gnueabihf@4.64.2':
optional: true
'@rollup/rollup-linux-arm-musleabihf@4.64.2':
optional: true
'@rollup/rollup-linux-arm64-gnu@4.64.2':
optional: true
'@rollup/rollup-linux-arm64-musl@4.64.2':
optional: true
'@rollup/rollup-linux-loong64-gnu@4.64.2':
optional: true
'@rollup/rollup-linux-loong64-musl@4.64.2':
optional: true
'@rollup/rollup-linux-ppc64-gnu@4.64.2':
optional: true
'@rollup/rollup-linux-ppc64-musl@4.64.2':
optional: true
'@rollup/rollup-linux-riscv64-gnu@4.64.2':
optional: true
'@rollup/rollup-linux-riscv64-musl@4.64.2':
optional: true
'@rollup/rollup-linux-s390x-gnu@4.64.2':
optional: true
'@rollup/rollup-linux-x64-gnu@4.64.2':
optional: true
'@rollup/rollup-linux-x64-musl@4.64.2':
optional: true
'@rollup/rollup-openbsd-x64@4.64.2':
optional: true
'@rollup/rollup-openharmony-arm64@4.64.2':
optional: true
'@rollup/rollup-win32-arm64-msvc@4.64.2':
optional: true
'@rollup/rollup-win32-ia32-msvc@4.64.2':
optional: true
'@rollup/rollup-win32-x64-gnu@4.64.2':
optional: true
'@rollup/rollup-win32-x64-msvc@4.64.2':
optional: true
'@types/estree@1.0.9': {}
'@types/node@20.19.43':
dependencies:
undici-types: 6.21.0
'@vitest/expect@2.1.9':
dependencies:
'@vitest/spy': 2.1.9
'@vitest/utils': 2.1.9
chai: 5.3.3
tinyrainbow: 1.2.0
'@vitest/mocker@2.1.9(vite@5.4.21(@types/node@20.19.43))':
dependencies:
'@vitest/spy': 2.1.9
estree-walker: 3.0.3
magic-string: 0.30.21
optionalDependencies:
vite: 5.4.21(@types/node@20.19.43)
'@vitest/pretty-format@2.1.9':
dependencies:
tinyrainbow: 1.2.0
'@vitest/runner@2.1.9':
dependencies:
'@vitest/utils': 2.1.9
pathe: 1.1.2
'@vitest/snapshot@2.1.9':
dependencies:
'@vitest/pretty-format': 2.1.9
magic-string: 0.30.21
pathe: 1.1.2
'@vitest/spy@2.1.9':
dependencies:
tinyspy: 3.0.2
'@vitest/utils@2.1.9':
dependencies:
'@vitest/pretty-format': 2.1.9
loupe: 3.2.1
tinyrainbow: 1.2.0
assertion-error@2.0.1: {}
cac@6.7.14: {}
chai@5.3.3:
dependencies:
assertion-error: 2.0.1
check-error: 2.1.3
deep-eql: 5.0.2
loupe: 3.2.1
pathval: 2.0.1
check-error@2.1.3: {}
debug@4.4.3:
dependencies:
ms: 2.1.3
deep-eql@5.0.2: {}
es-module-lexer@1.7.0: {}
esbuild@0.21.5:
optionalDependencies:
'@esbuild/aix-ppc64': 0.21.5
'@esbuild/android-arm': 0.21.5
'@esbuild/android-arm64': 0.21.5
'@esbuild/android-x64': 0.21.5
'@esbuild/darwin-arm64': 0.21.5
'@esbuild/darwin-x64': 0.21.5
'@esbuild/freebsd-arm64': 0.21.5
'@esbuild/freebsd-x64': 0.21.5
'@esbuild/linux-arm': 0.21.5
'@esbuild/linux-arm64': 0.21.5
'@esbuild/linux-ia32': 0.21.5
'@esbuild/linux-loong64': 0.21.5
'@esbuild/linux-mips64el': 0.21.5
'@esbuild/linux-ppc64': 0.21.5
'@esbuild/linux-riscv64': 0.21.5
'@esbuild/linux-s390x': 0.21.5
'@esbuild/linux-x64': 0.21.5
'@esbuild/netbsd-x64': 0.21.5
'@esbuild/openbsd-x64': 0.21.5
'@esbuild/sunos-x64': 0.21.5
'@esbuild/win32-arm64': 0.21.5
'@esbuild/win32-ia32': 0.21.5
'@esbuild/win32-x64': 0.21.5
estree-walker@3.0.3:
dependencies:
'@types/estree': 1.0.9
expect-type@1.4.0: {}
fsevents@2.3.3:
optional: true
loupe@3.2.1: {}
magic-string@0.30.21:
dependencies:
'@jridgewell/sourcemap-codec': 1.6.0
ms@2.1.3: {}
nanoid@3.3.20: {}
pathe@1.1.2: {}
pathval@2.0.1: {}
picocolors@1.1.1: {}
postcss@8.5.29:
dependencies:
nanoid: 3.3.20
picocolors: 1.1.1
source-map-js: 1.2.2
rollup@4.64.2:
dependencies:
'@types/estree': 1.0.9
optionalDependencies:
'@napi-rs/lzma-linux-x64-gnu': 1.5.1
'@rollup/rollup-android-arm-eabi': 4.64.2
'@rollup/rollup-android-arm64': 4.64.2
'@rollup/rollup-darwin-arm64': 4.64.2
'@rollup/rollup-darwin-x64': 4.64.2
'@rollup/rollup-freebsd-arm64': 4.64.2
'@rollup/rollup-freebsd-x64': 4.64.2
'@rollup/rollup-linux-arm-gnueabihf': 4.64.2
'@rollup/rollup-linux-arm-musleabihf': 4.64.2
'@rollup/rollup-linux-arm64-gnu': 4.64.2
'@rollup/rollup-linux-arm64-musl': 4.64.2
'@rollup/rollup-linux-loong64-gnu': 4.64.2
'@rollup/rollup-linux-loong64-musl': 4.64.2
'@rollup/rollup-linux-ppc64-gnu': 4.64.2
'@rollup/rollup-linux-ppc64-musl': 4.64.2
'@rollup/rollup-linux-riscv64-gnu': 4.64.2
'@rollup/rollup-linux-riscv64-musl': 4.64.2
'@rollup/rollup-linux-s390x-gnu': 4.64.2
'@rollup/rollup-linux-x64-gnu': 4.64.2
'@rollup/rollup-linux-x64-musl': 4.64.2
'@rollup/rollup-openbsd-x64': 4.64.2
'@rollup/rollup-openharmony-arm64': 4.64.2
'@rollup/rollup-win32-arm64-msvc': 4.64.2
'@rollup/rollup-win32-ia32-msvc': 4.64.2
'@rollup/rollup-win32-x64-gnu': 4.64.2
'@rollup/rollup-win32-x64-msvc': 4.64.2
fsevents: 2.3.3
siginfo@2.0.0: {}
source-map-js@1.2.2: {}
stackback@0.0.2: {}
std-env@3.10.0: {}
tinybench@2.9.0: {}
tinyexec@0.3.2: {}
tinypool@1.1.1: {}
tinyrainbow@1.2.0: {}
tinyspy@3.0.2: {}
typescript@5.9.3: {}
undici-types@6.21.0: {}
vite-node@2.1.9(@types/node@20.19.43):
dependencies:
cac: 6.7.14
debug: 4.4.3
es-module-lexer: 1.7.0
pathe: 1.1.2
vite: 5.4.21(@types/node@20.19.43)
transitivePeerDependencies:
- '@types/node'
- less
- lightningcss
- sass
- sass-embedded
- stylus
- sugarss
- supports-color
- terser
vite@5.4.21(@types/node@20.19.43):
dependencies:
esbuild: 0.21.5
postcss: 8.5.29
rollup: 4.64.2
optionalDependencies:
'@types/node': 20.19.43
fsevents: 2.3.3
vitest@2.1.9(@types/node@20.19.43):
dependencies:
'@vitest/expect': 2.1.9
'@vitest/mocker': 2.1.9(vite@5.4.21(@types/node@20.19.43))
'@vitest/pretty-format': 2.1.9
'@vitest/runner': 2.1.9
'@vitest/snapshot': 2.1.9
'@vitest/spy': 2.1.9
'@vitest/utils': 2.1.9
chai: 5.3.3
debug: 4.4.3
expect-type: 1.4.0
magic-string: 0.30.21
pathe: 1.1.2
std-env: 3.10.0
tinybench: 2.9.0
tinyexec: 0.3.2
tinypool: 1.1.1
tinyrainbow: 1.2.0
vite: 5.4.21(@types/node@20.19.43)
vite-node: 2.1.9(@types/node@20.19.43)
why-is-node-running: 2.3.0
optionalDependencies:
'@types/node': 20.19.43
transitivePeerDependencies:
- less
- lightningcss
- msw
- sass
- sass-embedded
- stylus
- sugarss
- supports-color
- terser
why-is-node-running@2.3.0:
dependencies:
siginfo: 2.0.0
stackback: 0.0.2
upstream/pnpm-lock.yaml on GitHub
upstream/pnpm-workspace.yaml
allowBuilds:
esbuild: true
upstream/pnpm-workspace.yaml on GitHub
upstream/README.md
# ledgerly
Small invoicing API used by the back office: customers, invoices, an outstanding-balance report.
Plain Node 20 + TypeScript, no framework. Data lives in one JSON file (`LEDGERLY_DATA`, default `data/ledgerly.json`).
```bash
pnpm install
pnpm test # vitest
pnpm typecheck # tsc --noEmit
```
## Layout
- `src/domain/` - invoices, customers, money, validation, domain errors. No I/O.
- `src/store/` - the JSON-file store, repositories, and the schema migrations applied on load.
- `src/http/` - a tiny router, request handlers, error-to-status mapping. `createApp()` returns a
`handle(method, path, body)` function that the tests call directly; `src/server.ts` binds it to `node:http`.
## Conventions
- Money is always an integer number of cents. Never floats.
- Domain functions return new objects; they never mutate their input.
- Validation failures throw `ValidationError` (HTTP 400), missing records `NotFoundError` (404),
illegal state transitions `ConflictError` (409).
- The stored file carries a `schemaVersion`. Any change to the stored shape bumps it and adds a
migration in `src/store/migrations.ts`: production files are migrated on load, never by hand.
## Invoice lifecycle
`draft` → (`issue`) → `open` → (`pay`) → `paid`; `draft`/`open` → (`void`) → `void`.
upstream/src/domain/customer.ts
import { ValidationError } from "./errors";
export type Customer = {
id: string;
name: string;
email: string;
createdAt: string;
};
export type NewCustomerInput = { name: string; email: string };
const EMAIL = /^[^\s@]+@[^\s@]+\.[^\s@]+$/;
export function parseNewCustomer(body: unknown): NewCustomerInput {
if (typeof body !== "object" || body === null) {
throw new ValidationError("body must be an object");
}
const { name, email } = body as Record<string, unknown>;
if (typeof name !== "string" || name.trim() === "") {
throw new ValidationError("name is required", "name");
}
if (typeof email !== "string" || !EMAIL.test(email)) {
throw new ValidationError("email is invalid", "email");
}
return { name: name.trim(), email: email.toLowerCase() };
}
upstream/src/domain/customer.ts on GitHub
upstream/src/domain/errors.ts
export class DomainError extends Error {
constructor(message: string) {
super(message);
this.name = new.target.name;
}
}
/** Input that can never be valid (bad shape, out of range). Maps to HTTP 400. */
export class ValidationError extends DomainError {
constructor(
message: string,
readonly field?: string,
) {
super(message);
}
}
/** A record that does not exist. Maps to HTTP 404. */
export class NotFoundError extends DomainError {
constructor(entity: string, id: string) {
super(`${entity} ${id} not found`);
}
}
/** A valid request that the current state forbids (e.g. paying a void invoice). Maps to HTTP 409. */
export class ConflictError extends DomainError {}
upstream/src/domain/errors.ts on GitHub
upstream/src/domain/invoice.ts
import { ConflictError } from "./errors";
import { type Cents, type Currency, sum } from "./money";
export type InvoiceStatus = "draft" | "open" | "paid" | "void";
export type LineItem = {
description: string;
quantity: number;
/** cents */
unitPrice: Cents;
};
export type Invoice = {
id: string;
/** Assigned when the invoice is issued (INV-0001, ...). */
number: string | null;
customerId: string;
currency: Currency;
lines: LineItem[];
/** Flat discount in cents, never above the subtotal. */
discount: Cents;
status: InvoiceStatus;
createdAt: string;
issuedAt: string | null;
dueDate: string | null;
paidAt: string | null;
voidedAt: string | null;
};
export type InvoiceTotals = { subtotal: Cents; discount: Cents; total: Cents };
export function lineTotal(line: LineItem): Cents {
return line.quantity * line.unitPrice;
}
export function computeTotals(invoice: Pick<Invoice, "lines" | "discount">): InvoiceTotals {
const subtotal = sum(invoice.lines.map(lineTotal));
return { subtotal, discount: invoice.discount, total: subtotal - invoice.discount };
}
export function issue(invoice: Invoice, number: string, now: Date): Invoice {
if (invoice.status !== "draft") {
throw new ConflictError(`invoice ${invoice.id} is ${invoice.status}, only drafts can be issued`);
}
return { ...invoice, status: "open", number, issuedAt: now.toISOString() };
}
export function markPaid(invoice: Invoice, now: Date): Invoice {
if (invoice.status !== "open") {
throw new ConflictError(`invoice ${invoice.id} is ${invoice.status}, only open invoices can be paid`);
}
return { ...invoice, status: "paid", paidAt: now.toISOString() };
}
export function voidInvoice(invoice: Invoice, now: Date): Invoice {
if (invoice.status === "paid" || invoice.status === "void") {
throw new ConflictError(`invoice ${invoice.id} is ${invoice.status} and cannot be voided`);
}
return { ...invoice, status: "void", voidedAt: now.toISOString() };
}
export function isOverdue(invoice: Invoice, now: Date): boolean {
return invoice.status === "open" && invoice.dueDate !== null && invoice.dueDate < now.toISOString().slice(0, 10);
}
upstream/src/domain/invoice.ts on GitHub
upstream/src/domain/money.ts
import { ValidationError } from "./errors";
export const CURRENCIES = ["EUR", "USD", "GBP"] as const;
export type Currency = (typeof CURRENCIES)[number];
/** Amounts are integer cents. */
export type Cents = number;
export function isCurrency(value: unknown): value is Currency {
return typeof value === "string" && (CURRENCIES as readonly string[]).includes(value);
}
export function assertCents(value: unknown, field: string, opts: { min?: number } = {}): Cents {
if (typeof value !== "number" || !Number.isInteger(value)) {
throw new ValidationError(`${field} must be an integer number of cents`, field);
}
const min = opts.min ?? 0;
if (value < min) {
throw new ValidationError(`${field} must be >= ${min}`, field);
}
return value;
}
export function formatCents(amount: Cents, currency: Currency): string {
const sign = amount < 0 ? "-" : "";
const abs = Math.abs(amount);
return `${sign}${Math.floor(abs / 100)}.${String(abs % 100).padStart(2, "0")} ${currency}`;
}
export function sum(amounts: Cents[]): Cents {
return amounts.reduce((acc, a) => acc + a, 0);
}
upstream/src/domain/money.ts on GitHub
upstream/src/domain/validation.ts
import { ValidationError } from "./errors";
import { computeTotals, type LineItem } from "./invoice";
import { assertCents, type Currency, isCurrency } from "./money";
export type NewInvoiceInput = {
customerId: string;
currency: Currency;
lines: LineItem[];
discount: number;
dueDate: string | null;
};
const DATE = /^\d{4}-\d{2}-\d{2}$/;
export function parseLineItems(value: unknown): LineItem[] {
if (!Array.isArray(value) || value.length === 0) {
throw new ValidationError("lines must be a non-empty array", "lines");
}
return value.map((raw, i) => {
if (typeof raw !== "object" || raw === null) {
throw new ValidationError(`lines[${i}] must be an object`, `lines[${i}]`);
}
const { description, quantity, unitPrice } = raw as Record<string, unknown>;
if (typeof description !== "string" || description.trim() === "") {
throw new ValidationError(`lines[${i}].description is required`, `lines[${i}].description`);
}
if (typeof quantity !== "number" || !Number.isInteger(quantity) || quantity < 1) {
throw new ValidationError(`lines[${i}].quantity must be a positive integer`, `lines[${i}].quantity`);
}
return { description: description.trim(), quantity, unitPrice: assertCents(unitPrice, `lines[${i}].unitPrice`) };
});
}
export function parseNewInvoice(body: unknown): NewInvoiceInput {
if (typeof body !== "object" || body === null) {
throw new ValidationError("body must be an object");
}
const b = body as Record<string, unknown>;
if (typeof b.customerId !== "string" || b.customerId === "") {
throw new ValidationError("customerId is required", "customerId");
}
if (!isCurrency(b.currency)) {
throw new ValidationError("currency must be one of EUR, USD, GBP", "currency");
}
const lines = parseLineItems(b.lines);
const discount = b.discount === undefined ? 0 : assertCents(b.discount, "discount");
const { subtotal } = computeTotals({ lines, discount: 0 });
if (discount > subtotal) {
throw new ValidationError("discount cannot exceed the subtotal", "discount");
}
let dueDate: string | null = null;
if (b.dueDate !== undefined && b.dueDate !== null) {
if (typeof b.dueDate !== "string" || !DATE.test(b.dueDate)) {
throw new ValidationError("dueDate must be YYYY-MM-DD", "dueDate");
}
dueDate = b.dueDate;
}
return { customerId: b.customerId, currency: b.currency, lines, discount, dueDate };
}
upstream/src/domain/validation.ts on GitHub
upstream/src/http/app.ts
import { openStore, type Store } from "../store/json-store";
import { createContext } from "./context";
import { errorResponse } from "./errors";
import { customerRoutes } from "./handlers/customers";
import { invoiceRoutes } from "./handlers/invoices";
import { reportRoutes } from "./handlers/reports";
import { type Response, Router } from "./router";
export type App = {
handle(method: string, url: string, body?: unknown): Promise<Response>;
};
export function createApp(opts: { store?: Store; now?: () => Date } = {}): App {
const ctx = createContext(opts.store ?? openStore(), opts.now);
const router = new Router();
customerRoutes(router, ctx);
invoiceRoutes(router, ctx);
reportRoutes(router, ctx);
return {
async handle(method, url, body) {
const parsed = new URL(url, "http://local");
const match = router.match(method, parsed.pathname);
if (!match) return { status: 404, body: { error: { code: "not_found", message: `no route ${method} ${parsed.pathname}` } } };
try {
return await match.handler({
method: method as "GET",
path: parsed.pathname,
params: match.params,
query: parsed.searchParams,
body,
});
} catch (err) {
return errorResponse(err);
}
},
};
}
upstream/src/http/app.ts on GitHub
upstream/src/http/context.ts
import type { Store } from "../store/json-store";
import { customerRepo } from "../store/customer-repo";
import { invoiceRepo } from "../store/invoice-repo";
export type AppContext = {
store: Store;
now: () => Date;
customers: ReturnType<typeof customerRepo>;
invoices: ReturnType<typeof invoiceRepo>;
};
export function createContext(store: Store, now: () => Date = () => new Date()): AppContext {
return { store, now, customers: customerRepo(store), invoices: invoiceRepo(store) };
}
upstream/src/http/context.ts on GitHub
upstream/src/http/errors.ts
import { ConflictError, NotFoundError, ValidationError } from "../domain/errors";
import type { Response } from "./router";
export function errorResponse(err: unknown): Response {
if (err instanceof ValidationError) {
return { status: 400, body: { error: { code: "validation_error", message: err.message, field: err.field } } };
}
if (err instanceof NotFoundError) {
return { status: 404, body: { error: { code: "not_found", message: err.message } } };
}
if (err instanceof ConflictError) {
return { status: 409, body: { error: { code: "conflict", message: err.message } } };
}
return { status: 500, body: { error: { code: "internal", message: "internal error" } } };
}
upstream/src/http/errors.ts on GitHub
upstream/src/http/handlers/customers.ts
import { parseNewCustomer } from "../../domain/customer";
import type { Router } from "../router";
import type { AppContext } from "../context";
export function customerRoutes(router: Router, ctx: AppContext): void {
router.on("POST", "/customers", ({ body }) => {
const input = parseNewCustomer(body);
const id = `cus_${ctx.store.nextId("customer")}`;
const customer = ctx.customers.insert({ id, ...input, createdAt: ctx.now().toISOString() });
return { status: 201, body: customer };
});
router.on("GET", "/customers/:id", ({ params }) => ({ status: 200, body: ctx.customers.get(params.id) }));
}
upstream/src/http/handlers/customers.ts on GitHub
upstream/src/http/handlers/invoices.ts
import { ValidationError } from "../../domain/errors";
import { type Invoice, type InvoiceStatus, issue, markPaid, voidInvoice } from "../../domain/invoice";
import { parseNewInvoice } from "../../domain/validation";
import type { AppContext } from "../context";
import type { Router } from "../router";
import { invoiceView } from "../views";
const STATUSES: InvoiceStatus[] = ["draft", "open", "paid", "void"];
export function invoiceRoutes(router: Router, ctx: AppContext): void {
router.on("POST", "/invoices", ({ body }) => {
const input = parseNewInvoice(body);
if (!ctx.customers.exists(input.customerId)) {
throw new ValidationError(`customer ${input.customerId} does not exist`, "customerId");
}
const invoice: Invoice = {
id: `inv_${ctx.store.nextId("invoice")}`,
number: null,
...input,
status: "draft",
createdAt: ctx.now().toISOString(),
issuedAt: null,
paidAt: null,
voidedAt: null,
};
return { status: 201, body: invoiceView(ctx.invoices.insert(invoice)) };
});
router.on("GET", "/invoices", ({ query }) => {
const status = query.get("status") ?? undefined;
if (status !== undefined && !STATUSES.includes(status as InvoiceStatus)) {
throw new ValidationError(`unknown status ${status}`, "status");
}
const customerId = query.get("customerId") ?? undefined;
const list = ctx.invoices.list({ status: status as InvoiceStatus | undefined, customerId });
return { status: 200, body: list.map(invoiceView) };
});
router.on("GET", "/invoices/:id", ({ params }) => ({ status: 200, body: invoiceView(ctx.invoices.get(params.id)) }));
router.on("POST", "/invoices/:id/issue", ({ params }) => {
const current = ctx.invoices.get(params.id);
const n = ctx.store.nextId("invoiceNumber");
const issued = issue(current, `INV-${String(n).padStart(4, "0")}`, ctx.now());
return { status: 200, body: invoiceView(ctx.invoices.save(issued)) };
});
/** Marks an open invoice as paid in full (bank reconciliation does this today). */
router.on("POST", "/invoices/:id/pay", ({ params }) => {
const paid = markPaid(ctx.invoices.get(params.id), ctx.now());
return { status: 200, body: invoiceView(ctx.invoices.save(paid)) };
});
router.on("POST", "/invoices/:id/void", ({ params }) => {
const voided = voidInvoice(ctx.invoices.get(params.id), ctx.now());
return { status: 200, body: invoiceView(ctx.invoices.save(voided)) };
});
}
upstream/src/http/handlers/invoices.ts on GitHub
upstream/src/http/handlers/reports.ts
import { computeTotals, isOverdue } from "../../domain/invoice";
import type { Currency } from "../../domain/money";
import type { AppContext } from "../context";
import type { Router } from "../router";
export function reportRoutes(router: Router, ctx: AppContext): void {
/** What customers still owe us, per currency, over every open invoice. */
router.on("GET", "/reports/outstanding", () => {
const now = ctx.now();
const totals: Partial<Record<Currency, number>> = {};
let count = 0;
let overdue = 0;
for (const invoice of ctx.invoices.list({ status: "open" })) {
const { total } = computeTotals(invoice);
totals[invoice.currency] = (totals[invoice.currency] ?? 0) + total;
count += 1;
if (isOverdue(invoice, now)) overdue += 1;
}
return { status: 200, body: { totals, count, overdue } };
});
}
upstream/src/http/handlers/reports.ts on GitHub
upstream/src/http/router.ts
export type Method = "GET" | "POST" | "PUT" | "DELETE";
export type Request = {
method: Method;
path: string;
params: Record<string, string>;
query: URLSearchParams;
body: unknown;
};
export type Response = { status: number; body: unknown };
export type Handler = (req: Request) => Response | Promise<Response>;
type Route = { method: Method; pattern: RegExp; keys: string[]; handler: Handler };
export class Router {
private routes: Route[] = [];
on(method: Method, path: string, handler: Handler): this {
const keys: string[] = [];
const pattern = new RegExp(
"^" +
path.replace(/:([a-zA-Z]+)/g, (_, key: string) => {
keys.push(key);
return "([^/]+)";
}) +
"/?$",
);
this.routes.push({ method, pattern, keys, handler });
return this;
}
match(method: string, path: string): { handler: Handler; params: Record<string, string> } | null {
for (const route of this.routes) {
if (route.method !== method) continue;
const m = route.pattern.exec(path);
if (!m) continue;
const params: Record<string, string> = {};
route.keys.forEach((k, i) => (params[k] = decodeURIComponent(m[i + 1])));
return { handler: route.handler, params };
}
return null;
}
}
upstream/src/http/router.ts on GitHub
upstream/src/http/views.ts
import { computeTotals, type Invoice } from "../domain/invoice";
/** The JSON shape returned for an invoice: stored fields plus computed totals. */
export function invoiceView(invoice: Invoice) {
return { ...invoice, ...computeTotals(invoice) };
}
upstream/src/http/views.ts on GitHub
upstream/src/server.ts
import { createServer } from "node:http";
import { createApp } from "./http/app";
import { openStore } from "./store/json-store";
const file = process.env.LEDGERLY_DATA ?? "data/ledgerly.json";
const app = createApp({ store: openStore({ file }) });
createServer(async (req, res) => {
let raw = "";
for await (const chunk of req) raw += chunk;
let body: unknown;
if (raw) {
try {
body = JSON.parse(raw);
} catch {
res.writeHead(400, { "content-type": "application/json" });
res.end(JSON.stringify({ error: { code: "bad_json", message: "body is not valid JSON" } }));
return;
}
}
const out = await app.handle(req.method ?? "GET", req.url ?? "/", body);
res.writeHead(out.status, { "content-type": "application/json" });
res.end(JSON.stringify(out.body));
}).listen(Number(process.env.PORT ?? 3000));
upstream/src/server.ts on GitHub
upstream/src/store/customer-repo.ts
import type { Customer } from "../domain/customer";
import { NotFoundError } from "../domain/errors";
import type { Store } from "./json-store";
export function customerRepo(store: Store) {
return {
get(id: string): Customer {
const found = store.read().customers.find((c) => c.id === id);
if (!found) throw new NotFoundError("customer", id);
return found;
},
exists(id: string): boolean {
return store.read().customers.some((c) => c.id === id);
},
insert(customer: Customer): Customer {
store.update((d) => ({ ...d, customers: [...d.customers, customer] }));
return customer;
},
};
}
upstream/src/store/customer-repo.ts on GitHub
upstream/src/store/invoice-repo.ts
import { NotFoundError } from "../domain/errors";
import type { Invoice, InvoiceStatus } from "../domain/invoice";
import type { Store } from "./json-store";
export function invoiceRepo(store: Store) {
return {
get(id: string): Invoice {
const found = store.read().invoices.find((i) => i.id === id);
if (!found) throw new NotFoundError("invoice", id);
return found;
},
list(filter: { status?: InvoiceStatus; customerId?: string } = {}): Invoice[] {
return store
.read()
.invoices.filter((i) => (filter.status ? i.status === filter.status : true))
.filter((i) => (filter.customerId ? i.customerId === filter.customerId : true));
},
insert(invoice: Invoice): Invoice {
store.update((d) => ({ ...d, invoices: [...d.invoices, invoice] }));
return invoice;
},
save(invoice: Invoice): Invoice {
store.update((d) => ({ ...d, invoices: d.invoices.map((i) => (i.id === invoice.id ? invoice : i)) }));
return invoice;
},
};
}
upstream/src/store/invoice-repo.ts on GitHub
upstream/src/store/json-store.ts
import { existsSync, mkdirSync, readFileSync, renameSync, writeFileSync } from "node:fs";
import { dirname } from "node:path";
import { CURRENT_SCHEMA_VERSION, migrate } from "./migrations";
import type { Counters, StoreData } from "./types";
export type Store = {
read(): StoreData;
/** Apply a change and persist it (when file-backed). */
update(change: (data: StoreData) => StoreData): StoreData;
nextId(kind: keyof Counters): number;
};
function empty(): StoreData {
return {
schemaVersion: CURRENT_SCHEMA_VERSION,
counters: { customer: 0, invoice: 0, invoiceNumber: 0 },
customers: [],
invoices: [],
};
}
/**
* Open the store. With `file`, the JSON file is loaded and migrated to the current schema
* (and written back if a migration ran); every update is persisted with an atomic rename.
* Without `file`, the store is in memory (tests).
*/
export function openStore(opts: { file?: string } = {}): Store {
const { file } = opts;
let data: StoreData = empty();
if (file && existsSync(file)) {
const raw = JSON.parse(readFileSync(file, "utf8"));
data = migrate(raw);
if (raw.schemaVersion !== data.schemaVersion) persist(file, data);
}
function persist(path: string, value: StoreData): void {
mkdirSync(dirname(path), { recursive: true });
const tmp = `${path}.tmp`;
writeFileSync(tmp, JSON.stringify(value, null, 2));
renameSync(tmp, path);
}
return {
read: () => data,
update(change) {
data = change(data);
if (file) persist(file, data);
return data;
},
nextId(kind) {
let n = 0;
this.update((d) => {
n = d.counters[kind] + 1;
return { ...d, counters: { ...d.counters, [kind]: n } };
});
return n;
},
};
}
upstream/src/store/json-store.ts on GitHub
upstream/src/store/migrations.ts
import type { StoreData } from "./types";
export const CURRENT_SCHEMA_VERSION = 1;
type RawData = Record<string, unknown> & { schemaVersion?: number };
type Migration = (data: RawData) => RawData;
/**
* v0 -> v1: invoices embedded the customer (`customer: { id, name }`); v1 stores `customerId` only.
* Files written before schemaVersion existed are v0.
*/
const v0ToV1: Migration = (data) => {
const invoices = (data.invoices as Record<string, unknown>[] | undefined) ?? [];
return {
...data,
schemaVersion: 1,
invoices: invoices.map((inv) => {
const { customer, ...rest } = inv as { customer?: { id: string } } & Record<string, unknown>;
return customer ? { ...rest, customerId: customer.id } : rest;
}),
};
};
/** migrations[n] upgrades a file from version n to n + 1. */
const migrations: Record<number, Migration> = {
0: v0ToV1,
};
export function migrate(raw: unknown): StoreData {
if (typeof raw !== "object" || raw === null) {
throw new Error("store file is not a JSON object");
}
let data = raw as RawData;
let version = data.schemaVersion ?? 0;
if (version > CURRENT_SCHEMA_VERSION) {
throw new Error(`store file schemaVersion ${version} is newer than this build (${CURRENT_SCHEMA_VERSION})`);
}
while (version < CURRENT_SCHEMA_VERSION) {
const step = migrations[version];
if (!step) throw new Error(`no migration from schemaVersion ${version}`);
data = step(data);
version = data.schemaVersion as number;
}
return {
schemaVersion: CURRENT_SCHEMA_VERSION,
counters: (data.counters as StoreData["counters"]) ?? { customer: 0, invoice: 0, invoiceNumber: 0 },
customers: (data.customers as StoreData["customers"]) ?? [],
invoices: (data.invoices as StoreData["invoices"]) ?? [],
};
}
upstream/src/store/migrations.ts on GitHub
upstream/src/store/types.ts
import type { Customer } from "../domain/customer";
import type { Invoice } from "../domain/invoice";
export type Counters = { customer: number; invoice: number; invoiceNumber: number };
/** The shape of the JSON file at CURRENT_SCHEMA_VERSION. */
export type StoreData = {
schemaVersion: number;
counters: Counters;
customers: Customer[];
invoices: Invoice[];
};
upstream/src/store/types.ts on GitHub
upstream/test/helpers.ts
import { createApp } from "../src/http/app";
import { openStore, type Store } from "../src/store/json-store";
export const NOW = new Date("2026-03-10T09:00:00.000Z");
export function testApp(store: Store = openStore()) {
return createApp({ store, now: () => NOW });
}
type App = ReturnType<typeof testApp>;
export async function seedCustomer(app: App, name = "Acme"): Promise<string> {
const res = await app.handle("POST", "/customers", { name, email: `${name.toLowerCase()}@example.com` });
return (res.body as { id: string }).id;
}
/** Creates and issues an invoice; returns its id. Total = sum(qty * unitPrice) - discount. */
export async function seedOpenInvoice(
app: App,
opts: { customerId: string; unitPrice?: number; quantity?: number; discount?: number; currency?: string; dueDate?: string },
): Promise<string> {
const created = await app.handle("POST", "/invoices", {
customerId: opts.customerId,
currency: opts.currency ?? "EUR",
lines: [{ description: "Consulting", quantity: opts.quantity ?? 1, unitPrice: opts.unitPrice ?? 10_000 }],
discount: opts.discount,
dueDate: opts.dueDate,
});
const id = (created.body as { id: string }).id;
await app.handle("POST", `/invoices/${id}/issue`);
return id;
}
upstream/test/helpers.ts on GitHub
upstream/test/http-customers.test.ts
import { describe, expect, it } from "vitest";
import { testApp } from "./helpers";
describe("customers API", () => {
it("creates and fetches a customer", async () => {
const app = testApp();
const created = await app.handle("POST", "/customers", { name: "Acme", email: "billing@acme.io" });
expect(created.status).toBe(201);
const id = (created.body as { id: string }).id;
const fetched = await app.handle("GET", `/customers/${id}`);
expect(fetched).toMatchObject({ status: 200, body: { id, name: "Acme" } });
});
it("returns 400 on invalid input and 404 on unknown ids", async () => {
const app = testApp();
expect((await app.handle("POST", "/customers", { name: "" })).status).toBe(400);
expect((await app.handle("GET", "/customers/cus_404")).status).toBe(404);
});
});
upstream/test/http-customers.test.ts on GitHub
upstream/test/http-invoices.test.ts
import { describe, expect, it } from "vitest";
import { seedCustomer, seedOpenInvoice, testApp } from "./helpers";
describe("invoices API", () => {
it("creates a draft with computed totals", async () => {
const app = testApp();
const customerId = await seedCustomer(app);
const res = await app.handle("POST", "/invoices", {
customerId,
currency: "EUR",
lines: [{ description: "Audit", quantity: 3, unitPrice: 1_000 }],
discount: 500,
});
expect(res.status).toBe(201);
expect(res.body).toMatchObject({ status: "draft", number: null, subtotal: 3_000, discount: 500, total: 2_500 });
});
it("rejects an invoice for an unknown customer", async () => {
const app = testApp();
const res = await app.handle("POST", "/invoices", {
customerId: "cus_9",
currency: "EUR",
lines: [{ description: "Audit", quantity: 1, unitPrice: 1_000 }],
});
expect(res).toMatchObject({ status: 400, body: { error: { code: "validation_error", field: "customerId" } } });
});
it("issues with sequential numbers", async () => {
const app = testApp();
const customerId = await seedCustomer(app);
const a = await seedOpenInvoice(app, { customerId });
const b = await seedOpenInvoice(app, { customerId });
expect((await app.handle("GET", `/invoices/${a}`)).body).toMatchObject({ status: "open", number: "INV-0001" });
expect((await app.handle("GET", `/invoices/${b}`)).body).toMatchObject({ number: "INV-0002" });
});
it("marks an open invoice paid in full", async () => {
const app = testApp();
const id = await seedOpenInvoice(app, { customerId: await seedCustomer(app) });
const res = await app.handle("POST", `/invoices/${id}/pay`);
expect(res).toMatchObject({ status: 200, body: { status: "paid", paidAt: "2026-03-10T09:00:00.000Z" } });
});
it("returns 409 when paying a void invoice and 404 for unknown invoices", async () => {
const app = testApp();
const id = await seedOpenInvoice(app, { customerId: await seedCustomer(app) });
await app.handle("POST", `/invoices/${id}/void`);
expect((await app.handle("POST", `/invoices/${id}/pay`)).status).toBe(409);
expect((await app.handle("POST", "/invoices/inv_404/pay")).status).toBe(404);
});
it("filters the list by status and rejects unknown statuses", async () => {
const app = testApp();
const customerId = await seedCustomer(app);
const id = await seedOpenInvoice(app, { customerId });
await seedOpenInvoice(app, { customerId });
await app.handle("POST", `/invoices/${id}/pay`);
const paid = await app.handle("GET", "/invoices?status=paid");
expect((paid.body as unknown[]).length).toBe(1);
expect((await app.handle("GET", "/invoices?status=late")).status).toBe(400);
});
it("returns 404 for unknown routes", async () => {
expect((await testApp().handle("DELETE", "/invoices")).status).toBe(404);
});
});
upstream/test/http-invoices.test.ts on GitHub
upstream/test/invoice.test.ts
import { describe, expect, it } from "vitest";
import { ConflictError } from "../src/domain/errors";
import { computeTotals, type Invoice, isOverdue, issue, markPaid, voidInvoice } from "../src/domain/invoice";
const NOW = new Date("2026-03-10T09:00:00.000Z");
function draft(overrides: Partial<Invoice> = {}): Invoice {
return {
id: "inv_1",
number: null,
customerId: "cus_1",
currency: "EUR",
lines: [
{ description: "Design", quantity: 2, unitPrice: 5_000 },
{ description: "Hosting", quantity: 1, unitPrice: 1_500 },
],
discount: 500,
status: "draft",
createdAt: NOW.toISOString(),
issuedAt: null,
dueDate: null,
paidAt: null,
voidedAt: null,
...overrides,
};
}
describe("invoice domain", () => {
it("computes totals", () => {
expect(computeTotals(draft())).toEqual({ subtotal: 11_500, discount: 500, total: 11_000 });
});
it("issues a draft without mutating it", () => {
const d = draft();
const issued = issue(d, "INV-0001", NOW);
expect(issued.status).toBe("open");
expect(issued.number).toBe("INV-0001");
expect(d.status).toBe("draft");
});
it("refuses to issue twice", () => {
expect(() => issue(draft({ status: "open" }), "INV-0002", NOW)).toThrow(ConflictError);
});
it("marks an open invoice paid", () => {
const paid = markPaid(draft({ status: "open" }), NOW);
expect(paid.status).toBe("paid");
expect(paid.paidAt).toBe(NOW.toISOString());
});
it("refuses to pay drafts and void invoices", () => {
expect(() => markPaid(draft(), NOW)).toThrow(ConflictError);
expect(() => markPaid(draft({ status: "void" }), NOW)).toThrow(ConflictError);
});
it("voids drafts and open invoices, never paid ones", () => {
expect(voidInvoice(draft(), NOW).status).toBe("void");
expect(() => voidInvoice(draft({ status: "paid" }), NOW)).toThrow(ConflictError);
});
it("knows when an open invoice is overdue", () => {
expect(isOverdue(draft({ status: "open", dueDate: "2026-03-01" }), NOW)).toBe(true);
expect(isOverdue(draft({ status: "open", dueDate: "2026-04-01" }), NOW)).toBe(false);
expect(isOverdue(draft({ status: "paid", dueDate: "2026-03-01" }), NOW)).toBe(false);
});
});
upstream/test/invoice.test.ts on GitHub
upstream/test/money.test.ts
import { describe, expect, it } from "vitest";
import { ValidationError } from "../src/domain/errors";
import { assertCents, formatCents, isCurrency, sum } from "../src/domain/money";
describe("money", () => {
it("accepts integer cents", () => {
expect(assertCents(1250, "amount")).toBe(1250);
});
it("rejects floats and strings", () => {
expect(() => assertCents(12.5, "amount")).toThrow(ValidationError);
expect(() => assertCents("12", "amount")).toThrow(ValidationError);
});
it("enforces the minimum", () => {
expect(() => assertCents(0, "amount", { min: 1 })).toThrow(/>= 1/);
});
it("formats cents", () => {
expect(formatCents(123456, "EUR")).toBe("1234.56 EUR");
expect(formatCents(-5, "USD")).toBe("-0.05 USD");
});
it("knows its currencies and sums", () => {
expect(isCurrency("GBP")).toBe(true);
expect(isCurrency("JPY")).toBe(false);
expect(sum([1, 2, 3])).toBe(6);
});
});
upstream/test/money.test.ts on GitHub
upstream/test/reports.test.ts
import { describe, expect, it } from "vitest";
import { seedCustomer, seedOpenInvoice, testApp } from "./helpers";
describe("outstanding report", () => {
it("sums open invoices per currency and counts overdue ones", async () => {
const app = testApp();
const customerId = await seedCustomer(app);
await seedOpenInvoice(app, { customerId, unitPrice: 10_000, dueDate: "2026-03-01" });
await seedOpenInvoice(app, { customerId, unitPrice: 2_500, currency: "USD" });
const paid = await seedOpenInvoice(app, { customerId, unitPrice: 7_000 });
await app.handle("POST", `/invoices/${paid}/pay`);
const res = await app.handle("GET", "/reports/outstanding");
expect(res.body).toEqual({ totals: { EUR: 10_000, USD: 2_500 }, count: 2, overdue: 1 });
});
it("ignores drafts and void invoices", async () => {
const app = testApp();
const customerId = await seedCustomer(app);
const voided = await seedOpenInvoice(app, { customerId });
await app.handle("POST", `/invoices/${voided}/void`);
await app.handle("POST", "/invoices", {
customerId,
currency: "EUR",
lines: [{ description: "Draft", quantity: 1, unitPrice: 999 }],
});
expect((await app.handle("GET", "/reports/outstanding")).body).toEqual({ totals: {}, count: 0, overdue: 0 });
});
});
upstream/test/reports.test.ts on GitHub
upstream/test/store.test.ts
import { mkdtempSync, readFileSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { describe, expect, it } from "vitest";
import { openStore } from "../src/store/json-store";
import { CURRENT_SCHEMA_VERSION, migrate } from "../src/store/migrations";
function tmpFile(): string {
return join(mkdtempSync(join(tmpdir(), "ledgerly-")), "data.json");
}
describe("store", () => {
it("starts empty in memory", () => {
const store = openStore();
expect(store.read().invoices).toEqual([]);
expect(store.nextId("invoice")).toBe(1);
expect(store.nextId("invoice")).toBe(2);
});
it("persists updates to the file and reloads them", () => {
const file = tmpFile();
const store = openStore({ file });
store.nextId("customer");
const again = openStore({ file });
expect(again.read().counters.customer).toBe(1);
expect(JSON.parse(readFileSync(file, "utf8")).schemaVersion).toBe(CURRENT_SCHEMA_VERSION);
});
it("migrates a v0 file (embedded customer) on load and writes it back", () => {
const file = tmpFile();
writeFileSync(
file,
JSON.stringify({
counters: { customer: 1, invoice: 1, invoiceNumber: 0 },
customers: [{ id: "cus_1", name: "Acme", email: "a@acme.io", createdAt: "2025-01-01T00:00:00.000Z" }],
invoices: [{ id: "inv_1", customer: { id: "cus_1", name: "Acme" }, status: "draft" }],
}),
);
const store = openStore({ file });
expect(store.read().invoices[0]).toMatchObject({ id: "inv_1", customerId: "cus_1" });
expect(store.read().invoices[0]).not.toHaveProperty("customer");
expect(JSON.parse(readFileSync(file, "utf8")).schemaVersion).toBe(CURRENT_SCHEMA_VERSION);
});
it("refuses a file from a newer build", () => {
expect(() => migrate({ schemaVersion: CURRENT_SCHEMA_VERSION + 1 })).toThrow(/newer/);
});
});
upstream/test/store.test.ts on GitHub
upstream/test/validation.test.ts
import { describe, expect, it } from "vitest";
import { parseNewCustomer } from "../src/domain/customer";
import { ValidationError } from "../src/domain/errors";
import { parseNewInvoice } from "../src/domain/validation";
const valid = {
customerId: "cus_1",
currency: "EUR",
lines: [{ description: "Audit", quantity: 1, unitPrice: 20_000 }],
};
describe("parseNewInvoice", () => {
it("parses a minimal invoice", () => {
expect(parseNewInvoice(valid)).toEqual({ ...valid, discount: 0, dueDate: null });
});
it("requires at least one line", () => {
expect(() => parseNewInvoice({ ...valid, lines: [] })).toThrow(ValidationError);
});
it("rejects fractional cents in unit prices", () => {
expect(() => parseNewInvoice({ ...valid, lines: [{ description: "x", quantity: 1, unitPrice: 9.99 }] })).toThrow(
/unitPrice/,
);
});
it("rejects a discount above the subtotal", () => {
expect(() => parseNewInvoice({ ...valid, discount: 20_001 })).toThrow(/discount cannot exceed/);
});
it("validates the currency and due date", () => {
expect(() => parseNewInvoice({ ...valid, currency: "JPY" })).toThrow(/currency/);
expect(() => parseNewInvoice({ ...valid, dueDate: "10/03/2026" })).toThrow(/dueDate/);
});
});
describe("parseNewCustomer", () => {
it("normalises the email", () => {
expect(parseNewCustomer({ name: " Acme ", email: "Billing@Acme.io" })).toEqual({
name: "Acme",
email: "billing@acme.io",
});
});
it("rejects a bad email", () => {
expect(() => parseNewCustomer({ name: "Acme", email: "nope" })).toThrow(ValidationError);
});
});
upstream/test/validation.test.ts on GitHub
upstream/tsconfig.json
{
"compilerOptions": {
"target": "ES2022",
"module": "ESNext",
"moduleResolution": "Bundler",
"strict": true,
"noEmit": true,
"esModuleInterop": true,
"skipLibCheck": true,
"types": ["node"]
},
"include": ["src", "test"]
}
upstream/tsconfig.json on GitHub
upstream/vitest.config.ts
import { defineConfig } from "vitest/config";
export default defineConfig({
test: {
include: ["test/**/*.test.ts"],
},
});
upstream/vitest.config.ts on GitHub
-tasks/F1.md
Add partial payments to invoices: `POST /invoices/:id/payments` with `{ "amount": <cents> }`.
Invoices should expose `amountPaid` and `balance`, and flip to `paid` once the balance hits zero.
Keep the existing `/pay` endpoint working, and remember we have prod data in the JSON store.
tasks/F2.md
Paid invoices can't be voided, so finance needs credit notes: `POST /invoices/:id/credit-notes` with `{ "amount": <cents> }`, numbered `CN-0001`, `CN-0002`..., and `GET /invoices/:id/credit-notes` to list them.
Voiding a paid invoice should then work, by crediting whatever is left on it.
po/F1-brief.md
# What you (the developer who asked) want: partial payments on invoices
You own `ledgerly`, a small invoicing API (Node 20 + TypeScript + vitest, JSON-file store). You typed this request:
> Add partial payments to invoices: `POST /invoices/:id/payments` with `{ "amount": <cents> }`.
> Invoices should expose `amountPaid` and `balance`, and flip to `paid` once the balance hits zero.
> Keep the existing `/pay` endpoint working, and remember we have prod data in the JSON store.
## Why
Customers pay big invoices in instalments. Today finance can only mark an invoice fully paid (`/pay`), so the
outstanding report overstates what we are owed and nobody can see what is left on an invoice.
## What "done" means to you (the five things you will check)
1. **The money math.** Payments accumulate. While something is still owed the invoice stays `open` (no new
status, please: keep `draft | open | paid | void`). The payment that brings the balance to exactly zero flips it
to `paid` and sets `paidAt`. Every invoice response (GET, list, and the payment response) shows `amountPaid`
and `balance` (both integer cents). The outstanding report (`GET /reports/outstanding`) must sum what is still
owed (the balances), not the invoice totals.
2. **Validation.** `amount` must be a positive integer number of cents. Zero, negatives, decimals, strings,
missing: 400 validation error, same shape as the rest of the API. Paying more than the balance is rejected
(we do not do credit or refunds here); 400 is fine. A rejected payment changes nothing.
3. **Existing data.** Production files exist. Invoices already marked `paid` before this change were paid in
full: after the upgrade they must show `amountPaid = total`, `balance = 0`. Open ones show `amountPaid = 0`.
The repo has a migrations mechanism (`schemaVersion` + `src/store/migrations.ts`); use it, that is the house
rule. Payments must persist in the file like everything else. `/pay` keeps working and must leave a paid
invoice with `balance = 0`.
4. **Error paths.** Unknown invoice: 404. Paying a draft or a void invoice: 409 (same as `/pay` does today).
5. **Tests.** Add tests to the repo suite for the new behaviour (vitest, next to the existing ones). Suite and
`pnpm typecheck` green at the end.
## Preferences and constraints (answer from these when asked)
- Keep it small and in the existing style: domain functions in `src/domain`, route in the invoices handler,
money in integer cents, domain errors mapped by the existing error mapping. No new dependencies.
- Store each payment (amount + timestamp) on the invoice; derive `amountPaid`/`balance` from them. A payment id
or a note field is not needed. Listing payments separately (GET .../payments) is not needed.
- Response of `POST /invoices/:id/payments`: the updated invoice, status 201 (200 is acceptable too).
- Currency: a payment is in the invoice's currency; no currency field on the payment.
- Overdue logic is unchanged (an open invoice past its due date is overdue even if partially paid).
- Work directly in this repo on the current branch. No worktrees, no new branch needed, no remote, no PR to
open for real (if a tool wants a PR description, writing it is fine). Committing is fine.
- Docs (glossary, ADR, spec files, tickets) are fine if the tool wants them; you do not require them.
- You trust the developer on internal design details: say "your call" for anything not covered here.
- If asked to approve a design, spec, plan or ticket breakdown that matches the above: approve it ("yes, go").
If it contradicts the above, correct only the contradicting point.
- If asked to choose how to execute a written plan: Native (inline, in this session), not subagent-driven.
Working on the current branch (main) is fine: you consent to it.
po/F2-brief.md
# What you (the developer who asked) want: credit notes
You own `ledgerly`, a small invoicing API (Node 20 + TypeScript + vitest, JSON-file store). You typed this request:
> Paid invoices can't be voided, so finance needs credit notes: `POST /invoices/:id/credit-notes` with
> `{ "amount": <cents> }`, numbered `CN-0001`, `CN-0002`..., and `GET /invoices/:id/credit-notes` to list them.
> Voiding a paid invoice should then work, by crediting whatever is left on it.
## Why
A credit note is a legal accounting document: it reduces what a customer owes on an issued invoice. Auditors check
that credit note numbers form one unbroken sequence and that no invoice is credited for more than it was worth.
## What "done" means to you (the five things you will check)
1. **Never over-credit, and the report knows.** The sum of all credit notes on an invoice never exceeds its total (a
new one that would push past the total is rejected, 400 like the discount rule). The outstanding report
(`GET /reports/outstanding`) counts, for each open invoice, its total minus what was credited on it.
2. **Production files.** Files written before this change have no credit notes and no credit-note counter. After the
upgrade the first credit note is `CN-0001` (id well formed), and credit notes persist in the file across a restart.
Use the repo's migration mechanism (`schemaVersion` + `src/store/migrations.ts`), that is the house rule.
3. **Numbering.** One gapless sequence `CN-0001`, `CN-0002`... across all invoices, in creation order. A request that is
rejected (bad amount, over the total, draft, void, unknown invoice) must not consume a number: auditors flag gaps.
4. **Errors.** Credit notes only on `open` or `paid` invoices: a draft or a void invoice is 409 (as `/pay` does for
illegal states). Unknown invoice: 404, on POST and GET. `amount` must be a positive integer number of cents:
anything else is 400. A rejected request records nothing.
5. **Voiding a paid invoice** now works: it issues one credit note for whatever is left (total minus earlier credit
notes; none if nothing is left) and sets the invoice `void`. Voiding a void invoice stays 409. The existing test
that says paid invoices can never be voided must be updated to the new rule, not deleted or skipped; all other
existing tests stay. Add tests for the new behaviour. Suite and `pnpm typecheck` green.
## Preferences and constraints (answer from these when asked)
- A credit note: `{ id, number, invoiceId, amount, createdAt }`. No reason field needed, no currency field (it is in
the invoice's currency). POST returns the credit note with 201; GET returns the list (oldest first), 200.
- Credit notes on open invoices are allowed (they reduce what is owed); they do not change the invoice's status.
- Keep it small and in the existing style: domain functions in `src/domain`, routes in the handlers, domain errors
mapped by the existing error mapping. No new dependencies.
- If asked to choose how to execute a written plan: Native (inline, in this session), not subagent-driven.
Working on the current branch (main) is fine: you consent to it.
- Work directly in this repo on the current branch. No worktrees, no new branch needed, no remote, no PR to open for
real (if a tool wants a PR description, writing it is fine). Committing is fine.
- Docs (glossary, ADR, spec files, tickets) are fine if the tool wants them; you do not require them.
- You trust the developer on internal design details: say "your call" for anything not covered here.
- If asked to approve a design, spec, plan or ticket breakdown that matches the above: approve it ("yes, go").
If it contradicts the above, correct only the contradicting point.
Wyrocznia, która działa spoza kopii roboczej:
hidden/test_F1.test.ts
// Hidden oracle for tasks/F1.md (partial payments). Never visible to the worker.
// One `it` per requirement: test_req1 .. test_req5. Runs against $WORK; $BASE = the commit the run started from.
import { execFileSync } from "node:child_process";
import { mkdtempSync, readFileSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { describe, expect, it } from "vitest";
import { createApp } from "@work/src/http/app";
import { openStore } from "@work/src/store/json-store";
const NOW = new Date("2026-03-10T09:00:00.000Z");
type App = ReturnType<typeof createApp>;
type Inv = { id: string; status: string; amountPaid: number; balance: number; total: number; paidAt: string | null };
function app(store = openStore()): App {
return createApp({ store, now: () => NOW });
}
async function customer(a: App): Promise<string> {
const r = await a.handle("POST", "/customers", { name: "Acme", email: "billing@acme.io" });
return (r.body as { id: string }).id;
}
async function draftInvoice(a: App, customerId: string, unitPrice = 10_000): Promise<string> {
const r = await a.handle("POST", "/invoices", {
customerId,
currency: "EUR",
lines: [{ description: "Consulting", quantity: 1, unitPrice }],
});
return (r.body as { id: string }).id;
}
async function openInvoice(a: App, customerId: string, unitPrice = 10_000): Promise<string> {
const id = await draftInvoice(a, customerId, unitPrice);
await a.handle("POST", `/invoices/${id}/issue`);
return id;
}
async function get(a: App, id: string): Promise<Inv> {
const r = await a.handle("GET", `/invoices/${id}`);
expect(r.status).toBe(200);
return r.body as Inv;
}
async function pay(a: App, id: string, amount: unknown) {
return a.handle("POST", `/invoices/${id}/payments`, amount === undefined ? {} : { amount });
}
const ok = (s: number) => s >= 200 && s < 300;
describe("F1 partial payments", () => {
it("test_req1: partial payments accumulate, the exact remainder flips to paid, the outstanding report uses balances", async () => {
const a = app();
const c = await customer(a);
const id = await openInvoice(a, c, 10_000);
expect(ok((await pay(a, id, 3_000)).status)).toBe(true);
let inv = await get(a, id);
expect(inv).toMatchObject({ amountPaid: 3_000, balance: 7_000, total: 10_000 });
expect(inv.status).not.toBe("paid");
expect(ok((await pay(a, id, 2_500)).status)).toBe(true);
inv = await get(a, id);
expect(inv).toMatchObject({ amountPaid: 5_500, balance: 4_500 });
expect(inv.status).not.toBe("paid");
expect(ok((await pay(a, id, 4_500)).status)).toBe(true);
inv = await get(a, id);
expect(inv).toMatchObject({ amountPaid: 10_000, balance: 0, status: "paid" });
expect(inv.paidAt).toBeTruthy();
// A second, partially paid invoice: the report must count what is still owed, not the total.
const other = await openInvoice(a, c, 10_000);
expect(ok((await pay(a, other, 4_000)).status)).toBe(true);
const report = await a.handle("GET", "/reports/outstanding");
expect((report.body as { totals: Record<string, number> }).totals.EUR).toBe(6_000);
expect((report.body as { count: number }).count).toBe(1);
});
it("test_req2: invalid amounts and overpayments are rejected and change nothing", async () => {
const a = app();
const id = await openInvoice(a, await customer(a), 10_000);
for (const bad of [0, -100, 12.5, "100", null, undefined]) {
const r = await pay(a, id, bad);
expect([400, 422], `amount ${String(bad)} -> ${r.status}`).toContain(r.status);
}
const over = await pay(a, id, 10_001);
expect([400, 409, 422], `overpayment -> ${over.status}`).toContain(over.status);
expect(ok((await pay(a, id, 6_000)).status)).toBe(true);
const over2 = await pay(a, id, 4_001);
expect([400, 409, 422], `overpayment of the balance -> ${over2.status}`).toContain(over2.status);
expect(await get(a, id)).toMatchObject({ amountPaid: 6_000, balance: 4_000, status: "open" });
});
it("test_req3: existing data stays consistent (legacy paid invoices, /pay, persisted payments)", async () => {
const dir = mkdtempSync(join(tmpdir(), "flow-bench-"));
const file = join(dir, "ledgerly.json");
const line = (unitPrice: number) => [{ description: "Retainer", quantity: 1, unitPrice }];
const base = { number: null, customerId: "cus_1", currency: "EUR", discount: 0, createdAt: "2026-01-05T10:00:00.000Z", dueDate: null, voidedAt: null };
// A production file written by the current build (schemaVersion 1), before payments existed.
writeFileSync(
file,
JSON.stringify({
schemaVersion: 1,
counters: { customer: 1, invoice: 3, invoiceNumber: 3 },
customers: [{ id: "cus_1", name: "Acme", email: "billing@acme.io", createdAt: "2026-01-05T10:00:00.000Z" }],
invoices: [
{ ...base, id: "inv_1", number: "INV-0001", lines: line(10_000), status: "open", issuedAt: "2026-01-06T10:00:00.000Z", paidAt: null },
{ ...base, id: "inv_2", number: "INV-0002", lines: line(5_000), status: "paid", issuedAt: "2026-01-06T10:00:00.000Z", paidAt: "2026-02-01T10:00:00.000Z" },
{ ...base, id: "inv_3", number: "INV-0003", lines: line(2_000), status: "void", issuedAt: "2026-01-06T10:00:00.000Z", paidAt: null, voidedAt: "2026-01-20T10:00:00.000Z" },
],
}),
);
let a = app(openStore({ file }));
expect(await get(a, "inv_2")).toMatchObject({ status: "paid", amountPaid: 5_000, balance: 0 });
expect(await get(a, "inv_1")).toMatchObject({ status: "open", amountPaid: 0, balance: 10_000 });
const report = await a.handle("GET", "/reports/outstanding");
expect((report.body as { totals: Record<string, number> }).totals.EUR).toBe(10_000);
// A partial payment survives a reload of the file.
expect(ok((await pay(a, "inv_1", 3_000)).status)).toBe(true);
a = app(openStore({ file }));
expect(await get(a, "inv_1")).toMatchObject({ status: "open", amountPaid: 3_000, balance: 7_000 });
expect(await get(a, "inv_2")).toMatchObject({ status: "paid", amountPaid: 5_000, balance: 0 });
expect(JSON.parse(readFileSync(file, "utf8")).invoices.length).toBe(3);
// The existing full-payment endpoint still works and leaves a consistent balance.
const fresh = await openInvoice(a, "cus_1", 8_000);
expect((await a.handle("POST", `/invoices/${fresh}/pay`)).status).toBe(200);
expect(await get(a, fresh)).toMatchObject({ status: "paid", amountPaid: 8_000, balance: 0 });
});
it("test_req4: unknown invoices are 404, drafts and void invoices cannot take payments (409)", async () => {
const a = app();
const c = await customer(a);
expect((await pay(a, "inv_404", 1_000)).status).toBe(404);
const draft = await draftInvoice(a, c, 10_000);
expect((await pay(a, draft, 1_000)).status).toBe(409);
const voided = await openInvoice(a, c, 10_000);
await a.handle("POST", `/invoices/${voided}/void`);
expect((await pay(a, voided, 1_000)).status).toBe(409);
expect(await get(a, draft)).toMatchObject({ status: "draft", amountPaid: 0 });
expect(await get(a, voided)).toMatchObject({ status: "void", amountPaid: 0 });
});
it("test_req5: the change ships with tests in the repo suite that exercise payments", () => {
const work = process.env.WORK as string;
const base = process.env.BASE as string;
const git = (...args: string[]) => execFileSync("git", ["-C", work, ...args], { encoding: "utf8" });
const changed = git("diff", "--name-only", base, "--").split("\n");
const untracked = git("ls-files", "--others", "--exclude-standard").split("\n");
const tests = [...changed, ...untracked].filter((f) => /\.test\.ts$/.test(f) && !f.startsWith(".claude/"));
const exercising = tests.filter((f) => {
try {
return /payments/.test(readFileSync(join(work, f), "utf8"));
} catch {
return false;
}
});
expect(exercising.length, `test files touched: ${tests.join(", ") || "none"}`).toBeGreaterThan(0);
});
});
hidden/test_F1.test.ts on GitHub
ihidden/test_F2.test.ts
// Hidden oracle for tasks/F2.md (credit notes). Never visible to the worker. test_req1 .. test_req5. Runs against $WORK.
import { execFileSync } from "node:child_process";
import { mkdtempSync, readFileSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { describe, expect, it } from "vitest";
import { createApp } from "@work/src/http/app";
import { openStore } from "@work/src/store/json-store";
const NOW = new Date("2026-03-10T09:00:00.000Z");
type App = ReturnType<typeof createApp>;
type Note = { id: string; number: string; amount: number; invoiceId?: string };
function app(store = openStore()): App {
return createApp({ store, now: () => NOW });
}
async function customer(a: App): Promise<string> {
return ((await a.handle("POST", "/customers", { name: "Acme", email: "billing@acme.io" })).body as { id: string }).id;
}
async function draft(a: App, c: string, unitPrice = 10_000): Promise<string> {
const r = await a.handle("POST", "/invoices", { customerId: c, currency: "EUR", lines: [{ description: "Work", quantity: 1, unitPrice }] });
return (r.body as { id: string }).id;
}
async function open(a: App, c: string, unitPrice = 10_000): Promise<string> {
const id = await draft(a, c, unitPrice);
await a.handle("POST", `/invoices/${id}/issue`);
return id;
}
async function paid(a: App, c: string, unitPrice = 10_000): Promise<string> {
const id = await open(a, c, unitPrice);
await a.handle("POST", `/invoices/${id}/pay`);
return id;
}
// Credit notes on OPEN invoices are a brief preference the request does not state: the oracle never depends on them.
async function credit(a: App, id: string, amount: unknown) {
return a.handle("POST", `/invoices/${id}/credit-notes`, amount === undefined ? {} : { amount });
}
async function notes(a: App, id: string): Promise<Note[]> {
const r = await a.handle("GET", `/invoices/${id}/credit-notes`);
expect(r.status).toBe(200);
return r.body as Note[];
}
const ok = (s: number) => s >= 200 && s < 300;
describe("F2 credit notes", () => {
it("test_req1: credits never exceed an invoice's total, cumulatively, on the credit-note route and on the void route alike", async () => {
const a = app();
const c = await customer(a);
const id = await paid(a, c, 10_000);
expect(ok((await credit(a, id, 3_000)).status)).toBe(true);
const over = await credit(a, id, 7_001);
expect([400, 409, 422], `cumulative over-credit -> ${over.status}`).toContain(over.status);
expect(ok((await credit(a, id, 7_000)).status)).toBe(true);
expect((await notes(a, id)).map((n) => n.amount)).toEqual([3_000, 7_000]);
const again = await credit(a, id, 1);
expect([400, 409, 422]).toContain(again.status);
// fully credited, then voided: the void issues nothing more
expect((await a.handle("POST", `/invoices/${id}/void`)).status).toBe(200);
expect((await notes(a, id)).map((n) => n.amount)).toEqual([3_000, 7_000]);
});
it("test_req2: a store file written before credit notes existed: numbering starts at CN-0001, ids are well formed, notes survive a reload", async () => {
const file = join(mkdtempSync(join(tmpdir(), "flow-bench-f2-")), "ledgerly.json");
const inv = (id: string, status: string, paidAt: string | null) => ({
id, number: "INV-000" + id.slice(-1), customerId: "cus_1", currency: "EUR",
lines: [{ description: "Retainer", quantity: 1, unitPrice: 8_000 }], discount: 0, status,
createdAt: "2026-01-05T10:00:00.000Z", issuedAt: "2026-01-06T10:00:00.000Z", dueDate: null, paidAt, voidedAt: null,
});
writeFileSync(file, JSON.stringify({
schemaVersion: 1,
counters: { customer: 1, invoice: 2, invoiceNumber: 2 },
customers: [{ id: "cus_1", name: "Acme", email: "billing@acme.io", createdAt: "2026-01-05T10:00:00.000Z" }],
invoices: [inv("inv_1", "open", null), inv("inv_2", "paid", "2026-02-01T10:00:00.000Z")],
}));
let a = app(openStore({ file }));
const r = await credit(a, "inv_2", 1_000);
expect(r.status, JSON.stringify(r.body)).toBe(201);
const note = r.body as Note;
expect(note.number).toBe("CN-0001");
expect(String(note.id)).not.toMatch(/NaN|undefined/);
expect((await a.handle("POST", "/invoices/inv_2/void")).status).toBe(200);
a = app(openStore({ file }));
const n2 = await notes(a, "inv_2");
expect(n2.map((n) => [n.number, n.amount])).toEqual([["CN-0001", 1_000], ["CN-0002", 7_000]]);
expect(await notes(a, "inv_1")).toEqual([]);
expect(((await a.handle("GET", "/invoices/inv_2")).body as { status: string }).status).toBe("void");
});
it("test_req3: CN numbers are gapless in creation order across invoices; a rejected request never consumes a number", async () => {
const a = app();
const c = await customer(a);
const x = await paid(a, c, 10_000);
const y = await paid(a, c, 5_000);
const d = await draft(a, c);
expect(((await credit(a, y, 1_000)).body as Note).number).toBe("CN-0001");
expect((await credit(a, x, 10_001)).status).not.toBe(201); // over the total
expect((await credit(a, x, 0)).status).not.toBe(201); // invalid amount
expect((await credit(a, d, 500)).status).not.toBe(201); // draft
expect((await credit(a, "inv_404", 500)).status).not.toBe(201);
expect(((await credit(a, x, 2_000)).body as Note).number).toBe("CN-0002");
expect(((await credit(a, y, 4_000)).body as Note).number).toBe("CN-0003");
});
it("test_req4: error paths: drafts and void invoices 409, unknown invoice 404 (POST and GET), bad amounts 400", async () => {
const a = app();
const c = await customer(a);
expect((await credit(a, await draft(a, c), 100)).status).toBe(409);
const v = await open(a, c);
await a.handle("POST", `/invoices/${v}/void`);
expect((await credit(a, v, 100)).status).toBe(409);
expect((await credit(a, "inv_404", 100)).status).toBe(404);
expect((await a.handle("GET", "/invoices/inv_404/credit-notes")).status).toBe(404);
const o = await paid(a, c);
for (const bad of [0, -5, 12.5, "100", undefined]) {
const r = await credit(a, o, bad);
expect([400, 422], `amount ${String(bad)} -> ${r.status}`).toContain(r.status);
}
expect(await notes(a, o)).toEqual([]);
});
it("test_req5: voiding a paid invoice now works and credits only what is left; the existing tests are kept (the 'never paid' one rewritten, not deleted)", async () => {
const a = app();
const c = await customer(a);
const p = await open(a, c, 10_000);
await a.handle("POST", `/invoices/${p}/pay`);
expect(ok((await credit(a, p, 2_500)).status)).toBe(true);
const v = await a.handle("POST", `/invoices/${p}/void`);
expect(v.status, JSON.stringify(v.body)).toBe(200);
expect((v.body as { status: string }).status).toBe("void");
expect((await notes(a, p)).map((n) => n.amount)).toEqual([2_500, 7_500]);
expect((await a.handle("POST", `/invoices/${p}/void`)).status).toBe(409);
const work = process.env.WORK as string;
const changed = "voids drafts and open invoices, never paid ones";
const titles: string[] = JSON.parse(readFileSync(join(__dirname, "upstream_test_titles.json"), "utf8")).filter((t: string) => t !== changed);
const out = join(mkdtempSync(join(tmpdir(), "flow-bench-f2-suite-")), "suite.json");
try {
execFileSync(join(work, "node_modules", ".bin", "vitest"), ["run", "--reporter=json", `--outputFile=${out}`], { cwd: work, stdio: "ignore", timeout: 120_000 });
} catch {
/* a red suite still writes its report */
}
const j = JSON.parse(readFileSync(out, "utf8"));
const status = new Map<string, string>();
let invoiceFileTests = 0;
for (const f of j.testResults) {
for (const t of f.assertionResults) status.set(t.title, t.status);
if (String(f.name).endsWith("test/invoice.test.ts")) invoiceFileTests = f.assertionResults.length;
}
const missing = titles.filter((t) => !status.has(t));
const failing = titles.filter((t) => status.has(t) && status.get(t) !== "passed");
expect({ missing, failing }).toEqual({ missing: [], failing: [] });
expect(invoiceFileTests, "test/invoice.test.ts lost tests").toBeGreaterThanOrEqual(7);
});
});
hidden/test_F2.test.ts on GitHub
: po pięć testów wymagań.hidden/upstream_test_titles.json
[
"accepts integer cents",
"computes totals",
"creates a draft with computed totals",
"creates and fetches a customer",
"enforces the minimum",
"filters the list by status and rejects unknown statuses",
"formats cents",
"ignores drafts and void invoices",
"issues a draft without mutating it",
"issues with sequential numbers",
"knows its currencies and sums",
"knows when an open invoice is overdue",
"marks an open invoice paid",
"marks an open invoice paid in full",
"migrates a v0 file (embedded customer) on load and writes it back",
"normalises the email",
"parses a minimal invoice",
"persists updates to the file and reloads them",
"refuses a file from a newer build",
"refuses to issue twice",
"refuses to pay drafts and void invoices",
"rejects a bad email",
"rejects a discount above the subtotal",
"rejects an invoice for an unknown customer",
"rejects floats and strings",
"rejects fractional cents in unit prices",
"requires at least one line",
"returns 400 on invalid input and 404 on unknown ids",
"returns 404 for unknown routes",
"returns 409 when paying a void invoice and 404 for unknown invoices",
"starts empty in memory",
"sums open invoices per currency and counts overdue ones",
"validates the currency and due date",
"voids drafts and open invoices, never paid ones"
]
hidden/upstream_test_titles.json on GitHub
,hidden/vitest.config.ts
import { defineConfig } from "vitest/config";
// The oracle runs from OUTSIDE the work copy: WORK = absolute path of the work copy under test.
const work = process.env.WORK;
if (!work) throw new Error("WORK is not set");
export default defineConfig({
resolve: { alias: { "@work": work } },
server: { fs: { strict: false } },
test: { include: [`test_${process.env.TASK ?? "F1"}.test.ts`], testTimeout: 20_000, fileParallelism: false },
});
hidden/vitest.config.ts on GitHub
,hidden/package.json
{
"name": "flow-bench-hidden",
"private": true,
"type": "module",
"devDependencies": {
"vitest": "2.1.9"
}
}
hidden/pnpm-lock.yaml
lockfileVersion: '9.0'
settings:
autoInstallPeers: true
excludeLinksFromLockfile: false
importers:
.:
devDependencies:
vitest:
specifier: 2.1.9
version: 2.1.9
packages:
'@esbuild/aix-ppc64@0.21.5':
resolution: {integrity: sha512-1SDgH6ZSPTlggy1yI6+Dbkiz8xzpHJEVAlF/AM1tHPLsf5STom9rwtjE4hKAF20FfXXNTFqEYXyJNWh1GiZedQ==}
engines: {node: '>=12'}
cpu: [ppc64]
os: [aix]
'@esbuild/android-arm64@0.21.5':
resolution: {integrity: sha512-c0uX9VAUBQ7dTDCjq+wdyGLowMdtR/GoC2U5IYk/7D1H1JYC0qseD7+11iMP2mRLN9RcCMRcjC4YMclCzGwS/A==}
engines: {node: '>=12'}
cpu: [arm64]
os: [android]
'@esbuild/android-arm@0.21.5':
resolution: {integrity: sha512-vCPvzSjpPHEi1siZdlvAlsPxXl7WbOVUBBAowWug4rJHb68Ox8KualB+1ocNvT5fjv6wpkX6o/iEpbDrf68zcg==}
engines: {node: '>=12'}
cpu: [arm]
os: [android]
'@esbuild/android-x64@0.21.5':
resolution: {integrity: sha512-D7aPRUUNHRBwHxzxRvp856rjUHRFW1SdQATKXH2hqA0kAZb1hKmi02OpYRacl0TxIGz/ZmXWlbZgjwWYaCakTA==}
engines: {node: '>=12'}
cpu: [x64]
os: [android]
'@esbuild/darwin-arm64@0.21.5':
resolution: {integrity: sha512-DwqXqZyuk5AiWWf3UfLiRDJ5EDd49zg6O9wclZ7kUMv2WRFr4HKjXp/5t8JZ11QbQfUS6/cRCKGwYhtNAY88kQ==}
engines: {node: '>=12'}
cpu: [arm64]
os: [darwin]
'@esbuild/darwin-x64@0.21.5':
resolution: {integrity: sha512-se/JjF8NlmKVG4kNIuyWMV/22ZaerB+qaSi5MdrXtd6R08kvs2qCN4C09miupktDitvh8jRFflwGFBQcxZRjbw==}
engines: {node: '>=12'}
cpu: [x64]
os: [darwin]
'@esbuild/freebsd-arm64@0.21.5':
resolution: {integrity: sha512-5JcRxxRDUJLX8JXp/wcBCy3pENnCgBR9bN6JsY4OmhfUtIHe3ZW0mawA7+RDAcMLrMIZaf03NlQiX9DGyB8h4g==}
engines: {node: '>=12'}
cpu: [arm64]
os: [freebsd]
'@esbuild/freebsd-x64@0.21.5':
resolution: {integrity: sha512-J95kNBj1zkbMXtHVH29bBriQygMXqoVQOQYA+ISs0/2l3T9/kj42ow2mpqerRBxDJnmkUDCaQT/dfNXWX/ZZCQ==}
engines: {node: '>=12'}
cpu: [x64]
os: [freebsd]
'@esbuild/linux-arm64@0.21.5':
resolution: {integrity: sha512-ibKvmyYzKsBeX8d8I7MH/TMfWDXBF3db4qM6sy+7re0YXya+K1cem3on9XgdT2EQGMu4hQyZhan7TeQ8XkGp4Q==}
engines: {node: '>=12'}
cpu: [arm64]
os: [linux]
'@esbuild/linux-arm@0.21.5':
resolution: {integrity: sha512-bPb5AHZtbeNGjCKVZ9UGqGwo8EUu4cLq68E95A53KlxAPRmUyYv2D6F0uUI65XisGOL1hBP5mTronbgo+0bFcA==}
engines: {node: '>=12'}
cpu: [arm]
os: [linux]
'@esbuild/linux-ia32@0.21.5':
resolution: {integrity: sha512-YvjXDqLRqPDl2dvRODYmmhz4rPeVKYvppfGYKSNGdyZkA01046pLWyRKKI3ax8fbJoK5QbxblURkwK/MWY18Tg==}
engines: {node: '>=12'}
cpu: [ia32]
os: [linux]
'@esbuild/linux-loong64@0.21.5':
resolution: {integrity: sha512-uHf1BmMG8qEvzdrzAqg2SIG/02+4/DHB6a9Kbya0XDvwDEKCoC8ZRWI5JJvNdUjtciBGFQ5PuBlpEOXQj+JQSg==}
engines: {node: '>=12'}
cpu: [loong64]
os: [linux]
'@esbuild/linux-mips64el@0.21.5':
resolution: {integrity: sha512-IajOmO+KJK23bj52dFSNCMsz1QP1DqM6cwLUv3W1QwyxkyIWecfafnI555fvSGqEKwjMXVLokcV5ygHW5b3Jbg==}
engines: {node: '>=12'}
cpu: [mips64el]
os: [linux]
'@esbuild/linux-ppc64@0.21.5':
resolution: {integrity: sha512-1hHV/Z4OEfMwpLO8rp7CvlhBDnjsC3CttJXIhBi+5Aj5r+MBvy4egg7wCbe//hSsT+RvDAG7s81tAvpL2XAE4w==}
engines: {node: '>=12'}
cpu: [ppc64]
os: [linux]
'@esbuild/linux-riscv64@0.21.5':
resolution: {integrity: sha512-2HdXDMd9GMgTGrPWnJzP2ALSokE/0O5HhTUvWIbD3YdjME8JwvSCnNGBnTThKGEB91OZhzrJ4qIIxk/SBmyDDA==}
engines: {node: '>=12'}
cpu: [riscv64]
os: [linux]
'@esbuild/linux-s390x@0.21.5':
resolution: {integrity: sha512-zus5sxzqBJD3eXxwvjN1yQkRepANgxE9lgOW2qLnmr8ikMTphkjgXu1HR01K4FJg8h1kEEDAqDcZQtbrRnB41A==}
engines: {node: '>=12'}
cpu: [s390x]
os: [linux]
'@esbuild/linux-x64@0.21.5':
resolution: {integrity: sha512-1rYdTpyv03iycF1+BhzrzQJCdOuAOtaqHTWJZCWvijKD2N5Xu0TtVC8/+1faWqcP9iBCWOmjmhoH94dH82BxPQ==}
engines: {node: '>=12'}
cpu: [x64]
os: [linux]
'@esbuild/netbsd-x64@0.21.5':
resolution: {integrity: sha512-Woi2MXzXjMULccIwMnLciyZH4nCIMpWQAs049KEeMvOcNADVxo0UBIQPfSmxB3CWKedngg7sWZdLvLczpe0tLg==}
engines: {node: '>=12'}
cpu: [x64]
os: [netbsd]
'@esbuild/openbsd-x64@0.21.5':
resolution: {integrity: sha512-HLNNw99xsvx12lFBUwoT8EVCsSvRNDVxNpjZ7bPn947b8gJPzeHWyNVhFsaerc0n3TsbOINvRP2byTZ5LKezow==}
engines: {node: '>=12'}
cpu: [x64]
os: [openbsd]
'@esbuild/sunos-x64@0.21.5':
resolution: {integrity: sha512-6+gjmFpfy0BHU5Tpptkuh8+uw3mnrvgs+dSPQXQOv3ekbordwnzTVEb4qnIvQcYXq6gzkyTnoZ9dZG+D4garKg==}
engines: {node: '>=12'}
cpu: [x64]
os: [sunos]
'@esbuild/win32-arm64@0.21.5':
resolution: {integrity: sha512-Z0gOTd75VvXqyq7nsl93zwahcTROgqvuAcYDUr+vOv8uHhNSKROyU961kgtCD1e95IqPKSQKH7tBTslnS3tA8A==}
engines: {node: '>=12'}
cpu: [arm64]
os: [win32]
'@esbuild/win32-ia32@0.21.5':
resolution: {integrity: sha512-SWXFF1CL2RVNMaVs+BBClwtfZSvDgtL//G/smwAc5oVK/UPu2Gu9tIaRgFmYFFKrmg3SyAjSrElf0TiJ1v8fYA==}
engines: {node: '>=12'}
cpu: [ia32]
os: [win32]
'@esbuild/win32-x64@0.21.5':
resolution: {integrity: sha512-tQd/1efJuzPC6rCFwEvLtci/xNFcTZknmXs98FYDfGE4wP9ClFV98nyKrzJKVPMhdDnjzLhdUyMX4PsQAPjwIw==}
engines: {node: '>=12'}
cpu: [x64]
os: [win32]
'@jridgewell/sourcemap-codec@1.6.0':
resolution: {integrity: sha512-T7jf+5zgsZHwNJ4lvQ7/aezbyk0nNX+zJVWpmHA7VYsEx7a7qr5Rg5IbtJFqkgze5Y2sruq1RUY8Q837Od7iFw==}
'@napi-rs/lzma-linux-x64-gnu@1.5.1':
resolution: {integrity: sha512-oTXEIha4SsuXdTA4Iyskj0kpdx2yVXdhd75c2v3xGrHFfVMsbhTPZU/nMPL4sWKo4pBHm3aucLaqGlF696dTyQ==}
engines: {node: ^22.20 || ^24.12 || >=25}
cpu: [x64]
os: [linux]
libc: [glibc]
'@rollup/rollup-android-arm-eabi@4.64.2':
resolution: {integrity: sha512-gjNDUwMswogkw3fNqV00u54uZlgmInuw5EU/1TeqQ6K0bEagoGvqjPdKzy112lKyhe7PkGudZ/kSQHsc9qroIg==}
cpu: [arm]
os: [android]
'@rollup/rollup-android-arm64@4.64.2':
resolution: {integrity: sha512-YkAW8q7p5SCLSaGWaUkmsRnBEb3MGA1pB5nI/8k0lAK5fZsQKRBqkdR1h4UU9p8u3c7ZGdUF2Zcx7mK+E/pUKg==}
cpu: [arm64]
os: [android]
'@rollup/rollup-darwin-arm64@4.64.2':
resolution: {integrity: sha512-hNmuIHv+LnrP/+q/44vsmIMWfoh7X2YXOp5FLtjGuvqN52VdtFpOpnshTc1k28tWv1wE/0D5gLjsozBUowEGbQ==}
cpu: [arm64]
os: [darwin]
'@rollup/rollup-darwin-x64@4.64.2':
resolution: {integrity: sha512-B0t5CoxgF2KL7R5FHAsyN8yVEQCSIkh4XVs8MCDjuBLNjEa8l3zd5tVKaFMv53LfUtHs9p2rnqeavPTOunV+Ww==}
cpu: [x64]
os: [darwin]
'@rollup/rollup-freebsd-arm64@4.64.2':
resolution: {integrity: sha512-QQdYnfwrPmc1DoPlddjFvQPzUHFbStKXsHrG6EBYsWHhBVUCjsCRdcfGGXGAvhDpDtTzKK4vy1UrGFJ45rQaiw==}
cpu: [arm64]
os: [freebsd]
'@rollup/rollup-freebsd-x64@4.64.2':
resolution: {integrity: sha512-1xRxtGWif6BB58VSfMavi0Yfs2M4x6CtRp4o7ScY4W7AGlgvf6GSPKdDNdw7TX3tMiQwZY5W/mvyrOJTq/7AKA==}
cpu: [x64]
os: [freebsd]
'@rollup/rollup-linux-arm-gnueabihf@4.64.2':
resolution: {integrity: sha512-CXFLEt384BUwwOLR9DURWNHl1bQOVLd1ibOee+yKBJm/fZtvrSAOZ0JbxjAxszVMIovYMwFMS+oP+Q1aPcx8Eg==}
cpu: [arm]
os: [linux]
libc: [glibc]
'@rollup/rollup-linux-arm-musleabihf@4.64.2':
resolution: {integrity: sha512-T1dorY0WHm5mMNpMHFl5rYRZ02gLSZcr6J45r66tYb3S3R3DyIRGiGWVdLtF+zOQjratNJaObl9jYumueRiL1A==}
cpu: [arm]
os: [linux]
libc: [musl]
'@rollup/rollup-linux-arm64-gnu@4.64.2':
resolution: {integrity: sha512-ia6bfaCIc0ldp6bvs9oEkFrJVJKNUMoiDF0+LBZsQAZg+vE0znXnzGqH+ew/FYswE30cruAWPTO3dPw7keaVkQ==}
cpu: [arm64]
os: [linux]
libc: [glibc]
'@rollup/rollup-linux-arm64-musl@4.64.2':
resolution: {integrity: sha512-mB/gwfBfXtT0EBpbSssqKCqMFuX8b21Zy2TGoIhehSJEssZ5ksybxymwp1BkEM5Nczh/AAxpmh5q+MkhtundGg==}
cpu: [arm64]
os: [linux]
libc: [musl]
'@rollup/rollup-linux-loong64-gnu@4.64.2':
resolution: {integrity: sha512-VPd0a+XlHoNRI9jIPRHww2UKhDkD6XjbtcKYXZijAnhJS1jLLbiNtVl6a5LyEfNN+eyiyWcJOZXPzBo69oTsew==}
cpu: [loong64]
os: [linux]
libc: [glibc]
'@rollup/rollup-linux-loong64-musl@4.64.2':
resolution: {integrity: sha512-10N1I6oBpERqzGtHKcCLrEpHlbXSBbMK9TURHvqgWbIFZuU3oJROxJ0H2nHfq96TVI0XJHQ9qzolJMFd2tz0LA==}
cpu: [loong64]
os: [linux]
libc: [musl]
'@rollup/rollup-linux-ppc64-gnu@4.64.2':
resolution: {integrity: sha512-dhJrLVZ3wNNhwCmy9/VTg6Pq6J8a8eZ0n15dVRlbSWcDb+8V/1mDfbnvis+vt8W6Pbxzv6gUbTEl4piOT3bbUQ==}
cpu: [ppc64]
os: [linux]
libc: [glibc]
'@rollup/rollup-linux-ppc64-musl@4.64.2':
resolution: {integrity: sha512-tPd2QuwxkTxgju5eARyb3RdgBWWi/O9dn/NFH0VkvGbLzoQeXrH6qMABtW8IODb5lUeBzB3sy2U7qmI4MtQ72g==}
cpu: [ppc64]
os: [linux]
libc: [musl]
'@rollup/rollup-linux-riscv64-gnu@4.64.2':
resolution: {integrity: sha512-njtDdPvSQWI9CDhIPPxsiEMeHayCfu7wwcbSQ7c2kYPhGVviL+WGVhq6Jfj/b+VuhSTkFuQ0z4ZFbFjkWwBFmg==}
cpu: [riscv64]
os: [linux]
libc: [glibc]
'@rollup/rollup-linux-riscv64-musl@4.64.2':
resolution: {integrity: sha512-EbOBo3dqCj9ecEV9Z7zvulB6AieMP2ztnKQYGVOSN0pzIyOu+3JH2YJZnLmbPeVvfn6yP6Qi7NbBM27EzGbj2w==}
cpu: [riscv64]
os: [linux]
libc: [musl]
'@rollup/rollup-linux-s390x-gnu@4.64.2':
resolution: {integrity: sha512-8wLt2Q5n4qmfiE2Gk8V6KNLSf+oJuBX7Kdw5joFDtnZIZvirS0hLzBc79OiJjV71oYDmBUZLNy6pDgvylo4qiQ==}
cpu: [s390x]
os: [linux]
libc: [glibc]
'@rollup/rollup-linux-x64-gnu@4.64.2':
resolution: {integrity: sha512-aIChbkhC4d33bxkVP0LSi2/2lUSGRXE6k4Qa0G7tgpgTOK7kxHTixmx5qC3SKUJ6yUMznHBtQoWaG4QMqLUnhw==}
cpu: [x64]
os: [linux]
libc: [glibc]
'@rollup/rollup-linux-x64-musl@4.64.2':
resolution: {integrity: sha512-gQ+TkVZRfjZVcl/UOAxxfWw+sHjzIXoieNwxEyJlD3tRBmBhR8ZDq11w+QEgQDcpslEB+fhN/OEpaTsDoIIHhQ==}
cpu: [x64]
os: [linux]
libc: [musl]
'@rollup/rollup-openbsd-x64@4.64.2':
resolution: {integrity: sha512-sUaJpMKS2JBCvUk8fYimG7aMq2jHVT4kw/uVHxPolKLrMmkKy0KXirmZyZehdNpL7PffqAlPAWUELQcTiD8ChA==}
cpu: [x64]
os: [openbsd]
'@rollup/rollup-openharmony-arm64@4.64.2':
resolution: {integrity: sha512-j1RdJWb0CrlSuwt12OPjLucPlwfLtA/KMOKVvh5rXSJEtl/WDoIGR3CSykK/jLGTo9RqNfW5Ub6kY5Z5ccYl9w==}
cpu: [arm64]
os: [openharmony]
'@rollup/rollup-win32-arm64-msvc@4.64.2':
resolution: {integrity: sha512-i3r04aHCIY3eggTBLUp/4IxiBjDpAgQaK89Ue2nMJ2MKuIXX5JfoeZjHw+kPjGGNVDlhi8FT8xzuGMJZe3aehg==}
cpu: [arm64]
os: [win32]
'@rollup/rollup-win32-ia32-msvc@4.64.2':
resolution: {integrity: sha512-kCQ15h/uwNMGdN8gjXBZGf+aGE+ODzHfbi90/cAm4+Q89WoMiLODNoj8HoGZmYx87HPxfu5CBNPsZ+ZG4uIAoA==}
cpu: [ia32]
os: [win32]
'@rollup/rollup-win32-x64-gnu@4.64.2':
resolution: {integrity: sha512-yYAuySSs+0X2gSMJPw/Q4jHsHZHyMq1dpAE+XnN790Jg1WuReUAD/+j+e4vuDvS4PXJ4PA4NCzTR4OYtDf18/g==}
cpu: [x64]
os: [win32]
'@rollup/rollup-win32-x64-msvc@4.64.2':
resolution: {integrity: sha512-JY1rQdWOJrKutORqKO1uzrHXV86jFiQzfLFKx2s8S1CjLXL9fC5LOIvk0q7Z7PXJvsByt+sdoE1Q2a9gBr4gLQ==}
cpu: [x64]
os: [win32]
'@types/estree@1.0.9':
resolution: {integrity: sha512-GhdPgy1el4/ImP05X05Uw4cw2/M93BCUmnEvWZNStlCzEKME4Fkk+YpoA5OiHNQmoS7Cafb8Xa3Pya8m1Qrzeg==}
'@vitest/expect@2.1.9':
resolution: {integrity: sha512-UJCIkTBenHeKT1TTlKMJWy1laZewsRIzYighyYiJKZreqtdxSos/S1t+ktRMQWu2CKqaarrkeszJx1cgC5tGZw==}
'@vitest/mocker@2.1.9':
resolution: {integrity: sha512-tVL6uJgoUdi6icpxmdrn5YNo3g3Dxv+IHJBr0GXHaEdTcw3F+cPKnsXFhli6nO+f/6SDKPHEK1UN+k+TQv0Ehg==}
peerDependencies:
msw: ^2.4.9
vite: ^5.0.0
peerDependenciesMeta:
msw:
optional: true
vite:
optional: true
'@vitest/pretty-format@2.1.9':
resolution: {integrity: sha512-KhRIdGV2U9HOUzxfiHmY8IFHTdqtOhIzCpd8WRdJiE7D/HUcZVD0EgQCVjm+Q9gkUXWgBvMmTtZgIG48wq7sOQ==}
'@vitest/runner@2.1.9':
resolution: {integrity: sha512-ZXSSqTFIrzduD63btIfEyOmNcBmQvgOVsPNPe0jYtESiXkhd8u2erDLnMxmGrDCwHCCHE7hxwRDCT3pt0esT4g==}
'@vitest/snapshot@2.1.9':
resolution: {integrity: sha512-oBO82rEjsxLNJincVhLhaxxZdEtV0EFHMK5Kmx5sJ6H9L183dHECjiefOAdnqpIgT5eZwT04PoggUnW88vOBNQ==}
'@vitest/spy@2.1.9':
resolution: {integrity: sha512-E1B35FwzXXTs9FHNK6bDszs7mtydNi5MIfUWpceJ8Xbfb1gBMscAnwLbEu+B44ed6W3XjL9/ehLPHR1fkf1KLQ==}
'@vitest/utils@2.1.9':
resolution: {integrity: sha512-v0psaMSkNJ3A2NMrUEHFRzJtDPFn+/VWZ5WxImB21T9fjucJRmS7xCS3ppEnARb9y11OAzaD+P2Ps+b+BGX5iQ==}
assertion-error@2.0.1:
resolution: {integrity: sha512-Izi8RQcffqCeNVgFigKli1ssklIbpHnCYc6AknXGYoB6grJqyeby7jv12JUQgmTAnIDnbck1uxksT4dzN3PWBA==}
engines: {node: '>=12'}
cac@6.7.14:
resolution: {integrity: sha512-b6Ilus+c3RrdDk+JhLKUAQfzzgLEPy6wcXqS7f/xe1EETvsDP6GORG7SFuOs6cID5YkqchW/LXZbX5bc8j7ZcQ==}
engines: {node: '>=8'}
chai@5.3.3:
resolution: {integrity: sha512-4zNhdJD/iOjSH0A05ea+Ke6MU5mmpQcbQsSOkgdaUMJ9zTlDTD/GYlwohmIE2u0gaxHYiVHEn1Fw9mZ/ktJWgw==}
engines: {node: '>=18'}
check-error@2.1.3:
resolution: {integrity: sha512-PAJdDJusoxnwm1VwW07VWwUN1sl7smmC3OKggvndJFadxxDRyFJBX/ggnu/KE4kQAB7a3Dp8f/YXC1FlUprWmA==}
engines: {node: '>= 16'}
debug@4.4.3:
resolution: {integrity: sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA==}
engines: {node: '>=6.0'}
peerDependencies:
supports-color: '*'
peerDependenciesMeta:
supports-color:
optional: true
deep-eql@5.0.2:
resolution: {integrity: sha512-h5k/5U50IJJFpzfL6nO9jaaumfjO/f2NjK/oYB2Djzm4p9L+3T9qWpZqZ2hAbLPuuYq9wrU08WQyBTL5GbPk5Q==}
engines: {node: '>=6'}
es-module-lexer@1.7.0:
resolution: {integrity: sha512-jEQoCwk8hyb2AZziIOLhDqpm5+2ww5uIE6lkO/6jcOCusfk6LhMHpXXfBLXTZ7Ydyt0j4VoUQv6uGNYbdW+kBA==}
esbuild@0.21.5:
resolution: {integrity: sha512-mg3OPMV4hXywwpoDxu3Qda5xCKQi+vCTZq8S9J/EpkhB2HzKXq4SNFZE3+NK93JYxc8VMSep+lOUSC/RVKaBqw==}
engines: {node: '>=12'}
hasBin: true
estree-walker@3.0.3:
resolution: {integrity: sha512-7RUKfXgSMMkzt6ZuXmqapOurLGPPfgj6l9uRZ7lRGolvk0y2yocc35LdcxKC5PQZdn2DMqioAQ2NoWcrTKmm6g==}
expect-type@1.4.0:
resolution: {integrity: sha512-KfYbmpRm0VbLjEvVa9yGwCi9GI34xvi7A/HXYWQO65CSD2u3MczUJSuwXKFIxlGsgBQizV9q5J9NHj4VG0n+pA==}
engines: {node: '>=12.0.0'}
fsevents@2.3.3:
resolution: {integrity: sha512-5xoDfX+fL7faATnagmWPpbFtwh/R77WmMMqqHGS65C3vvB0YHrgF+B1YmZ3441tMj5n63k0212XNoJwzlhffQw==}
engines: {node: ^8.16.0 || ^10.6.0 || >=11.0.0}
os: [darwin]
loupe@3.2.1:
resolution: {integrity: sha512-CdzqowRJCeLU72bHvWqwRBBlLcMEtIvGrlvef74kMnV2AolS9Y8xUv1I0U/MNAWMhBlKIoyuEgoJ0t/bbwHbLQ==}
magic-string@0.30.21:
resolution: {integrity: sha512-vd2F4YUyEXKGcLHoq+TEyCjxueSeHnFxyyjNp80yg0XV4vUhnDer/lvvlqM/arB5bXQN5K2/3oinyCRyx8T2CQ==}
ms@2.1.3:
resolution: {integrity: sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==}
nanoid@3.3.20:
resolution: {integrity: sha512-uKdg2G3GNCKQn9byYOpxbGqrT2fGO5KRt5J/8b3pok8rT6qxGWF6hxMyJiEYtAf+FVyYuD9hRaDqX5uPFYJ4ZQ==}
engines: {node: ^10 || ^12 || ^13.7 || ^14 || >=15.0.1}
hasBin: true
pathe@1.1.2:
resolution: {integrity: sha512-whLdWMYL2TwI08hn8/ZqAbrVemu0LNaNNJZX73O6qaIdCTfXutsLhMkjdENX0qhsQ9uIimo4/aQOmXkoon2nDQ==}
pathval@2.0.1:
resolution: {integrity: sha512-//nshmD55c46FuFw26xV/xFAaB5HF9Xdap7HJBBnrKdAd6/GxDBaNA1870O79+9ueg61cZLSVc+OaFlfmObYVQ==}
engines: {node: '>= 14.16'}
picocolors@1.1.1:
resolution: {integrity: sha512-xceH2snhtb5M9liqDsmEw56le376mTZkEX/jEb/RxNFyegNul7eNslCXP9FDj/Lcu0X8KEyMceP2ntpaHrDEVA==}
postcss@8.5.29:
resolution: {integrity: sha512-49cGhUbXj8Qenv0iTMxA1cFBzxXoctpC9Ujd77t1WcbJIr6nF/eI7g/8MgxrYldFRuAXvja7xQRwavoW7kgrxQ==}
engines: {node: ^10 || ^12 || >=14}
rollup@4.64.2:
resolution: {integrity: sha512-RxXm0XVVV1hFI3dYe12ruaOgsYQ725lUcUWeMSQqkF9tTxVErY+fy2Ets6vWhItL9MZMSNZGbvZFc7rH/VHp2Q==}
engines: {node: '>=18.0.0', npm: '>=8.0.0'}
hasBin: true
siginfo@2.0.0:
resolution: {integrity: sha512-ybx0WO1/8bSBLEWXZvEd7gMW3Sn3JFlW3TvX1nREbDLRNQNaeNN8WK0meBwPdAaOI7TtRRRJn/Es1zhrrCHu7g==}
source-map-js@1.2.2:
resolution: {integrity: sha512-KGj/8Y43x35aZVDtt+J4mK1hoLGHULMYfSkODJNQjNDC3oW1PqPoxMwo0pLUsWM/UEGzON/NxeHywEfNXNP3Vw==}
engines: {node: '>=0.10.0'}
stackback@0.0.2:
resolution: {integrity: sha512-1XMJE5fQo1jGH6Y/7ebnwPOBEkIEnT4QF32d5R1+VXdXveM0IBMJt8zfaxX1P3QhVwrYe+576+jkANtSS2mBbw==}
std-env@3.10.0:
resolution: {integrity: sha512-5GS12FdOZNliM5mAOxFRg7Ir0pWz8MdpYm6AY6VPkGpbA7ZzmbzNcBJQ0GPvvyWgcY7QAhCgf9Uy89I03faLkg==}
tinybench@2.9.0:
resolution: {integrity: sha512-0+DUvqWMValLmha6lr4kD8iAMK1HzV0/aKnCtWb9v9641TnP/MFb7Pc2bxoxQjTXAErryXVgUOfv2YqNllqGeg==}
tinyexec@0.3.2:
resolution: {integrity: sha512-KQQR9yN7R5+OSwaK0XQoj22pwHoTlgYqmUscPYoknOoWCWfj/5/ABTMRi69FrKU5ffPVh5QcFikpWJI/P1ocHA==}
tinypool@1.1.1:
resolution: {integrity: sha512-Zba82s87IFq9A9XmjiX5uZA/ARWDrB03OHlq+Vw1fSdt0I+4/Kutwy8BP4Y/y/aORMo61FQ0vIb5j44vSo5Pkg==}
engines: {node: ^18.0.0 || >=20.0.0}
tinyrainbow@1.2.0:
resolution: {integrity: sha512-weEDEq7Z5eTHPDh4xjX789+fHfF+P8boiFB+0vbWzpbnbsEr/GRaohi/uMKxg8RZMXnl1ItAi/IUHWMsjDV7kQ==}
engines: {node: '>=14.0.0'}
tinyspy@3.0.2:
resolution: {integrity: sha512-n1cw8k1k0x4pgA2+9XrOkFydTerNcJ1zWCO5Nn9scWHTD+5tp8dghT2x1uduQePZTZgd3Tupf+x9BxJjeJi77Q==}
engines: {node: '>=14.0.0'}
vite-node@2.1.9:
resolution: {integrity: sha512-AM9aQ/IPrW/6ENLQg3AGY4K1N2TGZdR5e4gu/MmmR2xR3Ll1+dib+nook92g4TV3PXVyeyxdWwtaCAiUL0hMxA==}
engines: {node: ^18.0.0 || >=20.0.0}
hasBin: true
vite@5.4.21:
resolution: {integrity: sha512-o5a9xKjbtuhY6Bi5S3+HvbRERmouabWbyUcpXXUA1u+GNUKoROi9byOJ8M0nHbHYHkYICiMlqxkg1KkYmm25Sw==}
engines: {node: ^18.0.0 || >=20.0.0}
hasBin: true
peerDependencies:
'@types/node': ^18.0.0 || >=20.0.0
less: '*'
lightningcss: ^1.21.0
sass: '*'
sass-embedded: '*'
stylus: '*'
sugarss: '*'
terser: ^5.4.0
peerDependenciesMeta:
'@types/node':
optional: true
less:
optional: true
lightningcss:
optional: true
sass:
optional: true
sass-embedded:
optional: true
stylus:
optional: true
sugarss:
optional: true
terser:
optional: true
vitest@2.1.9:
resolution: {integrity: sha512-MSmPM9REYqDGBI8439mA4mWhV5sKmDlBKWIYbA3lRb2PTHACE0mgKwA8yQ2xq9vxDTuk4iPrECBAEW2aoFXY0Q==}
engines: {node: ^18.0.0 || >=20.0.0}
hasBin: true
peerDependencies:
'@edge-runtime/vm': '*'
'@types/node': ^18.0.0 || >=20.0.0
'@vitest/browser': 2.1.9
'@vitest/ui': 2.1.9
happy-dom: '*'
jsdom: '*'
peerDependenciesMeta:
'@edge-runtime/vm':
optional: true
'@types/node':
optional: true
'@vitest/browser':
optional: true
'@vitest/ui':
optional: true
happy-dom:
optional: true
jsdom:
optional: true
why-is-node-running@2.3.0:
resolution: {integrity: sha512-hUrmaWBdVDcxvYqnyh09zunKzROWjbZTiNy8dBEjkS7ehEDQibXJ7XvlmtbwuTclUiIyN+CyXQD4Vmko8fNm8w==}
engines: {node: '>=8'}
hasBin: true
snapshots:
'@esbuild/aix-ppc64@0.21.5':
optional: true
'@esbuild/android-arm64@0.21.5':
optional: true
'@esbuild/android-arm@0.21.5':
optional: true
'@esbuild/android-x64@0.21.5':
optional: true
'@esbuild/darwin-arm64@0.21.5':
optional: true
'@esbuild/darwin-x64@0.21.5':
optional: true
'@esbuild/freebsd-arm64@0.21.5':
optional: true
'@esbuild/freebsd-x64@0.21.5':
optional: true
'@esbuild/linux-arm64@0.21.5':
optional: true
'@esbuild/linux-arm@0.21.5':
optional: true
'@esbuild/linux-ia32@0.21.5':
optional: true
'@esbuild/linux-loong64@0.21.5':
optional: true
'@esbuild/linux-mips64el@0.21.5':
optional: true
'@esbuild/linux-ppc64@0.21.5':
optional: true
'@esbuild/linux-riscv64@0.21.5':
optional: true
'@esbuild/linux-s390x@0.21.5':
optional: true
'@esbuild/linux-x64@0.21.5':
optional: true
'@esbuild/netbsd-x64@0.21.5':
optional: true
'@esbuild/openbsd-x64@0.21.5':
optional: true
'@esbuild/sunos-x64@0.21.5':
optional: true
'@esbuild/win32-arm64@0.21.5':
optional: true
'@esbuild/win32-ia32@0.21.5':
optional: true
'@esbuild/win32-x64@0.21.5':
optional: true
'@jridgewell/sourcemap-codec@1.6.0': {}
'@napi-rs/lzma-linux-x64-gnu@1.5.1':
optional: true
'@rollup/rollup-android-arm-eabi@4.64.2':
optional: true
'@rollup/rollup-android-arm64@4.64.2':
optional: true
'@rollup/rollup-darwin-arm64@4.64.2':
optional: true
'@rollup/rollup-darwin-x64@4.64.2':
optional: true
'@rollup/rollup-freebsd-arm64@4.64.2':
optional: true
'@rollup/rollup-freebsd-x64@4.64.2':
optional: true
'@rollup/rollup-linux-arm-gnueabihf@4.64.2':
optional: true
'@rollup/rollup-linux-arm-musleabihf@4.64.2':
optional: true
'@rollup/rollup-linux-arm64-gnu@4.64.2':
optional: true
'@rollup/rollup-linux-arm64-musl@4.64.2':
optional: true
'@rollup/rollup-linux-loong64-gnu@4.64.2':
optional: true
'@rollup/rollup-linux-loong64-musl@4.64.2':
optional: true
'@rollup/rollup-linux-ppc64-gnu@4.64.2':
optional: true
'@rollup/rollup-linux-ppc64-musl@4.64.2':
optional: true
'@rollup/rollup-linux-riscv64-gnu@4.64.2':
optional: true
'@rollup/rollup-linux-riscv64-musl@4.64.2':
optional: true
'@rollup/rollup-linux-s390x-gnu@4.64.2':
optional: true
'@rollup/rollup-linux-x64-gnu@4.64.2':
optional: true
'@rollup/rollup-linux-x64-musl@4.64.2':
optional: true
'@rollup/rollup-openbsd-x64@4.64.2':
optional: true
'@rollup/rollup-openharmony-arm64@4.64.2':
optional: true
'@rollup/rollup-win32-arm64-msvc@4.64.2':
optional: true
'@rollup/rollup-win32-ia32-msvc@4.64.2':
optional: true
'@rollup/rollup-win32-x64-gnu@4.64.2':
optional: true
'@rollup/rollup-win32-x64-msvc@4.64.2':
optional: true
'@types/estree@1.0.9': {}
'@vitest/expect@2.1.9':
dependencies:
'@vitest/spy': 2.1.9
'@vitest/utils': 2.1.9
chai: 5.3.3
tinyrainbow: 1.2.0
'@vitest/mocker@2.1.9(vite@5.4.21)':
dependencies:
'@vitest/spy': 2.1.9
estree-walker: 3.0.3
magic-string: 0.30.21
optionalDependencies:
vite: 5.4.21
'@vitest/pretty-format@2.1.9':
dependencies:
tinyrainbow: 1.2.0
'@vitest/runner@2.1.9':
dependencies:
'@vitest/utils': 2.1.9
pathe: 1.1.2
'@vitest/snapshot@2.1.9':
dependencies:
'@vitest/pretty-format': 2.1.9
magic-string: 0.30.21
pathe: 1.1.2
'@vitest/spy@2.1.9':
dependencies:
tinyspy: 3.0.2
'@vitest/utils@2.1.9':
dependencies:
'@vitest/pretty-format': 2.1.9
loupe: 3.2.1
tinyrainbow: 1.2.0
assertion-error@2.0.1: {}
cac@6.7.14: {}
chai@5.3.3:
dependencies:
assertion-error: 2.0.1
check-error: 2.1.3
deep-eql: 5.0.2
loupe: 3.2.1
pathval: 2.0.1
check-error@2.1.3: {}
debug@4.4.3:
dependencies:
ms: 2.1.3
deep-eql@5.0.2: {}
es-module-lexer@1.7.0: {}
esbuild@0.21.5:
optionalDependencies:
'@esbuild/aix-ppc64': 0.21.5
'@esbuild/android-arm': 0.21.5
'@esbuild/android-arm64': 0.21.5
'@esbuild/android-x64': 0.21.5
'@esbuild/darwin-arm64': 0.21.5
'@esbuild/darwin-x64': 0.21.5
'@esbuild/freebsd-arm64': 0.21.5
'@esbuild/freebsd-x64': 0.21.5
'@esbuild/linux-arm': 0.21.5
'@esbuild/linux-arm64': 0.21.5
'@esbuild/linux-ia32': 0.21.5
'@esbuild/linux-loong64': 0.21.5
'@esbuild/linux-mips64el': 0.21.5
'@esbuild/linux-ppc64': 0.21.5
'@esbuild/linux-riscv64': 0.21.5
'@esbuild/linux-s390x': 0.21.5
'@esbuild/linux-x64': 0.21.5
'@esbuild/netbsd-x64': 0.21.5
'@esbuild/openbsd-x64': 0.21.5
'@esbuild/sunos-x64': 0.21.5
'@esbuild/win32-arm64': 0.21.5
'@esbuild/win32-ia32': 0.21.5
'@esbuild/win32-x64': 0.21.5
estree-walker@3.0.3:
dependencies:
'@types/estree': 1.0.9
expect-type@1.4.0: {}
fsevents@2.3.3:
optional: true
loupe@3.2.1: {}
magic-string@0.30.21:
dependencies:
'@jridgewell/sourcemap-codec': 1.6.0
ms@2.1.3: {}
nanoid@3.3.20: {}
pathe@1.1.2: {}
pathval@2.0.1: {}
picocolors@1.1.1: {}
postcss@8.5.29:
dependencies:
nanoid: 3.3.20
picocolors: 1.1.1
source-map-js: 1.2.2
rollup@4.64.2:
dependencies:
'@types/estree': 1.0.9
optionalDependencies:
'@napi-rs/lzma-linux-x64-gnu': 1.5.1
'@rollup/rollup-android-arm-eabi': 4.64.2
'@rollup/rollup-android-arm64': 4.64.2
'@rollup/rollup-darwin-arm64': 4.64.2
'@rollup/rollup-darwin-x64': 4.64.2
'@rollup/rollup-freebsd-arm64': 4.64.2
'@rollup/rollup-freebsd-x64': 4.64.2
'@rollup/rollup-linux-arm-gnueabihf': 4.64.2
'@rollup/rollup-linux-arm-musleabihf': 4.64.2
'@rollup/rollup-linux-arm64-gnu': 4.64.2
'@rollup/rollup-linux-arm64-musl': 4.64.2
'@rollup/rollup-linux-loong64-gnu': 4.64.2
'@rollup/rollup-linux-loong64-musl': 4.64.2
'@rollup/rollup-linux-ppc64-gnu': 4.64.2
'@rollup/rollup-linux-ppc64-musl': 4.64.2
'@rollup/rollup-linux-riscv64-gnu': 4.64.2
'@rollup/rollup-linux-riscv64-musl': 4.64.2
'@rollup/rollup-linux-s390x-gnu': 4.64.2
'@rollup/rollup-linux-x64-gnu': 4.64.2
'@rollup/rollup-linux-x64-musl': 4.64.2
'@rollup/rollup-openbsd-x64': 4.64.2
'@rollup/rollup-openharmony-arm64': 4.64.2
'@rollup/rollup-win32-arm64-msvc': 4.64.2
'@rollup/rollup-win32-ia32-msvc': 4.64.2
'@rollup/rollup-win32-x64-gnu': 4.64.2
'@rollup/rollup-win32-x64-msvc': 4.64.2
fsevents: 2.3.3
siginfo@2.0.0: {}
source-map-js@1.2.2: {}
stackback@0.0.2: {}
std-env@3.10.0: {}
tinybench@2.9.0: {}
tinyexec@0.3.2: {}
tinypool@1.1.1: {}
tinyrainbow@1.2.0: {}
tinyspy@3.0.2: {}
vite-node@2.1.9:
dependencies:
cac: 6.7.14
debug: 4.4.3
es-module-lexer: 1.7.0
pathe: 1.1.2
vite: 5.4.21
transitivePeerDependencies:
- '@types/node'
- less
- lightningcss
- sass
- sass-embedded
- stylus
- sugarss
- supports-color
- terser
vite@5.4.21:
dependencies:
esbuild: 0.21.5
postcss: 8.5.29
rollup: 4.64.2
optionalDependencies:
fsevents: 2.3.3
vitest@2.1.9:
dependencies:
'@vitest/expect': 2.1.9
'@vitest/mocker': 2.1.9(vite@5.4.21)
'@vitest/pretty-format': 2.1.9
'@vitest/runner': 2.1.9
'@vitest/snapshot': 2.1.9
'@vitest/spy': 2.1.9
'@vitest/utils': 2.1.9
chai: 5.3.3
debug: 4.4.3
expect-type: 1.4.0
magic-string: 0.30.21
pathe: 1.1.2
std-env: 3.10.0
tinybench: 2.9.0
tinyexec: 0.3.2
tinypool: 1.1.1
tinyrainbow: 1.2.0
vite: 5.4.21
vite-node: 2.1.9
why-is-node-running: 2.3.0
transitivePeerDependencies:
- less
- lightningcss
- msw
- sass
- sass-embedded
- stylus
- sugarss
- supports-color
- terser
why-is-node-running@2.3.0:
dependencies:
siginfo: 2.0.0
stackback: 0.0.2
hidden/pnpm-lock.yaml on GitHub
ihidden/pnpm-workspace.yaml
allowBuilds:
esbuild: true
hidden/pnpm-workspace.yaml on GitHub
to jej konfiguracja. -reference/F1.patch
diff --git a/src/domain/invoice.ts b/src/domain/invoice.ts
index 5b69fc9..0b9b69f 100644
--- a/src/domain/invoice.ts
+++ b/src/domain/invoice.ts
@@ -1,4 +1,4 @@
-import { ConflictError } from "./errors";
+import { ConflictError, ValidationError } from "./errors";
import { type Cents, type Currency, sum } from "./money";
export type InvoiceStatus = "draft" | "open" | "paid" | "void";
@@ -10,6 +10,12 @@ export type LineItem = {
unitPrice: Cents;
};
+export type Payment = {
+ /** cents, > 0 */
+ amount: Cents;
+ receivedAt: string;
+};
+
export type Invoice = {
id: string;
/** Assigned when the invoice is issued (INV-0001, ...). */
@@ -25,6 +31,8 @@ export type Invoice = {
dueDate: string | null;
paidAt: string | null;
voidedAt: string | null;
+ /** Every payment received, oldest first. A paid invoice's payments add up to its total. */
+ payments: Payment[];
};
export type InvoiceTotals = { subtotal: Cents; discount: Cents; total: Cents };
@@ -38,6 +46,14 @@ export function computeTotals(invoice: Pick<Invoice, "lines" | "discount">): Inv
return { subtotal, discount: invoice.discount, total: subtotal - invoice.discount };
}
+export function amountPaid(invoice: Pick<Invoice, "payments">): Cents {
+ return sum(invoice.payments.map((p) => p.amount));
+}
+
+export function balance(invoice: Pick<Invoice, "lines" | "discount" | "payments">): Cents {
+ return computeTotals(invoice).total - amountPaid(invoice);
+}
+
export function issue(invoice: Invoice, number: string, now: Date): Invoice {
if (invoice.status !== "draft") {
throw new ConflictError(`invoice ${invoice.id} is ${invoice.status}, only drafts can be issued`);
@@ -45,11 +61,28 @@ export function issue(invoice: Invoice, number: string, now: Date): Invoice {
return { ...invoice, status: "open", number, issuedAt: now.toISOString() };
}
+/** Pays whatever is left on an open invoice in one go. */
export function markPaid(invoice: Invoice, now: Date): Invoice {
if (invoice.status !== "open") {
throw new ConflictError(`invoice ${invoice.id} is ${invoice.status}, only open invoices can be paid`);
}
- return { ...invoice, status: "paid", paidAt: now.toISOString() };
+ const remaining = balance(invoice);
+ const payments = remaining > 0 ? [...invoice.payments, { amount: remaining, receivedAt: now.toISOString() }] : invoice.payments;
+ return { ...invoice, payments, status: "paid", paidAt: now.toISOString() };
+}
+
+/** Records a partial (or final) payment; the invoice flips to paid when its balance reaches zero. */
+export function recordPayment(invoice: Invoice, amount: Cents, now: Date): Invoice {
+ if (invoice.status !== "open") {
+ throw new ConflictError(`invoice ${invoice.id} is ${invoice.status}, only open invoices can take payments`);
+ }
+ const remaining = balance(invoice);
+ if (amount > remaining) {
+ throw new ValidationError(`amount ${amount} exceeds the balance of ${remaining}`, "amount");
+ }
+ const payments = [...invoice.payments, { amount, receivedAt: now.toISOString() }];
+ const settled = amount === remaining;
+ return { ...invoice, payments, status: settled ? "paid" : "open", paidAt: settled ? now.toISOString() : null };
}
export function voidInvoice(invoice: Invoice, now: Date): Invoice {
diff --git a/src/domain/validation.ts b/src/domain/validation.ts
index 08dfa92..f2af418 100644
--- a/src/domain/validation.ts
+++ b/src/domain/validation.ts
@@ -10,6 +10,13 @@ export type NewInvoiceInput = {
dueDate: string | null;
};
+export function parsePayment(body: unknown): { amount: number } {
+ if (typeof body !== "object" || body === null) {
+ throw new ValidationError("body must be an object");
+ }
+ return { amount: assertCents((body as Record<string, unknown>).amount, "amount", { min: 1 }) };
+}
+
const DATE = /^\d{4}-\d{2}-\d{2}$/;
export function parseLineItems(value: unknown): LineItem[] {
diff --git a/src/http/handlers/invoices.ts b/src/http/handlers/invoices.ts
index 09959e1..ac2768e 100644
--- a/src/http/handlers/invoices.ts
+++ b/src/http/handlers/invoices.ts
@@ -1,6 +1,6 @@
import { ValidationError } from "../../domain/errors";
-import { type Invoice, type InvoiceStatus, issue, markPaid, voidInvoice } from "../../domain/invoice";
-import { parseNewInvoice } from "../../domain/validation";
+import { type Invoice, type InvoiceStatus, issue, markPaid, recordPayment, voidInvoice } from "../../domain/invoice";
+import { parseNewInvoice, parsePayment } from "../../domain/validation";
import type { AppContext } from "../context";
import type { Router } from "../router";
import { invoiceView } from "../views";
@@ -22,6 +22,7 @@ export function invoiceRoutes(router: Router, ctx: AppContext): void {
issuedAt: null,
paidAt: null,
voidedAt: null,
+ payments: [],
};
return { status: 201, body: invoiceView(ctx.invoices.insert(invoice)) };
});
@@ -51,6 +52,14 @@ export function invoiceRoutes(router: Router, ctx: AppContext): void {
return { status: 200, body: invoiceView(ctx.invoices.save(paid)) };
});
+ /** Records a partial payment; the last one (balance reaches zero) marks the invoice paid. */
+ router.on("POST", "/invoices/:id/payments", ({ params, body }) => {
+ const current = ctx.invoices.get(params.id);
+ const { amount } = parsePayment(body);
+ const updated = recordPayment(current, amount, ctx.now());
+ return { status: 201, body: invoiceView(ctx.invoices.save(updated)) };
+ });
+
router.on("POST", "/invoices/:id/void", ({ params }) => {
const voided = voidInvoice(ctx.invoices.get(params.id), ctx.now());
return { status: 200, body: invoiceView(ctx.invoices.save(voided)) };
diff --git a/src/http/handlers/reports.ts b/src/http/handlers/reports.ts
index 19262c9..656969e 100644
--- a/src/http/handlers/reports.ts
+++ b/src/http/handlers/reports.ts
@@ -1,18 +1,17 @@
-import { computeTotals, isOverdue } from "../../domain/invoice";
+import { balance, isOverdue } from "../../domain/invoice";
import type { Currency } from "../../domain/money";
import type { AppContext } from "../context";
import type { Router } from "../router";
export function reportRoutes(router: Router, ctx: AppContext): void {
- /** What customers still owe us, per currency, over every open invoice. */
+ /** What customers still owe us (balances, net of partial payments), per currency, over every open invoice. */
router.on("GET", "/reports/outstanding", () => {
const now = ctx.now();
const totals: Partial<Record<Currency, number>> = {};
let count = 0;
let overdue = 0;
for (const invoice of ctx.invoices.list({ status: "open" })) {
- const { total } = computeTotals(invoice);
- totals[invoice.currency] = (totals[invoice.currency] ?? 0) + total;
+ totals[invoice.currency] = (totals[invoice.currency] ?? 0) + balance(invoice);
count += 1;
if (isOverdue(invoice, now)) overdue += 1;
}
diff --git a/src/http/views.ts b/src/http/views.ts
index 856944c..5b11bff 100644
--- a/src/http/views.ts
+++ b/src/http/views.ts
@@ -1,6 +1,6 @@
-import { computeTotals, type Invoice } from "../domain/invoice";
+import { amountPaid, balance, computeTotals, type Invoice } from "../domain/invoice";
/** The JSON shape returned for an invoice: stored fields plus computed totals. */
export function invoiceView(invoice: Invoice) {
- return { ...invoice, ...computeTotals(invoice) };
+ return { ...invoice, ...computeTotals(invoice), amountPaid: amountPaid(invoice), balance: balance(invoice) };
}
diff --git a/src/store/migrations.ts b/src/store/migrations.ts
index c8b73a9..1eab4df 100644
--- a/src/store/migrations.ts
+++ b/src/store/migrations.ts
@@ -1,6 +1,7 @@
+import { computeTotals, type Invoice } from "../domain/invoice";
import type { StoreData } from "./types";
-export const CURRENT_SCHEMA_VERSION = 1;
+export const CURRENT_SCHEMA_VERSION = 2;
type RawData = Record<string, unknown> & { schemaVersion?: number };
type Migration = (data: RawData) => RawData;
@@ -21,9 +22,31 @@ const v0ToV1: Migration = (data) => {
};
};
+/**
+ * v1 -> v2: invoices carry `payments`. Invoices paid before payments existed were paid in full:
+ * they get one payment of their total at `paidAt`, so their balance is zero.
+ */
+const v1ToV2: Migration = (data) => {
+ const invoices = (data.invoices as Record<string, unknown>[] | undefined) ?? [];
+ return {
+ ...data,
+ schemaVersion: 2,
+ invoices: invoices.map((inv) => {
+ if (Array.isArray(inv.payments)) return inv;
+ const invoice = inv as unknown as Invoice;
+ const payments =
+ invoice.status === "paid"
+ ? [{ amount: computeTotals(invoice).total, receivedAt: invoice.paidAt ?? invoice.issuedAt ?? invoice.createdAt }]
+ : [];
+ return { ...inv, payments };
+ }),
+ };
+};
+
/** migrations[n] upgrades a file from version n to n + 1. */
const migrations: Record<number, Migration> = {
0: v0ToV1,
+ 1: v1ToV2,
};
export function migrate(raw: unknown): StoreData {
diff --git a/test/invoice.test.ts b/test/invoice.test.ts
index 85202f2..e043b5d 100644
--- a/test/invoice.test.ts
+++ b/test/invoice.test.ts
@@ -21,6 +21,7 @@ function draft(overrides: Partial<Invoice> = {}): Invoice {
dueDate: null,
paidAt: null,
voidedAt: null,
+ payments: [],
...overrides,
};
}
diff --git a/test/payments.test.ts b/test/payments.test.ts
new file mode 100644
index 0000000..baf9da7
--- /dev/null
+++ b/test/payments.test.ts
@@ -0,0 +1,53 @@
+import { describe, expect, it } from "vitest";
+import { openStore } from "../src/store/json-store";
+import { seedCustomer, seedOpenInvoice, testApp } from "./helpers";
+
+describe("partial payments", () => {
+ it("accumulates payments and flips to paid at a zero balance", async () => {
+ const app = testApp();
+ const id = await seedOpenInvoice(app, { customerId: await seedCustomer(app), unitPrice: 10_000 });
+ expect((await app.handle("POST", `/invoices/${id}/payments`, { amount: 4_000 })).body).toMatchObject({
+ status: "open",
+ amountPaid: 4_000,
+ balance: 6_000,
+ });
+ const last = await app.handle("POST", `/invoices/${id}/payments`, { amount: 6_000 });
+ expect(last).toMatchObject({ status: 201, body: { status: "paid", amountPaid: 10_000, balance: 0 } });
+ });
+
+ it("rejects bad amounts and overpayments with 400", async () => {
+ const app = testApp();
+ const id = await seedOpenInvoice(app, { customerId: await seedCustomer(app), unitPrice: 10_000 });
+ for (const amount of [0, -1, 1.5, "10", undefined]) {
+ expect((await app.handle("POST", `/invoices/${id}/payments`, { amount })).status).toBe(400);
+ }
+ expect((await app.handle("POST", `/invoices/${id}/payments`, { amount: 10_001 })).status).toBe(400);
+ });
+
+ it("refuses payments on drafts, void and unknown invoices", async () => {
+ const app = testApp();
+ const customerId = await seedCustomer(app);
+ const draft = await app.handle("POST", "/invoices", {
+ customerId,
+ currency: "EUR",
+ lines: [{ description: "x", quantity: 1, unitPrice: 100 }],
+ });
+ expect((await app.handle("POST", `/invoices/${(draft.body as { id: string }).id}/payments`, { amount: 10 })).status).toBe(409);
+ const voided = await seedOpenInvoice(app, { customerId });
+ await app.handle("POST", `/invoices/${voided}/void`);
+ expect((await app.handle("POST", `/invoices/${voided}/payments`, { amount: 10 })).status).toBe(409);
+ expect((await app.handle("POST", "/invoices/inv_404/payments", { amount: 10 })).status).toBe(404);
+ });
+
+ it("keeps /pay consistent and reports balances, not totals", async () => {
+ const app = testApp(openStore());
+ const customerId = await seedCustomer(app);
+ const a = await seedOpenInvoice(app, { customerId, unitPrice: 10_000 });
+ const b = await seedOpenInvoice(app, { customerId, unitPrice: 5_000 });
+ await app.handle("POST", `/invoices/${a}/payments`, { amount: 2_500 });
+ expect((await app.handle("GET", "/reports/outstanding")).body).toMatchObject({ totals: { EUR: 12_500 } });
+ expect((await app.handle("POST", `/invoices/${a}/pay`)).body).toMatchObject({ status: "paid", amountPaid: 10_000, balance: 0 });
+ expect((await app.handle("GET", "/reports/outstanding")).body).toMatchObject({ totals: { EUR: 5_000 }, count: 1 });
+ expect(b).toBeTruthy();
+ });
+});
diff --git a/test/store.test.ts b/test/store.test.ts
index a1bf345..3201e2a 100644
--- a/test/store.test.ts
+++ b/test/store.test.ts
@@ -42,6 +42,26 @@ describe("store", () => {
expect(JSON.parse(readFileSync(file, "utf8")).schemaVersion).toBe(CURRENT_SCHEMA_VERSION);
});
+ it("migrates a v1 file: invoices paid before payments existed get one payment of their total", () => {
+ const file = tmpFile();
+ const inv = { customerId: "cus_1", currency: "EUR", lines: [{ description: "x", quantity: 2, unitPrice: 1_500 }], discount: 0 };
+ writeFileSync(
+ file,
+ JSON.stringify({
+ schemaVersion: 1,
+ counters: { customer: 1, invoice: 2, invoiceNumber: 2 },
+ customers: [],
+ invoices: [
+ { ...inv, id: "inv_1", status: "paid", paidAt: "2026-02-01T00:00:00.000Z" },
+ { ...inv, id: "inv_2", status: "open", paidAt: null },
+ ],
+ }),
+ );
+ const [paid, open] = openStore({ file }).read().invoices;
+ expect(paid.payments).toEqual([{ amount: 3_000, receivedAt: "2026-02-01T00:00:00.000Z" }]);
+ expect(open.payments).toEqual([]);
+ });
+
it("refuses a file from a newer build", () => {
expect(() => migrate({ schemaVersion: CURRENT_SCHEMA_VERSION + 1 })).toThrow(/newer/);
});
reference/F2.patch
diff --git a/src/domain/credit-note.ts b/src/domain/credit-note.ts
new file mode 100644
index 0000000..7504c53
--- /dev/null
+++ b/src/domain/credit-note.ts
@@ -0,0 +1,44 @@
+import { ConflictError, ValidationError } from "./errors";
+import { computeTotals, type Invoice } from "./invoice";
+import { assertCents, type Cents, sum } from "./money";
+
+/** A numbered accounting document that reduces what is owed on an issued invoice. Never edited or deleted. */
+export type CreditNote = {
+ id: string;
+ /** CN-0001, ... gapless, in the order credit notes are created. */
+ number: string;
+ invoiceId: string;
+ amount: Cents;
+ createdAt: string;
+};
+
+export function creditedAmount(notes: Pick<CreditNote, "amount">[]): Cents {
+ return sum(notes.map((n) => n.amount));
+}
+
+/** What can still be credited on an invoice: its total minus every credit note already issued against it. */
+export function creditable(invoice: Invoice, notes: Pick<CreditNote, "amount">[]): Cents {
+ return computeTotals(invoice).total - creditedAmount(notes);
+}
+
+export function parseCreditAmount(body: unknown): Cents {
+ if (typeof body !== "object" || body === null) {
+ throw new ValidationError("body must be an object");
+ }
+ return assertCents((body as Record<string, unknown>).amount, "amount", { min: 1 });
+}
+
+/** Throws unless `amount` can be credited on `invoice` now. Call it BEFORE taking a number. */
+export function assertCreditable(invoice: Invoice, notes: Pick<CreditNote, "amount">[], amount: Cents): void {
+ if (invoice.status !== "open" && invoice.status !== "paid") {
+ throw new ConflictError(`invoice ${invoice.id} is ${invoice.status}: only open or paid invoices can be credited`);
+ }
+ const left = creditable(invoice, notes);
+ if (amount > left) {
+ throw new ValidationError(`amount ${amount} exceeds what is left to credit (${left})`, "amount");
+ }
+}
+
+export function creditNoteNumber(n: number): string {
+ return `CN-${String(n).padStart(4, "0")}`;
+}
diff --git a/src/domain/invoice.ts b/src/domain/invoice.ts
index 5b69fc9..2ee6fa4 100644
--- a/src/domain/invoice.ts
+++ b/src/domain/invoice.ts
@@ -52,8 +52,9 @@ export function markPaid(invoice: Invoice, now: Date): Invoice {
return { ...invoice, status: "paid", paidAt: now.toISOString() };
}
+/** Voids a draft, open or paid invoice. A paid one must be fully credited first (the handler issues that credit note). */
export function voidInvoice(invoice: Invoice, now: Date): Invoice {
- if (invoice.status === "paid" || invoice.status === "void") {
+ if (invoice.status === "void") {
throw new ConflictError(`invoice ${invoice.id} is ${invoice.status} and cannot be voided`);
}
return { ...invoice, status: "void", voidedAt: now.toISOString() };
diff --git a/src/http/context.ts b/src/http/context.ts
index c0a89d3..003b7af 100644
--- a/src/http/context.ts
+++ b/src/http/context.ts
@@ -1,4 +1,5 @@
import type { Store } from "../store/json-store";
+import { creditNoteRepo } from "../store/credit-note-repo";
import { customerRepo } from "../store/customer-repo";
import { invoiceRepo } from "../store/invoice-repo";
@@ -7,8 +8,9 @@ export type AppContext = {
now: () => Date;
customers: ReturnType<typeof customerRepo>;
invoices: ReturnType<typeof invoiceRepo>;
+ creditNotes: ReturnType<typeof creditNoteRepo>;
};
export function createContext(store: Store, now: () => Date = () => new Date()): AppContext {
- return { store, now, customers: customerRepo(store), invoices: invoiceRepo(store) };
+ return { store, now, customers: customerRepo(store), invoices: invoiceRepo(store), creditNotes: creditNoteRepo(store) };
}
diff --git a/src/http/handlers/invoices.ts b/src/http/handlers/invoices.ts
index 09959e1..469a402 100644
--- a/src/http/handlers/invoices.ts
+++ b/src/http/handlers/invoices.ts
@@ -1,3 +1,4 @@
+import { assertCreditable, type CreditNote, creditNoteNumber, creditable, parseCreditAmount } from "../../domain/credit-note";
import { ValidationError } from "../../domain/errors";
import { type Invoice, type InvoiceStatus, issue, markPaid, voidInvoice } from "../../domain/invoice";
import { parseNewInvoice } from "../../domain/validation";
@@ -51,8 +52,37 @@ export function invoiceRoutes(router: Router, ctx: AppContext): void {
return { status: 200, body: invoiceView(ctx.invoices.save(paid)) };
});
+ /** Validation first, number last: a rejected credit note never consumes a CN number (numbers stay gapless). */
+ function credit(invoice: Invoice, amount: number): CreditNote {
+ assertCreditable(invoice, ctx.creditNotes.forInvoice(invoice.id), amount);
+ const n = ctx.store.nextId("creditNote");
+ return ctx.creditNotes.insert({
+ id: `cn_${n}`,
+ number: creditNoteNumber(n),
+ invoiceId: invoice.id,
+ amount,
+ createdAt: ctx.now().toISOString(),
+ });
+ }
+
+ router.on("POST", "/invoices/:id/credit-notes", ({ params, body }) => {
+ const invoice = ctx.invoices.get(params.id);
+ return { status: 201, body: credit(invoice, parseCreditAmount(body)) };
+ });
+
+ router.on("GET", "/invoices/:id/credit-notes", ({ params }) => {
+ ctx.invoices.get(params.id);
+ return { status: 200, body: ctx.creditNotes.forInvoice(params.id) };
+ });
+
+ /** Voiding a paid invoice credits whatever is left on it first. */
router.on("POST", "/invoices/:id/void", ({ params }) => {
- const voided = voidInvoice(ctx.invoices.get(params.id), ctx.now());
+ const invoice = ctx.invoices.get(params.id);
+ const voided = voidInvoice(invoice, ctx.now());
+ if (invoice.status === "paid") {
+ const left = creditable(invoice, ctx.creditNotes.forInvoice(invoice.id));
+ if (left > 0) credit(invoice, left);
+ }
return { status: 200, body: invoiceView(ctx.invoices.save(voided)) };
});
}
diff --git a/src/http/handlers/reports.ts b/src/http/handlers/reports.ts
index 19262c9..f980c9b 100644
--- a/src/http/handlers/reports.ts
+++ b/src/http/handlers/reports.ts
@@ -1,4 +1,5 @@
-import { computeTotals, isOverdue } from "../../domain/invoice";
+import { creditable } from "../../domain/credit-note";
+import { isOverdue } from "../../domain/invoice";
import type { Currency } from "../../domain/money";
import type { AppContext } from "../context";
import type { Router } from "../router";
@@ -11,8 +12,8 @@ export function reportRoutes(router: Router, ctx: AppContext): void {
let count = 0;
let overdue = 0;
for (const invoice of ctx.invoices.list({ status: "open" })) {
- const { total } = computeTotals(invoice);
- totals[invoice.currency] = (totals[invoice.currency] ?? 0) + total;
+ const owed = creditable(invoice, ctx.creditNotes.forInvoice(invoice.id));
+ totals[invoice.currency] = (totals[invoice.currency] ?? 0) + owed;
count += 1;
if (isOverdue(invoice, now)) overdue += 1;
}
diff --git a/src/store/credit-note-repo.ts b/src/store/credit-note-repo.ts
new file mode 100644
index 0000000..eac6632
--- /dev/null
+++ b/src/store/credit-note-repo.ts
@@ -0,0 +1,17 @@
+import type { CreditNote } from "../domain/credit-note";
+import type { Store } from "./json-store";
+
+export function creditNoteRepo(store: Store) {
+ return {
+ forInvoice(invoiceId: string): CreditNote[] {
+ return store.read().creditNotes.filter((n) => n.invoiceId === invoiceId);
+ },
+ all(): CreditNote[] {
+ return store.read().creditNotes;
+ },
+ insert(note: CreditNote): CreditNote {
+ store.update((d) => ({ ...d, creditNotes: [...d.creditNotes, note] }));
+ return note;
+ },
+ };
+}
diff --git a/src/store/json-store.ts b/src/store/json-store.ts
index b404c51..99205b7 100644
--- a/src/store/json-store.ts
+++ b/src/store/json-store.ts
@@ -13,9 +13,10 @@ export type Store = {
function empty(): StoreData {
return {
schemaVersion: CURRENT_SCHEMA_VERSION,
- counters: { customer: 0, invoice: 0, invoiceNumber: 0 },
+ counters: { customer: 0, invoice: 0, invoiceNumber: 0, creditNote: 0 },
customers: [],
invoices: [],
+ creditNotes: [],
};
}
diff --git a/src/store/migrations.ts b/src/store/migrations.ts
index c8b73a9..eb7abfb 100644
--- a/src/store/migrations.ts
+++ b/src/store/migrations.ts
@@ -1,6 +1,6 @@
import type { StoreData } from "./types";
-export const CURRENT_SCHEMA_VERSION = 1;
+export const CURRENT_SCHEMA_VERSION = 2;
type RawData = Record<string, unknown> & { schemaVersion?: number };
type Migration = (data: RawData) => RawData;
@@ -21,9 +21,18 @@ const v0ToV1: Migration = (data) => {
};
};
+/** v1 -> v2: credit notes (a collection and their own counter). */
+const v1ToV2: Migration = (data) => ({
+ ...data,
+ schemaVersion: 2,
+ counters: { ...(data.counters as Record<string, number> | undefined), creditNote: 0 },
+ creditNotes: [],
+});
+
/** migrations[n] upgrades a file from version n to n + 1. */
const migrations: Record<number, Migration> = {
0: v0ToV1,
+ 1: v1ToV2,
};
export function migrate(raw: unknown): StoreData {
@@ -43,8 +52,9 @@ export function migrate(raw: unknown): StoreData {
}
return {
schemaVersion: CURRENT_SCHEMA_VERSION,
- counters: (data.counters as StoreData["counters"]) ?? { customer: 0, invoice: 0, invoiceNumber: 0 },
+ counters: { customer: 0, invoice: 0, invoiceNumber: 0, creditNote: 0, ...(data.counters as Partial<StoreData["counters"]>) },
customers: (data.customers as StoreData["customers"]) ?? [],
invoices: (data.invoices as StoreData["invoices"]) ?? [],
+ creditNotes: (data.creditNotes as StoreData["creditNotes"]) ?? [],
};
}
diff --git a/src/store/types.ts b/src/store/types.ts
index 222b191..a9d7277 100644
--- a/src/store/types.ts
+++ b/src/store/types.ts
@@ -1,7 +1,8 @@
+import type { CreditNote } from "../domain/credit-note";
import type { Customer } from "../domain/customer";
import type { Invoice } from "../domain/invoice";
-export type Counters = { customer: number; invoice: number; invoiceNumber: number };
+export type Counters = { customer: number; invoice: number; invoiceNumber: number; creditNote: number };
/** The shape of the JSON file at CURRENT_SCHEMA_VERSION. */
export type StoreData = {
@@ -9,4 +10,5 @@ export type StoreData = {
counters: Counters;
customers: Customer[];
invoices: Invoice[];
+ creditNotes: CreditNote[];
};
diff --git a/test/credit-notes.test.ts b/test/credit-notes.test.ts
new file mode 100644
index 0000000..240f820
--- /dev/null
+++ b/test/credit-notes.test.ts
@@ -0,0 +1,52 @@
+import { mkdtempSync, writeFileSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+import { describe, expect, it } from "vitest";
+import { openStore } from "../src/store/json-store";
+import { seedCustomer, seedOpenInvoice, testApp } from "./helpers";
+
+describe("credit notes", () => {
+ it("numbers credit notes gaplessly; rejected ones take no number", async () => {
+ const app = testApp();
+ const id = await seedOpenInvoice(app, { customerId: await seedCustomer(app), unitPrice: 10_000 });
+ expect((await app.handle("POST", `/invoices/${id}/credit-notes`, { amount: 3_000 })).body).toMatchObject({ number: "CN-0001" });
+ expect((await app.handle("POST", `/invoices/${id}/credit-notes`, { amount: 7_001 })).status).toBe(400);
+ expect((await app.handle("POST", `/invoices/${id}/credit-notes`, { amount: 0 })).status).toBe(400);
+ expect((await app.handle("POST", `/invoices/${id}/credit-notes`, { amount: 7_000 })).body).toMatchObject({ number: "CN-0002" });
+ expect(((await app.handle("GET", `/invoices/${id}/credit-notes`)).body as unknown[]).length).toBe(2);
+ });
+
+ it("refuses drafts and void invoices (409) and unknown ones (404)", async () => {
+ const app = testApp();
+ const customerId = await seedCustomer(app);
+ const draft = await app.handle("POST", "/invoices", { customerId, currency: "EUR", lines: [{ description: "x", quantity: 1, unitPrice: 100 }] });
+ expect((await app.handle("POST", `/invoices/${(draft.body as { id: string }).id}/credit-notes`, { amount: 10 })).status).toBe(409);
+ expect((await app.handle("POST", "/invoices/inv_404/credit-notes", { amount: 10 })).status).toBe(404);
+ });
+
+ it("voids a paid invoice by crediting what is left", async () => {
+ const app = testApp();
+ const id = await seedOpenInvoice(app, { customerId: await seedCustomer(app), unitPrice: 10_000 });
+ await app.handle("POST", `/invoices/${id}/pay`);
+ await app.handle("POST", `/invoices/${id}/credit-notes`, { amount: 2_500 });
+ expect((await app.handle("POST", `/invoices/${id}/void`)).body).toMatchObject({ status: "void" });
+ const notes = (await app.handle("GET", `/invoices/${id}/credit-notes`)).body as { amount: number }[];
+ expect(notes.map((n) => n.amount)).toEqual([2_500, 7_500]);
+ });
+
+ it("nets credit notes out of the outstanding report", async () => {
+ const app = testApp();
+ const id = await seedOpenInvoice(app, { customerId: await seedCustomer(app), unitPrice: 10_000 });
+ await app.handle("POST", `/invoices/${id}/credit-notes`, { amount: 4_000 });
+ expect((await app.handle("GET", "/reports/outstanding")).body).toMatchObject({ totals: { EUR: 6_000 }, count: 1 });
+ });
+
+ it("works on a store file written before credit notes existed", async () => {
+ const file = join(mkdtempSync(join(tmpdir(), "ledgerly-")), "data.json");
+ writeFileSync(file, JSON.stringify({ schemaVersion: 1, counters: { customer: 0, invoice: 0, invoiceNumber: 0 }, customers: [], invoices: [] }));
+ const app = testApp(openStore({ file }));
+ const id = await seedOpenInvoice(app, { customerId: await seedCustomer(app) });
+ expect((await app.handle("POST", `/invoices/${id}/credit-notes`, { amount: 100 })).body).toMatchObject({ id: "cn_1", number: "CN-0001" });
+ expect(openStore({ file }).read().creditNotes.length).toBe(1);
+ });
+});
diff --git a/test/invoice.test.ts b/test/invoice.test.ts
index 85202f2..4219226 100644
--- a/test/invoice.test.ts
+++ b/test/invoice.test.ts
@@ -54,8 +54,11 @@ describe("invoice domain", () => {
});
it("voids drafts and open invoices, never paid ones", () => {
+ // Meaning changed with credit notes: a paid invoice can now be voided (the handler credits what is left first);
+ // only an already void invoice is refused.
expect(voidInvoice(draft(), NOW).status).toBe("void");
- expect(() => voidInvoice(draft({ status: "paid" }), NOW)).toThrow(ConflictError);
+ expect(voidInvoice(draft({ status: "paid" }), NOW).status).toBe("void");
+ expect(() => voidInvoice(draft({ status: "void" }), NOW)).toThrow(ConflictError);
});
it("knows when an open invoice is overdue", () => {
Runner i driver:
run.sh
#!/usr/bin/env bash
# One run: fresh work copy + pack fixtures, the driver (worker turns + PO), then finalize.
# Usage: [TASK=F1|F2] run.sh <cfg> <rep> cfg: bare|sp|spd|mp|both
# RUN_TAG=x -> runs/<name>-x (smoke runs; analyze.py ignores tagged runs)
# The work copy lives OUTSIDE the repo tree ($FLOW_BENCH_WORK, default /private/tmp/flow-bench/<name>): Claude Code
# loads every CLAUDE.md of the parent directories, so a work copy under this repo read the repo's CLAUDE.md (and the
# instructions it pulls in) in every session. After finalize the copy (without node_modules) is archived to work/<name>.
set -uo pipefail
CFG="$1"; REP="${2:-1}"
HERE="$(cd "$(dirname "$0")" && pwd)"
. "$HERE/env.sh"
TASK="${TASK:-F1}"; export TASK
PFX=""; [ "$TASK" != "F1" ] && PFX="$(echo "$TASK" | tr "A-Z" "a-z")-"
NAME="${PFX}${CFG}-r${REP}${RUN_TAG:+-$RUN_TAG}"
WROOT="${FLOW_BENCH_WORK:-/private/tmp/flow-bench}"
OUT="$HERE/runs/$NAME"; W="$WROOT/$NAME"
rm -rf "$OUT"; mkdir -p "$OUT"
"$HERE/prep.sh" "$CFG" "$W" > "$OUT/prep.txt" 2>&1 || { echo "$NAME: prep failed"; cat "$OUT/prep.txt"; exit 2; }
BASE=$(tail -1 "$OUT/prep.txt")
cp "$W/.claude/settings.json" "$OUT/settings.json"
printf '{"task":"%s","cfg":"%s","rep":%s,"tag":"%s","work":"%s","name":"%s","base":"%s","started":"%s","claude_version":"%s"}\n' \
"$TASK" "$CFG" "$REP" "${RUN_TAG:-}" "$W" "$NAME" "$BASE" "$(date '+%F %T')" "$(claude --version | head -1)" > "$OUT/meta.json"
python3 "$HERE/lib/driver.py" "$CFG" "$OUT" "$W" "$BASE" > "$OUT/driver.out" 2> "$OUT/driver.err"
python3 "$HERE/lib/finalize.py" "$OUT"
rm -rf "$HERE/work/$NAME"; mkdir -p "$HERE/work"
rsync -a --exclude node_modules "$W/" "$HERE/work/$NAME/"
prep.sh
#!/usr/bin/env bash
# Fresh work copy of upstream (node_modules included) + the pack fixtures of <cfg>.
# Usage: prep.sh <cfg> <work dir> Prints the base commit (the oracle's BASE) on its last line.
set -euo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"
CFG="$1"; W="$2"
SP=$(python3 -c "import json;print(int(json.load(open('$HERE/configs.json'))['$CFG']['superpowers']))")
MP=$(python3 -c "import json;print(int(json.load(open('$HERE/configs.json'))['$CFG']['mattpocock']))")
rm -rf "$W"; mkdir -p "$(dirname "$W")"
cp -R "$HERE/upstream" "$W"
cd "$W"
# .claude/ never shows in git status or in any diff (hook config, installed skills).
echo ".claude/" >> .git/info/exclude
mkdir -p .claude/skills
if [ "$MP" = 1 ]; then
cp -R "$HERE/packs/mattpocock-skills/.claude/skills/." .claude/skills/
# One-time per-repo precondition of the pack (/setup-matt-pocock-skills), answered once with the defaults it
# recommends for a repo with no remote: local-markdown tracker, single-context domain docs, no triage labels
# (triage is installed, so Section B would ask; default labels kept). Committed so it never shows in the diff.
S="$HERE/packs/mattpocock-skills/.claude/skills/setup-matt-pocock-skills"
mkdir -p docs/agents
cp "$S/issue-tracker-local.md" docs/agents/issue-tracker.md
cp "$S/domain.md" docs/agents/domain.md
cp "$S/triage-labels.md" docs/agents/triage-labels.md
cat > CLAUDE.md <<'MD'
## Agent skills
### Issue tracker
Issues and specs live as local markdown files under `.scratch/`. See `docs/agents/issue-tracker.md`.
### Triage labels
Default labels (`needs-triage`, `needs-info`, `ready-for-agent`, `ready-for-human`, `wontfix`). See `docs/agents/triage-labels.md`.
### Domain docs
Single-context: one `GLOSSARY.md` + `docs/adr/` at the repo root. See `docs/agents/domain.md`.
MD
git add CLAUDE.md docs/agents
GIT_AUTHOR_DATE="2026-03-09T18:00:00" GIT_COMMITTER_DATE="2026-03-09T18:00:00" \
git -c user.name="Dana Whitfield" -c user.email="dana@ledgerly.dev" commit -q -m "chore: configure agent skills (local tracker)"
fi
if [ "$SP" = 1 ]; then
# The plugin root as Claude Code would hold it, so the hook script finds ../skills/using-superpowers.
mkdir -p .claude/plugins
cp -R "$HERE/packs/superpowers" .claude/plugins/superpowers
cp -R "$HERE/packs/superpowers/skills/." .claude/skills/
fi
python3 - "$HERE" "$SP" "$CFG" > .claude/settings.json <<'PY'
import json, sys
bench, sp, cfg = sys.argv[1], sys.argv[2] == "1", sys.argv[3]
# Oracle, reference, PO brief, other runs: denied by path (Read/Grep/Glob) and by command text (Bash).
deny = ["Read(/%s/hidden/**)" % bench, "Read(/%s/reference/**)" % bench, "Read(/%s/runs/**)" % bench,
"Read(/%s/po/**)" % bench, "Read(/%s/validation/**)" % bench,
"Bash(*hidden*)", "Bash(*reference/F*)", "Bash(*-brief.md*)", "Bash(*/runs/*)", "Bash(*../*)", "Bash(find *)"]
if cfg == "spd":
# executing-plans runs its sibling scripts as ../subagent-driven-development/scripts/...: a "../" deny would block the
# pack's own loop. The oracle stays protected by the Read denies and Bash(*hidden*).
deny.remove("Bash(*../*)")
settings = {"permissions": {"deny": deny}}
if sp:
# = packs/superpowers/hooks/hooks.json, with ${CLAUDE_PLUGIN_ROOT} provided the way the plugin loader provides it
# (the script also picks its JSON output shape from CLAUDE_PLUGIN_ROOT being set).
root = '$CLAUDE_PROJECT_DIR/.claude/plugins/superpowers'
hooks = json.load(open("%s/packs/superpowers/hooks/hooks.json" % bench))["hooks"]
for groups in hooks.values():
for g in groups:
for h in g["hooks"]:
h["command"] = 'CLAUDE_PLUGIN_ROOT="%s" %s' % (root, h["command"].replace("${CLAUDE_PLUGIN_ROOT}", root))
settings["hooks"] = hooks
print(json.dumps(settings, indent=2))
PY
[ "$(ls .claude/skills | wc -l)" -eq 0 ] && rmdir .claude/skills
test -z "$(git status --porcelain)" || { echo "work copy not clean"; git status --porcelain; exit 1; }
git rev-parse HEAD
env.sh
# Sourced by run.sh, gate.sh and validate.sh.
# 1. Scrub every CLAUDE* variable inherited from a launching Claude Code session
# (CLAUDE_EFFORT, CLAUDE_CODE_SUBAGENT_MODEL, CLAUDE_CODE_SESSION_ID, CLAUDECODE,
# CLAUDE_CODE_CHILD_SESSION, ...): they would silently change the worker's effort,
# subagent model and session bookkeeping.
for _v in $(env | sed -n 's/^\(CLAUDE[A-Za-z0-9_]*\)=.*/\1/p'); do unset "$_v"; done
unset _v
# 2. Auth: the user's default config dir (~/.claude, OAuth). See README traps (--bare cannot auth).
unset CLAUDE_CONFIG_DIR
# 3. Fresh work copies at the same path must not share auto-memory across reps.
export CLAUDE_CODE_DISABLE_AUTO_MEMORY=1
# 4. No bytecode in the shared hidden/ dir when 3 runs execute hidden tests in parallel.
export PYTHONDONTWRITEBYTECODE=1
# 5. The bench venv first: `python -m pytest` inside the work copy uses it.
GATE_BENCH="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
export GATE_BENCH
export PATH="$GATE_BENCH/.venv/bin:$PATH"
configs.json
{
"bare": {
"superpowers": false,
"mattpocock": false,
"flow": "bare",
"desc": "no skills, no hooks (project settings carry only the permission deny-list)"
},
"sp": {
"superpowers": true,
"mattpocock": false,
"flow": "sp",
"desc": "HOOK-ROUTED: obra/superpowers v6.4.2: skills in .claude/skills, SessionStart hook in .claude/settings.json as hooks/hooks.json wires it; turn 1 = the bare task, the hook decides"
},
"mp": {
"superpowers": false,
"mattpocock": true,
"flow": "mp",
"desc": "mattpocock/skills v1.3.1 via npx skills add (27 promoted skills, copied), setup-matt-pocock-skills output committed (local tracker)"
},
"both": {
"superpowers": true,
"mattpocock": true,
"flow": "mp",
"desc": "sp + mp in one project; driven with the mp slash-command flow (the stacking question: does the Superpowers hook hijack it?)"
},
"spd": {
"superpowers": true,
"mattpocock": false,
"flow": "spd",
"desc": "Superpowers, DRIVEN: same project install as sp (hook included); the driver types /brainstorming <task> -> /writing-plans -> /executing-plans (Native/inline); the pack's own scripts allowed in Bash"
}
}
lib/driver.py
"""One run of the flow bench: drives a headless Sonnet 5.5 session turn by turn, routes questions to the PO.
driver.py <cfg> <run dir> <work dir> <base sha>
Every worker turn: `claude -p <prompt> [--resume <sid>] --output-format stream-json --verbose --include-hook-events
--include-partial-messages ...` with cwd = the work copy. Per turn it writes turn-NN.stream.jsonl (raw stdout),
turn-NN.times (receive time of every stdout line), turn-NN.stderr, and appends one record to turns.jsonl.
Caps (SPEC): 12 driver turns, --max-turns 60 per claude turn, 40 min wall per run (PO time included).
"""
import json
import os
import subprocess
import sys
import threading
import time
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import po # noqa: E402
from accounting import account_turn # noqa: E402
BENCH = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
MODEL = "claude-sonnet-5-5"
MAX_DRIVER_TURNS = int(os.environ.get("BENCH_MAX_DRIVER_TURNS", "12"))
MAX_TURNS = os.environ.get("BENCH_MAX_TURNS", "60")
WALL_CAP = float(os.environ.get("BENCH_WALL_CAP_S", str(40 * 60)))
# SPEC allow-list. AskUserQuestion is removed (headless: questions come as plain text); plan mode and worktree
# tools are removed too (a headless session cannot leave plan mode, and a worktree would move the work out of
# the scored copy). Read-only shell commands are auto-approved by Claude Code whatever the list says.
ALLOWED = ",".join([
"Read", "Edit", "Write", "MultiEdit", "Glob", "Grep", "Skill", "TodoWrite", "Task", "Agent",
"Bash(node:*)", "Bash(npm:*)", "Bash(pnpm:*)", "Bash(npx vitest:*)", "Bash(npx tsc:*)", "Bash(tsc:*)",
"Bash(git status:*)", "Bash(git diff:*)", "Bash(git log:*)", "Bash(git add:*)", "Bash(git commit:*)",
"Bash(git checkout -b:*)", "Bash(ls:*)", "Bash(cat:*)", "Bash(mkdir:*)", "Bash(head:*)", "Bash(tail:*)",
"Bash(grep:*)", "Bash(wc:*)", "Bash(gh:*)",
])
# spd only: the scripts executing-plans / subagent-driven-development call (task-start, task-done, sdd-workspace,
# task-brief, review-package), wherever the model addresses them from.
SPD_EXTRA = ",".join("Bash(*scripts/%s*)" % x for x in ("task-start", "task-done", "sdd-workspace", "task-brief", "review-package"))
DISALLOWED = "AskUserQuestion,EnterPlanMode,ExitPlanMode,EnterWorktree,ExitWorktree,WebFetch,WebSearch"
PRE_IMPLEMENT = ("/grill-with-docs", "/to-spec", "/to-tickets")
SPD_PRE_IMPLEMENT = ("/brainstorming", "/writing-plans")
DONE_SUFFIX = "\n\nWhen you are done, stop and summarise what you changed."
def worker_env():
env = {k: v for k, v in os.environ.items() if not k.startswith("CLAUDE")}
env["CLAUDE_CODE_DISABLE_AUTO_MEMORY"] = "1"
return env
def run_claude(prompt, sid, work, out_prefix, timeout, allowed=ALLOWED):
cmd = ["claude", "-p", prompt]
if sid:
cmd += ["--resume", sid]
cmd += ["--model", MODEL, "--output-format", "stream-json", "--verbose", "--include-hook-events",
"--include-partial-messages", "--max-turns", MAX_TURNS, "--permission-mode", "acceptEdits",
"--allowedTools", allowed, "--disallowedTools", DISALLOWED,
"--setting-sources", "project", "--strict-mcp-config"]
t0 = time.time()
fo = open(out_prefix + ".stream.jsonl", "w")
ft = open(out_prefix + ".times", "w")
fe = open(out_prefix + ".stderr", "w")
p = subprocess.Popen(cmd, cwd=work, env=worker_env(), stdin=subprocess.DEVNULL, stdout=subprocess.PIPE,
stderr=fe, text=True, bufsize=1)
killed = {"v": False}
def watchdog():
try:
p.wait(timeout=timeout)
except subprocess.TimeoutExpired:
killed["v"] = True
p.kill()
th = threading.Thread(target=watchdog, daemon=True)
th.start()
for line in p.stdout:
fo.write(line)
ft.write("%.3f\n" % (time.time() - t0))
p.wait()
th.join(timeout=5)
for f in (fo, ft, fe):
f.close()
return {"exit_code": p.returncode, "wall_s": round(time.time() - t0, 2), "killed_wall_cap": killed["v"], "t0": t0}
def scratch_issues(work):
root = os.path.join(work, ".scratch")
if not os.path.isdir(root):
return None
for d in sorted(os.listdir(root)):
if os.path.isdir(os.path.join(root, d, "issues")):
return ".scratch/%s/issues/" % d
return None
def main():
cfg, run_dir, work, base = sys.argv[1:5]
conf = json.load(open(os.path.join(BENCH, "configs.json")))[cfg]
flow = conf["flow"]
task_id = os.environ.get("TASK", "F1")
task = open(os.path.join(BENCH, "tasks", task_id + ".md")).read().strip()
allowed = ALLOWED + ("," + SPD_EXTRA if flow == "spd" else "")
spd_stages = ["/brainstorming", "/writing-plans", "/executing-plans"]
spd_i = 0
t_run = time.time()
sid = None
prev = None # cumulative modelUsage / cost of the session so far
stage_in_force = "(none)"
caps = []
turns_log = open(os.path.join(run_dir, "turns.jsonl"), "a")
# mp flow: the slash commands in SPEC order; the implement stage is built when it is reached.
mp_stages = ["/grill-with-docs", "/to-spec", "/to-tickets", "/implement", "/code-review", "/pr"]
mp_i = 0
def mp_prompt(i):
s = mp_stages[i]
if s == "/grill-with-docs":
return "/grill-with-docs " + task
if s == "/implement":
issues = scratch_issues(work)
return "/implement " + ("the tickets in %s, all of them, blockers first" % issues if issues
else "the spec and tickets above")
if s == "/code-review":
return "/code-review since %s" % base
return s
if flow == "bare":
prompt, stage = task + DONE_SUFFIX, "bare"
elif flow == "spd":
prompt, stage = "/brainstorming " + task, spd_stages[0]
stage_in_force = "brainstorming"
elif flow == "sp":
prompt, stage = task, None
else:
prompt, stage = mp_prompt(0), mp_stages[0]
source = "driver"
end_reason = None
for n in range(1, MAX_DRIVER_TURNS + 1):
remaining = WALL_CAP - (time.time() - t_run)
if remaining <= 5:
caps.append("wall_40min")
end_reason = "cap_wall"
break
prefix = os.path.join(run_dir, "turn-%02d" % n)
r = run_claude(prompt, sid, work, prefix, remaining, allowed)
seg_mode = flow in ("sp", "spd") # stage split by skill in force inside the turn
acc = account_turn(prefix + ".stream.jsonl", prefix + ".times", prev,
None if seg_mode else stage, stage_in_force)
if acc.get("session_id"):
sid = acc["session_id"]
if acc.get("cumulative"):
prev = acc["cumulative"]
if seg_mode:
stage_in_force = acc["stage_at_end"]
rec = {"n": n, "flow": flow, "task": task_id, "stage": acc["stage_at_end"] if flow == "sp" else stage,
"prompt": prompt,
"prompt_source": source, **{k: v for k, v in r.items() if k != "t0"}, **{k: v for k, v in acc.items() if k != "cumulative"}}
if r["killed_wall_cap"]:
caps.append("wall_40min")
if acc.get("subtype") == "error_max_turns":
caps.append("max_turns_60@turn%d" % n)
last = acc.get("result_text") or acc.get("last_assistant_text") or ""
open(prefix + ".last.txt", "w").write(last)
if r["killed_wall_cap"]:
rec["po"] = None
turns_log.write(json.dumps(rec) + "\n")
turns_log.flush()
end_reason = "cap_wall"
break
po_stage = stage if flow != "sp" else acc["stage_at_end"]
ans = po.ask(run_dir, n, flow, po_stage, last, task_id)
rec["po"] = ans
turns_log.write(json.dumps(rec) + "\n")
turns_log.flush()
# Next prompt
if flow in ("bare", "sp"):
if ans["kind"] == "done":
end_reason = "done"
break
prompt = ans["reply"] if ans["kind"] == "question" else (
"go on" if flow == "bare" else "go on, proceed with the plan")
source = "po" if ans["kind"] == "question" else "driver"
elif flow == "spd":
if ans["kind"] == "question":
prompt, source = ans["reply"], "po"
if stage in SPD_PRE_IMPLEMENT and "implementing yet" not in prompt:
prompt += "\n\nDon't start implementing yet."
elif ans["kind"] == "waiting" or acc.get("subtype") == "error_max_turns":
prompt, source = "go on", "driver"
else:
spd_i += 1
if spd_i >= len(spd_stages):
end_reason = "done"
break
stage, prompt, source = spd_stages[spd_i], spd_stages[spd_i], "driver"
stage_in_force = stage.lstrip("/") # a typed slash command expands inline: no Skill event
else:
if ans["kind"] == "question":
prompt, source = ans["reply"], "po"
# A user following the Main Flow builds only at /implement. Without this line the worker took the
# answers to its last grilling questions as a go and coded inside /grill-with-docs (both-r1, twice).
if stage in PRE_IMPLEMENT and "implementing yet" not in prompt:
prompt += "\n\nDon't start implementing yet."
elif ans["kind"] == "waiting" or acc.get("subtype") == "error_max_turns":
prompt, source = "go on", "driver" # stage not finished: stay in it
else:
mp_i += 1
if mp_i >= len(mp_stages):
end_reason = "done"
break
stage, prompt, source = mp_stages[mp_i], mp_prompt(mp_i), "driver"
if n == MAX_DRIVER_TURNS:
caps.append("driver_turns_12")
end_reason = "cap_driver_turns"
meta = {"end_reason": end_reason, "caps": sorted(set(caps)), "session_id": sid,
"run_wall_s": round(time.time() - t_run, 1), "mp_stage_reached": mp_stages[min(mp_i, 5)] if flow == "mp" else None,
"spd_stage_reached": spd_stages[min(spd_i, 2)] if flow == "spd" else None, "task": task_id}
json.dump(meta, open(os.path.join(run_dir, "driver.json"), "w"), indent=2)
print(json.dumps(meta))
if __name__ == "__main__":
main()
lib/po.py
"""Product-owner simulator: a headless Sonnet 5.5 with NO tools that read po/F1-brief.md and answers the worker.
po.py <run dir> <turn n> <flow> <stage> <last message file> -> prints one JSON line, writes po-<n>.json
Output: {"kind": "question" | "waiting" | "done", "reply": str, "cost_usd", "wall_s", "error", "heuristic_question"}
- question: the worker asks the user something (a question, options, an approval request) -> reply answers it
- waiting: it stopped without asking (announced a next step, paused between phases) -> driver sends "go on"
- done: it reports the whole change finished -> driver stops (bare/sp)
Isolation (same as the previous benches' reviewers): --setting-sources "", disableAllHooks, --strict-mcp-config,
--tools "", CLAUDE_CODE_DISABLE_CLAUDE_MDS=1, CLAUDE_CODE_DISABLE_AUTO_MEMORY=1, scrubbed CLAUDE* env,
cwd = an empty temp dir (never the work copy).
"""
import json
import os
import re
import subprocess
import sys
import tempfile
import time
BENCH = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
MODEL = "claude-sonnet-5-5"
SCHEMA = {
"type": "object",
"properties": {
"kind": {"type": "string", "enum": ["question", "waiting", "done"]},
"reply": {"type": "string"},
},
"required": ["kind", "reply"],
"additionalProperties": False,
}
FLOW_NOTE = {
"bare": "The assistant was asked to implement the request and stop with a summary when done.",
"sp": "The assistant runs a process (brainstorm -> design -> written plan -> execution -> review -> finish). "
"It often stops to ask for approval of a design, a spec or a plan, or to ask how to finish the branch.",
"spd": "You are driving the assistant through the Superpowers stages by slash command, one stage at a time "
"(brainstorming -> writing-plans -> executing-plans); the current stage is shown below and YOU send the "
"next command once a stage is over. RULE FOR THIS FLOW: when the stage's output is delivered (a design "
"approved and written, a plan saved) and the assistant only asks you to approve it or whether to proceed, "
"answer what is open (approve it; if asked for an execution approach, choose Native / inline in this "
"session; consent to working on the current branch) and classify it kind=question only if something "
"beyond approval is open, otherwise kind=done. Never tell it to start implementing yourself before the "
"executing-plans stage. During executing-plans, approve its rulings and ask it to finish.",
"mp": "You are driving the assistant through slash commands, one stage at a time (grill -> spec -> tickets -> "
"implement -> code review -> PR); the current stage is shown below and YOU send the next command once "
"a stage is over. Interviews (grilling) come in rounds of numbered questions with a recommended answer "
"each. RULE FOR THIS FLOW: when the stage's output is delivered and the assistant only asks whether to "
"proceed, start building, move on, or confirms its own summary ('Ready to write the spec?', 'Shall I "
"start?', 'Does this capture it?' after a full recap), that is kind=done with an empty reply: the next "
"slash command will follow. Never tell it to start implementing or to move to another phase yourself; "
"answer only the open design questions. Before the /implement stage, if the assistant offers to build "
"next, end your reply with: \"Don't start implementing yet.\"",
}
def heuristic_question(text):
"""SPEC's driver rule: a turn that ends on a question mark or an explicit options list."""
tail = (text or "").strip()[-600:]
if tail.rstrip("*_` \n").endswith("?"):
return True
lines = [l.strip() for l in tail.splitlines() if l.strip()]
opts = [l for l in lines[-8:] if re.match(r"^(\(?[A-Da-d1-4][\).:]|[-*] \*\*Option)", l)]
return len(opts) >= 2 or "?" in "\n".join(lines[-3:])
def ask(run_dir, n, flow, stage, last_text, task="F1"):
brief = open(os.path.join(BENCH, "po", "%s-brief.md" % task)).read()
prompt = (
"You play the developer who asked a coding assistant for a change in your repo. You are busy: you answer "
"tersely (1 to 6 short sentences or a short numbered list), the way a dev types in a chat. You never write "
"code, never mention how you will check the work, never invent requirements beyond your brief. You answer "
"ONLY what the assistant asked: never volunteer a requirement, an edge case or a constraint it did not ask "
"about (a question it asked is fair game, even a broad one like 'anything else I should know?').\n\n"
"=== YOUR BRIEF (private, the assistant has not seen it) ===\n" + brief + "\n=== END BRIEF ===\n\n"
+ FLOW_NOTE.get(flow, "") + "\nCurrent stage: " + stage + "\n\n"
"=== THE ASSISTANT'S LAST MESSAGE (it is now waiting for you) ===\n" + (last_text or "(empty message)")[-12000:]
+ "\n=== END MESSAGE ===\n\n"
"Classify the message and write your reply:\n"
"- kind=question when it asks you anything or waits for a decision or an approval (numbered questions, "
"options, 'does this look right?', 'shall I proceed?'). Reply by answering EVERY open question in one go, "
"from your brief; where your brief is silent, accept the assistant's recommendation or say 'your call'. "
"Approve designs, specs, plans and ticket breakdowns that match your brief; correct only what contradicts it. "
"If it asks how to finish a branch (merge, PR, keep), say: keep it on the current branch, no PR, done. "
"If a code review it ran lists real defects, tell it to fix them.\n"
"- kind=waiting when it stopped without asking anything but the work is not finished (it announced a next "
"step, or paused between phases). Reply: 'go on'.\n"
"- kind=done when it reports the change finished (implemented, tests run) or, for a slash-command stage, "
"when that stage's output is delivered and nothing is asked. Reply: ''.\n"
)
env = {k: v for k, v in os.environ.items() if not k.startswith("CLAUDE")}
env.update({"CLAUDE_CODE_DISABLE_CLAUDE_MDS": "1", "CLAUDE_CODE_DISABLE_AUTO_MEMORY": "1"})
cmd = ["claude", "-p", prompt, "--model", MODEL, "--output-format", "json", "--json-schema", json.dumps(SCHEMA),
"--tools", "", "--max-turns", "4", "--permission-mode", "dontAsk", "--setting-sources", "",
"--settings", '{"disableAllHooks":true}', "--strict-mcp-config"]
out = {"kind": None, "reply": "", "cost_usd": 0.0, "wall_s": 0.0, "error": None,
"heuristic_question": heuristic_question(last_text)}
t0 = time.time()
for attempt in range(2):
try:
with tempfile.TemporaryDirectory(prefix="po-") as cwd:
p = subprocess.run(cmd, cwd=cwd, env=env, stdin=subprocess.DEVNULL, capture_output=True,
text=True, timeout=300)
res = json.loads(p.stdout.strip().splitlines()[-1])
out["cost_usd"] += res.get("total_cost_usd") or 0.0
out["models"] = sorted((res.get("modelUsage") or {}).keys())
so = res.get("structured_output")
if isinstance(so, str):
so = json.loads(so)
if not so or so.get("kind") not in ("question", "waiting", "done"):
raise ValueError("no structured_output: %s" % (res.get("result") or "")[:300])
out["kind"], out["reply"], out["error"] = so["kind"], so["reply"].strip(), None
break
except Exception as e: # noqa: BLE001
out["error"] = "%s: %s" % (type(e).__name__, e)
time.sleep(3)
if out["kind"] is None: # PO failed twice: fall back on the SPEC heuristic
out["kind"] = "question" if out["heuristic_question"] else "waiting"
out["reply"] = "Your call on all of these; go on." if out["kind"] == "question" else "go on"
out["fallback"] = True
if out["kind"] == "waiting" and not out["reply"]:
out["reply"] = "go on"
out["wall_s"] = round(time.time() - t0, 2)
json.dump(dict(out, n=n, stage=stage, prompt_chars=len(prompt)), open(os.path.join(run_dir, "po-%02d.json" % n), "w"),
indent=2)
return out
if __name__ == "__main__":
rd, n, flow, stage, f = sys.argv[1:6]
print(json.dumps(ask(rd, int(n), flow, stage, open(f).read(), os.environ.get("TASK", "F1"))))
lib/accounting.py
"""Per-turn and per-stage accounting from one claude -p stream-json (with --include-partial-messages).
Sources of truth (checked on 2.1.294, see README traps):
- result.total_cost_usd and result.modelUsage are CUMULATIVE over a resumed session and include subagents:
per-turn USD and tokens = delta against the previous turn's result.
- result.usage covers the main thread of this invocation only (no subagents): not used for totals.
- assistant events carry a usage SNAPSHOT taken when the message started (output_tokens partial);
stream_event message_start/message_delta (main thread only) carry the exact per-API-call usage.
- Subagent API calls emit only assistant snapshots (parent_tool_use_id = the Task/Agent tool_use id).
Stage split inside a turn (sp: the skill in force): main-thread calls are priced exactly from their own usage
(Sonnet 5.5 list prices, verified against total_cost_usd); the subagent remainder (turn delta - main calls) is
split over the Task calls that spawned them, weighted by their snapshot tokens. Wall per stage = the receive-time
gaps of the stdout lines, attributed to the stage in force.
"""
import json
from collections import Counter, defaultdict
# USD per token, claude-sonnet-5-5 list prices (derived from two probe turns: exact match to total_cost_usd).
PRICE = {"in": 2e-6, "out": 10e-6, "cr": 0.2e-6, "cw1h": 4e-6, "cw5m": 2.5e-6}
SP_INJECT_MARK = "You have superpowers"
EDIT_TOOLS = {"Edit", "Write", "MultiEdit", "NotebookEdit"}
DESIGN_STAGES = {"(none)", "using-superpowers", "brainstorming", "writing-plans"}
NO_SKILL_IMPL = "implementation (no skill)"
def _tok(u):
"""Normalise an API usage dict to in/out/cr/cw (+ cw split)."""
u = u or {}
cc = u.get("cache_creation") or {}
cw = u.get("cache_creation_input_tokens") or 0
cw1h = cc.get("ephemeral_1h_input_tokens")
cw5m = cc.get("ephemeral_5m_input_tokens")
if cw1h is None and cw5m is None:
cw1h, cw5m = cw, 0
return {"in": u.get("input_tokens") or 0, "out": u.get("output_tokens") or 0,
"cr": u.get("cache_read_input_tokens") or 0, "cw": cw, "cw1h": cw1h or 0, "cw5m": cw5m or 0}
def _usd(t):
return t["in"] * PRICE["in"] + t["out"] * PRICE["out"] + t["cr"] * PRICE["cr"] + \
t["cw1h"] * PRICE["cw1h"] + t["cw5m"] * PRICE["cw5m"]
def _mu_tokens(mu):
out = {}
for m, v in (mu or {}).items():
out[m] = {"in": v.get("inputTokens") or 0, "out": v.get("outputTokens") or 0,
"cr": v.get("cacheReadInputTokens") or 0, "cw": v.get("cacheCreationInputTokens") or 0,
"usd": v.get("costUSD") or 0.0}
return out
def _sub(a, b):
keys = set(a) | set(b)
return {m: {k: (a.get(m, {}).get(k, 0) - b.get(m, {}).get(k, 0)) for k in ("in", "out", "cr", "cw", "usd")}
for m in keys}
def _sum_models(per_model):
t = {k: 0 for k in ("in", "out", "cr", "cw", "usd")}
for v in per_model.values():
for k in t:
t[k] += v.get(k, 0)
return t
def account_turn(stream_path, times_path, prev, fixed_stage, stage_in_force):
lines, times = [], []
try:
raw = open(stream_path, errors="replace").read().splitlines()
tms = [float(x) for x in open(times_path).read().split()]
except FileNotFoundError:
raw, tms = [], []
for i, l in enumerate(raw):
l = l.strip()
if l.startswith("{"):
try:
lines.append(json.loads(l))
times.append(tms[i] if i < len(tms) else (times[-1] if times else 0.0))
except Exception:
pass
stage = fixed_stage or stage_in_force
out = {"session_id": None, "subtype": None, "is_error": None, "num_turns": None, "result_text": None,
"last_assistant_text": "", "stage_at_start": stage}
tools_main, tools_sub = Counter(), Counter()
skills, hooks, files = [], [], set()
skill_ids = {}
task_stage = {} # Task/Agent tool_use id -> stage when spawned
main_calls = {} # msg id -> {"stage", "tok"}
sub_snap = {} # msg id -> (parent id, tokens)
seg_wall = defaultdict(float)
order = [stage]
last_t = 0.0
result = None
denials = []
for o, t in zip(lines, times):
seg_wall[stage] += max(0.0, t - last_t)
last_t = t
typ = o.get("type")
if typ == "system" and o.get("subtype") == "init":
out["session_id"] = o.get("session_id")
out["init_model"] = o.get("model")
out["init_skills"] = o.get("skills")
out["init_plugins"] = [p.get("name") for p in o.get("plugins") or []]
elif typ == "system" and str(o.get("subtype", "")).startswith("hook"):
if o.get("subtype") == "hook_response":
outp = o.get("output") or ""
hooks.append({"event": o.get("hook_event"), "name": o.get("hook_name"),
"superpowers_injected": SP_INJECT_MARK in outp, "output_chars": len(outp),
"exit_code": o.get("exit_code"), "outcome": o.get("outcome")})
elif typ == "stream_event" and not o.get("parent_tool_use_id"):
ev = o.get("event") or {}
if ev.get("type") == "message_start":
m = ev.get("message") or {}
main_calls[m.get("id")] = {"stage": stage, "tok": _tok(m.get("usage")), "model": m.get("model")}
main_calls["__last__"] = m.get("id")
elif ev.get("type") == "message_delta":
mid = main_calls.get("__last__")
if mid in main_calls:
u = ev.get("usage") or {}
tk = main_calls[mid]["tok"]
tk["out"] = u.get("output_tokens", tk["out"]) or tk["out"]
elif typ == "assistant":
m = o.get("message") or {}
parent = o.get("parent_tool_use_id")
if parent:
sub_snap[m.get("id")] = (parent, _tok(m.get("usage")))
for c in m.get("content") or []:
if c.get("type") == "text" and not parent:
out["last_assistant_text"] = c.get("text") or out["last_assistant_text"]
if c.get("type") != "tool_use":
continue
name = c.get("name")
inp = c.get("input") or {}
(tools_sub if parent else tools_main)[name] += 1
if name in EDIT_TOOLS and inp.get("file_path"):
files.add(inp["file_path"])
# sp: code written while only a design skill is in force = implementation with no skill
if (not parent and not fixed_stage and stage in DESIGN_STAGES
and ("/src/" in inp["file_path"] or "/test/" in inp["file_path"])):
stage = NO_SKILL_IMPL
order.append(stage)
if name in ("Task", "Agent") and not parent:
task_stage[c.get("id")] = stage
if name == "Skill":
sk = inp.get("skill") or inp.get("command") or "?"
rec = {"skill": sk, "t": round(t, 2), "thread": "sub" if parent else "main", "ok": None,
"args": (inp.get("args") or "")[:200]}
skills.append(rec)
skill_ids[c.get("id")] = rec
if not parent and not fixed_stage:
stage = sk.split(":")[-1]
order.append(stage)
elif typ == "user":
for c in (o.get("message") or {}).get("content") or []:
if isinstance(c, dict) and c.get("type") == "tool_result" and c.get("tool_use_id") in skill_ids:
txt = c.get("content")
txt = txt if isinstance(txt, str) else json.dumps(txt)
skill_ids[c["tool_use_id"]]["ok"] = not c.get("is_error") and "Launching skill" in txt
skill_ids[c["tool_use_id"]]["result"] = txt[:200]
elif typ == "result":
result = o
main_calls.pop("__last__", None)
if result:
out.update({"subtype": result.get("subtype"), "is_error": result.get("is_error"),
"num_turns": result.get("num_turns"), "result_text": result.get("result"),
"terminal_reason": result.get("terminal_reason"), "stop_reason": result.get("stop_reason")})
denials = result.get("permission_denials") or []
cum = {"cost": result.get("total_cost_usd") or 0.0, "models": _mu_tokens(result.get("modelUsage"))}
pm = prev["models"] if prev else {}
delta_models = _sub(cum["models"], pm)
delta_cost = cum["cost"] - (prev["cost"] if prev else 0.0)
out["cumulative"] = cum
out["estimated"] = False
else: # killed or crashed: best effort from the stream itself
delta_models = {}
for c in main_calls.values():
d = delta_models.setdefault(c.get("model") or "?", {"in": 0, "out": 0, "cr": 0, "cw": 0, "usd": 0.0})
for k in ("in", "out", "cr", "cw"):
d[k] += c["tok"][k]
d["usd"] += _usd(c["tok"])
for _, (parent, tk) in sub_snap.items():
d = delta_models.setdefault("?sub", {"in": 0, "out": 0, "cr": 0, "cw": 0, "usd": 0.0})
for k in ("in", "out", "cr", "cw"):
d[k] += tk[k]
d["usd"] += _usd(tk)
delta_cost = sum(v["usd"] for v in delta_models.values())
out["cumulative"] = None
out["estimated"] = True
tot = _sum_models(delta_models)
out["usd"] = round(delta_cost, 6)
out["tokens"] = {k: tot[k] for k in ("in", "out", "cr", "cw")}
out["tokens"]["total"] = sum(out["tokens"].values())
out["models"] = {m: v for m, v in delta_models.items() if any(v[k] for k in ("in", "out", "cr", "cw"))}
# Stage segments
segs = defaultdict(lambda: {"in": 0, "out": 0, "cr": 0, "cw": 0, "usd": 0.0, "api_calls": 0, "wall_s": 0.0})
main_tok = {"in": 0, "out": 0, "cr": 0, "cw": 0}
main_usd = 0.0
for c in main_calls.values():
s = segs[c["stage"]]
for k in ("in", "out", "cr", "cw"):
s[k] += c["tok"][k]
main_tok[k] += c["tok"][k]
u = _usd(c["tok"])
s["usd"] += u
s["api_calls"] += 1
main_usd += u
rem = {k: max(0, out["tokens"][k] - main_tok[k]) for k in ("in", "out", "cr", "cw")}
rem_usd = max(0.0, delta_cost - main_usd) if result else 0.0
weights = defaultdict(float)
for _, (parent, tk) in sub_snap.items():
# subagent spend is its own segment, labelled with the stage that dispatched it
weights[task_stage.get(parent, stage) + " [subagent]"] += tk["in"] + tk["cr"] + tk["cw"] + tk["out"]
wsum = sum(weights.values())
if result and (wsum > 0 or rem_usd > 0):
if wsum == 0:
weights = {stage + " [subagent]": 1.0}
wsum = 1.0
for st, w in weights.items():
s = segs[st]
f = w / wsum
for k in ("in", "out", "cr", "cw"):
s[k] += int(round(rem[k] * f))
s["usd"] += rem_usd * f
for st, w in seg_wall.items():
segs[st]["wall_s"] += w
seen = []
for st in order:
if st not in seen:
seen.append(st)
for st in segs:
if st not in seen:
seen.append(st)
out["segments"] = [dict(stage=st, **{k: (round(v, 6) if isinstance(v, float) else v) for k, v in segs[st].items()},
total=segs[st]["in"] + segs[st]["out"] + segs[st]["cr"] + segs[st]["cw"])
for st in seen if st in segs]
out["pricing_check"] = {"main_calls_usd": round(main_usd, 6), "turn_usd": round(delta_cost, 6),
"subagent_usd": round(rem_usd, 6), "main_calls": len(main_calls),
"subagent_msgs": len(sub_snap)}
out["stage_at_end"] = stage
out["tools_main"] = dict(tools_main)
out["tools_sub"] = dict(tools_sub)
out["tool_calls"] = sum(tools_main.values()) + sum(tools_sub.values())
out["skills"] = skills
out["hooks"] = hooks
out["files_touched"] = sorted(files)
out["permission_denials"] = [{"tool": d.get("tool_name"), "input": json.dumps(d.get("tool_input"))[:200]}
for d in denials]
out["stream_lines"] = len(lines)
return out
lib/reaccount.py
"""Recompute the accounting fields of runs/<run>/turns.jsonl from the saved turn streams (after a change to
lib/accounting.py), keeping what the driver recorded (prompt, source, wall, exit code, PO answer). Then run finalize.py.
reaccount.py <run dir>
"""
import json
import os
import sys
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from accounting import account_turn # noqa: E402
KEEP = ("n", "flow", "prompt", "prompt_source", "exit_code", "wall_s", "killed_wall_cap", "po")
run = os.path.abspath(sys.argv[1])
turns = [json.loads(l) for l in open(os.path.join(run, "turns.jsonl"))]
prev, stage_in_force, out = None, "(none)", []
prev_slash = None
for t in turns:
prefix = os.path.join(run, "turn-%02d" % t["n"])
if t["flow"] == "spd" and t["stage"] != prev_slash: # a typed slash command starts its skill (no Skill event)
stage_in_force, prev_slash = t["stage"].lstrip("/"), t["stage"]
sp = t["flow"] in ("sp", "spd")
acc = account_turn(prefix + ".stream.jsonl", prefix + ".times", prev, None if sp else t["stage"], stage_in_force)
if acc.get("cumulative"):
prev = acc["cumulative"]
if sp:
stage_in_force = acc["stage_at_end"]
rec = {k: t[k] for k in KEEP if k in t}
rec["stage"] = acc["stage_at_end"] if t["flow"] == "sp" else t["stage"]
rec.update({k: v for k, v in acc.items() if k != "cumulative"})
out.append(rec)
with open(os.path.join(run, "turns.jsonl"), "w") as f:
for r in out:
f.write(json.dumps(r) + "\n")
print("%s: %d turns reaccounted" % (os.path.basename(run), len(out)))
lib/finalize.py
"""After the driver: final diff, suite, typecheck, hidden oracle, per-stage table -> score.json, stages.json.
finalize.py <run dir> (re-runnable; reads meta.json, turns.jsonl, driver.json)
"""
import json
import os
import re
import shutil
import subprocess
import sys
from collections import OrderedDict
BENCH = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
HIDDEN = os.path.join(BENCH, "hidden")
MODEL = "claude-sonnet-5-5"
UPSTREAM_TESTS = 34
def sh(args, cwd, env=None, timeout=600):
try:
p = subprocess.run(args, cwd=cwd, env=env, stdin=subprocess.DEVNULL, stdout=subprocess.PIPE,
stderr=subprocess.STDOUT, timeout=timeout)
return p.returncode, p.stdout.decode("utf-8", "replace")
except subprocess.TimeoutExpired as e:
return 124, (e.stdout or b"").decode("utf-8", "replace") + "\n[timeout]"
def clean_env(extra=None):
env = {k: v for k, v in os.environ.items() if not k.startswith("CLAUDE")}
env.update(extra or {})
return env
def work_diff(work, base):
"""Working tree vs the base commit (commits made by the worker included) + untracked files. Index untouched."""
_, d = sh(["git", "diff", base, "--no-color", "--", "."], work)
_, others = sh(["git", "ls-files", "--others", "--exclude-standard"], work)
for f in [l for l in others.splitlines() if l.strip()]:
_, nd = sh(["git", "diff", "--no-color", "--no-index", "--", "/dev/null", f], work)
d += nd
return d
def numstat(work, base):
rows = []
_, ns = sh(["git", "diff", base, "--numstat", "--", "."], work)
for l in ns.splitlines():
p = l.split("\t")
if len(p) == 3:
rows.append((int(p[0]) if p[0] != "-" else 0, int(p[1]) if p[1] != "-" else 0, p[2]))
_, others = sh(["git", "ls-files", "--others", "--exclude-standard"], work)
for f in [l for l in others.splitlines() if l.strip()]:
try:
rows.append((sum(1 for _ in open(os.path.join(work, f), errors="replace")), 0, f))
except Exception:
pass
return rows
def run_hidden(work, base, task="F1"):
rep = os.path.join(os.path.dirname(work), ".hidden-%s.json" % os.path.basename(work))
rc, txt = sh([os.path.join(HIDDEN, "node_modules", ".bin", "vitest"), "run", "--reporter=json",
"--outputFile=" + rep], HIDDEN, env=clean_env({"WORK": work, "BASE": base, "TASK": task}), timeout=300)
reqs = OrderedDict(("test_req%d" % i, False) for i in range(1, 6))
fails = {}
try:
j = json.load(open(rep))
for tr in j.get("testResults", []):
for a in tr.get("assertionResults", []):
m = re.match(r"(test_req\d)", a.get("title", ""))
if m:
reqs[m.group(1)] = a.get("status") == "passed"
if a.get("status") != "passed":
fails[m.group(1)] = (" ".join(a.get("failureMessages") or []))[:600]
if tr.get("status") == "failed" and not tr.get("assertionResults"):
fails["collect"] = (tr.get("message") or "")[:800]
os.remove(rep)
except Exception as e: # noqa: BLE001
fails["runner"] = "%s: %s" % (type(e).__name__, txt[-800:])
return reqs, fails, txt
def main():
out = os.path.abspath(sys.argv[1])
meta = json.load(open(os.path.join(out, "meta.json")))
work, base, cfg = meta["work"], meta["base"], meta["cfg"]
if not os.path.isdir(work): # tmp work copy gone: rescore the archived copy
work = os.path.join(BENCH, "work", meta["name"])
drv = json.load(open(os.path.join(out, "driver.json"))) if os.path.exists(os.path.join(out, "driver.json")) else {}
turns = [json.loads(l) for l in open(os.path.join(out, "turns.jsonl"))] if os.path.exists(
os.path.join(out, "turns.jsonl")) else []
have_nm = os.path.isdir(os.path.join(work, "node_modules"))
if not have_nm: # rescoring after cleanup: restore deps from upstream
shutil.copytree(os.path.join(BENCH, "upstream", "node_modules"), os.path.join(work, "node_modules"), symlinks=True)
diff = work_diff(work, base)
open(os.path.join(out, "post.diff"), "w").write(diff)
rows = numstat(work, base)
open(os.path.join(out, "numstat.txt"), "w").write("".join("%d\t%d\t%s\n" % r for r in rows))
code = [r for r in rows if r[2].startswith(("src/", "test/"))]
tests_files = [r for r in rows if r[2].endswith(".test.ts")]
rc_s, suite = sh([os.path.join(work, "node_modules", ".bin", "vitest"), "run"], work, env=clean_env(), timeout=300)
open(os.path.join(out, "suite.txt"), "w").write(suite)
clean = re.sub(r"\x1b\[[0-9;]*m", "", suite)
m = re.search(r"Tests\s+(?:(\d+) failed \| )?(\d+) passed(?: \| \d+ \w+)? \((\d+)\)", clean)
s_fail = int(m.group(1) or 0) if m else None
s_pass = int(m.group(2)) if m else 0
s_total = int(m.group(3)) if m else 0
if m is None:
mf = re.search(r"Tests\s+(\d+) failed \((\d+)\)", clean)
s_fail, s_total = (int(mf.group(1)), int(mf.group(2))) if mf else (None, 0)
rc_t, tc = sh([os.path.join(work, "node_modules", ".bin", "tsc"), "--noEmit"], work, env=clean_env(), timeout=300)
open(os.path.join(out, "typecheck.txt"), "w").write(tc)
task = meta.get("task", "F1")
reqs, fails, htxt = run_hidden(work, base, task)
open(os.path.join(out, "hidden.txt"), "w").write(htxt + "\n" + json.dumps(fails, indent=2))
_, commits = sh(["git", "log", "--oneline", "%s..HEAD" % base], work)
_, branch = sh(["git", "rev-parse", "--abbrev-ref", "HEAD"], work)
_, branches = sh(["git", "branch", "--format=%(refname:short)"], work)
added_its = 0
for l in diff.splitlines():
if re.match(r"^\+\s*(it|test)(\.each\([^)]*\))?\s*\(", l):
added_its += 1
# Per-stage table (sum of turn segments; for mp/bare/both a segment = the turn's stage)
stages = OrderedDict()
for t in turns:
for sgm in t.get("segments") or []:
st = sgm["stage"] if cfg in ("sp", "spd") else t["stage"]
s = stages.setdefault(st, {"turns": set(), "in": 0, "out": 0, "cr": 0, "cw": 0, "total": 0, "usd": 0.0,
"wall_s": 0.0, "api_calls": 0})
s["turns"].add(t["n"])
for k in ("in", "out", "cr", "cw", "total", "api_calls"):
s[k] += sgm.get(k, 0)
s["usd"] += sgm.get("usd", 0.0)
s["wall_s"] += sgm.get("wall_s", 0.0)
if cfg not in ("sp", "spd") and not t.get("segments"):
stages.setdefault(t["stage"], {"turns": {t["n"]}, "in": 0, "out": 0, "cr": 0, "cw": 0, "total": 0,
"usd": 0.0, "wall_s": t.get("wall_s", 0), "api_calls": 0})
# Turn-level wall/usd are exact; mp/bare stage wall = turn wall (segments only cover stdout gaps)
if cfg not in ("sp", "spd"):
for st, s in stages.items():
s["wall_s"] = sum(t.get("wall_s", 0) for t in turns if t["stage"] == st)
s["usd"] = sum(t.get("usd", 0) for t in turns if t["stage"] == st)
for s in stages.values():
s["turns"] = sorted(s["turns"])
s["usd"] = round(s["usd"], 4)
s["wall_s"] = round(s["wall_s"], 1)
json.dump(stages, open(os.path.join(out, "stages.json"), "w"), indent=2)
tok = {k: sum(t.get("tokens", {}).get(k, 0) for t in turns) for k in ("in", "out", "cr", "cw", "total")}
models = sorted({m for t in turns for m in (t.get("models") or {})})
skills = [dict(s, turn=t["n"]) for t in turns for s in t.get("skills") or []]
hooks = [dict(h, turn=t["n"]) for t in turns for h in t.get("hooks") or []]
po_cost = sum((t.get("po") or {}).get("cost_usd", 0) for t in turns)
files = sorted({f.replace(work + "/", "") for t in turns for f in t.get("files_touched") or []})
score = {
"run": meta["name"], "task": meta.get("task", "F1"), "cfg": cfg, "rep": meta["rep"], "tag": meta.get("tag"),
"hidden_pass": sum(reqs.values()), "hidden_reqs": reqs, "hidden_fail_msgs": fails,
"suite_green": rc_s == 0 and (s_fail or 0) == 0 and s_total > 0, "suite_pass": s_pass, "suite_total": s_total,
"tests_added_suite": max(0, s_total - UPSTREAM_TESTS), "tests_added_diff": added_its,
"typecheck_green": rc_t == 0,
"diff_files": len(rows), "diff_added": sum(r[0] for r in rows), "diff_removed": sum(r[1] for r in rows),
"code_diff_added": sum(r[0] for r in code), "code_diff_removed": sum(r[1] for r in code),
"code_files": len(code), "test_files_touched": len(tests_files),
"non_code_files": sorted(r[2] for r in rows if not r[2].startswith(("src/", "test/"))),
"files_touched_by_tools": files,
"commits": len([l for l in commits.splitlines() if l.strip()]), "branch_at_end": branch.strip(),
"branches": branches.split(),
"driver_turns": len(turns), "claude_turns_total": sum(t.get("num_turns") or 0 for t in turns),
"tool_calls": sum(t.get("tool_calls", 0) for t in turns),
"tokens": tok, "usd": round(sum(t.get("usd", 0) for t in turns), 4),
"usd_po": round(po_cost, 4),
"wall_s": round(sum(t.get("wall_s", 0) for t in turns), 1), "run_wall_s": drv.get("run_wall_s"),
"end_reason": drv.get("end_reason"), "caps": drv.get("caps", []),
"questions_routed_to_po": sum(1 for t in turns if (t.get("po") or {}).get("kind") == "question"),
"skills_invoked": [s["skill"] for s in skills],
"skills_failed": [s["skill"] for s in skills if s.get("ok") is False],
"skill_events": skills,
"hook_events": hooks,
"superpowers_injected": any(h.get("superpowers_injected") for h in hooks),
"init_plugins": (turns[0].get("init_plugins") if turns else None),
"models": models, "model_ok": bool(models) and set(models) <= {MODEL},
"permission_denials": sum(len(t.get("permission_denials") or []) for t in turns),
"estimated_turns": [t["n"] for t in turns if t.get("estimated")],
"spd_slash_stage_per_turn": [t["stage"] for t in turns] if cfg == "spd" else None,
"per_turn": [{"n": t["n"], "stage": t["stage"], "source": t.get("prompt_source"), "usd": t.get("usd"),
"tokens": t.get("tokens", {}).get("total"), "wall_s": t.get("wall_s"),
"num_turns": t.get("num_turns"), "subtype": t.get("subtype"),
"skills": [s["skill"] for s in t.get("skills") or []],
"po_kind": (t.get("po") or {}).get("kind")} for t in turns],
"stages": stages,
}
json.dump(score, open(os.path.join(out, "score.json"), "w"), indent=2)
if os.environ.get("KEEP_NODE_MODULES") != "1":
shutil.rmtree(os.path.join(work, "node_modules"), ignore_errors=True)
print("%s: hidden %d/5 suite %s (%d/%d) tsc %s diff +%d/-%d (code +%d/-%d) tokens %d usd %.2f wall %ss turns %d "
"end=%s caps=%s skills=%s" % (
meta["name"], score["hidden_pass"], "green" if score["suite_green"] else "RED", s_pass, s_total,
"ok" if score["typecheck_green"] else "FAIL", score["diff_added"], score["diff_removed"],
score["code_diff_added"], score["code_diff_removed"], tok["total"], score["usd"], score["wall_s"],
len(turns), score["end_reason"], ",".join(score["caps"]) or "-",
",".join(OrderedDict.fromkeys(score["skills_invoked"])) or "-"))
if __name__ == "__main__":
main()
analyze.py
#!/usr/bin/env python3
"""runs/*/score.json (untagged runs only) -> results.json + results.md."""
import json
import os
import statistics as st
from collections import OrderedDict, defaultdict
HERE = os.path.dirname(os.path.abspath(__file__))
CFG_ORDER = ["bare", "sp", "spd", "mp", "both"]
CFG_LABEL = {"bare": "bare", "sp": "sp (hook-routed)", "spd": "spd (Superpowers driven)", "mp": "mp (Main Flow)", "both": "both (sp+mp, mp flow)"}
TASKS = ["F1", "F2"]
SPD_ORDER = ["/brainstorming", "/writing-plans", "/executing-plans"]
MP_ORDER = ["/grill-with-docs", "/to-spec", "/to-tickets", "/implement", "/code-review", "/pr"]
# Superpowers phases (the raw skill-in-force stages are kept too)
SP_PHASE = {
"(none)": "before any skill", "using-superpowers": "before any skill",
"brainstorming": "brainstorming", "writing-plans": "writing-plans",
"executing-plans": "execution", "subagent-driven-development": "execution",
"test-driven-development": "execution", "systematic-debugging": "execution",
"verification-before-completion": "verification", "dispatching-parallel-agents": "execution",
"requesting-code-review": "review", "receiving-code-review": "review",
"finishing-a-development-branch": "finishing", "using-git-worktrees": "execution",
"implementation (no skill)": "implementation (no skill)",
}
PHASE_ORDER = ["before any skill", "brainstorming", "writing-plans", "implementation (no skill)", "execution", "verification", "review", "finishing", "subagents (reviewers, explorers)", "other"]
def mean(xs):
xs = [x for x in xs if x is not None]
return round(st.mean(xs), 3) if xs else None
def med(xs):
xs = [x for x in xs if x is not None]
return round(st.median(xs), 3) if xs else None
def load():
runs = []
for d in sorted(os.listdir(os.path.join(HERE, "runs"))):
p = os.path.join(HERE, "runs", d, "score.json")
if not os.path.exists(p):
continue
s = json.load(open(p))
if s.get("tag"):
continue
runs.append(s)
return runs
def stage_agg(runs, key_fn, order):
agg = defaultdict(lambda: defaultdict(list))
for s in runs:
per = defaultdict(lambda: {"total": 0, "usd": 0.0, "wall_s": 0.0, "out": 0})
for name, v in s["stages"].items():
k = key_fn(name)
for f in ("total", "usd", "wall_s", "out"):
per[k][f] += v.get(f, 0)
for k, v in per.items():
if not (v["total"] or v["usd"] or v["wall_s"]):
continue # empty subagent placeholder segment
for f in v:
agg[k][f].append(v[f])
names = [n for n in order if n in agg] + sorted(n for n in agg if n not in order)
out = OrderedDict()
for n in names:
a = agg[n]
out[n] = {"runs_with_stage": len(a["total"]), "tokens_mean": mean(a["total"]), "tokens_median": med(a["total"]),
"out_tokens_mean": mean(a["out"]), "usd_mean": mean(a["usd"]), "usd_median": med(a["usd"]),
"wall_s_mean": mean(a["wall_s"]), "wall_s_median": med(a["wall_s"])}
return out
def turn_stage_agg(runs, order):
"""Per slash-command stage from the per-turn records (exact turn USD / tokens / wall)."""
agg = defaultdict(lambda: defaultdict(list))
for s in runs:
per = defaultdict(lambda: {"total": 0, "usd": 0.0, "wall_s": 0.0, "out": 0})
for t in s["per_turn"]:
per[t["stage"]]["total"] += t.get("tokens") or 0
per[t["stage"]]["usd"] += t.get("usd") or 0
per[t["stage"]]["wall_s"] += t.get("wall_s") or 0
for k, v in per.items():
for f in v:
agg[k][f].append(v[f])
out = OrderedDict()
for n in [n for n in order if n in agg] + sorted(n for n in agg if n not in order):
a = agg[n]
out[n] = {"runs_with_stage": len(a["total"]), "tokens_mean": mean(a["total"]), "tokens_median": med(a["total"]),
"out_tokens_mean": None, "usd_mean": mean(a["usd"]), "usd_median": med(a["usd"]),
"wall_s_mean": mean(a["wall_s"]), "wall_s_median": med(a["wall_s"])}
return out
def main():
runs = load()
for s in runs:
s.setdefault("task", "F1")
by = OrderedDict(("%s/%s" % (t, c), [s for s in runs if s["cfg"] == c and s["task"] == t]) for t in TASKS for c in CFG_ORDER)
res = {"configs": OrderedDict(), "stages": OrderedDict(), "runs": []}
for key, rs in by.items():
if not rs:
continue
task, c = key.split("/")
m = lambda f: [f(s) for s in rs] # noqa: E731
res["configs"][key] = {
"task": task, "cfg": c, "label": CFG_LABEL[c],
"n": len(rs),
"hidden_mean": mean(m(lambda s: s["hidden_pass"])), "hidden_median": med(m(lambda s: s["hidden_pass"])),
"hidden_per_run": m(lambda s: s["hidden_pass"]),
"req_pass_rate": {r: sum(1 for s in rs if s["hidden_reqs"].get(r)) for r in ["test_req%d" % i for i in range(1, 6)]},
"suite_green": sum(1 for s in rs if s["suite_green"]), "typecheck_green": sum(1 for s in rs if s["typecheck_green"]),
"tests_added_mean": mean(m(lambda s: s["tests_added_suite"])),
"code_diff_mean": mean(m(lambda s: s["code_diff_added"] + s["code_diff_removed"])),
"code_diff_median": med(m(lambda s: s["code_diff_added"] + s["code_diff_removed"])),
"all_diff_mean": mean(m(lambda s: s["diff_added"] + s["diff_removed"])),
"non_code_files_mean": mean(m(lambda s: len(s["non_code_files"]))),
"tokens_mean": mean(m(lambda s: s["tokens"]["total"])), "tokens_median": med(m(lambda s: s["tokens"]["total"])),
"out_tokens_mean": mean(m(lambda s: s["tokens"]["out"])),
"usd_mean": mean(m(lambda s: s["usd"])), "usd_median": med(m(lambda s: s["usd"])),
"usd_po_mean": mean(m(lambda s: s["usd_po"])),
"wall_s_mean": mean(m(lambda s: s["wall_s"])), "wall_s_median": med(m(lambda s: s["wall_s"])),
"driver_turns_mean": mean(m(lambda s: s["driver_turns"])),
"questions_mean": mean(m(lambda s: s["questions_routed_to_po"])),
"caps": {s["run"]: s["caps"] for s in rs if s["caps"]},
"end_reasons": m(lambda s: s["end_reason"]),
"models": sorted({x for s in rs for x in s["models"]}),
"superpowers_injected": m(lambda s: s["superpowers_injected"]),
"skills_per_run": {s["run"]: list(OrderedDict.fromkeys(s["skills_invoked"])) for s in rs},
"skills_failed": {s["run"]: s["skills_failed"] for s in rs if s["skills_failed"]},
}
if c in ("sp", "spd"):
res["stages"]["%s/%s_skill" % (task, c)] = stage_agg(rs, lambda n: n, list(SP_PHASE))
res["stages"]["%s/%s_phase" % (task, c)] = stage_agg(rs, lambda n: "subagents (reviewers, explorers)" if n.endswith("[subagent]") else SP_PHASE.get(n, "other"), PHASE_ORDER)
if c == "spd":
res["stages"]["%s/spd_slash" % task] = turn_stage_agg(rs, SPD_ORDER)
if c in ("mp", "both"):
res["stages"]["%s/%s" % (task, c)] = stage_agg(rs, lambda n: n, MP_ORDER)
for s in runs:
res["runs"].append({k: s[k] for k in ("run", "task", "cfg", "hidden_pass", "hidden_reqs", "suite_green", "typecheck_green",
"code_diff_added", "code_diff_removed", "diff_added", "diff_removed",
"tests_added_suite", "tokens", "usd", "usd_po", "wall_s", "driver_turns",
"questions_routed_to_po", "end_reason", "caps", "skills_invoked",
"superpowers_injected", "commits", "models", "stages")})
json.dump(res, open(os.path.join(HERE, "results.json"), "w"), indent=2)
L = ["# flow-bench results", "",
"Untagged runs only (smoke/validation excluded). Tokens = input + output + cache read + cache write, all models, "
"subagents included. USD = list-price equivalent from `total_cost_usd` deltas (plan quota, not a bill). "
"Wall = sum of worker turn walls (PO time excluded). Diff = lines added+removed vs the base commit, src/ + test/.", "",
"## Per config (mean / median)", "",
"| task / config | n | hidden /5 (per run) | suite green | tsc green | code diff +/- | all files diff | tests added | tokens | USD | wall s | driver turns | questions | caps hit |",
"|---|---|---|---|---|---|---|---|---|---|---|---|---|---|"]
for c, v in res["configs"].items():
L.append("| %s | %d | %s / %s (%s) | %d/%d | %d/%d | %s / %s | %s | %s | %s / %s | %s / %s | %s / %s | %s | %s | %s |" % (
"%s / %s" % (v["task"], v["label"]), v["n"], v["hidden_mean"], v["hidden_median"], ",".join(map(str, v["hidden_per_run"])), v["suite_green"], v["n"],
v["typecheck_green"], v["n"], v["code_diff_mean"], v["code_diff_median"], v["all_diff_mean"], v["tests_added_mean"],
fmt(v["tokens_mean"]), fmt(v["tokens_median"]), v["usd_mean"], v["usd_median"], v["wall_s_mean"], v["wall_s_median"],
v["driver_turns_mean"], v["questions_mean"], "; ".join("%s: %s" % (k, ",".join(x)) for k, x in v["caps"].items()) or "none"))
L += ["", "Requirement pass counts (runs passing each hidden req):", ""]
for c, v in res["configs"].items():
L.append("- **%s %s**: %s" % (v["task"], v["label"], ", ".join("%s %d/%d" % (k.replace("test_", ""), n, v["n"]) for k, n in v["req_pass_rate"].items())))
titles = []
for t in TASKS:
titles += [("%s/spd_slash" % t, "%s spd (Superpowers driven) per slash-command stage (turn level, exact)" % t),
("%s/spd_skill" % t, "%s spd per skill in force (within-turn split)" % t),
("%s/sp_phase" % t, "%s sp (hook-routed) per phase" % t),
("%s/sp_skill" % t, "%s sp (hook-routed) per skill in force" % t),
("%s/mp" % t, "%s mp per slash-command stage" % t), ("%s/both" % t, "%s both per slash-command stage" % t)]
for key, title in titles:
if key not in res["stages"]:
continue
L += ["", "## %s (mean over the runs that had the stage)" % title, "",
"| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |",
"|---|---|---|---|---|---|---|---|---|"]
for n, v in res["stages"][key].items():
L.append("| %s | %d | %s | %s | %s | %s | %s | %s | %s |" % (n, v["runs_with_stage"], fmt(v["tokens_mean"]),
fmt(v["tokens_median"]), fmt(v["out_tokens_mean"]), v["usd_mean"], v["usd_median"], v["wall_s_mean"], v["wall_s_median"]))
L += ["", "## Per run", "",
"| run | task | hidden | reqs failed | suite | tsc | code +/- | tests added | tokens | USD | wall s | turns | end | skills (first use order) |",
"|---|---|---|---|---|---|---|---|---|---|---|---|---|---|"]
for s in sorted(runs, key=lambda s: (s["task"], CFG_ORDER.index(s["cfg"]), s["run"])):
L.append("| %s | %s | %d/5 | %s | %s | %s | +%d/-%d | %d | %s | %.2f | %.0f | %d | %s%s | %s |" % (
s["run"], s["task"], s["hidden_pass"], ",".join(k.replace("test_", "") for k, v in s["hidden_reqs"].items() if not v) or "-",
"green" if s["suite_green"] else "RED", "ok" if s["typecheck_green"] else "FAIL", s["code_diff_added"],
s["code_diff_removed"], s["tests_added_suite"], fmt(s["tokens"]["total"]), s["usd"], s["wall_s"], s["driver_turns"],
s["end_reason"], (" (" + ",".join(s["caps"]) + ")") if s["caps"] else "",
" > ".join(OrderedDict.fromkeys(x.split(":")[-1] for x in s["skills_invoked"])) or "-"))
open(os.path.join(HERE, "results.md"), "w").write("\n".join(L) + "\n")
print("\n".join(L))
def fmt(x):
if x is None:
return "-"
return "{:,.0f}".format(x)
if __name__ == "__main__":
main()
results.json
{
"configs": {
"F1/bare": {
"task": "F1",
"cfg": "bare",
"label": "bare",
"n": 3,
"hidden_mean": 5,
"hidden_median": 5,
"hidden_per_run": [
5,
5,
5
],
"req_pass_rate": {
"test_req1": 3,
"test_req2": 3,
"test_req3": 3,
"test_req4": 3,
"test_req5": 3
},
"suite_green": 3,
"typecheck_green": 3,
"tests_added_mean": 4.667,
"code_diff_mean": 141.667,
"code_diff_median": 137,
"all_diff_mean": 142.333,
"non_code_files_mean": 0.333,
"tokens_mean": 519113.667,
"tokens_median": 361842,
"out_tokens_mean": 12001,
"usd_mean": 0.343,
"usd_median": 0.286,
"usd_po_mean": 0.021,
"wall_s_mean": 84.5,
"wall_s_median": 71.7,
"driver_turns_mean": 1.333,
"questions_mean": 0.333,
"caps": {},
"end_reasons": [
"done",
"done",
"done"
],
"models": [
"claude-sonnet-5-5"
],
"superpowers_injected": [
false,
false,
false
],
"skills_per_run": {
"bare-r1": [],
"bare-r2": [],
"bare-r3": []
},
"skills_failed": {}
},
"F1/sp": {
"task": "F1",
"cfg": "sp",
"label": "sp (hook-routed)",
"n": 3,
"hidden_mean": 5,
"hidden_median": 5,
"hidden_per_run": [
5,
5,
5
],
"req_pass_rate": {
"test_req1": 3,
"test_req2": 3,
"test_req3": 3,
"test_req4": 3,
"test_req5": 3
},
"suite_green": 3,
"typecheck_green": 3,
"tests_added_mean": 12.333,
"code_diff_mean": 235,
"code_diff_median": 227,
"all_diff_mean": 237,
"non_code_files_mean": 0.667,
"tokens_mean": 724600.667,
"tokens_median": 765108,
"out_tokens_mean": 16869.333,
"usd_mean": 0.502,
"usd_median": 0.525,
"usd_po_mean": 0.041,
"wall_s_mean": 118.1,
"wall_s_median": 119.6,
"driver_turns_mean": 2.667,
"questions_mean": 1.667,
"caps": {},
"end_reasons": [
"done",
"done",
"done"
],
"models": [
"claude-sonnet-5-5"
],
"superpowers_injected": [
true,
true,
true
],
"skills_per_run": {
"sp-r1": [
"brainstorming"
],
"sp-r2": [
"brainstorming",
"test-driven-development"
],
"sp-r3": [
"brainstorming"
]
},
"skills_failed": {}
},
"F1/spd": {
"task": "F1",
"cfg": "spd",
"label": "spd (Superpowers driven)",
"n": 3,
"hidden_mean": 5,
"hidden_median": 5,
"hidden_per_run": [
5,
5,
5
],
"req_pass_rate": {
"test_req1": 3,
"test_req2": 3,
"test_req3": 3,
"test_req4": 3,
"test_req5": 3
},
"suite_green": 3,
"typecheck_green": 3,
"tests_added_mean": 27.667,
"code_diff_mean": 386,
"code_diff_median": 371,
"all_diff_mean": 582,
"non_code_files_mean": 2,
"tokens_mean": 4479876.333,
"tokens_median": 3887015,
"out_tokens_mean": 61684.667,
"usd_mean": 3.738,
"usd_median": 3.412,
"usd_po_mean": 0.116,
"wall_s_mean": 607.367,
"wall_s_median": 600.8,
"driver_turns_mean": 7,
"questions_mean": 4,
"caps": {},
"end_reasons": [
"done",
"done",
"done"
],
"models": [
"claude-fable-5-1",
"claude-sonnet-5-5"
],
"superpowers_injected": [
true,
true,
true
],
"skills_per_run": {
"spd-r1": [
"test-driven-development"
],
"spd-r2": [
"test-driven-development"
],
"spd-r3": [
"test-driven-development",
"finishing-a-development-branch"
]
},
"skills_failed": {}
},
"F1/mp": {
"task": "F1",
"cfg": "mp",
"label": "mp (Main Flow)",
"n": 3,
"hidden_mean": 5,
"hidden_median": 5,
"hidden_per_run": [
5,
5,
5
],
"req_pass_rate": {
"test_req1": 3,
"test_req2": 3,
"test_req3": 3,
"test_req4": 3,
"test_req5": 3
},
"suite_green": 3,
"typecheck_green": 3,
"tests_added_mean": 27.667,
"code_diff_mean": 372,
"code_diff_median": 369,
"all_diff_mean": 578,
"non_code_files_mean": 7.667,
"tokens_mean": 4476103.667,
"tokens_median": 4286993,
"out_tokens_mean": 56959,
"usd_mean": 2.043,
"usd_median": 2.102,
"usd_po_mean": 0.205,
"wall_s_mean": 439.033,
"wall_s_median": 472.4,
"driver_turns_mean": 12,
"questions_mean": 7,
"caps": {
"mp-r1": [
"driver_turns_12"
],
"mp-r3": [
"driver_turns_12"
]
},
"end_reasons": [
"cap_driver_turns",
"done",
"cap_driver_turns"
],
"models": [
"claude-sonnet-5-5"
],
"superpowers_injected": [
false,
false,
false
],
"skills_per_run": {
"mp-r1": [
"grilling",
"domain-modeling",
"tdd",
"code-review"
],
"mp-r2": [
"grilling",
"domain-modeling",
"tdd",
"code-review"
],
"mp-r3": [
"grilling",
"domain-modeling",
"tdd",
"code-review"
]
},
"skills_failed": {}
},
"F1/both": {
"task": "F1",
"cfg": "both",
"label": "both (sp+mp, mp flow)",
"n": 1,
"hidden_mean": 5,
"hidden_median": 5,
"hidden_per_run": [
5
],
"req_pass_rate": {
"test_req1": 1,
"test_req2": 1,
"test_req3": 1,
"test_req4": 1,
"test_req5": 1
},
"suite_green": 1,
"typecheck_green": 1,
"tests_added_mean": 22,
"code_diff_mean": 374,
"code_diff_median": 374,
"all_diff_mean": 561,
"non_code_files_mean": 8,
"tokens_mean": 5754354,
"tokens_median": 5754354,
"out_tokens_mean": 57632,
"usd_mean": 2.326,
"usd_median": 2.326,
"usd_po_mean": 0.218,
"wall_s_mean": 448.2,
"wall_s_median": 448.2,
"driver_turns_mean": 12,
"questions_mean": 6,
"caps": {},
"end_reasons": [
"done"
],
"models": [
"claude-sonnet-5-5"
],
"superpowers_injected": [
true
],
"skills_per_run": {
"both-r1": [
"grilling",
"domain-modeling",
"tdd",
"code-review"
]
},
"skills_failed": {}
},
"F2/bare": {
"task": "F2",
"cfg": "bare",
"label": "bare",
"n": 2,
"hidden_mean": 4.5,
"hidden_median": 4.5,
"hidden_per_run": [
4,
5
],
"req_pass_rate": {
"test_req1": 2,
"test_req2": 2,
"test_req3": 1,
"test_req4": 2,
"test_req5": 2
},
"suite_green": 2,
"typecheck_green": 2,
"tests_added_mean": 5,
"code_diff_mean": 206.5,
"code_diff_median": 206.5,
"all_diff_mean": 209.5,
"non_code_files_mean": 1,
"tokens_mean": 651322,
"tokens_median": 651322.0,
"out_tokens_mean": 15475,
"usd_mean": 0.413,
"usd_median": 0.413,
"usd_po_mean": 0.031,
"wall_s_mean": 103.15,
"wall_s_median": 103.15,
"driver_turns_mean": 2,
"questions_mean": 0.5,
"caps": {},
"end_reasons": [
"done",
"done"
],
"models": [
"claude-sonnet-5-5"
],
"superpowers_injected": [
false,
false
],
"skills_per_run": {
"f2-bare-r1": [],
"f2-bare-r2": []
},
"skills_failed": {}
},
"F2/spd": {
"task": "F2",
"cfg": "spd",
"label": "spd (Superpowers driven)",
"n": 2,
"hidden_mean": 5,
"hidden_median": 5.0,
"hidden_per_run": [
5,
5
],
"req_pass_rate": {
"test_req1": 2,
"test_req2": 2,
"test_req3": 2,
"test_req4": 2,
"test_req5": 2
},
"suite_green": 2,
"typecheck_green": 2,
"tests_added_mean": 33,
"code_diff_mean": 488.5,
"code_diff_median": 488.5,
"all_diff_mean": 750.5,
"non_code_files_mean": 2.5,
"tokens_mean": 4490365,
"tokens_median": 4490365.0,
"out_tokens_mean": 56644.5,
"usd_mean": 3.213,
"usd_median": 3.213,
"usd_po_mean": 0.101,
"wall_s_mean": 511.7,
"wall_s_median": 511.7,
"driver_turns_mean": 6.5,
"questions_mean": 3.5,
"caps": {},
"end_reasons": [
"done",
"done"
],
"models": [
"claude-fable-5-1",
"claude-sonnet-5-5"
],
"superpowers_injected": [
true,
true
],
"skills_per_run": {
"f2-spd-r1": [
"writing-plans",
"test-driven-development"
],
"f2-spd-r2": [
"test-driven-development"
]
},
"skills_failed": {}
},
"F2/mp": {
"task": "F2",
"cfg": "mp",
"label": "mp (Main Flow)",
"n": 2,
"hidden_mean": 5,
"hidden_median": 5.0,
"hidden_per_run": [
5,
5
],
"req_pass_rate": {
"test_req1": 2,
"test_req2": 2,
"test_req3": 2,
"test_req4": 2,
"test_req5": 2
},
"suite_green": 2,
"typecheck_green": 2,
"tests_added_mean": 31,
"code_diff_mean": 469,
"code_diff_median": 469.0,
"all_diff_mean": 658,
"non_code_files_mean": 6.5,
"tokens_mean": 4520638.5,
"tokens_median": 4520638.5,
"out_tokens_mean": 57142,
"usd_mean": 2.067,
"usd_median": 2.067,
"usd_po_mean": 0.203,
"wall_s_mean": 421.2,
"wall_s_median": 421.2,
"driver_turns_mean": 12,
"questions_mean": 6.5,
"caps": {
"f2-mp-r1": [
"driver_turns_12"
]
},
"end_reasons": [
"cap_driver_turns",
"done"
],
"models": [
"claude-sonnet-5-5"
],
"superpowers_injected": [
false,
false
],
"skills_per_run": {
"f2-mp-r1": [
"grilling",
"domain-modeling",
"tdd",
"code-review"
],
"f2-mp-r2": [
"grilling",
"domain-modeling",
"tdd",
"code-review"
]
},
"skills_failed": {}
}
},
"stages": {
"F1/sp_skill": {
"(none)": {
"runs_with_stage": 3,
"tokens_mean": 19549,
"tokens_median": 19538,
"out_tokens_mean": 238,
"usd_mean": 0.03,
"usd_median": 0.03,
"wall_s_mean": 5.333,
"wall_s_median": 5.4
},
"brainstorming": {
"runs_with_stage": 3,
"tokens_mean": 249260,
"tokens_median": 247900,
"out_tokens_mean": 8037.333,
"usd_mean": 0.219,
"usd_median": 0.244,
"wall_s_mean": 40.533,
"wall_s_median": 40.1
},
"test-driven-development": {
"runs_with_stage": 1,
"tokens_mean": 595318,
"tokens_median": 595318,
"out_tokens_mean": 15622,
"usd_mean": 0.405,
"usd_median": 0.405,
"wall_s_mean": 93.0,
"wall_s_median": 93.0
},
"implementation (no skill)": {
"runs_with_stage": 2,
"tokens_mean": 386028.5,
"tokens_median": 386028.5,
"out_tokens_mean": 5080,
"usd_mean": 0.176,
"usd_median": 0.176,
"wall_s_mean": 60.3,
"wall_s_median": 60.3
}
},
"F1/sp_phase": {
"before any skill": {
"runs_with_stage": 3,
"tokens_mean": 19549,
"tokens_median": 19538,
"out_tokens_mean": 238,
"usd_mean": 0.03,
"usd_median": 0.03,
"wall_s_mean": 5.333,
"wall_s_median": 5.4
},
"brainstorming": {
"runs_with_stage": 3,
"tokens_mean": 249260,
"tokens_median": 247900,
"out_tokens_mean": 8037.333,
"usd_mean": 0.219,
"usd_median": 0.244,
"wall_s_mean": 40.533,
"wall_s_median": 40.1
},
"implementation (no skill)": {
"runs_with_stage": 2,
"tokens_mean": 386028.5,
"tokens_median": 386028.5,
"out_tokens_mean": 5080,
"usd_mean": 0.176,
"usd_median": 0.176,
"wall_s_mean": 60.3,
"wall_s_median": 60.3
},
"execution": {
"runs_with_stage": 1,
"tokens_mean": 595318,
"tokens_median": 595318,
"out_tokens_mean": 15622,
"usd_mean": 0.405,
"usd_median": 0.405,
"wall_s_mean": 93.0,
"wall_s_median": 93.0
}
},
"F1/spd_skill": {
"brainstorming": {
"runs_with_stage": 3,
"tokens_mean": 182021.667,
"tokens_median": 160834,
"out_tokens_mean": 4160.667,
"usd_mean": 0.166,
"usd_median": 0.156,
"wall_s_mean": 38.8,
"wall_s_median": 33.6
},
"writing-plans": {
"runs_with_stage": 3,
"tokens_mean": 321354.667,
"tokens_median": 298267,
"out_tokens_mean": 11827,
"usd_mean": 0.266,
"usd_median": 0.274,
"wall_s_mean": 86.733,
"wall_s_median": 95.1
},
"executing-plans": {
"runs_with_stage": 3,
"tokens_mean": 66938,
"tokens_median": 68739,
"out_tokens_mean": 505,
"usd_mean": 0.047,
"usd_median": 0.047,
"wall_s_mean": 4.6,
"wall_s_median": 4.6
},
"test-driven-development": {
"runs_with_stage": 3,
"tokens_mean": 2927098.333,
"tokens_median": 2854967,
"out_tokens_mean": 22508.333,
"usd_mean": 0.974,
"usd_median": 0.936,
"wall_s_mean": 457.033,
"wall_s_median": 448.9
},
"finishing-a-development-branch": {
"runs_with_stage": 1,
"tokens_mean": 976505,
"tokens_median": 976505,
"out_tokens_mean": 5297,
"usd_mean": 0.282,
"usd_median": 0.282,
"wall_s_mean": 50.7,
"wall_s_median": 50.7
},
"test-driven-development [subagent]": {
"runs_with_stage": 3,
"tokens_mean": 656962,
"tokens_median": 565811,
"out_tokens_mean": 20918,
"usd_mean": 2.192,
"usd_median": 2.047,
"wall_s_mean": 0.0,
"wall_s_median": 0.0
}
},
"F1/spd_phase": {
"brainstorming": {
"runs_with_stage": 3,
"tokens_mean": 182021.667,
"tokens_median": 160834,
"out_tokens_mean": 4160.667,
"usd_mean": 0.166,
"usd_median": 0.156,
"wall_s_mean": 38.8,
"wall_s_median": 33.6
},
"writing-plans": {
"runs_with_stage": 3,
"tokens_mean": 321354.667,
"tokens_median": 298267,
"out_tokens_mean": 11827,
"usd_mean": 0.266,
"usd_median": 0.274,
"wall_s_mean": 86.733,
"wall_s_median": 95.1
},
"execution": {
"runs_with_stage": 3,
"tokens_mean": 2994036.333,
"tokens_median": 2916459,
"out_tokens_mean": 23013.333,
"usd_mean": 1.021,
"usd_median": 0.981,
"wall_s_mean": 461.633,
"wall_s_median": 453.5
},
"finishing": {
"runs_with_stage": 1,
"tokens_mean": 976505,
"tokens_median": 976505,
"out_tokens_mean": 5297,
"usd_mean": 0.282,
"usd_median": 0.282,
"wall_s_mean": 50.7,
"wall_s_median": 50.7
},
"subagents (reviewers, explorers)": {
"runs_with_stage": 3,
"tokens_mean": 656962,
"tokens_median": 565811,
"out_tokens_mean": 20918,
"usd_mean": 2.192,
"usd_median": 2.047,
"wall_s_mean": 0.0,
"wall_s_median": 0.0
}
},
"F1/spd_slash": {
"/brainstorming": {
"runs_with_stage": 3,
"tokens_mean": 182021.667,
"tokens_median": 160834,
"out_tokens_mean": null,
"usd_mean": 0.166,
"usd_median": 0.156,
"wall_s_mean": 40.283,
"wall_s_median": 35.44
},
"/writing-plans": {
"runs_with_stage": 3,
"tokens_mean": 321354.667,
"tokens_median": 298267,
"out_tokens_mean": null,
"usd_mean": 0.266,
"usd_median": 0.274,
"wall_s_mean": 88.133,
"wall_s_median": 95.82
},
"/executing-plans": {
"runs_with_stage": 3,
"tokens_mean": 3976500,
"tokens_median": 3427914,
"out_tokens_mean": null,
"usd_mean": 3.307,
"usd_median": 3.028,
"wall_s_mean": 478.95,
"wall_s_median": 453.89
}
},
"F1/mp": {
"/grill-with-docs": {
"runs_with_stage": 3,
"tokens_mean": 316507.333,
"tokens_median": 348862,
"out_tokens_mean": 5999,
"usd_mean": 0.204,
"usd_median": 0.215,
"wall_s_mean": 59.933,
"wall_s_median": 59.9
},
"/to-spec": {
"runs_with_stage": 3,
"tokens_mean": 163344,
"tokens_median": 167578,
"out_tokens_mean": 5134.667,
"usd_mean": 0.118,
"usd_median": 0.113,
"wall_s_mean": 42.1,
"wall_s_median": 40.8
},
"/to-tickets": {
"runs_with_stage": 3,
"tokens_mean": 257955.333,
"tokens_median": 268062,
"out_tokens_mean": 5867.333,
"usd_mean": 0.143,
"usd_median": 0.151,
"wall_s_mean": 44.4,
"wall_s_median": 47.8
},
"/implement": {
"runs_with_stage": 3,
"tokens_mean": 2655564.667,
"tokens_median": 2613062,
"out_tokens_mean": 30214.667,
"usd_mean": 1.125,
"usd_median": 1.195,
"wall_s_mean": 224.567,
"wall_s_median": 245.4
},
"/code-review": {
"runs_with_stage": 3,
"tokens_mean": 1047585.667,
"tokens_median": 1107555,
"out_tokens_mean": 9293.667,
"usd_mean": 0.439,
"usd_median": 0.445,
"wall_s_mean": 64.2,
"wall_s_median": 65.8
},
"/pr": {
"runs_with_stage": 1,
"tokens_mean": 105440,
"tokens_median": 105440,
"out_tokens_mean": 1349,
"usd_mean": 0.041,
"usd_median": 0.041,
"wall_s_mean": 11.6,
"wall_s_median": 11.6
}
},
"F1/both": {
"/grill-with-docs": {
"runs_with_stage": 1,
"tokens_mean": 344391,
"tokens_median": 344391,
"out_tokens_mean": 6193,
"usd_mean": 0.225,
"usd_median": 0.225,
"wall_s_mean": 61.8,
"wall_s_median": 61.8
},
"/to-spec": {
"runs_with_stage": 1,
"tokens_mean": 217572,
"tokens_median": 217572,
"out_tokens_mean": 4835,
"usd_mean": 0.123,
"usd_median": 0.123,
"wall_s_mean": 42.9,
"wall_s_median": 42.9
},
"/to-tickets": {
"runs_with_stage": 1,
"tokens_mean": 382953,
"tokens_median": 382953,
"out_tokens_mean": 7115,
"usd_mean": 0.196,
"usd_median": 0.196,
"wall_s_mean": 49.4,
"wall_s_median": 49.4
},
"/implement": {
"runs_with_stage": 1,
"tokens_mean": 3070579,
"tokens_median": 3070579,
"out_tokens_mean": 24618,
"usd_mean": 1.112,
"usd_median": 1.112,
"wall_s_mean": 194.2,
"wall_s_median": 194.2
},
"/code-review": {
"runs_with_stage": 1,
"tokens_mean": 1615265,
"tokens_median": 1615265,
"out_tokens_mean": 13410,
"usd_mean": 0.625,
"usd_median": 0.625,
"wall_s_mean": 86.8,
"wall_s_median": 86.8
},
"/pr": {
"runs_with_stage": 1,
"tokens_mean": 123594,
"tokens_median": 123594,
"out_tokens_mean": 1461,
"usd_mean": 0.045,
"usd_median": 0.045,
"wall_s_mean": 13.1,
"wall_s_median": 13.1
}
},
"F2/spd_skill": {
"brainstorming": {
"runs_with_stage": 2,
"tokens_mean": 253890,
"tokens_median": 253890.0,
"out_tokens_mean": 6038,
"usd_mean": 0.207,
"usd_median": 0.207,
"wall_s_mean": 53.05,
"wall_s_median": 53.05
},
"writing-plans": {
"runs_with_stage": 2,
"tokens_mean": 397486.5,
"tokens_median": 397486.5,
"out_tokens_mean": 14059.5,
"usd_mean": 0.307,
"usd_median": 0.307,
"wall_s_mean": 90.75,
"wall_s_median": 90.75
},
"executing-plans": {
"runs_with_stage": 2,
"tokens_mean": 69973,
"tokens_median": 69973.0,
"out_tokens_mean": 286.5,
"usd_mean": 0.046,
"usd_median": 0.046,
"wall_s_mean": 3.5,
"wall_s_median": 3.5
},
"test-driven-development": {
"runs_with_stage": 2,
"tokens_mean": 3304676.5,
"tokens_median": 3304676.5,
"out_tokens_mean": 23243.5,
"usd_mean": 1.052,
"usd_median": 1.052,
"wall_s_mean": 361.25,
"wall_s_median": 361.25
},
"test-driven-development [subagent]": {
"runs_with_stage": 2,
"tokens_mean": 464339,
"tokens_median": 464339.0,
"out_tokens_mean": 13017,
"usd_mean": 1.602,
"usd_median": 1.602,
"wall_s_mean": 0.0,
"wall_s_median": 0.0
}
},
"F2/spd_phase": {
"brainstorming": {
"runs_with_stage": 2,
"tokens_mean": 253890,
"tokens_median": 253890.0,
"out_tokens_mean": 6038,
"usd_mean": 0.207,
"usd_median": 0.207,
"wall_s_mean": 53.05,
"wall_s_median": 53.05
},
"writing-plans": {
"runs_with_stage": 2,
"tokens_mean": 397486.5,
"tokens_median": 397486.5,
"out_tokens_mean": 14059.5,
"usd_mean": 0.307,
"usd_median": 0.307,
"wall_s_mean": 90.75,
"wall_s_median": 90.75
},
"execution": {
"runs_with_stage": 2,
"tokens_mean": 3374649.5,
"tokens_median": 3374649.5,
"out_tokens_mean": 23530,
"usd_mean": 1.098,
"usd_median": 1.098,
"wall_s_mean": 364.75,
"wall_s_median": 364.75
},
"subagents (reviewers, explorers)": {
"runs_with_stage": 2,
"tokens_mean": 464339,
"tokens_median": 464339.0,
"out_tokens_mean": 13017,
"usd_mean": 1.602,
"usd_median": 1.602,
"wall_s_mean": 0.0,
"wall_s_median": 0.0
}
},
"F2/spd_slash": {
"/brainstorming": {
"runs_with_stage": 2,
"tokens_mean": 485364,
"tokens_median": 485364.0,
"out_tokens_mean": null,
"usd_mean": 0.354,
"usd_median": 0.354,
"wall_s_mean": 96.815,
"wall_s_median": 96.815
},
"/writing-plans": {
"runs_with_stage": 2,
"tokens_mean": 166012.5,
"tokens_median": 166012.5,
"out_tokens_mean": null,
"usd_mean": 0.159,
"usd_median": 0.159,
"wall_s_mean": 49.755,
"wall_s_median": 49.755
},
"/executing-plans": {
"runs_with_stage": 2,
"tokens_mean": 3838988.5,
"tokens_median": 3838988.5,
"out_tokens_mean": null,
"usd_mean": 2.7,
"usd_median": 2.7,
"wall_s_mean": 365.1,
"wall_s_median": 365.1
}
},
"F2/mp": {
"/grill-with-docs": {
"runs_with_stage": 2,
"tokens_mean": 232965,
"tokens_median": 232965.0,
"out_tokens_mean": 5946,
"usd_mean": 0.182,
"usd_median": 0.182,
"wall_s_mean": 56.7,
"wall_s_median": 56.7
},
"/to-spec": {
"runs_with_stage": 2,
"tokens_mean": 180084.5,
"tokens_median": 180084.5,
"out_tokens_mean": 4452,
"usd_mean": 0.115,
"usd_median": 0.115,
"wall_s_mean": 37.6,
"wall_s_median": 37.6
},
"/to-tickets": {
"runs_with_stage": 2,
"tokens_mean": 253688.5,
"tokens_median": 253688.5,
"out_tokens_mean": 5152.5,
"usd_mean": 0.131,
"usd_median": 0.131,
"wall_s_mean": 38.5,
"wall_s_median": 38.5
},
"/implement": {
"runs_with_stage": 2,
"tokens_mean": 2689071,
"tokens_median": 2689071.0,
"out_tokens_mean": 31452,
"usd_mean": 1.159,
"usd_median": 1.159,
"wall_s_mean": 218.35,
"wall_s_median": 218.35
},
"/code-review": {
"runs_with_stage": 2,
"tokens_mean": 1107122.5,
"tokens_median": 1107122.5,
"out_tokens_mean": 9453.5,
"usd_mean": 0.458,
"usd_median": 0.458,
"wall_s_mean": 63.85,
"wall_s_median": 63.85
},
"/pr": {
"runs_with_stage": 1,
"tokens_mean": 115414,
"tokens_median": 115414,
"out_tokens_mean": 1372,
"usd_mean": 0.043,
"usd_median": 0.043,
"wall_s_mean": 12.4,
"wall_s_median": 12.4
}
}
},
"runs": [
{
"run": "bare-r1",
"task": "F1",
"cfg": "bare",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 149,
"code_diff_removed": 11,
"diff_added": 150,
"diff_removed": 12,
"tests_added_suite": 6,
"tokens": {
"in": 42,
"out": 16265,
"cr": 786548,
"cw": 37896,
"total": 840751
},
"usd": 0.4716,
"usd_po": 0.0321,
"wall_s": 118.8,
"driver_turns": 2,
"questions_routed_to_po": 1,
"end_reason": "done",
"caps": [],
"skills_invoked": [],
"superpowers_injected": false,
"commits": 0,
"models": [
"claude-sonnet-5-5"
],
"stages": {
"bare": {
"turns": [
1,
2
],
"in": 42,
"out": 16265,
"cr": 786548,
"cw": 37896,
"total": 840751,
"usd": 0.4716,
"wall_s": 118.8,
"api_calls": 21
}
}
},
{
"run": "bare-r2",
"task": "F1",
"cfg": "bare",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 118,
"code_diff_removed": 10,
"diff_added": 118,
"diff_removed": 10,
"tests_added_suite": 4,
"tokens": {
"in": 22,
"out": 9513,
"cr": 316959,
"cw": 28254,
"total": 354748
},
"usd": 0.2716,
"usd_po": 0.016,
"wall_s": 63.0,
"driver_turns": 1,
"questions_routed_to_po": 0,
"end_reason": "done",
"caps": [],
"skills_invoked": [],
"superpowers_injected": false,
"commits": 0,
"models": [
"claude-sonnet-5-5"
],
"stages": {
"bare": {
"turns": [
1
],
"in": 22,
"out": 9513,
"cr": 316959,
"cw": 28254,
"total": 354748,
"usd": 0.2716,
"wall_s": 63.0,
"api_calls": 11
}
}
},
{
"run": "bare-r3",
"task": "F1",
"cfg": "bare",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 125,
"code_diff_removed": 12,
"diff_added": 125,
"diff_removed": 12,
"tests_added_suite": 4,
"tokens": {
"in": 22,
"out": 10225,
"cr": 321705,
"cw": 29890,
"total": 361842
},
"usd": 0.2862,
"usd_po": 0.0148,
"wall_s": 71.7,
"driver_turns": 1,
"questions_routed_to_po": 0,
"end_reason": "done",
"caps": [],
"skills_invoked": [],
"superpowers_injected": false,
"commits": 0,
"models": [
"claude-sonnet-5-5"
],
"stages": {
"bare": {
"turns": [
1
],
"in": 22,
"out": 10225,
"cr": 321705,
"cw": 29890,
"total": 361842,
"usd": 0.2862,
"wall_s": 71.7,
"api_calls": 11
}
}
},
{
"run": "both-r1",
"task": "F1",
"cfg": "both",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 356,
"code_diff_removed": 18,
"diff_added": 543,
"diff_removed": 18,
"tests_added_suite": 22,
"tokens": {
"in": 192,
"out": 57632,
"cr": 5502518,
"cw": 194012,
"total": 5754354
},
"usd": 2.3259,
"usd_po": 0.2185,
"wall_s": 448.2,
"driver_turns": 12,
"questions_routed_to_po": 6,
"end_reason": "done",
"caps": [],
"skills_invoked": [
"grilling",
"domain-modeling",
"tdd",
"code-review"
],
"superpowers_injected": true,
"commits": 7,
"models": [
"claude-sonnet-5-5"
],
"stages": {
"/grill-with-docs": {
"turns": [
1,
2,
3,
4
],
"in": 22,
"out": 6193,
"cr": 313031,
"cw": 25145,
"total": 344391,
"usd": 0.2252,
"wall_s": 61.8,
"api_calls": 11
},
"/to-spec": {
"turns": [
5,
6
],
"in": 12,
"out": 4835,
"cr": 204355,
"cw": 8370,
"total": 217572,
"usd": 0.1227,
"wall_s": 42.9,
"api_calls": 5
},
"/to-tickets": {
"turns": [
7,
8
],
"in": 16,
"out": 7115,
"cr": 362815,
"cw": 13007,
"total": 382953,
"usd": 0.1958,
"wall_s": 49.4,
"api_calls": 7
},
"/implement": {
"turns": [
9
],
"in": 92,
"out": 24618,
"cr": 2960003,
"cw": 85866,
"total": 3070579,
"usd": 1.1123,
"wall_s": 194.2,
"api_calls": 36
},
"/code-review": {
"turns": [
10,
11
],
"in": 46,
"out": 13410,
"cr": 1541663,
"cw": 60146,
"total": 1615265,
"usd": 0.6253,
"wall_s": 86.8,
"api_calls": 13
},
"/pr": {
"turns": [
12
],
"in": 4,
"out": 1461,
"cr": 120651,
"cw": 1478,
"total": 123594,
"usd": 0.0447,
"wall_s": 13.1,
"api_calls": 1
}
}
},
{
"run": "f2-bare-r1",
"task": "F2",
"cfg": "bare",
"hidden_pass": 4,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": false,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 204,
"code_diff_removed": 10,
"diff_added": 208,
"diff_removed": 10,
"tests_added_suite": 5,
"tokens": {
"in": 28,
"out": 15066,
"cr": 444890,
"cw": 34247,
"total": 494231
},
"usd": 0.3767,
"usd_po": 0.0167,
"wall_s": 91.6,
"driver_turns": 1,
"questions_routed_to_po": 0,
"end_reason": "done",
"caps": [],
"skills_invoked": [],
"superpowers_injected": false,
"commits": 0,
"models": [
"claude-sonnet-5-5"
],
"stages": {
"bare": {
"turns": [
1
],
"in": 28,
"out": 15066,
"cr": 444890,
"cw": 34247,
"total": 494231,
"usd": 0.3767,
"wall_s": 91.6,
"api_calls": 14
}
}
},
{
"run": "f2-bare-r2",
"task": "F2",
"cfg": "bare",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 189,
"code_diff_removed": 10,
"diff_added": 191,
"diff_removed": 10,
"tests_added_suite": 5,
"tokens": {
"in": 42,
"out": 15884,
"cr": 757517,
"cw": 34970,
"total": 808413
},
"usd": 0.4503,
"usd_po": 0.0453,
"wall_s": 114.7,
"driver_turns": 3,
"questions_routed_to_po": 1,
"end_reason": "done",
"caps": [],
"skills_invoked": [],
"superpowers_injected": false,
"commits": 1,
"models": [
"claude-sonnet-5-5"
],
"stages": {
"bare": {
"turns": [
1,
2,
3
],
"in": 42,
"out": 15884,
"cr": 757517,
"cw": 34970,
"total": 808413,
"usd": 0.4503,
"wall_s": 114.7,
"api_calls": 21
}
}
},
{
"run": "f2-mp-r1",
"task": "F2",
"cfg": "mp",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 460,
"code_diff_removed": 23,
"diff_added": 666,
"diff_removed": 25,
"tests_added_suite": 31,
"tokens": {
"in": 172,
"out": 60658,
"cr": 4559790,
"cw": 202896,
"total": 4823516
},
"usd": 2.1824,
"usd_po": 0.1989,
"wall_s": 443.0,
"driver_turns": 12,
"questions_routed_to_po": 7,
"end_reason": "cap_driver_turns",
"caps": [
"driver_turns_12"
],
"skills_invoked": [
"grilling",
"domain-modeling",
"tdd",
"code-review"
],
"superpowers_injected": false,
"commits": 7,
"models": [
"claude-sonnet-5-5"
],
"stages": {
"/grill-with-docs": {
"turns": [
1,
2,
3
],
"in": 18,
"out": 6627,
"cr": 229372,
"cw": 22326,
"total": 258343,
"usd": 0.2015,
"wall_s": 62.8,
"api_calls": 9
},
"/to-spec": {
"turns": [
4,
5
],
"in": 12,
"out": 5218,
"cr": 195137,
"cw": 10113,
"total": 210480,
"usd": 0.1317,
"wall_s": 45.4,
"api_calls": 5
},
"/to-tickets": {
"turns": [
6,
7,
8
],
"in": 16,
"out": 7288,
"cr": 356825,
"cw": 10505,
"total": 374634,
"usd": 0.1863,
"wall_s": 53.0,
"api_calls": 7
},
"/implement": {
"turns": [
9,
10
],
"in": 92,
"out": 34442,
"cr": 2978947,
"cw": 104921,
"total": 3118402,
"usd": 1.2795,
"wall_s": 238.8,
"api_calls": 36
},
"/code-review": {
"turns": [
11,
12
],
"in": 34,
"out": 7083,
"cr": 799509,
"cw": 55031,
"total": 861657,
"usd": 0.3835,
"wall_s": 43.1,
"api_calls": 6
}
}
},
{
"run": "f2-mp-r2",
"task": "F2",
"cfg": "mp",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 440,
"code_diff_removed": 15,
"diff_added": 607,
"diff_removed": 18,
"tests_added_suite": 31,
"tokens": {
"in": 152,
"out": 53626,
"cr": 3976936,
"cw": 187047,
"total": 4217761
},
"usd": 1.9511,
"usd_po": 0.2073,
"wall_s": 399.4,
"driver_turns": 12,
"questions_routed_to_po": 6,
"end_reason": "done",
"caps": [],
"skills_invoked": [
"grilling",
"domain-modeling",
"tdd",
"code-review"
],
"superpowers_injected": false,
"commits": 2,
"models": [
"claude-sonnet-5-5"
],
"stages": {
"/grill-with-docs": {
"turns": [
1,
2,
3
],
"in": 16,
"out": 5265,
"cr": 184055,
"cw": 18251,
"total": 207587,
"usd": 0.1625,
"wall_s": 50.6,
"api_calls": 8
},
"/to-spec": {
"turns": [
4,
5
],
"in": 10,
"out": 3686,
"cr": 137547,
"cw": 8446,
"total": 149689,
"usd": 0.0982,
"wall_s": 29.8,
"api_calls": 4
},
"/to-tickets": {
"turns": [
6,
7
],
"in": 8,
"out": 3017,
"cr": 124377,
"cw": 5341,
"total": 132743,
"usd": 0.0764,
"wall_s": 24.0,
"api_calls": 3
},
"/implement": {
"turns": [
8
],
"in": 74,
"out": 28462,
"cr": 2129926,
"cw": 101278,
"total": 2259740,
"usd": 1.0391,
"wall_s": 197.9,
"api_calls": 27
},
"/code-review": {
"turns": [
9,
10,
11
],
"in": 43,
"out": 11824,
"cr": 1288603,
"cw": 52118,
"total": 1352588,
"usd": 0.5323,
"wall_s": 84.6,
"api_calls": 11
},
"/pr": {
"turns": [
12
],
"in": 1,
"out": 1372,
"cr": 112428,
"cw": 1613,
"total": 115414,
"usd": 0.0427,
"wall_s": 12.4,
"api_calls": 1
}
}
},
{
"run": "f2-spd-r1",
"task": "F2",
"cfg": "spd",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 546,
"code_diff_removed": 14,
"diff_added": 856,
"diff_removed": 17,
"tests_added_suite": 46,
"tokens": {
"in": 380,
"out": 56881,
"cr": 5074733,
"cw": 178622,
"total": 5310616
},
"usd": 3.4244,
"usd_po": 0.09,
"wall_s": 506.3,
"driver_turns": 6,
"questions_routed_to_po": 3,
"end_reason": "done",
"caps": [],
"skills_invoked": [
"writing-plans",
"test-driven-development"
],
"superpowers_injected": true,
"commits": 8,
"models": [
"claude-fable-5-1",
"claude-sonnet-5-5"
],
"stages": {
"brainstorming": {
"turns": [
1,
2,
3
],
"in": 18,
"out": 8103,
"cr": 280105,
"cw": 28324,
"total": 316550,
"usd": 0.2504,
"wall_s": 64.3,
"api_calls": 9
},
"writing-plans": {
"turns": [
3,
4,
5
],
"in": 20,
"out": 13028,
"cr": 491569,
"cw": 24216,
"total": 528833,
"usd": 0.3255,
"wall_s": 89.2,
"api_calls": 9
},
"writing-plans [subagent]": {
"turns": [
4
],
"in": 0,
"out": 0,
"cr": 0,
"cw": 0,
"total": 0,
"usd": 0.0,
"wall_s": 0.0,
"api_calls": 0
},
"executing-plans": {
"turns": [
6
],
"in": 4,
"out": 291,
"cr": 65551,
"cw": 7617,
"total": 73463,
"usd": 0.0465,
"wall_s": 4.0,
"api_calls": 1
},
"test-driven-development": {
"turns": [
6
],
"in": 80,
"out": 24030,
"cr": 3786616,
"cw": 42748,
"total": 3853474,
"usd": 1.1688,
"wall_s": 345.9,
"api_calls": 40
},
"test-driven-development [subagent]": {
"turns": [
6
],
"in": 258,
"out": 11429,
"cr": 450892,
"cw": 75717,
"total": 538296,
"usd": 1.6332,
"wall_s": 0.0,
"api_calls": 0
}
}
},
{
"run": "f2-spd-r2",
"task": "F2",
"cfg": "spd",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 401,
"code_diff_removed": 16,
"diff_added": 611,
"diff_removed": 17,
"tests_added_suite": 20,
"tokens": {
"in": 316,
"out": 56408,
"cr": 3456411,
"cw": 156979,
"total": 3670114
},
"usd": 3.0022,
"usd_po": 0.1126,
"wall_s": 517.1,
"driver_turns": 7,
"questions_routed_to_po": 4,
"end_reason": "done",
"caps": [],
"skills_invoked": [
"test-driven-development"
],
"superpowers_injected": true,
"commits": 6,
"models": [
"claude-fable-5-1",
"claude-sonnet-5-5"
],
"stages": {
"brainstorming": {
"turns": [
1,
2,
3,
4
],
"in": 12,
"out": 3973,
"cr": 164538,
"cw": 22707,
"total": 191230,
"usd": 0.1635,
"wall_s": 41.8,
"api_calls": 6
},
"brainstorming [subagent]": {
"turns": [
1,
3
],
"in": 0,
"out": 0,
"cr": 0,
"cw": 0,
"total": 0,
"usd": 0.0,
"wall_s": 0.0,
"api_calls": 0
},
"writing-plans": {
"turns": [
5,
6
],
"in": 12,
"out": 15091,
"cr": 228188,
"cw": 22849,
"total": 266140,
"usd": 0.288,
"wall_s": 92.3,
"api_calls": 5
},
"writing-plans [subagent]": {
"turns": [
6
],
"in": 0,
"out": 0,
"cr": 0,
"cw": 0,
"total": 0,
"usd": 0.0,
"wall_s": 0.0,
"api_calls": 0
},
"executing-plans": {
"turns": [
7
],
"in": 4,
"out": 282,
"cr": 58567,
"cw": 7630,
"total": 66483,
"usd": 0.0451,
"wall_s": 3.0,
"api_calls": 1
},
"test-driven-development": {
"turns": [
7
],
"in": 62,
"out": 22457,
"cr": 2690340,
"cw": 43020,
"total": 2755879,
"usd": 0.9348,
"wall_s": 376.6,
"api_calls": 31
},
"test-driven-development [subagent]": {
"turns": [
7
],
"in": 226,
"out": 14605,
"cr": 314778,
"cw": 60773,
"total": 390382,
"usd": 1.5709,
"wall_s": 0.0,
"api_calls": 0
}
}
},
{
"run": "mp-r1",
"task": "F1",
"cfg": "mp",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 349,
"code_diff_removed": 20,
"diff_added": 524,
"diff_removed": 20,
"tests_added_suite": 29,
"tokens": {
"in": 181,
"out": 60902,
"cr": 4842442,
"cw": 194978,
"total": 5098503
},
"usd": 2.2217,
"usd_po": 0.2058,
"wall_s": 472.4,
"driver_turns": 12,
"questions_routed_to_po": 7,
"end_reason": "cap_driver_turns",
"caps": [
"driver_turns_12"
],
"skills_invoked": [
"grilling",
"domain-modeling",
"tdd",
"code-review"
],
"superpowers_injected": false,
"commits": 7,
"models": [
"claude-sonnet-5-5"
],
"stages": {
"/grill-with-docs": {
"turns": [
1,
2,
3
],
"in": 24,
"out": 6223,
"cr": 320351,
"cw": 22264,
"total": 348862,
"usd": 0.2154,
"wall_s": 59.9,
"api_calls": 12
},
"/to-spec": {
"turns": [
4,
5
],
"in": 10,
"out": 4516,
"cr": 153800,
"cw": 9252,
"total": 167578,
"usd": 0.1129,
"wall_s": 37.6,
"api_calls": 4
},
"/to-tickets": {
"turns": [
6,
7,
8
],
"in": 16,
"out": 7382,
"cr": 349965,
"cw": 10865,
"total": 368228,
"usd": 0.1873,
"wall_s": 55.8,
"api_calls": 7
},
"/implement": {
"turns": [
9,
10
],
"in": 92,
"out": 32290,
"cr": 2890397,
"cw": 100256,
"total": 3023035,
"usd": 1.226,
"wall_s": 251.2,
"api_calls": 35
},
"/code-review": {
"turns": [
11,
12
],
"in": 39,
"out": 10491,
"cr": 1127929,
"cw": 52341,
"total": 1190800,
"usd": 0.4801,
"wall_s": 67.9,
"api_calls": 9
}
}
},
{
"run": "mp-r2",
"task": "F1",
"cfg": "mp",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 314,
"code_diff_removed": 15,
"diff_added": 527,
"diff_removed": 15,
"tests_added_suite": 25,
"tokens": {
"in": 154,
"out": 48104,
"cr": 3826019,
"cw": 168538,
"total": 4042815
},
"usd": 1.8045,
"usd_po": 0.1988,
"wall_s": 365.0,
"driver_turns": 12,
"questions_routed_to_po": 6,
"end_reason": "done",
"caps": [],
"skills_invoked": [
"grilling",
"domain-modeling",
"tdd",
"code-review"
],
"superpowers_injected": false,
"commits": 7,
"models": [
"claude-sonnet-5-5"
],
"stages": {
"/grill-with-docs": {
"turns": [
1,
2,
3
],
"in": 16,
"out": 4727,
"cr": 187718,
"cw": 18567,
"total": 211028,
"usd": 0.1591,
"wall_s": 47.0,
"api_calls": 8
},
"/to-spec": {
"turns": [
4,
5
],
"in": 10,
"out": 4965,
"cr": 136863,
"cw": 8781,
"total": 150619,
"usd": 0.1122,
"wall_s": 40.8,
"api_calls": 4
},
"/to-tickets": {
"turns": [
6,
7
],
"in": 8,
"out": 4035,
"cr": 126982,
"cw": 6551,
"total": 137576,
"usd": 0.092,
"wall_s": 29.6,
"api_calls": 3
},
"/implement": {
"turns": [
8,
9
],
"in": 78,
"out": 24215,
"cr": 2224544,
"cw": 81760,
"total": 2330597,
"usd": 0.9551,
"wall_s": 177.1,
"api_calls": 31
},
"/code-review": {
"turns": [
10,
11
],
"in": 38,
"out": 8813,
"cr": 1047606,
"cw": 51098,
"total": 1107555,
"usd": 0.4451,
"wall_s": 58.9,
"api_calls": 10
},
"/pr": {
"turns": [
12
],
"in": 4,
"out": 1349,
"cr": 102306,
"cw": 1781,
"total": 105440,
"usd": 0.0411,
"wall_s": 11.6,
"api_calls": 1
}
}
},
{
"run": "mp-r3",
"task": "F1",
"cfg": "mp",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 395,
"code_diff_removed": 23,
"diff_added": 625,
"diff_removed": 23,
"tests_added_suite": 29,
"tokens": {
"in": 167,
"out": 61871,
"cr": 4014893,
"cw": 210062,
"total": 4286993
},
"usd": 2.1024,
"usd_po": 0.2105,
"wall_s": 479.7,
"driver_turns": 12,
"questions_routed_to_po": 8,
"end_reason": "cap_driver_turns",
"caps": [
"driver_turns_12"
],
"skills_invoked": [
"grilling",
"domain-modeling",
"tdd",
"code-review"
],
"superpowers_injected": false,
"commits": 5,
"models": [
"claude-sonnet-5-5"
],
"stages": {
"/grill-with-docs": {
"turns": [
1,
2,
3,
4
],
"in": 26,
"out": 7047,
"cr": 358800,
"cw": 23759,
"total": 389632,
"usd": 0.2373,
"wall_s": 72.9,
"api_calls": 13
},
"/to-spec": {
"turns": [
5,
6
],
"in": 10,
"out": 5923,
"cr": 156583,
"cw": 9319,
"total": 171835,
"usd": 0.1278,
"wall_s": 47.9,
"api_calls": 4
},
"/to-tickets": {
"turns": [
7,
8,
9
],
"in": 9,
"out": 6185,
"cr": 252224,
"cw": 9644,
"total": 268062,
"usd": 0.1509,
"wall_s": 47.8,
"api_calls": 5
},
"/implement": {
"turns": [
10,
11
],
"in": 82,
"out": 34139,
"cr": 2465960,
"cw": 112881,
"total": 2613062,
"usd": 1.1946,
"wall_s": 245.4,
"api_calls": 27
},
"/code-review": {
"turns": [
12
],
"in": 40,
"out": 8577,
"cr": 781326,
"cw": 54459,
"total": 844402,
"usd": 0.3917,
"wall_s": 65.8,
"api_calls": 5
}
}
},
{
"run": "sp-r1",
"task": "F1",
"cfg": "sp",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 182,
"code_diff_removed": 15,
"diff_added": 183,
"diff_removed": 16,
"tests_added_suite": 10,
"tokens": {
"in": 28,
"out": 14522,
"cr": 553287,
"cw": 45597,
"total": 613434
},
"usd": 0.4383,
"usd_po": 0.0424,
"wall_s": 99.4,
"driver_turns": 3,
"questions_routed_to_po": 2,
"end_reason": "done",
"caps": [],
"skills_invoked": [
"brainstorming"
],
"superpowers_injected": true,
"commits": 1,
"models": [
"claude-sonnet-5-5"
],
"stages": {
"(none)": {
"turns": [
1
],
"in": 2,
"out": 209,
"cr": 13011,
"cw": 6298,
"total": 19520,
"usd": 0.0299,
"wall_s": 5.4,
"api_calls": 1
},
"brainstorming": {
"turns": [
1,
2
],
"in": 14,
"out": 9666,
"cr": 212052,
"cw": 26168,
"total": 247900,
"usd": 0.2438,
"wall_s": 40.1,
"api_calls": 7
},
"implementation (no skill)": {
"turns": [
2,
3
],
"in": 12,
"out": 4647,
"cr": 328224,
"cw": 13131,
"total": 346014,
"usd": 0.1647,
"wall_s": 52.8,
"api_calls": 6
},
"implementation (no skill) [subagent]": {
"turns": [
2
],
"in": 0,
"out": 0,
"cr": 0,
"cw": 0,
"total": 0,
"usd": 0.0,
"wall_s": 0.0,
"api_calls": 0
}
}
},
{
"run": "sp-r2",
"task": "F1",
"cfg": "sp",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 212,
"code_diff_removed": 15,
"diff_added": 216,
"diff_removed": 15,
"tests_added_suite": 13,
"tokens": {
"in": 32,
"out": 18160,
"cr": 691441,
"cw": 55475,
"total": 765108
},
"usd": 0.5419,
"usd_po": 0.0305,
"wall_s": 119.6,
"driver_turns": 2,
"questions_routed_to_po": 1,
"end_reason": "done",
"caps": [],
"skills_invoked": [
"brainstorming",
"test-driven-development"
],
"superpowers_injected": true,
"commits": 0,
"models": [
"claude-sonnet-5-5"
],
"stages": {
"(none)": {
"turns": [
1
],
"in": 2,
"out": 227,
"cr": 13011,
"cw": 6298,
"total": 19538,
"usd": 0.0301,
"wall_s": 5.7,
"api_calls": 1
},
"brainstorming": {
"turns": [
1,
2
],
"in": 10,
"out": 2311,
"cr": 133731,
"cw": 14200,
"total": 150252,
"usd": 0.1067,
"wall_s": 20.4,
"api_calls": 5
},
"test-driven-development": {
"turns": [
2
],
"in": 20,
"out": 15622,
"cr": 544699,
"cw": 34977,
"total": 595318,
"usd": 0.4051,
"wall_s": 93.0,
"api_calls": 10
}
}
},
{
"run": "sp-r3",
"task": "F1",
"cfg": "sp",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 267,
"code_diff_removed": 14,
"diff_added": 267,
"diff_removed": 14,
"tests_added_suite": 14,
"tokens": {
"in": 34,
"out": 17926,
"cr": 727283,
"cw": 50017,
"total": 795260
},
"usd": 0.5249,
"usd_po": 0.0489,
"wall_s": 135.3,
"driver_turns": 3,
"questions_routed_to_po": 2,
"end_reason": "done",
"caps": [],
"skills_invoked": [
"brainstorming"
],
"superpowers_injected": true,
"commits": 1,
"models": [
"claude-sonnet-5-5"
],
"stages": {
"(none)": {
"turns": [
1
],
"in": 2,
"out": 278,
"cr": 13011,
"cw": 6298,
"total": 19589,
"usd": 0.0306,
"wall_s": 4.9,
"api_calls": 1
},
"brainstorming": {
"turns": [
1,
2
],
"in": 18,
"out": 12135,
"cr": 306378,
"cw": 31097,
"total": 349628,
"usd": 0.307,
"wall_s": 61.1,
"api_calls": 9
},
"implementation (no skill)": {
"turns": [
2,
3
],
"in": 14,
"out": 5513,
"cr": 407894,
"cw": 12622,
"total": 426043,
"usd": 0.1872,
"wall_s": 67.8,
"api_calls": 7
},
"implementation (no skill) [subagent]": {
"turns": [
2
],
"in": 0,
"out": 0,
"cr": 0,
"cw": 0,
"total": 0,
"usd": 0.0,
"wall_s": 0.0,
"api_calls": 0
}
}
},
{
"run": "spd-r1",
"task": "F1",
"cfg": "spd",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 350,
"code_diff_removed": 21,
"diff_added": 516,
"diff_removed": 21,
"tests_added_suite": 21,
"tokens": {
"in": 340,
"out": 59243,
"cr": 3428307,
"cw": 179396,
"total": 3667286
},
"usd": 3.4118,
"usd_po": 0.1084,
"wall_s": 600.8,
"driver_turns": 6,
"questions_routed_to_po": 3,
"end_reason": "done",
"caps": [],
"skills_invoked": [
"test-driven-development"
],
"superpowers_injected": true,
"commits": 4,
"models": [
"claude-fable-5-1",
"claude-sonnet-5-5"
],
"stages": {
"brainstorming": {
"turns": [
1,
2,
3
],
"in": 14,
"out": 5191,
"cr": 198912,
"cw": 25310,
"total": 229427,
"usd": 0.193,
"wall_s": 49.8,
"api_calls": 7
},
"brainstorming [subagent]": {
"turns": [
2
],
"in": 0,
"out": 0,
"cr": 0,
"cw": 0,
"total": 0,
"usd": 0.0,
"wall_s": 0.0,
"api_calls": 0
},
"writing-plans": {
"turns": [
4,
5
],
"in": 12,
"out": 13010,
"cr": 243336,
"cw": 23856,
"total": 280214,
"usd": 0.2742,
"wall_s": 95.1,
"api_calls": 5
},
"executing-plans": {
"turns": [
6
],
"in": 4,
"out": 614,
"cr": 62177,
"cw": 7788,
"total": 70583,
"usd": 0.0497,
"wall_s": 4.6,
"api_calls": 1
},
"test-driven-development": {
"turns": [
6
],
"in": 52,
"out": 22476,
"cr": 2455319,
"cw": 43404,
"total": 2521251,
"usd": 0.8895,
"wall_s": 448.9,
"api_calls": 26
},
"test-driven-development [subagent]": {
"turns": [
6
],
"in": 258,
"out": 17952,
"cr": 468563,
"cw": 79038,
"total": 565811,
"usd": 2.0053,
"wall_s": 0.0,
"api_calls": 0
}
}
},
{
"run": "spd-r2",
"task": "F1",
"cfg": "spd",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 317,
"code_diff_removed": 16,
"diff_added": 527,
"diff_removed": 16,
"tests_added_suite": 36,
"tokens": {
"in": 354,
"out": 53270,
"cr": 3666473,
"cw": 166918,
"total": 3887015
},
"usd": 3.3947,
"usd_po": 0.1118,
"wall_s": 520.0,
"driver_turns": 7,
"questions_routed_to_po": 4,
"end_reason": "done",
"caps": [],
"skills_invoked": [
"test-driven-development"
],
"superpowers_injected": true,
"commits": 5,
"models": [
"claude-fable-5-1",
"claude-sonnet-5-5"
],
"stages": {
"brainstorming": {
"turns": [
1,
2,
3
],
"in": 10,
"out": 3724,
"cr": 134220,
"cw": 22880,
"total": 160834,
"usd": 0.1556,
"wall_s": 33.6,
"api_calls": 5
},
"writing-plans": {
"turns": [
4,
5,
6
],
"in": 14,
"out": 8546,
"cr": 271872,
"cw": 17835,
"total": 298267,
"usd": 0.2112,
"wall_s": 63.9,
"api_calls": 6
},
"writing-plans [subagent]": {
"turns": [
6
],
"in": 0,
"out": 0,
"cr": 0,
"cw": 0,
"total": 0,
"usd": 0.0,
"wall_s": 0.0,
"api_calls": 0
},
"executing-plans": {
"turns": [
7
],
"in": 4,
"out": 385,
"cr": 53726,
"cw": 7377,
"total": 61492,
"usd": 0.0441,
"wall_s": 3.7,
"api_calls": 1
},
"test-driven-development": {
"turns": [
7
],
"in": 68,
"out": 21023,
"cr": 2791978,
"cw": 41898,
"total": 2854967,
"usd": 0.9364,
"wall_s": 415.1,
"api_calls": 34
},
"test-driven-development [subagent]": {
"turns": [
7
],
"in": 258,
"out": 19592,
"cr": 414677,
"cw": 76928,
"total": 511455,
"usd": 2.0474,
"wall_s": 0.0,
"api_calls": 0
}
}
},
{
"run": "spd-r3",
"task": "F1",
"cfg": "spd",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 434,
"code_diff_removed": 20,
"diff_added": 646,
"diff_removed": 20,
"tests_added_suite": 26,
"tokens": {
"in": 504,
"out": 72541,
"cr": 5615970,
"cw": 196313,
"total": 5885328
},
"usd": 4.4087,
"usd_po": 0.1286,
"wall_s": 701.3,
"driver_turns": 8,
"questions_routed_to_po": 5,
"end_reason": "done",
"caps": [],
"skills_invoked": [
"test-driven-development",
"finishing-a-development-branch"
],
"superpowers_injected": true,
"commits": 6,
"models": [
"claude-fable-5-1",
"claude-sonnet-5-5"
],
"stages": {
"brainstorming": {
"turns": [
1,
2,
3
],
"in": 10,
"out": 3567,
"cr": 130512,
"cw": 21715,
"total": 155804,
"usd": 0.1487,
"wall_s": 33.0,
"api_calls": 5
},
"writing-plans": {
"turns": [
4,
5,
6
],
"in": 16,
"out": 13925,
"cr": 345667,
"cw": 25975,
"total": 385583,
"usd": 0.3123,
"wall_s": 101.2,
"api_calls": 7
},
"writing-plans [subagent]": {
"turns": [
5,
6
],
"in": 0,
"out": 0,
"cr": 0,
"cw": 0,
"total": 0,
"usd": 0.0,
"wall_s": 0.0,
"api_calls": 0
},
"executing-plans": {
"turns": [
7
],
"in": 4,
"out": 516,
"cr": 60701,
"cw": 7518,
"total": 68739,
"usd": 0.0474,
"wall_s": 5.5,
"api_calls": 1
},
"test-driven-development": {
"turns": [
7
],
"in": 72,
"out": 24026,
"cr": 3334129,
"cw": 46850,
"total": 3405077,
"usd": 1.0946,
"wall_s": 507.1,
"api_calls": 36
},
"finishing-a-development-branch": {
"turns": [
7,
8
],
"in": 16,
"out": 5297,
"cr": 962018,
"cw": 9174,
"total": 976505,
"usd": 0.2821,
"wall_s": 50.7,
"api_calls": 8
},
"test-driven-development [subagent]": {
"turns": [
7
],
"in": 386,
"out": 25210,
"cr": 782943,
"cw": 85081,
"total": 893620,
"usd": 2.5236,
"wall_s": 0.0,
"api_calls": 0
},
"finishing-a-development-branch [subagent]": {
"turns": [
8
],
"in": 0,
"out": 0,
"cr": 0,
"cw": 0,
"total": 0,
"usd": 0.0,
"wall_s": 0.0,
"api_calls": 0
}
}
}
]
}
results.md
# flow-bench results
Untagged runs only (smoke/validation excluded). Tokens = input + output + cache read + cache write, all models, subagents included. USD = list-price equivalent from `total_cost_usd` deltas (plan quota, not a bill). Wall = sum of worker turn walls (PO time excluded). Diff = lines added+removed vs the base commit, src/ + test/.
## Per config (mean / median)
| task / config | n | hidden /5 (per run) | suite green | tsc green | code diff +/- | all files diff | tests added | tokens | USD | wall s | driver turns | questions | caps hit |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| F1 / bare | 3 | 5 / 5 (5,5,5) | 3/3 | 3/3 | 141.667 / 137 | 142.333 | 4.667 | 519,114 / 361,842 | 0.343 / 0.286 | 84.5 / 71.7 | 1.333 | 0.333 | none |
| F1 / sp (hook-routed) | 3 | 5 / 5 (5,5,5) | 3/3 | 3/3 | 235 / 227 | 237 | 12.333 | 724,601 / 765,108 | 0.502 / 0.525 | 118.1 / 119.6 | 2.667 | 1.667 | none |
| F1 / spd (Superpowers driven) | 3 | 5 / 5 (5,5,5) | 3/3 | 3/3 | 386 / 371 | 582 | 27.667 | 4,479,876 / 3,887,015 | 3.738 / 3.412 | 607.367 / 600.8 | 7 | 4 | none |
| F1 / mp (Main Flow) | 3 | 5 / 5 (5,5,5) | 3/3 | 3/3 | 372 / 369 | 578 | 27.667 | 4,476,104 / 4,286,993 | 2.043 / 2.102 | 439.033 / 472.4 | 12 | 7 | mp-r1: driver_turns_12; mp-r3: driver_turns_12 |
| F1 / both (sp+mp, mp flow) | 1 | 5 / 5 (5) | 1/1 | 1/1 | 374 / 374 | 561 | 22 | 5,754,354 / 5,754,354 | 2.326 / 2.326 | 448.2 / 448.2 | 12 | 6 | none |
| F2 / bare | 2 | 4.5 / 4.5 (4,5) | 2/2 | 2/2 | 206.5 / 206.5 | 209.5 | 5 | 651,322 / 651,322 | 0.413 / 0.413 | 103.15 / 103.15 | 2 | 0.5 | none |
| F2 / spd (Superpowers driven) | 2 | 5 / 5.0 (5,5) | 2/2 | 2/2 | 488.5 / 488.5 | 750.5 | 33 | 4,490,365 / 4,490,365 | 3.213 / 3.213 | 511.7 / 511.7 | 6.5 | 3.5 | none |
| F2 / mp (Main Flow) | 2 | 5 / 5.0 (5,5) | 2/2 | 2/2 | 469 / 469.0 | 658 | 31 | 4,520,638 / 4,520,638 | 2.067 / 2.067 | 421.2 / 421.2 | 12 | 6.5 | f2-mp-r1: driver_turns_12 |
Requirement pass counts (runs passing each hidden req):
- **F1 bare**: req1 3/3, req2 3/3, req3 3/3, req4 3/3, req5 3/3
- **F1 sp (hook-routed)**: req1 3/3, req2 3/3, req3 3/3, req4 3/3, req5 3/3
- **F1 spd (Superpowers driven)**: req1 3/3, req2 3/3, req3 3/3, req4 3/3, req5 3/3
- **F1 mp (Main Flow)**: req1 3/3, req2 3/3, req3 3/3, req4 3/3, req5 3/3
- **F1 both (sp+mp, mp flow)**: req1 1/1, req2 1/1, req3 1/1, req4 1/1, req5 1/1
- **F2 bare**: req1 2/2, req2 2/2, req3 1/2, req4 2/2, req5 2/2
- **F2 spd (Superpowers driven)**: req1 2/2, req2 2/2, req3 2/2, req4 2/2, req5 2/2
- **F2 mp (Main Flow)**: req1 2/2, req2 2/2, req3 2/2, req4 2/2, req5 2/2
## F1 spd (Superpowers driven) per slash-command stage (turn level, exact) (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| /brainstorming | 3 | 182,022 | 160,834 | - | 0.166 | 0.156 | 40.283 | 35.44 |
| /writing-plans | 3 | 321,355 | 298,267 | - | 0.266 | 0.274 | 88.133 | 95.82 |
| /executing-plans | 3 | 3,976,500 | 3,427,914 | - | 3.307 | 3.028 | 478.95 | 453.89 |
## F1 spd per skill in force (within-turn split) (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| brainstorming | 3 | 182,022 | 160,834 | 4,161 | 0.166 | 0.156 | 38.8 | 33.6 |
| writing-plans | 3 | 321,355 | 298,267 | 11,827 | 0.266 | 0.274 | 86.733 | 95.1 |
| executing-plans | 3 | 66,938 | 68,739 | 505 | 0.047 | 0.047 | 4.6 | 4.6 |
| test-driven-development | 3 | 2,927,098 | 2,854,967 | 22,508 | 0.974 | 0.936 | 457.033 | 448.9 |
| finishing-a-development-branch | 1 | 976,505 | 976,505 | 5,297 | 0.282 | 0.282 | 50.7 | 50.7 |
| test-driven-development [subagent] | 3 | 656,962 | 565,811 | 20,918 | 2.192 | 2.047 | 0.0 | 0.0 |
## F1 sp (hook-routed) per phase (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| before any skill | 3 | 19,549 | 19,538 | 238 | 0.03 | 0.03 | 5.333 | 5.4 |
| brainstorming | 3 | 249,260 | 247,900 | 8,037 | 0.219 | 0.244 | 40.533 | 40.1 |
| implementation (no skill) | 2 | 386,028 | 386,028 | 5,080 | 0.176 | 0.176 | 60.3 | 60.3 |
| execution | 1 | 595,318 | 595,318 | 15,622 | 0.405 | 0.405 | 93.0 | 93.0 |
## F1 sp (hook-routed) per skill in force (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| (none) | 3 | 19,549 | 19,538 | 238 | 0.03 | 0.03 | 5.333 | 5.4 |
| brainstorming | 3 | 249,260 | 247,900 | 8,037 | 0.219 | 0.244 | 40.533 | 40.1 |
| test-driven-development | 1 | 595,318 | 595,318 | 15,622 | 0.405 | 0.405 | 93.0 | 93.0 |
| implementation (no skill) | 2 | 386,028 | 386,028 | 5,080 | 0.176 | 0.176 | 60.3 | 60.3 |
## F1 mp per slash-command stage (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| /grill-with-docs | 3 | 316,507 | 348,862 | 5,999 | 0.204 | 0.215 | 59.933 | 59.9 |
| /to-spec | 3 | 163,344 | 167,578 | 5,135 | 0.118 | 0.113 | 42.1 | 40.8 |
| /to-tickets | 3 | 257,955 | 268,062 | 5,867 | 0.143 | 0.151 | 44.4 | 47.8 |
| /implement | 3 | 2,655,565 | 2,613,062 | 30,215 | 1.125 | 1.195 | 224.567 | 245.4 |
| /code-review | 3 | 1,047,586 | 1,107,555 | 9,294 | 0.439 | 0.445 | 64.2 | 65.8 |
| /pr | 1 | 105,440 | 105,440 | 1,349 | 0.041 | 0.041 | 11.6 | 11.6 |
## F1 both per slash-command stage (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| /grill-with-docs | 1 | 344,391 | 344,391 | 6,193 | 0.225 | 0.225 | 61.8 | 61.8 |
| /to-spec | 1 | 217,572 | 217,572 | 4,835 | 0.123 | 0.123 | 42.9 | 42.9 |
| /to-tickets | 1 | 382,953 | 382,953 | 7,115 | 0.196 | 0.196 | 49.4 | 49.4 |
| /implement | 1 | 3,070,579 | 3,070,579 | 24,618 | 1.112 | 1.112 | 194.2 | 194.2 |
| /code-review | 1 | 1,615,265 | 1,615,265 | 13,410 | 0.625 | 0.625 | 86.8 | 86.8 |
| /pr | 1 | 123,594 | 123,594 | 1,461 | 0.045 | 0.045 | 13.1 | 13.1 |
## F2 spd (Superpowers driven) per slash-command stage (turn level, exact) (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| /brainstorming | 2 | 485,364 | 485,364 | - | 0.354 | 0.354 | 96.815 | 96.815 |
| /writing-plans | 2 | 166,012 | 166,012 | - | 0.159 | 0.159 | 49.755 | 49.755 |
| /executing-plans | 2 | 3,838,988 | 3,838,988 | - | 2.7 | 2.7 | 365.1 | 365.1 |
## F2 spd per skill in force (within-turn split) (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| brainstorming | 2 | 253,890 | 253,890 | 6,038 | 0.207 | 0.207 | 53.05 | 53.05 |
| writing-plans | 2 | 397,486 | 397,486 | 14,060 | 0.307 | 0.307 | 90.75 | 90.75 |
| executing-plans | 2 | 69,973 | 69,973 | 286 | 0.046 | 0.046 | 3.5 | 3.5 |
| test-driven-development | 2 | 3,304,676 | 3,304,676 | 23,244 | 1.052 | 1.052 | 361.25 | 361.25 |
| test-driven-development [subagent] | 2 | 464,339 | 464,339 | 13,017 | 1.602 | 1.602 | 0.0 | 0.0 |
## F2 mp per slash-command stage (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| /grill-with-docs | 2 | 232,965 | 232,965 | 5,946 | 0.182 | 0.182 | 56.7 | 56.7 |
| /to-spec | 2 | 180,084 | 180,084 | 4,452 | 0.115 | 0.115 | 37.6 | 37.6 |
| /to-tickets | 2 | 253,688 | 253,688 | 5,152 | 0.131 | 0.131 | 38.5 | 38.5 |
| /implement | 2 | 2,689,071 | 2,689,071 | 31,452 | 1.159 | 1.159 | 218.35 | 218.35 |
| /code-review | 2 | 1,107,122 | 1,107,122 | 9,454 | 0.458 | 0.458 | 63.85 | 63.85 |
| /pr | 1 | 115,414 | 115,414 | 1,372 | 0.043 | 0.043 | 12.4 | 12.4 |
## Per run
| run | task | hidden | reqs failed | suite | tsc | code +/- | tests added | tokens | USD | wall s | turns | end | skills (first use order) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bare-r1 | F1 | 5/5 | - | green | ok | +149/-11 | 6 | 840,751 | 0.47 | 119 | 2 | done | - |
| bare-r2 | F1 | 5/5 | - | green | ok | +118/-10 | 4 | 354,748 | 0.27 | 63 | 1 | done | - |
| bare-r3 | F1 | 5/5 | - | green | ok | +125/-12 | 4 | 361,842 | 0.29 | 72 | 1 | done | - |
| sp-r1 | F1 | 5/5 | - | green | ok | +182/-15 | 10 | 613,434 | 0.44 | 99 | 3 | done | brainstorming |
| sp-r2 | F1 | 5/5 | - | green | ok | +212/-15 | 13 | 765,108 | 0.54 | 120 | 2 | done | brainstorming > test-driven-development |
| sp-r3 | F1 | 5/5 | - | green | ok | +267/-14 | 14 | 795,260 | 0.52 | 135 | 3 | done | brainstorming |
| spd-r1 | F1 | 5/5 | - | green | ok | +350/-21 | 21 | 3,667,286 | 3.41 | 601 | 6 | done | test-driven-development |
| spd-r2 | F1 | 5/5 | - | green | ok | +317/-16 | 36 | 3,887,015 | 3.39 | 520 | 7 | done | test-driven-development |
| spd-r3 | F1 | 5/5 | - | green | ok | +434/-20 | 26 | 5,885,328 | 4.41 | 701 | 8 | done | test-driven-development > finishing-a-development-branch |
| mp-r1 | F1 | 5/5 | - | green | ok | +349/-20 | 29 | 5,098,503 | 2.22 | 472 | 12 | cap_driver_turns (driver_turns_12) | grilling > domain-modeling > tdd > code-review |
| mp-r2 | F1 | 5/5 | - | green | ok | +314/-15 | 25 | 4,042,815 | 1.80 | 365 | 12 | done | grilling > domain-modeling > tdd > code-review |
| mp-r3 | F1 | 5/5 | - | green | ok | +395/-23 | 29 | 4,286,993 | 2.10 | 480 | 12 | cap_driver_turns (driver_turns_12) | grilling > domain-modeling > tdd > code-review |
| both-r1 | F1 | 5/5 | - | green | ok | +356/-18 | 22 | 5,754,354 | 2.33 | 448 | 12 | done | grilling > domain-modeling > tdd > code-review |
| f2-bare-r1 | F2 | 4/5 | req3 | green | ok | +204/-10 | 5 | 494,231 | 0.38 | 92 | 1 | done | - |
| f2-bare-r2 | F2 | 5/5 | - | green | ok | +189/-10 | 5 | 808,413 | 0.45 | 115 | 3 | done | - |
| f2-spd-r1 | F2 | 5/5 | - | green | ok | +546/-14 | 46 | 5,310,616 | 3.42 | 506 | 6 | done | writing-plans > test-driven-development |
| f2-spd-r2 | F2 | 5/5 | - | green | ok | +401/-16 | 20 | 3,670,114 | 3.00 | 517 | 7 | done | test-driven-development |
| f2-mp-r1 | F2 | 5/5 | - | green | ok | +460/-23 | 31 | 4,823,516 | 2.18 | 443 | 12 | cap_driver_turns (driver_turns_12) | grilling > domain-modeling > tdd > code-review |
| f2-mp-r2 | F2 | 5/5 | - | green | ok | +440/-15 | 31 | 4,217,761 | 1.95 | 399 | 12 | done | grilling > domain-modeling > tdd > code-review |
matrix.sh
#!/usr/bin/env bash
# Run matrix.txt ("<cfg> <rep>" lines), MATRIX_JOBS (default 2) at a time, each run in its own work copy.
# Appends to matrix.log. Resumable: SKIP_DONE=1. Usage: nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 &
set -uo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"; cd "$HERE"
LIST="${1:-matrix.txt}"; JOBS="${MATRIX_JOBS:-2}"
mkdir "$HERE/.matrix.lock" 2>/dev/null || { echo "another matrix.sh is running (remove .matrix.lock if stale)" >&2; exit 1; }
trap 'rmdir "$HERE/.matrix.lock"' EXIT
echo "=== $(date '+%F %T') start $LIST jobs=$JOBS pid=$$" >> matrix.log
grep -vE '^[[:space:]]*(#|$)' "$LIST" | xargs -P "$JOBS" -L 1 ./matrix-one.sh
echo "=== $(date '+%F %T') done $LIST" >> matrix.log
matrix-one.sh
#!/usr/bin/env bash
# One matrix cell (called by matrix.sh through xargs). Args: <cfg> <rep>
HERE="$(cd "$(dirname "$0")" && pwd)"; cd "$HERE"
CFG="$1"; REP="$2"; export TASK="${3:-F1}"
PFX=""; [ "$TASK" != "F1" ] && PFX="$(echo "$TASK" | tr "A-Z" "a-z")-"
if [ "${SKIP_DONE:-0}" = 1 ] && [ -f "runs/${PFX}${CFG}-r${REP}/score.json" ]; then
echo "$(date '+%F %T') skip ${PFX}${CFG}-r${REP}" >> matrix.log; exit 0
fi
echo "$(date '+%F %T') start ${PFX}${CFG}-r${REP}" >> matrix.log
line=$(./run.sh "$CFG" "$REP" < /dev/null 2>&1 | tail -1)
echo "$(date '+%F %T') $line" >> matrix.log
echo "$line"
matrix.txt
# <cfg> <rep>, run 2 at a time in this order (rep 1 of every config first)
bare 1
sp 1
mp 1
both 1
sp 2
mp 2
bare 2
mp 3
sp 3
bare 3
matrix2.txt
# <cfg> <rep> <task>; 2 lanes; long runs interleaved with short ones
spd 1 F1
mp 1 F1
bare 1 F1
sp 1 F1
spd 1 F2
mp 1 F2
bare 1 F2
spd 2 F1
mp 2 F1
bare 2 F1
sp 2 F1
spd 2 F2
mp 2 F2
bare 2 F2
both 1 F1
spd 3 F1
mp 3 F1
bare 3 F1
sp 3 F1
matrix-mp2.txt
mp 1
both 1
mp 2
mp 3
validate.sh
#!/usr/bin/env bash
# The 5 validation steps of SPEC.md, appended to validation.log.
# Usage: ./validate.sh [steps] default "1 2 3 4 5"; steps 3-5 (smoke runs, RUN_TAG=smoke) run in parallel.
set -uo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"; cd "$HERE"
. "$HERE/env.sh"
STEPS="${*:-1 2 3 4 5}"
LOG="$HERE/validation.log"
log(){ echo "$(date '+%F %T') $*" | tee -a "$LOG"; }
static_run(){ # <name> <patch or ""> [task] : prep a bare copy, optionally apply a patch, score it with finalize.py
local name="$1" patch="$2" task="${3:-F1}" out="$HERE/runs/$1" w="$HERE/work/$1"
rm -rf "$out"; mkdir -p "$out"
local base; base=$("$HERE/prep.sh" bare "$w" | tail -1)
if [ -n "$patch" ]; then ( cd "$w" && git apply "$patch" ) || { log "FAIL $name: git apply"; return 1; }; fi
printf '{"task":"%s","cfg":"bare","rep":0,"tag":"validation","work":"%s","name":"%s","base":"%s"}\n' "$task" "$w" "$name" "$base" > "$out/meta.json"
: > "$out/turns.jsonl"; echo '{"end_reason":"static"}' > "$out/driver.json"
python3 "$HERE/lib/finalize.py" "$out"
}
check(){ # <run dir> <python expr over s (score.json)> <label>
python3 - "$1" "$2" "$3" <<'PY'
import json, sys
s = json.load(open(sys.argv[1] + "/score.json"))
ok = eval(sys.argv[2], {"s": s})
print(("PASS " if ok else "FAIL ") + sys.argv[3])
sys.exit(0 if ok else 1)
PY
}
log "=== validate.sh steps: $STEPS (claude $(claude --version | head -1))"
for st in $STEPS; do case $st in
1) log "step 1: reference/F1.patch -> hidden 5/5, suite green, typecheck green"
static_run val1-reference "$HERE/reference/F1.patch" | tee -a "$LOG"
check runs/val1-reference 's["hidden_pass"]==5 and s["suite_green"] and s["typecheck_green"]' "step 1" | tee -a "$LOG" ;;
2) log "step 2: upstream unchanged -> hidden 0/5 (honest number recorded), suite green"
static_run val2-upstream "" | tee -a "$LOG"
check runs/val2-upstream 's["hidden_pass"]==0 and s["suite_green"] and s["suite_total"]==34' "step 2" | tee -a "$LOG" ;;
f2-1) log "F2 step 1: reference/F2.patch -> hidden 5/5, suite green, typecheck green"
static_run val1-f2-reference "$HERE/reference/F2.patch" F2 | tee -a "$LOG"
check runs/val1-f2-reference 's["hidden_pass"]==5 and s["suite_green"] and s["typecheck_green"]' "F2 step 1" | tee -a "$LOG" ;;
f2-2) log "F2 step 2: upstream unchanged -> hidden 0/5, suite green"
static_run val2-f2-upstream "" F2 | tee -a "$LOG"
check runs/val2-f2-upstream 's["hidden_pass"]==0 and s["suite_green"]' "F2 step 2" | tee -a "$LOG" ;;
esac; done
pids=""
for st in $STEPS; do case $st in
3) log "step 3: bare smoke (RUN_TAG=smoke) started"; ( RUN_TAG=smoke ./run.sh bare 1 > runs/.smoke-bare.out 2>&1 ) & pids="$pids $!" ;;
4) log "step 4: sp smoke (RUN_TAG=smoke) started"; ( RUN_TAG=smoke ./run.sh sp 1 > runs/.smoke-sp.out 2>&1 ) & pids="$pids $!" ;;
5) log "step 5: mp smoke (RUN_TAG=smoke) started"; ( RUN_TAG=smoke ./run.sh mp 1 > runs/.smoke-mp.out 2>&1 ) & pids="$pids $!" ;;
f2-3) log "F2 step 3: bare smoke on F2 (RUN_TAG=smoke): bare must score < 5"; ( TASK=F2 RUN_TAG=smoke ./run.sh bare 1 > runs/.smoke-f2-bare.out 2>&1 ) & pids="$pids $!" ;;
spd) log "spd smoke on F1 (RUN_TAG=smoke)"; ( RUN_TAG=smoke ./run.sh spd 1 > runs/.smoke-spd.out 2>&1 ) & pids="$pids $!" ;;
esac; done
[ -n "$pids" ] && wait $pids
for st in $STEPS; do case $st in
3) tail -1 runs/.smoke-bare.out | tee -a "$LOG"
check runs/bare-r1-smoke 's["end_reason"] in ("done","cap_driver_turns") and len(s["per_turn"])>=1 and all(t["po_kind"] for t in s["per_turn"])' "step 3a: bare ran end to end, the PO classified every turn" | tee -a "$LOG"
check runs/bare-r1-smoke 'not s["superpowers_injected"] and not s["skill_events"] and "superpowers" not in (s["init_plugins"] or [])' "step 3b: isolation: no Superpowers hook output, no Skill events, user plugin not loaded" | tee -a "$LOG" ;;
4) tail -1 runs/.smoke-sp.out | tee -a "$LOG"
check runs/sp-r1-smoke 's["superpowers_injected"] and any(h["event"]=="SessionStart" for h in s["hook_events"])' "step 4a: SessionStart hook fired and injected using-superpowers" | tee -a "$LOG"
check runs/sp-r1-smoke '"brainstorming" in [x.split(":")[-1] for x in s["skills_invoked"]]' "step 4b: brainstorming invoked (Skill event)" | tee -a "$LOG" ;;
5) tail -1 runs/.smoke-mp.out | tee -a "$LOG"
check runs/mp-r1-smoke 's["per_turn"][0]["stage"]=="/grill-with-docs" and {"grilling","domain-modeling"} <= set(s["per_turn"][0]["skills"])' "step 5: /grill-with-docs resolved (turn 1 loads grilling + domain-modeling, as its SKILL.md instructs)" | tee -a "$LOG" ;;
f2-3) tail -1 runs/.smoke-f2-bare.out | tee -a "$LOG"
check runs/f2-bare-r1-smoke 's["hidden_pass"] < 5' "F2 step 3: bare scores < 5 on F2" | tee -a "$LOG" ;;
spd) tail -1 runs/.smoke-spd.out | tee -a "$LOG"
check runs/spd-r1-smoke '{"/brainstorming","/writing-plans","/executing-plans"} <= set(s["spd_slash_stage_per_turn"] or [])' "spd: the three slash stages were all reached" | tee -a "$LOG" ;;
esac; done
log "=== validate.sh done"
Aby uruchomić obie paczki na własnej funkcji, idź za README:
- Wstaw swoje repo w miejsce
upstream/, napisz zlecenie wtasks/, a brief product ownera wpo/prostymi słowami, nigdy jako testy. - Napisz plik ukrytych testów z jednym testem na każde wymaganie, którego oczekiwałby senior, plus patch referencyjny, i sprawdź wyrocznię przez
./validate.sh 1 2. - Zainstaluj paczki tak, jak pokazuje
prep.sh: Superpowers do.claude/skillsz jego hookiem SessionStart skopiowanym do.claude/settings.json, paczkę Pococka przeznpx skills add, a potem jednorazowo zmaterializuj/setup-matt-pocock-skills. - Uruchom jedną komórkę przez
RUN_TAG=x ./run.sh <cfg> <rep>. Przebiegi z tagiem są ignorowane przez analizę, więc używaj tagu do smoke runów. - Uruchom macierz przez
nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 &, wznów zSKIP_DONE=1, a potem uruchompython3 analyze.py, by dostać tabele na konfigurację i na etap.
Trzy pułapki z README warto znać przed startem. Uruchamiaj kopię roboczą poza każdym drzewem katalogów, w którym jest CLAUDE.md, bo Claude Code ładuje CLAUDE.md każdego katalogu nadrzędnego, a nasza pierwsza runda poszła do kosza właśnie z tego powodu. Nigdy nie używaj --bare przy logowaniu subskrypcyjnym. A hook Superpowers odpala się tylko w pierwszej turze, więc przebieg headless routuje raz; sam routing różnił się między przebiegami, więc policz, który skill wybrał.
Zrób to w poniedziałek
- Sprawdź, którą wersję Pococka masz. Jeśli zainstalowałeś plugin Claude, jest przypięty na 1.2.3 i nie ma implement-spec, pr ani retro; zainstaluj ponownie installerem, jeśli chcesz v1.3.1.
- Jeśli używasz Superpowers, otwórz instrukcję recenzenta w executing-plans i sprawdź, o jaki model prosi. Przypnij tańszy, potem zmierz.
- Wpisz /brainstorming sam, gdy zlecenie jest mgliste. Zostawiony swojemu hookowi, brainstorming ruszył w 3 z 3 przebiegów kierowanych hookiem, ale routing nie jest gwarantowany.
- Wypróbuj grill-with-docs, to-spec i to-tickets na następnym mglistym zleceniu i przeczytaj, ile kosztują na twoim repo.
- Nie łącz obu paczek jako routerów bez testu. Na naszym jednym przebiegu z obiema nic się nie zepsuło, ale to jeden przebieg.
- Skopiuj bench na funkcję, którą wypuściłeś w zeszłym miesiącu, z własnymi ukrytymi testami, i puść bare, Superpowers i Main Flow po razie.
- Czytaj tabelę na etap, nie sumę. Znajdź jeden etap, który jest większością twojego rachunku, zanim zmienisz cokolwiek innego.
- Zmień CONTEXT.md na GLOSSARY.md, jeśli przyjmujesz v1.3, bo skille szukają tylko nowych nazw.
Pomiary
Źródło:
results.md
# flow-bench results
Untagged runs only (smoke/validation excluded). Tokens = input + output + cache read + cache write, all models, subagents included. USD = list-price equivalent from `total_cost_usd` deltas (plan quota, not a bill). Wall = sum of worker turn walls (PO time excluded). Diff = lines added+removed vs the base commit, src/ + test/.
## Per config (mean / median)
| task / config | n | hidden /5 (per run) | suite green | tsc green | code diff +/- | all files diff | tests added | tokens | USD | wall s | driver turns | questions | caps hit |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| F1 / bare | 3 | 5 / 5 (5,5,5) | 3/3 | 3/3 | 141.667 / 137 | 142.333 | 4.667 | 519,114 / 361,842 | 0.343 / 0.286 | 84.5 / 71.7 | 1.333 | 0.333 | none |
| F1 / sp (hook-routed) | 3 | 5 / 5 (5,5,5) | 3/3 | 3/3 | 235 / 227 | 237 | 12.333 | 724,601 / 765,108 | 0.502 / 0.525 | 118.1 / 119.6 | 2.667 | 1.667 | none |
| F1 / spd (Superpowers driven) | 3 | 5 / 5 (5,5,5) | 3/3 | 3/3 | 386 / 371 | 582 | 27.667 | 4,479,876 / 3,887,015 | 3.738 / 3.412 | 607.367 / 600.8 | 7 | 4 | none |
| F1 / mp (Main Flow) | 3 | 5 / 5 (5,5,5) | 3/3 | 3/3 | 372 / 369 | 578 | 27.667 | 4,476,104 / 4,286,993 | 2.043 / 2.102 | 439.033 / 472.4 | 12 | 7 | mp-r1: driver_turns_12; mp-r3: driver_turns_12 |
| F1 / both (sp+mp, mp flow) | 1 | 5 / 5 (5) | 1/1 | 1/1 | 374 / 374 | 561 | 22 | 5,754,354 / 5,754,354 | 2.326 / 2.326 | 448.2 / 448.2 | 12 | 6 | none |
| F2 / bare | 2 | 4.5 / 4.5 (4,5) | 2/2 | 2/2 | 206.5 / 206.5 | 209.5 | 5 | 651,322 / 651,322 | 0.413 / 0.413 | 103.15 / 103.15 | 2 | 0.5 | none |
| F2 / spd (Superpowers driven) | 2 | 5 / 5.0 (5,5) | 2/2 | 2/2 | 488.5 / 488.5 | 750.5 | 33 | 4,490,365 / 4,490,365 | 3.213 / 3.213 | 511.7 / 511.7 | 6.5 | 3.5 | none |
| F2 / mp (Main Flow) | 2 | 5 / 5.0 (5,5) | 2/2 | 2/2 | 469 / 469.0 | 658 | 31 | 4,520,638 / 4,520,638 | 2.067 / 2.067 | 421.2 / 421.2 | 12 | 6.5 | f2-mp-r1: driver_turns_12 |
Requirement pass counts (runs passing each hidden req):
- **F1 bare**: req1 3/3, req2 3/3, req3 3/3, req4 3/3, req5 3/3
- **F1 sp (hook-routed)**: req1 3/3, req2 3/3, req3 3/3, req4 3/3, req5 3/3
- **F1 spd (Superpowers driven)**: req1 3/3, req2 3/3, req3 3/3, req4 3/3, req5 3/3
- **F1 mp (Main Flow)**: req1 3/3, req2 3/3, req3 3/3, req4 3/3, req5 3/3
- **F1 both (sp+mp, mp flow)**: req1 1/1, req2 1/1, req3 1/1, req4 1/1, req5 1/1
- **F2 bare**: req1 2/2, req2 2/2, req3 1/2, req4 2/2, req5 2/2
- **F2 spd (Superpowers driven)**: req1 2/2, req2 2/2, req3 2/2, req4 2/2, req5 2/2
- **F2 mp (Main Flow)**: req1 2/2, req2 2/2, req3 2/2, req4 2/2, req5 2/2
## F1 spd (Superpowers driven) per slash-command stage (turn level, exact) (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| /brainstorming | 3 | 182,022 | 160,834 | - | 0.166 | 0.156 | 40.283 | 35.44 |
| /writing-plans | 3 | 321,355 | 298,267 | - | 0.266 | 0.274 | 88.133 | 95.82 |
| /executing-plans | 3 | 3,976,500 | 3,427,914 | - | 3.307 | 3.028 | 478.95 | 453.89 |
## F1 spd per skill in force (within-turn split) (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| brainstorming | 3 | 182,022 | 160,834 | 4,161 | 0.166 | 0.156 | 38.8 | 33.6 |
| writing-plans | 3 | 321,355 | 298,267 | 11,827 | 0.266 | 0.274 | 86.733 | 95.1 |
| executing-plans | 3 | 66,938 | 68,739 | 505 | 0.047 | 0.047 | 4.6 | 4.6 |
| test-driven-development | 3 | 2,927,098 | 2,854,967 | 22,508 | 0.974 | 0.936 | 457.033 | 448.9 |
| finishing-a-development-branch | 1 | 976,505 | 976,505 | 5,297 | 0.282 | 0.282 | 50.7 | 50.7 |
| test-driven-development [subagent] | 3 | 656,962 | 565,811 | 20,918 | 2.192 | 2.047 | 0.0 | 0.0 |
## F1 sp (hook-routed) per phase (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| before any skill | 3 | 19,549 | 19,538 | 238 | 0.03 | 0.03 | 5.333 | 5.4 |
| brainstorming | 3 | 249,260 | 247,900 | 8,037 | 0.219 | 0.244 | 40.533 | 40.1 |
| implementation (no skill) | 2 | 386,028 | 386,028 | 5,080 | 0.176 | 0.176 | 60.3 | 60.3 |
| execution | 1 | 595,318 | 595,318 | 15,622 | 0.405 | 0.405 | 93.0 | 93.0 |
## F1 sp (hook-routed) per skill in force (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| (none) | 3 | 19,549 | 19,538 | 238 | 0.03 | 0.03 | 5.333 | 5.4 |
| brainstorming | 3 | 249,260 | 247,900 | 8,037 | 0.219 | 0.244 | 40.533 | 40.1 |
| test-driven-development | 1 | 595,318 | 595,318 | 15,622 | 0.405 | 0.405 | 93.0 | 93.0 |
| implementation (no skill) | 2 | 386,028 | 386,028 | 5,080 | 0.176 | 0.176 | 60.3 | 60.3 |
## F1 mp per slash-command stage (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| /grill-with-docs | 3 | 316,507 | 348,862 | 5,999 | 0.204 | 0.215 | 59.933 | 59.9 |
| /to-spec | 3 | 163,344 | 167,578 | 5,135 | 0.118 | 0.113 | 42.1 | 40.8 |
| /to-tickets | 3 | 257,955 | 268,062 | 5,867 | 0.143 | 0.151 | 44.4 | 47.8 |
| /implement | 3 | 2,655,565 | 2,613,062 | 30,215 | 1.125 | 1.195 | 224.567 | 245.4 |
| /code-review | 3 | 1,047,586 | 1,107,555 | 9,294 | 0.439 | 0.445 | 64.2 | 65.8 |
| /pr | 1 | 105,440 | 105,440 | 1,349 | 0.041 | 0.041 | 11.6 | 11.6 |
## F1 both per slash-command stage (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| /grill-with-docs | 1 | 344,391 | 344,391 | 6,193 | 0.225 | 0.225 | 61.8 | 61.8 |
| /to-spec | 1 | 217,572 | 217,572 | 4,835 | 0.123 | 0.123 | 42.9 | 42.9 |
| /to-tickets | 1 | 382,953 | 382,953 | 7,115 | 0.196 | 0.196 | 49.4 | 49.4 |
| /implement | 1 | 3,070,579 | 3,070,579 | 24,618 | 1.112 | 1.112 | 194.2 | 194.2 |
| /code-review | 1 | 1,615,265 | 1,615,265 | 13,410 | 0.625 | 0.625 | 86.8 | 86.8 |
| /pr | 1 | 123,594 | 123,594 | 1,461 | 0.045 | 0.045 | 13.1 | 13.1 |
## F2 spd (Superpowers driven) per slash-command stage (turn level, exact) (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| /brainstorming | 2 | 485,364 | 485,364 | - | 0.354 | 0.354 | 96.815 | 96.815 |
| /writing-plans | 2 | 166,012 | 166,012 | - | 0.159 | 0.159 | 49.755 | 49.755 |
| /executing-plans | 2 | 3,838,988 | 3,838,988 | - | 2.7 | 2.7 | 365.1 | 365.1 |
## F2 spd per skill in force (within-turn split) (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| brainstorming | 2 | 253,890 | 253,890 | 6,038 | 0.207 | 0.207 | 53.05 | 53.05 |
| writing-plans | 2 | 397,486 | 397,486 | 14,060 | 0.307 | 0.307 | 90.75 | 90.75 |
| executing-plans | 2 | 69,973 | 69,973 | 286 | 0.046 | 0.046 | 3.5 | 3.5 |
| test-driven-development | 2 | 3,304,676 | 3,304,676 | 23,244 | 1.052 | 1.052 | 361.25 | 361.25 |
| test-driven-development [subagent] | 2 | 464,339 | 464,339 | 13,017 | 1.602 | 1.602 | 0.0 | 0.0 |
## F2 mp per slash-command stage (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| /grill-with-docs | 2 | 232,965 | 232,965 | 5,946 | 0.182 | 0.182 | 56.7 | 56.7 |
| /to-spec | 2 | 180,084 | 180,084 | 4,452 | 0.115 | 0.115 | 37.6 | 37.6 |
| /to-tickets | 2 | 253,688 | 253,688 | 5,152 | 0.131 | 0.131 | 38.5 | 38.5 |
| /implement | 2 | 2,689,071 | 2,689,071 | 31,452 | 1.159 | 1.159 | 218.35 | 218.35 |
| /code-review | 2 | 1,107,122 | 1,107,122 | 9,454 | 0.458 | 0.458 | 63.85 | 63.85 |
| /pr | 1 | 115,414 | 115,414 | 1,372 | 0.043 | 0.043 | 12.4 | 12.4 |
## Per run
| run | task | hidden | reqs failed | suite | tsc | code +/- | tests added | tokens | USD | wall s | turns | end | skills (first use order) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bare-r1 | F1 | 5/5 | - | green | ok | +149/-11 | 6 | 840,751 | 0.47 | 119 | 2 | done | - |
| bare-r2 | F1 | 5/5 | - | green | ok | +118/-10 | 4 | 354,748 | 0.27 | 63 | 1 | done | - |
| bare-r3 | F1 | 5/5 | - | green | ok | +125/-12 | 4 | 361,842 | 0.29 | 72 | 1 | done | - |
| sp-r1 | F1 | 5/5 | - | green | ok | +182/-15 | 10 | 613,434 | 0.44 | 99 | 3 | done | brainstorming |
| sp-r2 | F1 | 5/5 | - | green | ok | +212/-15 | 13 | 765,108 | 0.54 | 120 | 2 | done | brainstorming > test-driven-development |
| sp-r3 | F1 | 5/5 | - | green | ok | +267/-14 | 14 | 795,260 | 0.52 | 135 | 3 | done | brainstorming |
| spd-r1 | F1 | 5/5 | - | green | ok | +350/-21 | 21 | 3,667,286 | 3.41 | 601 | 6 | done | test-driven-development |
| spd-r2 | F1 | 5/5 | - | green | ok | +317/-16 | 36 | 3,887,015 | 3.39 | 520 | 7 | done | test-driven-development |
| spd-r3 | F1 | 5/5 | - | green | ok | +434/-20 | 26 | 5,885,328 | 4.41 | 701 | 8 | done | test-driven-development > finishing-a-development-branch |
| mp-r1 | F1 | 5/5 | - | green | ok | +349/-20 | 29 | 5,098,503 | 2.22 | 472 | 12 | cap_driver_turns (driver_turns_12) | grilling > domain-modeling > tdd > code-review |
| mp-r2 | F1 | 5/5 | - | green | ok | +314/-15 | 25 | 4,042,815 | 1.80 | 365 | 12 | done | grilling > domain-modeling > tdd > code-review |
| mp-r3 | F1 | 5/5 | - | green | ok | +395/-23 | 29 | 4,286,993 | 2.10 | 480 | 12 | cap_driver_turns (driver_turns_12) | grilling > domain-modeling > tdd > code-review |
| both-r1 | F1 | 5/5 | - | green | ok | +356/-18 | 22 | 5,754,354 | 2.33 | 448 | 12 | done | grilling > domain-modeling > tdd > code-review |
| f2-bare-r1 | F2 | 4/5 | req3 | green | ok | +204/-10 | 5 | 494,231 | 0.38 | 92 | 1 | done | - |
| f2-bare-r2 | F2 | 5/5 | - | green | ok | +189/-10 | 5 | 808,413 | 0.45 | 115 | 3 | done | - |
| f2-spd-r1 | F2 | 5/5 | - | green | ok | +546/-14 | 46 | 5,310,616 | 3.42 | 506 | 6 | done | writing-plans > test-driven-development |
| f2-spd-r2 | F2 | 5/5 | - | green | ok | +401/-16 | 20 | 3,670,114 | 3.00 | 517 | 7 | done | test-driven-development |
| f2-mp-r1 | F2 | 5/5 | - | green | ok | +460/-23 | 31 | 4,823,516 | 2.18 | 443 | 12 | cap_driver_turns (driver_turns_12) | grilling > domain-modeling > tdd > code-review |
| f2-mp-r2 | F2 | 5/5 | - | green | ok | +440/-15 | 31 | 4,217,761 | 1.95 | 399 | 12 | done | grilling > domain-modeling > tdd > code-review |
| Zadanie i konfiguracja | Przebiegi | Ukryte /5 (na przebieg) | Średnie USD | Średni wall s | Średnie tokeny | Średni diff kodu +/- |
|---|---|---|---|---|---|---|
| F1 bare | 3 | 5 (5,5,5) | 0.343 | 84.5 | 519,114 | 141.667 |
| F1 sp (kierowany hookiem) | 3 | 5 (5,5,5) | 0.502 | 118.1 | 724,601 | 235 |
| F1 spd (Superpowers prowadzony) | 3 | 5 (5,5,5) | 3.738 | 607.367 | 4,479,876 | 386 |
| F1 mp (Main Flow) | 3 | 5 (5,5,5) | 2.043 | 439.033 | 4,476,104 | 372 |
| F1 both (sp+mp, przepływ mp) | 1 | 5 (5) | 2.326 | 448.2 | 5,754,354 | 374 |
| F2 bare | 2 | 4.5 (4,5) | 0.413 | 103.15 | 651,322 | 206.5 |
| F2 spd (Superpowers prowadzony) | 2 | 5 (5,5) | 3.213 | 511.7 | 4,490,365 | 488.5 |
| F2 mp (Main Flow) | 2 | 5 (5,5) | 2.067 | 421.2 | 4,520,638 | 469 |
Superpowers prowadzony, F1, na etap (średnia z 3 przebiegów):
| Etap | Tokeny | USD | Wall s |
|---|---|---|---|
| brainstorming | 182,022 | 0.166 | 40.283 |
| writing-plans | 321,355 | 0.266 | 88.133 |
| executing-plans | 3,976,500 | 3.307 | 478.95 |
| z czego subagent-recenzent | 656,962 | 2.192 | n/a |
Wiersz recenzenta to reszta subagenta wewnątrz executing-plans, w pliku wyników opisana jako "test-driven-development [subagent]". Budowa w sesji wewnątrz executing-plans (test-driven-development) wyniosła średnio $0.974.
Main Flow, F1, na etap (średnia z przebiegów, w których był etap; /pr ruszył w 1 z 3):
| Etap | Tokeny | USD | Wall s |
|---|---|---|---|
| grill-with-docs | 316,507 | 0.204 | 59.933 |
| to-spec | 163,344 | 0.118 | 42.1 |
| to-tickets | 257,955 | 0.143 | 44.4 |
| implement | 2,655,565 | 1.125 | 224.567 |
| code-review | 1,047,586 | 0.439 | 64.2 |
| pr | 105,440 | 0.041 | 11.6 |
Superpowers kierowany hookiem, F1: przed jakimkolwiek skillem $0.03, brainstorming $0.219, implementacja bez skilla $0.176 (2 przebiegi), wykonanie z test-driven-development $0.405 (1 przebieg). Brainstorming ruszył we wszystkich 3 przebiegach, a writing-plans w żadnym. Prowadzona pętla kosztowała 7.4 razy więcej niż kierowana hookiem w średnich F1 ($3.738 wobec $0.502). Trzy przebiegi Main Flow (mp-r1, mp-r3 i f2-mp-r1) dobiły do limitu 12 tur drivera przed /pr i są tak zapisane.
Zastrzeżenia z
FINDINGS.md
# flow-bench findings (2026-10-09, round 2: isolated work copies, Sonnet 5.5 headless, 19 counted runs)
**F1 (partial payments): quality did not separate.** All 13 F1 runs hidden 5/5, suite and tsc green.
Cost per run: bare $0.27 (bare-r2) to $0.47 (bare-r1); sp, hook-routed, $0.44 (sp-r1) to $0.54 (sp-r2); mp $1.80
(mp-r2) to $2.22 (mp-r1); spd, Superpowers driven, $3.39 (spd-r2) to $4.41 (spd-r3).
Wall: mp 365 to 480 s (mp-r2, mp-r3), spd 520 to 701 s (spd-r2, spd-r3).
**Where the money goes.** mp-r2 (complete flow): grill $0.16, spec $0.11, tickets $0.09, implement $0.96, code-review
$0.45, pr $0.04 (F1 means: implement $1.13, code-review $0.44). spd: brainstorming $0.17, writing-plans $0.27,
executing-plans $3.31 (F1 means). Inside executing-plans the build itself (TDD in session) costs $0.97 on average; the
rest is the one whole-branch reviewer the skill dispatches "on the most capable model", which ran on claude-fable-5-1
in all five spd runs: $2.01 of spd-r1's $3.41, $2.52 of spd-r3's $4.41, $1.57 of f2-spd-r2's $3.00.
**Hook vs driven.** Left to its hook, Superpowers ran brainstorming in sp-r1, sp-r2 and sp-r3 and then coded; writing-
plans ran 0 times in 3. Driven by slash commands, the full loop ran in every spd run and cost 7.4 times more (F1 means: spd $3.74, sp $0.50). The cost is the final review, not the planning.
**F2 (credit notes) separated, barely.** f2-bare-r1 shipped one missing requirement, R3: credit-note numbers are not
gapless. It took the number before validating, like the existing `/issue` route, so rejected requests burned numbers
(CN-0004 where CN-0002 was due; the F2 bare smoke failed the same way). f2-bare-r2, f2-spd-r1, f2-spd-r2, f2-mp-r1 and
f2-mp-r2 scored 5/5. Costs: bare $0.38/$0.45, mp $2.18/$1.95, spd $3.42/$3.00 (r1/r2). Caveat: the first F2 oracle
also required credit notes on open invoices; both bare runs allowed paid invoices only, which the request's wording
permits, so that check was removed and the F2 runs rescored.
**Stacking broke nothing.** both-r1: hook injected, no Superpowers skill invoked, $2.33, inside the mp range.
**Caps.** mp-r1, mp-r3 and f2-mp-r1 hit the 12-driver-turn cap before /pr. spd never needed more than 8 turns.
**Caveats.** Round-1 runs read this repo's CLAUDE.md (parent-directory loading), so they were discarded and rerun outside the repo. A simulated PO answers each round in one message. The SessionStart hook fires on turn 1 only.
/implement covers every ticket in one turn. 5 to 11 compound shell lines per spd run were denied. PO cost is excluded.
Idź dalej
- Bench debugowania. Post na r/claudeskills przeprowadził 15 bugów z historii jednego repo, po trzy przebiegi. Wszystkie trzy warianty naprawiły 9 z 15, dodatkowe skille dodały około 3 minut na buga, a skill debugowania Pococka robił najmniejsze zmiany, mediana 32 linii wobec 41 i 48, bez psucia innego testu s8. Testuje tylko skille debugowania, nie pełne przepływy, a post nie mówi, które z 41 i 48 należy do którego wariantu. Powtórz go harnessem benchu na własnej historii bugów.
- Blogi o przesiadkach. Przeczytaj tezę, że orkiestracja stała się podatkiem s9 i historię migracji, która usunęła hooki Superpowers i zostawiła grill-with-docs s10, a potem porównaj ich twierdzenia z tabelą na etap powyżej.
- Inne paczki. Pstack i paczka Osmaniego to dwa kolejne zestawy skilli, których nie uruchamialiśmy. Ten sam harness przyjmie każdą paczkę instalowaną do
.claude/skills; dodaj konfigurację wconfigs.json
{
"bare": {
"superpowers": false,
"mattpocock": false,
"flow": "bare",
"desc": "no skills, no hooks (project settings carry only the permission deny-list)"
},
"sp": {
"superpowers": true,
"mattpocock": false,
"flow": "sp",
"desc": "HOOK-ROUTED: obra/superpowers v6.4.2: skills in .claude/skills, SessionStart hook in .claude/settings.json as hooks/hooks.json wires it; turn 1 = the bare task, the hook decides"
},
"mp": {
"superpowers": false,
"mattpocock": true,
"flow": "mp",
"desc": "mattpocock/skills v1.3.1 via npx skills add (27 promoted skills, copied), setup-matt-pocock-skills output committed (local tracker)"
},
"both": {
"superpowers": true,
"mattpocock": true,
"flow": "mp",
"desc": "sp + mp in one project; driven with the mp slash-command flow (the stacking question: does the Superpowers hook hijack it?)"
},
"spd": {
"superpowers": true,
"mattpocock": false,
"flow": "spd",
"desc": "Superpowers, DRIVEN: same project install as sp (hook included); the driver types /brainstorming <task> -> /writing-plans -> /executing-plans (Native/inline); the pack's own scripts allowed in Bash"
}
}
prep.sh
#!/usr/bin/env bash
# Fresh work copy of upstream (node_modules included) + the pack fixtures of <cfg>.
# Usage: prep.sh <cfg> <work dir> Prints the base commit (the oracle's BASE) on its last line.
set -euo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"
CFG="$1"; W="$2"
SP=$(python3 -c "import json;print(int(json.load(open('$HERE/configs.json'))['$CFG']['superpowers']))")
MP=$(python3 -c "import json;print(int(json.load(open('$HERE/configs.json'))['$CFG']['mattpocock']))")
rm -rf "$W"; mkdir -p "$(dirname "$W")"
cp -R "$HERE/upstream" "$W"
cd "$W"
# .claude/ never shows in git status or in any diff (hook config, installed skills).
echo ".claude/" >> .git/info/exclude
mkdir -p .claude/skills
if [ "$MP" = 1 ]; then
cp -R "$HERE/packs/mattpocock-skills/.claude/skills/." .claude/skills/
# One-time per-repo precondition of the pack (/setup-matt-pocock-skills), answered once with the defaults it
# recommends for a repo with no remote: local-markdown tracker, single-context domain docs, no triage labels
# (triage is installed, so Section B would ask; default labels kept). Committed so it never shows in the diff.
S="$HERE/packs/mattpocock-skills/.claude/skills/setup-matt-pocock-skills"
mkdir -p docs/agents
cp "$S/issue-tracker-local.md" docs/agents/issue-tracker.md
cp "$S/domain.md" docs/agents/domain.md
cp "$S/triage-labels.md" docs/agents/triage-labels.md
cat > CLAUDE.md <<'MD'
## Agent skills
### Issue tracker
Issues and specs live as local markdown files under `.scratch/`. See `docs/agents/issue-tracker.md`.
### Triage labels
Default labels (`needs-triage`, `needs-info`, `ready-for-agent`, `ready-for-human`, `wontfix`). See `docs/agents/triage-labels.md`.
### Domain docs
Single-context: one `GLOSSARY.md` + `docs/adr/` at the repo root. See `docs/agents/domain.md`.
MD
git add CLAUDE.md docs/agents
GIT_AUTHOR_DATE="2026-03-09T18:00:00" GIT_COMMITTER_DATE="2026-03-09T18:00:00" \
git -c user.name="Dana Whitfield" -c user.email="dana@ledgerly.dev" commit -q -m "chore: configure agent skills (local tracker)"
fi
if [ "$SP" = 1 ]; then
# The plugin root as Claude Code would hold it, so the hook script finds ../skills/using-superpowers.
mkdir -p .claude/plugins
cp -R "$HERE/packs/superpowers" .claude/plugins/superpowers
cp -R "$HERE/packs/superpowers/skills/." .claude/skills/
fi
python3 - "$HERE" "$SP" "$CFG" > .claude/settings.json <<'PY'
import json, sys
bench, sp, cfg = sys.argv[1], sys.argv[2] == "1", sys.argv[3]
# Oracle, reference, PO brief, other runs: denied by path (Read/Grep/Glob) and by command text (Bash).
deny = ["Read(/%s/hidden/**)" % bench, "Read(/%s/reference/**)" % bench, "Read(/%s/runs/**)" % bench,
"Read(/%s/po/**)" % bench, "Read(/%s/validation/**)" % bench,
"Bash(*hidden*)", "Bash(*reference/F*)", "Bash(*-brief.md*)", "Bash(*/runs/*)", "Bash(*../*)", "Bash(find *)"]
if cfg == "spd":
# executing-plans runs its sibling scripts as ../subagent-driven-development/scripts/...: a "../" deny would block the
# pack's own loop. The oracle stays protected by the Read denies and Bash(*hidden*).
deny.remove("Bash(*../*)")
settings = {"permissions": {"deny": deny}}
if sp:
# = packs/superpowers/hooks/hooks.json, with ${CLAUDE_PLUGIN_ROOT} provided the way the plugin loader provides it
# (the script also picks its JSON output shape from CLAUDE_PLUGIN_ROOT being set).
root = '$CLAUDE_PROJECT_DIR/.claude/plugins/superpowers'
hooks = json.load(open("%s/packs/superpowers/hooks/hooks.json" % bench))["hooks"]
for groups in hooks.values():
for g in groups:
for h in g["hooks"]:
h["command"] = 'CLAUDE_PLUGIN_ROOT="%s" %s' % (root, h["command"].replace("${CLAUDE_PLUGIN_ROOT}", root))
settings["hooks"] = hooks
print(json.dumps(settings, indent=2))
PY
[ "$(ls .claude/skills | wc -l)" -eq 0 ] && rmdir .claude/skills
test -z "$(git status --porcelain)" || { echo "work copy not clean"; git status --porcelain; exit 1; }
git rev-parse HEAD
FAQ
Czy któraś paczka dostarczyła lepszy kod?
Nie przy funkcji 1: wszystkie 13 przebiegów dało 5 z 5, zestaw testów i typecheck zielone. Przy funkcji 2 jeden przebieg bare nie spełnił jednego wymagania, ciągłego numeru noty kredytowej, bo wziął numer przed walidacją zlecenia, tak jak robi to istniejąca trasa wystawiania faktur. Obie pętle napisały test na numerację przed kodem. Drugi przebieg bare zrobił to dobrze.
Dlaczego pętla Superpowers jest tu o tyle droższa niż Main Flow?
Przez jeden etap. executing-plans kosztował średnio $3.307, a subagent-recenzent wewnątrz niego $2.192. Recenzent działał na claude-fable-5-1 we wszystkich pięciu prowadzonych przebiegach, a skill prosi o najmocniejszy model. Etapy planowania kosztowały $0.166 i $0.266. Nie powtarzaliśmy przebiegu z tańszym recenzentem, więc nie mamy zmierzonej liczby dla tej zmiany.
Czy instalacja obu paczek coś psuje?
Nie na naszym jednym przebiegu z obiema: hook został wstrzyknięty, żaden skill Superpowers nie został wywołany, komendy slash wygrały, a przebieg kosztował $2.326 przy 5 z 5. Skille Pococka wywoływane przez użytkownika mają flagę, która nie pozwala modelowi ładować ich samemu, a obie paczki nie mają wspólnej nazwy katalogu skilla. To jeden przebieg, więc przetestuj to u siebie.
Dlaczego po prostu nie użyć pluginu Claude dla skilli Pococka?
Bo marketplace przypina go na 1.2.3, podczas gdy upstream jest w v1.3.1 s32. Installer pobiera wydanie, o które prosisz.
Czy to uczciwy test jakości Main Flow?
To uczciwy test tego, ile kosztuje każda paczka i czy dostarcza pięć wymagań na małym repo. Artefakty Main Flow, spec i tickety, nie są tu oceniane, a większa funkcja albo zespół mogą cenić je wyżej.