AIDiveTL;DR ve karar
- Aynı özellik, uydurma tek bir repoda çıplak bir Claude Code oturumu, Superpowers ve Matt Pocock'un Main Flow'u tarafından inşa edildi; headless Sonnet 5.5, 19 sayılan çalıştırma. Token, süre ve liste fiyatı maliyeti çalıştırma başına değil, aşama başına okundu.
- Aşama haritası: Superpowers'ı hook ile yönlendirilmiş haliyle tutun. İstek bulanıkken Pocock'un ön yarısını, grill-with-docs, to-spec, to-tickets, ödünç alın. executing-plans reviewer'ındaki model satırını değiştirin, çünkü paketler arasındaki farkın neredeyse tamamı o tek aşama.
- Özellik 1 (kısmi ödemeler), konfigürasyon başına 3 çalıştırmanın ortalaması: bare $0.343 ve 84.5 s, hook'una bırakılan Superpowers $0.502 ve 118.1 s, tam döngüyle sürülen Superpowers $3.738 ve 607.367 s, Main Flow $2.043 ve 439.033 s. İki paketin üst üste kurulduğu bir çalıştırma: $2.326. Bu çalıştırmaların hepsi gizli testlerin 5 üzerinden 5'ini geçti.
- Sürülen Superpowers döngüsünde executing-plans toplam $3.738'in $3.307'sini yedi ve bunun $2.192'si, skill'in en yetenekli modele gönderdiği tek bir reviewer subagent'ıydı. Brainstorming $0.166, writing-plans $0.266 tuttu.
- Özellik 2 (credit note'lar), daha zor: bare $0.413 ve iki çalıştırmada bir kaçırma, Superpowers driven $3.213, Main Flow $2.067, son ikisi 5 üzerinden 5. 19 çalıştırmada kaçırılan tek gereksinim, bir bare çalıştırmasının gönderdiği boşluksuz credit note numarasıydı.
- Sınırlar: tek uydurma repo, tek worker modeli, iki özellik, senaryolu bir product owner, hücre başına küçük n. Burada paketleri kalite değil, maliyet ayırdı.
Kaynaklar ne diyor
Çalıştırma sırasındaki sürümler: Superpowers v6.4.2, 2026-09-25'te yayımlandı s20 ve mattpocock/skills v1.3.1, 2026-10-04'te yayımlandı s26. Superpowers 15 skill getiriyor, Pocock'un repo'su 27 shipped skill listeliyor s1 s3. 4 ortak, 10 yalnızca Superpowers'a ait, 12 yalnızca Pocock'a ait diyen eski örtüşme sayımı Superpowers 6.0.3 ile Pocock 1.0.1 arasında, 14'e karşı 17 skill ile yapılmıştı, yani güncel değil s6.
İki paket zıt bahisler. Superpowers, startup, clear ve compact sırasında using-superpowers skill'ini bir EXTREMELY_IMPORTANT etiketine sarıp enjekte eden bir SessionStart hook'u bağlıyor s23. Metni: "If you think there is even a 1% chance a skill might apply to what you are doing, you ABSOLUTELY MUST invoke the skill." s22 Ayrıca netleştirici sorular dahil, herhangi bir cevap ya da eylemden önce ilgili skill'lerin çağrılmasını söylüyor s22. Pocock'un paketinde oturum hook'u yok: "There is no session hook; nothing gets injected at startup." s6 Repo ağacında da hook dosyası yok s3.
Skill dosyaları kısa. grill-with-docs 35 kelime ve tek yaptığı Skill tool'unu iki kez çağırmak, grilling ve domain-modeling için s27. implement 103 kelime, disable-model-invocation taşıyor ve code-review çağırarak bitiyor s28. Matt'in duyurduğu yüzde 63 rakamı, bir çalıştırmanın değil, skill açıklamalarının token maliyetindeki azalma s34 s35.
Router skill'in dizdiği haliyle Main Flow, bir pull request adımından geçiyor ve döngüyü kapatan bir retro ile bitiyor s29. Genel bakış sayfası Main Flow'da 6 skill listeliyor ve /pr'ı reference skill'lerin altına koyuyor s1. Matt, kendi v1.3 notlarında şunu yazıyor: "The agent loop is worse than a deterministic loop. But it is a good way to start with AFK work" s2. Pürüzleri de sıralıyor: "worktrees do not remove collisions, they move them to merge time" s2. v1.3, CONTEXT.md'yi GLOSSARY.md, CONTEXT-MAP.md'yi GLOSSARY-MAP.md olarak yeniden adlandırıyor s2 s26.
Kurulum yolları. README bir plugin komutu ve bir installer komutu veriyor, claude plugin install mattpocock-skills@claude-plugins-official ve npx skills@latest add mattpocock/skills -a <agent>, ve agent başına birini seçmenizi söylüyor, çünkü ikisini birden kurmak her skill'i iki kez getiriyor s3. Marketplace kopyası geride kalıyor: resmi Claude plugin marketplace'i mattpocock-skills'i 1.2.3'te sabitliyor ve sürüm yükseltme isteği hâlâ açık s32. Bizim kontrolümüzde upstream main bu sabitlemenin 71 commit önündeydi ve sabitlenmiş kopyada implement-spec, pr ve retro yoktu s31 s32. Aynı eskiliğe dair bir kullanıcı raporu Pocock'un repo'sunda açık s31 ve 2026-10-08'de Matt, X'te cache'lenmiş sürümün yükseltilmesini istedi: "All my users are stuck on v1.2." s33 Aynı kontrolde marketplace Superpowers'ı 6.4.1'de sabitlemişti, v6.4.2'nin bir patch gerisinde s20.
Superpowers tarafında, 6.4.1 executing-plans'i yerel, inline bir modla yeniden kurdu, "the cheapest way to run a plan" s21 ve 6.4.2 writing-plans'i daha yalın hale getirdi s20. İnsanların andığı açık issue'lar: skill kurulduktan sonra yavaşlama s24 ve 5 saatlik bütçenin tamamını tüketen bir Codex subagent-driven çalıştırması; bu, bir 6.4 sürümünde değil, Superpowers 5.0.7 ve Codex'te bildirildi s25. Pocock'un /implement token maliyeti hakkındaki issue'su bir kullanıcı tartışması ve Matt onu bir skill hatası olarak değil, tartışma olarak kapattı s30.
İnsanların ölçmeden söyledikleri. Geçiş yapan biri, Pocock'un Q&A'sının Superpowers'ın brainstorming'inden daha az token kullandığını buldu, "with no worse results" s10. Bir başkası, zorunlu orkestrasyonun artık bir güvenlik ağı olmaktan çıkıp vergiye dönüştüğünü yazıyor s9. Bir Hacker News thread'i, Superpowers'ı hiç kullanmamaya karşı benchmark istiyor, yanında evals ile geliştirildiği savunması da var s13. Grilling üzerine bir thread, kısa bir implementasyondan önce "5 hours wayfinding and spec reading" bildiriyor ve bir yorumcu, daha yeni modellerle grilling'in "feels like a hinderance" olduğunu söylüyor s7. İki paketi router olarak üst üste kurmakla ilgili bir yazı, slash komutlarının ve oturum hook'larının çakıştığını iddia ediyor s11. Bunların hiçbiri aşama başına token rakamı vermiyor; bu bench'in doldurduğu boşluk bu.
Bench
Bench, ledgerly adlı küçük, uydurma bir TypeScript fatura API'si: bir domain katmanı, şema migration'lı bir JSON dosya deposu, bir HTTP katmanı, 34 geçen test, 4 commit. Her özellik isteği üç satır ve bir geliştiricinin yazacağı gibi yazılmış. Bir senior mühendis, isteğin hiç yazmadığı beş şey bekler ve bu beşi, worker'ın hiç görmediği gizli testler. Worker headless Sonnet 5.5. İkinci bir headless model product owner'ı oynuyor, aracı yok, özel bir brief okuyor ve worker'ın sorularını kısa cevaplıyor. Her tur token, liste fiyatı maliyeti, duvar süresi, araç çağrıları ve skill çağrılarıyla kaydediliyor. Maliyet CLI'ın total_cost_usd değeri, yani liste fiyatı karşılığı, bir fatura değil. Hiçbiri kimsenin gerçek kodunda çalıştırılmadı, bu yüzden kendi repo'nuz için bir şablon olarak okuyun.
README.md
# flow-bench: Superpowers loop vs Matt Pocock's Main Flow, one feature, one repo
Built 2026-10-09 from `SPEC.md` (the fixed decisions are implemented as written; deviations and their reasons are
listed under "Decisions taken where SPEC left a choice"). Worker = headless `claude-sonnet-5-5` (`claude -p`, OAuth
from `~/.claude`), Claude Code 2.1.294.
## Commands (from this dir)
```bash
./validate.sh # SPEC validation steps 1-5 -> validation.log (3-5 are smoke runs, RUN_TAG=smoke)
./validate.sh 1 2 # only the static steps (reference patch, untouched upstream)
RUN_TAG=x ./run.sh <cfg> <rep> # one run -> runs/<cfg>-r<rep>-x/, work copy work/<same>/ ; tagged runs are ignored by analyze.py
nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 & # the 10-run matrix, 2 lanes (MATRIX_JOBS=2)
SKIP_DONE=1 nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 & # resume
python3 analyze.py # results.json + results.md
python3 lib/finalize.py runs/<run> # rescore one run (restores node_modules into the work copy if it was removed)
python3 lib/reaccount.py runs/<run> # recompute turns.jsonl accounting from the saved streams (then finalize)
```
## Layout
| path | what |
|---|---|
| `upstream/` | **ledgerly**, the invented repo: Node 20 + TS + vitest + pnpm, 18 source files (domain / store / http layers), 34 tests in 7 files, 4 commits (HEAD 980d542), `test` + `typecheck` scripts, `node_modules` installed (pnpm 11) |
| `tasks/F1.md` | the task, verbatim (3 lines, the way a dev types it) |
| `po/F1-brief.md` | the product owner's full intent: the 5 requirements in plain words, preferences, constraints (never the tests) |
| `hidden/` | `test_F1.test.ts` (the oracle: `test_req1..5`), its own `vitest.config.ts` (alias `@work` -> `$WORK`) and `node_modules` (vitest 2.1.9). Runs from OUTSIDE the work copy |
| `reference/F1.patch` | reference solution: hidden 5/5, suite 39/39, tsc green |
| `configs.json` | `bare`, `sp`, `mp`, `both` |
| `packs/` | `superpowers/` (plugin root of obra/superpowers v6.4.2), `mattpocock-skills/.claude/skills/` (27 skills of mattpocock/skills v1.3.1 installed by `npx skills add`), `VERSIONS.md`, `mattpocock-install.log`; `*-src/` = the clones (gitignored) |
| `env.sh` | copied from an earlier bench of the same kind: scrubs inherited `CLAUDE*` vars, `CLAUDE_CODE_DISABLE_AUTO_MEMORY=1` |
| `prep.sh <cfg> <work>` | fresh work copy + the pack fixtures of the config + `.claude/settings.json`; prints the base commit |
| `run.sh <cfg> <rep>` | prep -> `lib/driver.py` -> `lib/finalize.py` |
| `lib/driver.py` | the turn loop (flows below), caps, `turns.jsonl` |
| `lib/accounting.py` | per-turn and per-stage tokens / USD / wall / tools / Skill events / hook events from one turn's stream |
| `lib/po.py` | the product-owner simulator |
| `lib/finalize.py` | diff vs base, suite, typecheck, hidden oracle, stages -> `score.json`, `stages.json` |
| `analyze.py` | `results.json`, `results.md` |
| `validate.sh`, `validation.log`, `validation/routing-probes/` | pre-matrix checks (+ 6 one-turn probes: plugin wiring vs project wiring) |
| `matrix.txt`, `matrix.sh`, `matrix-one.sh`, `matrix.log` | 10 runs, 2 at a time |
### The feature and the oracle
`tasks/F1.md`: partial payments, `POST /invoices/:id/payments` with an amount in cents, `amountPaid`/`balance` on
invoices, flip to `paid` at zero balance, keep `/pay` working, "we have prod data in the JSON store".
The 5 hidden requirements (one `it` each) and the codebase precedent that makes each one a senior's expectation:
| req | checks | precedent in the repo |
|---|---|---|
| R1 edge cases | payments accumulate; status stays non-`paid` until the exact remainder; `paidAt` set; `/reports/outstanding` sums balances, not totals | the report sums `total` of open invoices |
| R2 validation | 0, negative, 12.5, "100", null, missing -> 400/422; overpayment (of the total, then of the balance) -> 400/409/422; state unchanged | `assertCents`, "discount cannot exceed the subtotal" |
| R3 migration of existing data | a schemaVersion-1 file: legacy `paid` invoice shows amountPaid=total, balance 0; open shows 0/total; payments persist across a reload; `/pay` leaves balance 0 | `schemaVersion` + `src/store/migrations.ts` (v0->v1), README house rule |
| R4 error path | unknown id 404; draft 409; void 409; nothing recorded | `markPaid` throws `ConflictError` on draft/void |
| R5 a test | a `*.test.ts` changed or added since the base commit mentions `payments` | 34 existing tests |
Lenient where the brief would be a guess (status codes for overpayment accept 400/409/422, payment response 2xx,
response body never inspected: state is read back with GET).
## Configs and how each pack is wired (`prep.sh`)
- Every config: `cp -R upstream`, `.claude/` in `.git/info/exclude`, `.claude/settings.json` with a `permissions.deny`
list (Read of `hidden/ reference/ runs/ po/ validation/` of this bench; Bash commands containing `hidden`,
`reference/F1`, `F1-brief`, `/runs/`, `../`, and `find`). Bare has nothing else: no skills, no hooks.
- `sp`: the plugin root is copied to `.claude/plugins/superpowers/` (the hook script reads
`../skills/using-superpowers/SKILL.md` relative to itself), the 15 skills to `.claude/skills/<name>/`, and
`hooks/hooks.json` is reproduced in `.claude/settings.json` verbatim (`SessionStart`, matcher
`startup|clear|compact`, `"shell": "bash"`, `"async": false`) with the command prefixed by
`CLAUDE_PLUGIN_ROOT="$CLAUDE_PROJECT_DIR/.claude/plugins/superpowers"`: Claude Code sets that variable for plugin hooks,
and `hooks/session-start` uses it to pick the `hookSpecificOutput.additionalContext` output shape. Superpowers ships no
other hook (`hooks-cursor.json` is Cursor's).
- `mp`: the 27 skills to `.claude/skills/`. Pocock ships **no hook** (verified: no settings or hooks file in the install;
`git-guardrails-claude-code`, a skill that writes hooks, is outside the promoted set and was trimmed). The pack's
per-repo precondition `/setup-matt-pocock-skills` is materialised once: `docs/agents/issue-tracker.md` (local
markdown under `.scratch/`), `docs/agents/domain.md`, `docs/agents/triage-labels.md` from the skill's own seed
templates, and a `CLAUDE.md` "## Agent skills" block, committed as "chore: configure agent skills (local tracker)"
so the diff only shows the feature. Base commit for mp/both = that commit.
- `both`: sp + mp in the same `.claude/` (no skill-name collision between the two packs), driven with the mp flow.
## Flows (`lib/driver.py`)
Every worker turn: `claude -p <prompt> [--resume <sid>] --model claude-sonnet-5-5 --output-format stream-json --verbose
--include-hook-events --include-partial-messages --max-turns 60 --permission-mode acceptEdits --allowedTools <SPEC list>
--disallowedTools AskUserQuestion,EnterPlanMode,ExitPlanMode,EnterWorktree,ExitWorktree,WebFetch,WebSearch
--setting-sources project --strict-mcp-config`, cwd = work copy, stdin /dev/null, scrubbed env.
Allow-list: `Read,Edit,Write,MultiEdit,Glob,Grep,Skill,TodoWrite,Task,Agent`, Bash `node npm pnpm npx vitest npx tsc tsc
git status|diff|log|add|commit|checkout -b ls cat mkdir head tail grep wc gh`.
- `bare`: turn 1 = task + "When you are done, stop and summarise what you changed."
- `sp`: turn 1 = the task alone (the SessionStart hook is supposed to route it).
- `mp` / `both`: `/grill-with-docs <task>` -> `/to-spec` -> `/to-tickets` -> `/implement the tickets in
.scratch/<slug>/issues/, all of them, blockers first` -> `/code-review since <base sha>` -> `/pr`.
- After every turn the PO reads the last assistant message and classifies it: `question` (its reply is the next
prompt), `waiting` (driver sends "go on" / "go on, proceed with the plan"; mp: stays in the stage), `done` (bare/sp:
stop; mp: next slash command; after `/pr`: stop). If the PO call fails twice the SPEC heuristic (ends on "?" or an
options list) decides.
- Caps: 12 driver turns, 60 `--max-turns` per claude turn (a turn that hits it gets "go on"), 40 min wall per run
(the claude process is killed at the deadline). Caps hit are recorded in `driver.json` / `score.json`, never discarded.
### PO simulator (`lib/po.py`)
Headless `claude-sonnet-5-5`, `--tools ""`, `--setting-sources ""`, `--settings '{"disableAllHooks":true}'`,
`--strict-mcp-config`, `CLAUDE_CODE_DISABLE_CLAUDE_MDS=1`, `CLAUDE_CODE_DISABLE_AUTO_MEMORY=1`, cwd = an empty temp dir,
`--json-schema {kind, reply}`. Prompt = role + `po/F1-brief.md` + flow note + stage + the last message. Rules: terse,
answer every open question at once, approve what matches the brief, correct only contradictions, "your call" where
the brief is silent, **answer only what was asked** (first smoke: the PO volunteered the migration requirement nobody
had asked about; the rule was added before any counted run), how to finish a branch: "keep it on the current branch,
no PR". mp rule: a stage whose output is delivered and that only asks "shall I proceed / start?" is `done` (first mp
smoke: the PO's "yes, go" after the grilling recap made the worker start implementing inside `/grill-with-docs`; that
smoke was killed and rerun). PO cost is recorded apart (`usd_po`), never in the worker's USD.
## Accounting (the per-turn / per-stage numbers)
Checked on 2.1.294 with probes before writing `lib/accounting.py`:
- `result.total_cost_usd` and `result.modelUsage` are **cumulative over a resumed session** and **include subagents**:
per-turn USD and tokens = delta against the previous turn's result line.
- `result.usage` = main thread of the invocation only (a Task subagent's tokens are missing from it): not used.
- `assistant` events carry the usage snapshot of the message START (`output_tokens` partial, e.g. 3 for a 783-token
message): useless for output. `--include-partial-messages` adds `stream_event` `message_start` / `message_delta`
with the exact per-API-call usage, main thread only.
- Sonnet 5.5 list prices derived from two probe turns and matched to the cent: $2/M input, $10/M output, $0.20/M cache
read, $4/M cache write (1 h TTL, what Claude Code uses here); 5-min writes priced $2.5/M (unseen). In every turn without
subagents, the per-call priced sum equals the `total_cost_usd` delta exactly (`pricing_check` in `turns.jsonl`).
- Stage = slash command of the turn (mp/both, exact) or the skill in force (sp: switches at each main-thread `Skill`
tool_use; turn 1 starts at `(none)`). Inside a turn, main-thread calls are attributed exactly; the subagent remainder
(turn delta minus main calls) goes to the stage that spawned the Task, weighted by the subagents' snapshot tokens.
Wall per stage = receive-time gaps of the stdout lines (`turn-NN.times`) attributed to the stage in force.
- `analyze.py` also groups sp skills into phases (brainstorming / writing-plans / execution incl. TDD /
verification / review / finishing).
- Tokens reported = input + output + cache read + cache write. Cache reads dominate (re-reading the context each call).
### Per run (`runs/<cfg>-r<rep>/`)
`meta.json`, `prep.txt`, `settings.json` (copy of the work copy's), per turn `turn-NN.stream.jsonl` (raw stdout, every
line), `turn-NN.times`, `turn-NN.stderr`, `turn-NN.last.txt`, `po-NN.json`; `turns.jsonl` (one accounting record per
turn: prompt, source, wall, subtype, USD, tokens per model, segments, tools, Skill events, hook events, files, denials,
PO answer), `driver.json` (end reason, caps), `post.diff`, `numstat.txt`, `suite.txt`, `typecheck.txt`, `hidden.txt`,
`stages.json`, `score.json`. Work copies stay in `work/` without `node_modules` (49 MB each; finalize removes it,
`KEEP_NODE_MODULES=1` keeps it).
## Decisions taken where SPEC left a choice
- **Implement stage = `/implement`, not `/implement-spec`.** v1.3.1 ships both. `/implement-spec` requires implementer
subagents "each in its own worktree on its own branch" plus merger subagents: `git worktree`, `git merge`, `git branch`
are outside the SPEC Bash allow-list, so it could not run as designed. `/implement` is sent once with the ticket
directory ("all of them, blockers first"), since one session cannot `/clear` between tickets as `ask-matt` suggests.
- `/code-review` gets `since <base sha>` (the skill reviews "the diff since a fixed point"); Pocock's `code-review`
replaces Claude Code's bundled `code-review` skill of the same name in the session (init lists it once, the Skill
event loads `.claude/skills/code-review`).
- `/retro` not run (optional in SPEC).
- `both` uses the mp flow: the stacking question is whether the Superpowers SessionStart routing hijacks the slash
commands.
- Reps / order: `matrix.txt` (rep 1 of every config first), 2 lanes.
## Traps met
- **Superpowers routing is stochastic headless.** On turn 1 the hook always fires, but the model picks
`test-driven-development` instead of `brainstorming` part of the time. Six one-turn probes on the F1 prompt
(`validation/routing-probes/`): real plugin wiring (`--plugin-dir packs/superpowers`, namespaced
`superpowers:*`) 1/3 brainstorming, 2/3 TDD; project wiring (this bench) 2/3 brainstorming, 1/3 TDD. So the project
wiring is not the cause; the first sp smoke went TDD-only (one turn, hidden 5/5). A second sp smoke (`smoke2`) was run
to see the brainstorming path end to end.
- **The SessionStart hook fires only on turn 1.** Its matcher is `startup|clear|compact`; every later turn is
`--resume` (source `resume`), so the injection is not repeated; it stays in the transcript.
- **User plugins do not leak** with `--setting-sources project`: the init event lists only Claude Code's built-in
plugins (`cc-plugin-agents-md`, `cc-plugin-telemetry`, `cc-plugin-plugin-authoring`), never the user's Superpowers
6.4.1; bare smoke: no hook event, no Skill event. Built-in skills (`verify`, `debug`, `simplify`, `code-review`, ...)
are still listed in every config.
- **`--bare` is never used** (cannot authenticate on this OAuth account; see the previous bench).
- **Read-only shell commands bypass the allow-list.** Claude Code auto-approves read-only commands (`git ls-files`,
`sed -n`, pipes of read-only commands) whatever `--allowedTools` says; `python3` edits are denied (the bare smoke had 4
`permission_denials`, all `python3` heredoc edits; the worker fell back to Edit). `cat > /tmp/x <<EOF` matches
`Bash(cat:*)`: the allow-list is a prefix match, not a sandbox.
- **pnpm 11** refuses to install without approving esbuild's build script: `pnpm-workspace.yaml` with
`allowBuilds: {esbuild: true}` (pnpm 11 no longer reads the `pnpm` field of package.json). No `packages:` key, so it
is not a monorepo signal for Pocock's setup.
- **`npx skills add` ignores `--skill a,b` and `--skill "a b"`** ("No matching skills found"); repeated `--skill x`
flags installed every skill of the repo (37). Trimmed to the plugin's 27 (`packs/VERSIONS.md`). Pinning works with
the `https://github.com/mattpocock/skills/tree/v1.3.1` source (main is past the release).
- **mattpocock skills are mostly user-invoked** (`disable-model-invocation: true` on grill-with-docs, to-spec,
to-tickets, implement, implement-spec, ...): reachable as slash commands only; `/grill-with-docs` itself is one line
("Call the Skill tool twice, for grilling and domain-modeling"), so the proof that it resolved is those two Skill
events on turn 1.
- **The model mistypes the bench path.** Sonnet 5.5 regularly writes absolute paths with
`superpowers-vs-matt-pocock-skins-...` (or `skips`) instead of `skills`: 0 to 10 tool calls per run (0 to 7% of calls;
e.g. mp-r3-po1 10/145, mp-r2 0/124, sp-r2 3/48, bare-r1 0/28), in every config. Edits on the wrong path fail and are
retried (a small, roughly proportional token tax); **Writes succeed outside the work copy** because `Write` in
`--allowedTools` is not path-limited: the runs left stray files under a mistyped copy of the bench path (a spec
copy, an `x`) and three 1-byte placeholder files, deleted after the matrix
(nothing else in the repo was touched; `git status` checked). A future bench should put work copies on a short
path without the word `skills` and deny `Write(//**)` outside the work copy.
- **Shared plan meter.** Workers and PO bill the user's plan quota; USD = the CLI's list-price equivalent.
- **rtk** rewrites interactive `diff`/`grep` in the launching session only; scripts call binaries directly.
## Validation (2026-10-09, `validation.log`)
1. `reference/F1.patch` -> hidden 5/5, suite 39/39 green, tsc green: PASS.
2. upstream unchanged -> hidden 0/5, suite 34/34 green: PASS.
3. `bare-r1-smoke` -> 2 driver turns (the PO answered one closing question), hidden 5/5, $0.52, 125 s; isolation: no
hook event, no Skill event, only built-in plugins in init: PASS.
4. `sp-r1-smoke` -> hook fired and injected `using-superpowers` (PASS) but the model went straight to
`test-driven-development` (brainstorming FAIL in that run). Retry `sp-r1-smoke2`: hook fired, `brainstorming` on
turn 1, PO approved the design, then `test-driven-development`: PASS. Routing probes: see Traps.
5. `mp-r1-smoke` -> turn 1 loads `grilling` + `domain-modeling` (what `/grill-with-docs` says to do): PASS. The run then
used all 12 driver turns (grill 4, to-spec 2, to-tickets 2, implement 2, code-review 2) and never reached `/pr`.
Fixes made during validation, before any counted run: the PO volunteering unasked requirements; the PO telling the
worker "yes, go" at the end of the grilling (the worker started implementing inside `/grill-with-docs`; that first mp
smoke was killed at turn 4 and rerun).
## Matrix launch
`nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 &` at 2026-10-09 10:41:33, 2 lanes, order in `matrix.txt`.
Done 11:06:29 (10 runs, 25 min).
### Reruns of the mp-flow runs (harness fixes, `matrix.log`)
- `both-r1` (first): the PO answered the grilling recap's branch question with an approval and the worker coded inside
`/grill-with-docs` (turn 4, 12 src/test files). Kept as `runs/both-r1-pofault` (tag `pofault`, ignored by analyze).
PO rule added (mp flow: before `/implement`, end with "Don't start implementing yet." when the worker offers to build).
- `both-r1` (second): the PO did not apply that rule and the worker coded inside `/grill-with-docs` again (turn 3, after
"say so and I'll start with failing tests"). The rule was made deterministic in `lib/driver.py`: in `/grill-with-docs`,
`/to-spec`, `/to-tickets`, the driver appends "Don't start implementing yet." to every PO answer. Since that changes
the conditions of all mp-flow runs, mp-r1..r3 and both-r1 were moved to `runs/*-po1` (tag `po1`; mp-r1..r3-po1 never
coded early, hidden 5/5 each, $2.22 to $2.57) and rerun with `matrix-mp2.txt` (11:18 to 11:40). The counted mp/both
runs are the reruns.
- sp stage labels: brainstorming's SKILL.md ends at the approved design, but in sp-r1 and sp-r3 the model then coded with
no new Skill call, so "skill in force" would bill the whole build to brainstorming. `lib/accounting.py` now switches the
stage to `implementation (no skill)` at the first main-thread Edit/Write under `src/` or `test/` while only a design
skill (`(none)`, `using-superpowers`, `brainstorming`, `writing-plans`) is in force; the sp runs were re-accounted from
their saved streams with `lib/reaccount.py` (tokens, USD and wall totals unchanged, only the split).
## Results
`results.md` / `results.json` (`python3 analyze.py`), reading in `FINDINGS.md`.
---
# Round 2 (2026-10-09 afternoon): `spd`, feature F2, isolation fix, full rerun
## Isolation leak found and fixed (all round-1 runs superseded)
The first `spd` smoke said "the most capable model your CLAUDE.md permits for subagents": the workers were reading
CLAUDE.md files. `--setting-sources project` does keep `~/.claude/CLAUDE.md` out (probe in an empty dir outside the
repo: "NONE"), but Claude Code also loads **every CLAUDE.md of the parent directories**, and the work copies lived under
the host repository, so every round-1 session carried that repository's CLAUDE.md and what it imports
(a probe in a round-1 work copy showed the host rules present; the round-1 mp code
reviews even cited them). Fix: `run.sh` builds the work copy in
`/private/tmp/flow-bench/<run>` (outside any CLAUDE.md tree) and archives it to `work/<run>` (without node_modules)
after finalize; probe in an mp copy there: only the pack's own "## Agent skills" CLAUDE.md block appears.
Every round-1 run was renamed `runs/*-leak` (tag `leak`, ignored by analyze) and the whole matrix rerun
(`matrix2.txt`). The short path also removes the "pocock-skins" typo surface (see Traps).
## `spd`: Superpowers, driven
Same project install as `sp` (skills in `.claude/skills/`, SessionStart hook). Installed in the project, the skills
resolve **without the plugin namespace**: init `slash_commands` lists `brainstorming`, `writing-plans`,
`executing-plans` (with `--plugin-dir` they are `superpowers:brainstorming`, ...; see `validation/routing-probes/`),
so the driver types `/brainstorming <task>` -> `/writing-plans` -> `/executing-plans`. writing-plans ends by asking
"Subagent-driven or Native?": the PO answers Native (added to both briefs: "If asked to choose how to execute a written
plan: Native (inline, in this session) ... working on main is fine: you consent"). Before `/executing-plans` the driver
appends "Don't start implementing yet." to PO answers (same rule as mp). A stage whose output is delivered and that
only asks for approval/go is `done` -> next command; after `/executing-plans` the run ends when the PO says done
(executing-plans itself runs the final whole-branch review and finishing-a-development-branch).
- Allow-list: the SPEC list + `Bash(*scripts/task-start*)`, `task-done`, `sdd-workspace`, `task-brief`,
`review-package` (executing-plans drives its loop with these pack scripts), and the `Bash(*../*)` deny is dropped for
spd (the skill calls `../subagent-driven-development/scripts/...`); the oracle stays behind the Read denies and
`Bash(*hidden*)`.
- Stages: per slash command (turn level, exact) AND per skill in force inside the turn (a typed slash command starts
its skill; later Skill events switch). Subagent spend is now its own segment `<dispatching stage> [subagent]`
(round 1 folded it into the dispatching stage); executing-plans' final reviewer is dispatched "on the most capable
available model": it ran on `claude-fable-5-1` in the spd smoke ($2.19 of $3.88).
- `sp` is unchanged and labelled **hook-routed** in results.
## F2: credit notes (`tasks/F2.md`, `po/F2-brief.md`, `hidden/test_F2.test.ts`, `reference/F2.patch`)
Request (2 lines): credit notes `POST/GET /invoices/:id/credit-notes`, numbered `CN-0001`..., and "voiding a paid
invoice should then work, by crediting whatever is left on it". Implicit requirements (hidden, one `it` each):
| req | category | checks | why a careless build misses it |
|---|---|---|---|
| R1 | cross-cutting invariant, 2 layers | credits never exceed the total cumulatively (domain) and `/reports/outstanding` owes total minus credits (report) | the report is not in the request |
| R2 | stored-data compat | a schemaVersion-1 file (no `creditNotes`, no credit-note counter): first note `CN-0001`, id not `NaN`, notes survive a reload; a legacy paid invoice can be voided | `migrate()` rebuilds only known keys and casts `counters`: a new counter is `undefined + 1` |
| R3 | ordering rule | one gapless CN sequence across invoices; rejected requests (over-credit, bad amount, draft, unknown) consume no number | the existing `/issue` route takes `nextId()` before validating: copying it burns numbers |
| R4 | error path, specific status | draft/void 409, unknown 404 on POST and GET, bad amounts 400, nothing recorded | |
| R5 | existing test changes meaning | void of a paid invoice: 200, credits only the remainder, void again 409; the 33 other upstream tests present and green and `test/invoice.test.ts` keeps >= 7 tests ("voids drafts and open invoices, never paid ones" must be rewritten, not deleted) | |
Validation (`validation.log`): reference 5/5, suite 39/39, tsc green; upstream 0/5, suite 34/34; bare smoke (isolated)
**4/5** (R3 failed: `CN-0004` where `CN-0002` was due, rejected requests burned numbers). A first F2 (payment terms,
`validation/f2-v1/`) was dropped: its bare smoke scored 5/5.
`hidden/upstream_test_titles.json` = the 34 upstream test titles (for R5). `hidden/vitest.config.ts` picks
`test_${TASK}.test.ts`. Run names: F1 `<cfg>-r<rep>`, F2 `f2-<cfg>-r<rep>`; `TASK=F2 ./run.sh <cfg> <rep>`.
## Matrix round 2 (`matrix2.txt`, 2 lanes, started 12:09:53)
F1: bare x3, sp x3, spd x3, mp x3, both x1. F2: bare x2, spd x2, mp x2. Same caps (12 driver turns, 60 max-turns,
40 min).
Done 13:0x (19 runs). Then `analyze.py` (results.md / results.json carry F1 and F2, five configs).
### After the matrix
- **F2 oracle v3.** Both F2 bare runs allowed credit notes on `paid` invoices only (409 on open ones), a reading the
request's framing ("paid invoices can't be voided, so finance needs credit notes") permits; oracle v2 ran R1-R3 on
open invoices, so that one choice failed three requirements. R1-R4 now run on paid invoices; credit notes on open
invoices and the report netting are out of the oracle (the brief still prefers them). v2 kept in
`validation/test_F2.v2-open-invoices.test.ts`; reference 5/5 and upstream 0/5 re-checked; the 7 F2 runs rescored
with `lib/finalize.py` (worker runs untouched). With v3: f2-bare-r1 4/5 (R3, numbers burned), every other F2 run 5/5.
- **spd permission denials:** 5 to 11 per run, mostly compound shell lines (`W=.superpowers/sdd/...; mkdir ...; cat ...`)
that match no allow rule; the pack scripts themselves ran (task-start/task-done in every spd run).
- **"pocock-skins" path typos** (Traps) concern round 1 only: round-2 work copies live at `/private/tmp/flow-bench/<run>`,
a path without the word `skills`; no stray file was found in the repo after round 2.
- Zero-cost `[subagent]` placeholder segments are skipped by analyze.py.
SPEC.md
# flow-bench - Superpowers loop vs Matt Pocock's Main Flow, same feature, one repo
Question the video must answer: on one small invented repo, for one bounded brownfield feature,
what does each pack cost PER STAGE (tokens, wall-clock, USD) and what does it deliver (hidden
acceptance tests passed, repo suite green, diff size, bugs shipped), against a bare Claude Code run?
Secondary question: does stacking both packs in one project break (hook / slash conflicts)?
## Fixed decisions (2026-10-09)
- Worker model: `claude-sonnet-5-5`, headless (`claude -p`), OAuth from `~/.claude` (see traps below).
- The repo `upstream/`: INVENTED, built here, a small TypeScript project (about 12-20 source files,
vitest, pnpm or npm, a green suite of ~25-40 tests): a plausible brownfield service, e.g. an
invoicing API or a booking CLI with a store layer, a domain layer and an HTTP or CLI layer. It is a
git repo with a clean history (3-6 commits), a short README and a `package.json` with
`test` and `typecheck` scripts. No framework exotica; plain Node 20 + TS + vitest.
- ONE feature task `tasks/F1.md`: a terse developer request (headline ask + one or two constraints,
2-4 lines, the way a dev types it), whose honest completion needs 5 requirements that a senior
would expect (edge cases, a validation, a migration of existing data, an error path, a test).
`hidden/test_F1.test.ts` is the oracle (5 requirement tests, never visible to the worker, run
from OUTSIDE the work copy against it). `reference/F1.patch` passes all 5 and the repo suite.
- Configs (`configs.json`):
- `bare`: no skills, no hooks.
- `sp`: obra/superpowers installed in the PROJECT (not the user's plugin): clone the repo at the
latest release tag (record it), expose its skills under `<work>/.claude/skills/` and its
SessionStart hook + any hooks it ships under `<work>/.claude/settings.json` exactly the way the
plugin wires them (read its `hooks/` and `plugin.json`; reproduce, don't improvise). Verify in a
smoke run that the `using-superpowers` text actually appears in the session (stream-json shows
the hook output) and that `brainstorming` is invoked.
- `mp`: mattpocock/skills installed in the project via the official installer
(`npx skills add mattpocock/skills` or the README's exact command, latest release, record the
version); verify the Main Flow slash commands resolve (`/grill-with-docs`, `/to-spec`,
`/to-tickets`, `/implement` or `/implement-spec`, `/code-review`, `/pr`, `/retro`; use the names
the installed version really has).
- `both`: `sp` + `mp` together (one or two runs only: the stacking question).
- Flows (the driver `run.sh <cfg> <rep>` / `lib/driver.py`):
- `bare`: one session, the task prompt, "when done, stop and summarise".
- `sp`: one session; turn 1 = the task prompt (the hook is supposed to route into brainstorming).
Then the Superpowers loop as it unfolds (brainstorming → writing-plans → executing-plans →
review); the driver only answers questions and says "go on / proceed with the plan" when the
assistant ends a turn waiting. Record which skills were invoked and when (Skill tool events).
- `mp`: one session, one stage per driver turn, in this order: `/grill-with-docs <task>` →
`/to-spec` → `/to-tickets` → the implement stage (`/implement` or `/implement-spec`, whichever
v1.3 ships for the loop over tickets) → `/code-review` → `/pr`. (`/retro` optional, one run.)
The driver answers questions between turns.
- Every turn is `claude -p … --resume <session_id> --output-format stream-json --verbose
--include-hook-events`; the first turn creates the session. Per TURN: tokens in/out/cache
read/cache write, USD (`result.total_cost_usd` deltas), wall-clock, tool calls, Skill
invocations, files touched. Per STAGE: the sum of its turns. A "stage" for `sp` is the skill in
force (from the Skill events), for `mp` the slash command of the turn.
- **Product-owner simulator** (`lib/po.py`): both packs interrogate the user. A headless
`claude-sonnet-5-5` with NO tools, `--setting-sources ""`, `CLAUDE_CODE_DISABLE_CLAUDE_MDS=1`,
reads `po/F1-brief.md` (the full intent, the 5 requirements in plain words, preferences and
constraints; never the tests) and answers the worker's questions tersely, as a busy dev would.
When the worker ends a turn without a question, the PO replies "go on" (or the next slash
command for `mp`). Hard caps: 12 driver turns per run, 60 `--max-turns` per claude turn, 40 min
wall per run. A run that hits a cap is recorded as such, not discarded.
- Reps: 3 per config for `bare`, `sp`, `mp`; 1 for `both`. 10 runs. Parallel 2.
- Scoring (`lib/finalize.py`): hidden 0-5, repo suite pass/fail, typecheck pass/fail, `git diff
--stat` lines added/removed, files touched, tests added, plus the per-stage table. `analyze.py`
→ `results.json` + `results.md` (per config: mean/median per metric; per stage: tokens, USD, wall).
- Validation BEFORE the matrix (`validate.sh`, logged): (1) `reference/F1.patch` → hidden 5/5 and
suite green; (2) upstream unchanged → hidden 0/5 (or the honest number, recorded) and suite green;
(3) one `bare` smoke run end to end with the PO loop exercised; (4) one `sp` smoke proving the
hook fired and brainstorming ran; (5) one `mp` smoke proving `/grill-with-docs` resolved. A
smoke run is tagged (`RUN_TAG=smoke`) and ignored by analyze.py.
## Traps already known (from a previous bench of the same kind)
- `claude -p --bare` cannot authenticate on an OAuth (Max plan) account: never use `--bare`. Use
`--setting-sources project` for workers so the user's own plugins (Superpowers 6.4.1 is installed
at user scope on this machine) never leak into a `bare` or `mp` run. VERIFY this isolation in the
smoke: a `bare` run must show no Superpowers hook output and no Skill events.
- Scrub inherited `CLAUDE*` env vars (copy `env.sh` from the previous bench), set
`CLAUDE_CODE_DISABLE_AUTO_MEMORY=1`.
- Fresh work copy per run at a unique path; `.claude/` excluded via `.git/info/exclude` so diffs
stay about the feature.
- A worker may try to read the hidden tests: the hidden dir lives outside the work copy and the
allow-list has no `find`/`ls` outside cwd.
- Allowed tools: Read,Edit,Write,MultiEdit,Glob,Grep,Skill,TodoWrite,Task,AskUserQuestion is NOT
available headless (questions come as plain text; the driver detects a turn that ends on a
question mark or an explicit "?"/options list and routes it to the PO); Bash limited to
node/npm/pnpm/npx vitest/tsc/git status|diff|log|add|commit|checkout -b/ls/cat/mkdir/head/tail/
grep/wc. Pocock's `/pr` wants `gh`: allow `Bash(gh:*)` but with no remote the PR step records
its output text as the artefact (no network push).
## Deliverables
`README.md` (how to run, layout, every trap met), `SPEC.md` (this), `upstream/`, `tasks/F1.md`,
`po/F1-brief.md`, `hidden/`, `reference/`, `configs.json`, `env.sh`, `prep.sh`, `run.sh`,
`validate.sh`, `matrix.sh`, `lib/{driver.py,po.py,finalize.py}`, `analyze.py`, `validation.log`,
`runs/<cfg>-r<rep>/` (stream.jsonl per turn, stages.json, score.json), `results.json`,
`results.md`, and `FINDINGS.md`: the honest reading in 300 words, every number with its run name.
FINDINGS.md
# flow-bench findings (2026-10-09, round 2: isolated work copies, Sonnet 5.5 headless, 19 counted runs)
**F1 (partial payments): quality did not separate.** All 13 F1 runs hidden 5/5, suite and tsc green.
Cost per run: bare $0.27 (bare-r2) to $0.47 (bare-r1); sp, hook-routed, $0.44 (sp-r1) to $0.54 (sp-r2); mp $1.80
(mp-r2) to $2.22 (mp-r1); spd, Superpowers driven, $3.39 (spd-r2) to $4.41 (spd-r3).
Wall: mp 365 to 480 s (mp-r2, mp-r3), spd 520 to 701 s (spd-r2, spd-r3).
**Where the money goes.** mp-r2 (complete flow): grill $0.16, spec $0.11, tickets $0.09, implement $0.96, code-review
$0.45, pr $0.04 (F1 means: implement $1.13, code-review $0.44). spd: brainstorming $0.17, writing-plans $0.27,
executing-plans $3.31 (F1 means). Inside executing-plans the build itself (TDD in session) costs $0.97 on average; the
rest is the one whole-branch reviewer the skill dispatches "on the most capable model", which ran on claude-fable-5-1
in all five spd runs: $2.01 of spd-r1's $3.41, $2.52 of spd-r3's $4.41, $1.57 of f2-spd-r2's $3.00.
**Hook vs driven.** Left to its hook, Superpowers ran brainstorming in sp-r1, sp-r2 and sp-r3 and then coded; writing-
plans ran 0 times in 3. Driven by slash commands, the full loop ran in every spd run and cost 7.4 times more (F1 means: spd $3.74, sp $0.50). The cost is the final review, not the planning.
**F2 (credit notes) separated, barely.** f2-bare-r1 shipped one missing requirement, R3: credit-note numbers are not
gapless. It took the number before validating, like the existing `/issue` route, so rejected requests burned numbers
(CN-0004 where CN-0002 was due; the F2 bare smoke failed the same way). f2-bare-r2, f2-spd-r1, f2-spd-r2, f2-mp-r1 and
f2-mp-r2 scored 5/5. Costs: bare $0.38/$0.45, mp $2.18/$1.95, spd $3.42/$3.00 (r1/r2). Caveat: the first F2 oracle
also required credit notes on open invoices; both bare runs allowed paid invoices only, which the request's wording
permits, so that check was removed and the F2 runs rescored.
**Stacking broke nothing.** both-r1: hook injected, no Superpowers skill invoked, $2.33, inside the mp range.
**Caps.** mp-r1, mp-r3 and f2-mp-r1 hit the 12-driver-turn cap before /pr. spd never needed more than 8 turns.
**Caveats.** Round-1 runs read this repo's CLAUDE.md (parent-directory loading), so they were discarded and rerun outside the repo. A simulated PO answers each round in one message. The SessionStart hook fires on turn 1 only.
/implement covers every ticket in one turn. 5 to 11 compound shell lines per spd run were denied. PO cost is excluded.
Repo ve görevler:
upstream/package.json
{
"name": "ledgerly",
"version": "0.4.0",
"private": true,
"description": "Small invoicing API: customers, invoices, reports. JSON-file store.",
"type": "module",
"engines": {
"node": ">=20"
},
"scripts": {
"test": "vitest run",
"typecheck": "tsc --noEmit"
},
"devDependencies": {
"@types/node": "^20.14.0",
"typescript": "^5.6.0",
"vitest": "^2.1.0"
}
}
upstream/package.json on GitHub
upstream/pnpm-lock.yaml
lockfileVersion: '9.0'
settings:
autoInstallPeers: true
excludeLinksFromLockfile: false
importers:
.:
devDependencies:
'@types/node':
specifier: ^20.14.0
version: 20.19.43
typescript:
specifier: ^5.6.0
version: 5.9.3
vitest:
specifier: ^2.1.0
version: 2.1.9(@types/node@20.19.43)
packages:
'@esbuild/aix-ppc64@0.21.5':
resolution: {integrity: sha512-1SDgH6ZSPTlggy1yI6+Dbkiz8xzpHJEVAlF/AM1tHPLsf5STom9rwtjE4hKAF20FfXXNTFqEYXyJNWh1GiZedQ==}
engines: {node: '>=12'}
cpu: [ppc64]
os: [aix]
'@esbuild/android-arm64@0.21.5':
resolution: {integrity: sha512-c0uX9VAUBQ7dTDCjq+wdyGLowMdtR/GoC2U5IYk/7D1H1JYC0qseD7+11iMP2mRLN9RcCMRcjC4YMclCzGwS/A==}
engines: {node: '>=12'}
cpu: [arm64]
os: [android]
'@esbuild/android-arm@0.21.5':
resolution: {integrity: sha512-vCPvzSjpPHEi1siZdlvAlsPxXl7WbOVUBBAowWug4rJHb68Ox8KualB+1ocNvT5fjv6wpkX6o/iEpbDrf68zcg==}
engines: {node: '>=12'}
cpu: [arm]
os: [android]
'@esbuild/android-x64@0.21.5':
resolution: {integrity: sha512-D7aPRUUNHRBwHxzxRvp856rjUHRFW1SdQATKXH2hqA0kAZb1hKmi02OpYRacl0TxIGz/ZmXWlbZgjwWYaCakTA==}
engines: {node: '>=12'}
cpu: [x64]
os: [android]
'@esbuild/darwin-arm64@0.21.5':
resolution: {integrity: sha512-DwqXqZyuk5AiWWf3UfLiRDJ5EDd49zg6O9wclZ7kUMv2WRFr4HKjXp/5t8JZ11QbQfUS6/cRCKGwYhtNAY88kQ==}
engines: {node: '>=12'}
cpu: [arm64]
os: [darwin]
'@esbuild/darwin-x64@0.21.5':
resolution: {integrity: sha512-se/JjF8NlmKVG4kNIuyWMV/22ZaerB+qaSi5MdrXtd6R08kvs2qCN4C09miupktDitvh8jRFflwGFBQcxZRjbw==}
engines: {node: '>=12'}
cpu: [x64]
os: [darwin]
'@esbuild/freebsd-arm64@0.21.5':
resolution: {integrity: sha512-5JcRxxRDUJLX8JXp/wcBCy3pENnCgBR9bN6JsY4OmhfUtIHe3ZW0mawA7+RDAcMLrMIZaf03NlQiX9DGyB8h4g==}
engines: {node: '>=12'}
cpu: [arm64]
os: [freebsd]
'@esbuild/freebsd-x64@0.21.5':
resolution: {integrity: sha512-J95kNBj1zkbMXtHVH29bBriQygMXqoVQOQYA+ISs0/2l3T9/kj42ow2mpqerRBxDJnmkUDCaQT/dfNXWX/ZZCQ==}
engines: {node: '>=12'}
cpu: [x64]
os: [freebsd]
'@esbuild/linux-arm64@0.21.5':
resolution: {integrity: sha512-ibKvmyYzKsBeX8d8I7MH/TMfWDXBF3db4qM6sy+7re0YXya+K1cem3on9XgdT2EQGMu4hQyZhan7TeQ8XkGp4Q==}
engines: {node: '>=12'}
cpu: [arm64]
os: [linux]
'@esbuild/linux-arm@0.21.5':
resolution: {integrity: sha512-bPb5AHZtbeNGjCKVZ9UGqGwo8EUu4cLq68E95A53KlxAPRmUyYv2D6F0uUI65XisGOL1hBP5mTronbgo+0bFcA==}
engines: {node: '>=12'}
cpu: [arm]
os: [linux]
'@esbuild/linux-ia32@0.21.5':
resolution: {integrity: sha512-YvjXDqLRqPDl2dvRODYmmhz4rPeVKYvppfGYKSNGdyZkA01046pLWyRKKI3ax8fbJoK5QbxblURkwK/MWY18Tg==}
engines: {node: '>=12'}
cpu: [ia32]
os: [linux]
'@esbuild/linux-loong64@0.21.5':
resolution: {integrity: sha512-uHf1BmMG8qEvzdrzAqg2SIG/02+4/DHB6a9Kbya0XDvwDEKCoC8ZRWI5JJvNdUjtciBGFQ5PuBlpEOXQj+JQSg==}
engines: {node: '>=12'}
cpu: [loong64]
os: [linux]
'@esbuild/linux-mips64el@0.21.5':
resolution: {integrity: sha512-IajOmO+KJK23bj52dFSNCMsz1QP1DqM6cwLUv3W1QwyxkyIWecfafnI555fvSGqEKwjMXVLokcV5ygHW5b3Jbg==}
engines: {node: '>=12'}
cpu: [mips64el]
os: [linux]
'@esbuild/linux-ppc64@0.21.5':
resolution: {integrity: sha512-1hHV/Z4OEfMwpLO8rp7CvlhBDnjsC3CttJXIhBi+5Aj5r+MBvy4egg7wCbe//hSsT+RvDAG7s81tAvpL2XAE4w==}
engines: {node: '>=12'}
cpu: [ppc64]
os: [linux]
'@esbuild/linux-riscv64@0.21.5':
resolution: {integrity: sha512-2HdXDMd9GMgTGrPWnJzP2ALSokE/0O5HhTUvWIbD3YdjME8JwvSCnNGBnTThKGEB91OZhzrJ4qIIxk/SBmyDDA==}
engines: {node: '>=12'}
cpu: [riscv64]
os: [linux]
'@esbuild/linux-s390x@0.21.5':
resolution: {integrity: sha512-zus5sxzqBJD3eXxwvjN1yQkRepANgxE9lgOW2qLnmr8ikMTphkjgXu1HR01K4FJg8h1kEEDAqDcZQtbrRnB41A==}
engines: {node: '>=12'}
cpu: [s390x]
os: [linux]
'@esbuild/linux-x64@0.21.5':
resolution: {integrity: sha512-1rYdTpyv03iycF1+BhzrzQJCdOuAOtaqHTWJZCWvijKD2N5Xu0TtVC8/+1faWqcP9iBCWOmjmhoH94dH82BxPQ==}
engines: {node: '>=12'}
cpu: [x64]
os: [linux]
'@esbuild/netbsd-x64@0.21.5':
resolution: {integrity: sha512-Woi2MXzXjMULccIwMnLciyZH4nCIMpWQAs049KEeMvOcNADVxo0UBIQPfSmxB3CWKedngg7sWZdLvLczpe0tLg==}
engines: {node: '>=12'}
cpu: [x64]
os: [netbsd]
'@esbuild/openbsd-x64@0.21.5':
resolution: {integrity: sha512-HLNNw99xsvx12lFBUwoT8EVCsSvRNDVxNpjZ7bPn947b8gJPzeHWyNVhFsaerc0n3TsbOINvRP2byTZ5LKezow==}
engines: {node: '>=12'}
cpu: [x64]
os: [openbsd]
'@esbuild/sunos-x64@0.21.5':
resolution: {integrity: sha512-6+gjmFpfy0BHU5Tpptkuh8+uw3mnrvgs+dSPQXQOv3ekbordwnzTVEb4qnIvQcYXq6gzkyTnoZ9dZG+D4garKg==}
engines: {node: '>=12'}
cpu: [x64]
os: [sunos]
'@esbuild/win32-arm64@0.21.5':
resolution: {integrity: sha512-Z0gOTd75VvXqyq7nsl93zwahcTROgqvuAcYDUr+vOv8uHhNSKROyU961kgtCD1e95IqPKSQKH7tBTslnS3tA8A==}
engines: {node: '>=12'}
cpu: [arm64]
os: [win32]
'@esbuild/win32-ia32@0.21.5':
resolution: {integrity: sha512-SWXFF1CL2RVNMaVs+BBClwtfZSvDgtL//G/smwAc5oVK/UPu2Gu9tIaRgFmYFFKrmg3SyAjSrElf0TiJ1v8fYA==}
engines: {node: '>=12'}
cpu: [ia32]
os: [win32]
'@esbuild/win32-x64@0.21.5':
resolution: {integrity: sha512-tQd/1efJuzPC6rCFwEvLtci/xNFcTZknmXs98FYDfGE4wP9ClFV98nyKrzJKVPMhdDnjzLhdUyMX4PsQAPjwIw==}
engines: {node: '>=12'}
cpu: [x64]
os: [win32]
'@jridgewell/sourcemap-codec@1.6.0':
resolution: {integrity: sha512-T7jf+5zgsZHwNJ4lvQ7/aezbyk0nNX+zJVWpmHA7VYsEx7a7qr5Rg5IbtJFqkgze5Y2sruq1RUY8Q837Od7iFw==}
'@napi-rs/lzma-linux-x64-gnu@1.5.1':
resolution: {integrity: sha512-oTXEIha4SsuXdTA4Iyskj0kpdx2yVXdhd75c2v3xGrHFfVMsbhTPZU/nMPL4sWKo4pBHm3aucLaqGlF696dTyQ==}
engines: {node: ^22.20 || ^24.12 || >=25}
cpu: [x64]
os: [linux]
libc: [glibc]
'@rollup/rollup-android-arm-eabi@4.64.2':
resolution: {integrity: sha512-gjNDUwMswogkw3fNqV00u54uZlgmInuw5EU/1TeqQ6K0bEagoGvqjPdKzy112lKyhe7PkGudZ/kSQHsc9qroIg==}
cpu: [arm]
os: [android]
'@rollup/rollup-android-arm64@4.64.2':
resolution: {integrity: sha512-YkAW8q7p5SCLSaGWaUkmsRnBEb3MGA1pB5nI/8k0lAK5fZsQKRBqkdR1h4UU9p8u3c7ZGdUF2Zcx7mK+E/pUKg==}
cpu: [arm64]
os: [android]
'@rollup/rollup-darwin-arm64@4.64.2':
resolution: {integrity: sha512-hNmuIHv+LnrP/+q/44vsmIMWfoh7X2YXOp5FLtjGuvqN52VdtFpOpnshTc1k28tWv1wE/0D5gLjsozBUowEGbQ==}
cpu: [arm64]
os: [darwin]
'@rollup/rollup-darwin-x64@4.64.2':
resolution: {integrity: sha512-B0t5CoxgF2KL7R5FHAsyN8yVEQCSIkh4XVs8MCDjuBLNjEa8l3zd5tVKaFMv53LfUtHs9p2rnqeavPTOunV+Ww==}
cpu: [x64]
os: [darwin]
'@rollup/rollup-freebsd-arm64@4.64.2':
resolution: {integrity: sha512-QQdYnfwrPmc1DoPlddjFvQPzUHFbStKXsHrG6EBYsWHhBVUCjsCRdcfGGXGAvhDpDtTzKK4vy1UrGFJ45rQaiw==}
cpu: [arm64]
os: [freebsd]
'@rollup/rollup-freebsd-x64@4.64.2':
resolution: {integrity: sha512-1xRxtGWif6BB58VSfMavi0Yfs2M4x6CtRp4o7ScY4W7AGlgvf6GSPKdDNdw7TX3tMiQwZY5W/mvyrOJTq/7AKA==}
cpu: [x64]
os: [freebsd]
'@rollup/rollup-linux-arm-gnueabihf@4.64.2':
resolution: {integrity: sha512-CXFLEt384BUwwOLR9DURWNHl1bQOVLd1ibOee+yKBJm/fZtvrSAOZ0JbxjAxszVMIovYMwFMS+oP+Q1aPcx8Eg==}
cpu: [arm]
os: [linux]
libc: [glibc]
'@rollup/rollup-linux-arm-musleabihf@4.64.2':
resolution: {integrity: sha512-T1dorY0WHm5mMNpMHFl5rYRZ02gLSZcr6J45r66tYb3S3R3DyIRGiGWVdLtF+zOQjratNJaObl9jYumueRiL1A==}
cpu: [arm]
os: [linux]
libc: [musl]
'@rollup/rollup-linux-arm64-gnu@4.64.2':
resolution: {integrity: sha512-ia6bfaCIc0ldp6bvs9oEkFrJVJKNUMoiDF0+LBZsQAZg+vE0znXnzGqH+ew/FYswE30cruAWPTO3dPw7keaVkQ==}
cpu: [arm64]
os: [linux]
libc: [glibc]
'@rollup/rollup-linux-arm64-musl@4.64.2':
resolution: {integrity: sha512-mB/gwfBfXtT0EBpbSssqKCqMFuX8b21Zy2TGoIhehSJEssZ5ksybxymwp1BkEM5Nczh/AAxpmh5q+MkhtundGg==}
cpu: [arm64]
os: [linux]
libc: [musl]
'@rollup/rollup-linux-loong64-gnu@4.64.2':
resolution: {integrity: sha512-VPd0a+XlHoNRI9jIPRHww2UKhDkD6XjbtcKYXZijAnhJS1jLLbiNtVl6a5LyEfNN+eyiyWcJOZXPzBo69oTsew==}
cpu: [loong64]
os: [linux]
libc: [glibc]
'@rollup/rollup-linux-loong64-musl@4.64.2':
resolution: {integrity: sha512-10N1I6oBpERqzGtHKcCLrEpHlbXSBbMK9TURHvqgWbIFZuU3oJROxJ0H2nHfq96TVI0XJHQ9qzolJMFd2tz0LA==}
cpu: [loong64]
os: [linux]
libc: [musl]
'@rollup/rollup-linux-ppc64-gnu@4.64.2':
resolution: {integrity: sha512-dhJrLVZ3wNNhwCmy9/VTg6Pq6J8a8eZ0n15dVRlbSWcDb+8V/1mDfbnvis+vt8W6Pbxzv6gUbTEl4piOT3bbUQ==}
cpu: [ppc64]
os: [linux]
libc: [glibc]
'@rollup/rollup-linux-ppc64-musl@4.64.2':
resolution: {integrity: sha512-tPd2QuwxkTxgju5eARyb3RdgBWWi/O9dn/NFH0VkvGbLzoQeXrH6qMABtW8IODb5lUeBzB3sy2U7qmI4MtQ72g==}
cpu: [ppc64]
os: [linux]
libc: [musl]
'@rollup/rollup-linux-riscv64-gnu@4.64.2':
resolution: {integrity: sha512-njtDdPvSQWI9CDhIPPxsiEMeHayCfu7wwcbSQ7c2kYPhGVviL+WGVhq6Jfj/b+VuhSTkFuQ0z4ZFbFjkWwBFmg==}
cpu: [riscv64]
os: [linux]
libc: [glibc]
'@rollup/rollup-linux-riscv64-musl@4.64.2':
resolution: {integrity: sha512-EbOBo3dqCj9ecEV9Z7zvulB6AieMP2ztnKQYGVOSN0pzIyOu+3JH2YJZnLmbPeVvfn6yP6Qi7NbBM27EzGbj2w==}
cpu: [riscv64]
os: [linux]
libc: [musl]
'@rollup/rollup-linux-s390x-gnu@4.64.2':
resolution: {integrity: sha512-8wLt2Q5n4qmfiE2Gk8V6KNLSf+oJuBX7Kdw5joFDtnZIZvirS0hLzBc79OiJjV71oYDmBUZLNy6pDgvylo4qiQ==}
cpu: [s390x]
os: [linux]
libc: [glibc]
'@rollup/rollup-linux-x64-gnu@4.64.2':
resolution: {integrity: sha512-aIChbkhC4d33bxkVP0LSi2/2lUSGRXE6k4Qa0G7tgpgTOK7kxHTixmx5qC3SKUJ6yUMznHBtQoWaG4QMqLUnhw==}
cpu: [x64]
os: [linux]
libc: [glibc]
'@rollup/rollup-linux-x64-musl@4.64.2':
resolution: {integrity: sha512-gQ+TkVZRfjZVcl/UOAxxfWw+sHjzIXoieNwxEyJlD3tRBmBhR8ZDq11w+QEgQDcpslEB+fhN/OEpaTsDoIIHhQ==}
cpu: [x64]
os: [linux]
libc: [musl]
'@rollup/rollup-openbsd-x64@4.64.2':
resolution: {integrity: sha512-sUaJpMKS2JBCvUk8fYimG7aMq2jHVT4kw/uVHxPolKLrMmkKy0KXirmZyZehdNpL7PffqAlPAWUELQcTiD8ChA==}
cpu: [x64]
os: [openbsd]
'@rollup/rollup-openharmony-arm64@4.64.2':
resolution: {integrity: sha512-j1RdJWb0CrlSuwt12OPjLucPlwfLtA/KMOKVvh5rXSJEtl/WDoIGR3CSykK/jLGTo9RqNfW5Ub6kY5Z5ccYl9w==}
cpu: [arm64]
os: [openharmony]
'@rollup/rollup-win32-arm64-msvc@4.64.2':
resolution: {integrity: sha512-i3r04aHCIY3eggTBLUp/4IxiBjDpAgQaK89Ue2nMJ2MKuIXX5JfoeZjHw+kPjGGNVDlhi8FT8xzuGMJZe3aehg==}
cpu: [arm64]
os: [win32]
'@rollup/rollup-win32-ia32-msvc@4.64.2':
resolution: {integrity: sha512-kCQ15h/uwNMGdN8gjXBZGf+aGE+ODzHfbi90/cAm4+Q89WoMiLODNoj8HoGZmYx87HPxfu5CBNPsZ+ZG4uIAoA==}
cpu: [ia32]
os: [win32]
'@rollup/rollup-win32-x64-gnu@4.64.2':
resolution: {integrity: sha512-yYAuySSs+0X2gSMJPw/Q4jHsHZHyMq1dpAE+XnN790Jg1WuReUAD/+j+e4vuDvS4PXJ4PA4NCzTR4OYtDf18/g==}
cpu: [x64]
os: [win32]
'@rollup/rollup-win32-x64-msvc@4.64.2':
resolution: {integrity: sha512-JY1rQdWOJrKutORqKO1uzrHXV86jFiQzfLFKx2s8S1CjLXL9fC5LOIvk0q7Z7PXJvsByt+sdoE1Q2a9gBr4gLQ==}
cpu: [x64]
os: [win32]
'@types/estree@1.0.9':
resolution: {integrity: sha512-GhdPgy1el4/ImP05X05Uw4cw2/M93BCUmnEvWZNStlCzEKME4Fkk+YpoA5OiHNQmoS7Cafb8Xa3Pya8m1Qrzeg==}
'@types/node@20.19.43':
resolution: {integrity: sha512-6oYBAi5ikg4Pl+kGsoYtawUMBT2zZMCvPNF7pVLnHZfd1zf38DRiWn/gT01RYCdUqkv7Fhr+C9ot4/tb+2sVvA==}
'@vitest/expect@2.1.9':
resolution: {integrity: sha512-UJCIkTBenHeKT1TTlKMJWy1laZewsRIzYighyYiJKZreqtdxSos/S1t+ktRMQWu2CKqaarrkeszJx1cgC5tGZw==}
'@vitest/mocker@2.1.9':
resolution: {integrity: sha512-tVL6uJgoUdi6icpxmdrn5YNo3g3Dxv+IHJBr0GXHaEdTcw3F+cPKnsXFhli6nO+f/6SDKPHEK1UN+k+TQv0Ehg==}
peerDependencies:
msw: ^2.4.9
vite: ^5.0.0
peerDependenciesMeta:
msw:
optional: true
vite:
optional: true
'@vitest/pretty-format@2.1.9':
resolution: {integrity: sha512-KhRIdGV2U9HOUzxfiHmY8IFHTdqtOhIzCpd8WRdJiE7D/HUcZVD0EgQCVjm+Q9gkUXWgBvMmTtZgIG48wq7sOQ==}
'@vitest/runner@2.1.9':
resolution: {integrity: sha512-ZXSSqTFIrzduD63btIfEyOmNcBmQvgOVsPNPe0jYtESiXkhd8u2erDLnMxmGrDCwHCCHE7hxwRDCT3pt0esT4g==}
'@vitest/snapshot@2.1.9':
resolution: {integrity: sha512-oBO82rEjsxLNJincVhLhaxxZdEtV0EFHMK5Kmx5sJ6H9L183dHECjiefOAdnqpIgT5eZwT04PoggUnW88vOBNQ==}
'@vitest/spy@2.1.9':
resolution: {integrity: sha512-E1B35FwzXXTs9FHNK6bDszs7mtydNi5MIfUWpceJ8Xbfb1gBMscAnwLbEu+B44ed6W3XjL9/ehLPHR1fkf1KLQ==}
'@vitest/utils@2.1.9':
resolution: {integrity: sha512-v0psaMSkNJ3A2NMrUEHFRzJtDPFn+/VWZ5WxImB21T9fjucJRmS7xCS3ppEnARb9y11OAzaD+P2Ps+b+BGX5iQ==}
assertion-error@2.0.1:
resolution: {integrity: sha512-Izi8RQcffqCeNVgFigKli1ssklIbpHnCYc6AknXGYoB6grJqyeby7jv12JUQgmTAnIDnbck1uxksT4dzN3PWBA==}
engines: {node: '>=12'}
cac@6.7.14:
resolution: {integrity: sha512-b6Ilus+c3RrdDk+JhLKUAQfzzgLEPy6wcXqS7f/xe1EETvsDP6GORG7SFuOs6cID5YkqchW/LXZbX5bc8j7ZcQ==}
engines: {node: '>=8'}
chai@5.3.3:
resolution: {integrity: sha512-4zNhdJD/iOjSH0A05ea+Ke6MU5mmpQcbQsSOkgdaUMJ9zTlDTD/GYlwohmIE2u0gaxHYiVHEn1Fw9mZ/ktJWgw==}
engines: {node: '>=18'}
check-error@2.1.3:
resolution: {integrity: sha512-PAJdDJusoxnwm1VwW07VWwUN1sl7smmC3OKggvndJFadxxDRyFJBX/ggnu/KE4kQAB7a3Dp8f/YXC1FlUprWmA==}
engines: {node: '>= 16'}
debug@4.4.3:
resolution: {integrity: sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA==}
engines: {node: '>=6.0'}
peerDependencies:
supports-color: '*'
peerDependenciesMeta:
supports-color:
optional: true
deep-eql@5.0.2:
resolution: {integrity: sha512-h5k/5U50IJJFpzfL6nO9jaaumfjO/f2NjK/oYB2Djzm4p9L+3T9qWpZqZ2hAbLPuuYq9wrU08WQyBTL5GbPk5Q==}
engines: {node: '>=6'}
es-module-lexer@1.7.0:
resolution: {integrity: sha512-jEQoCwk8hyb2AZziIOLhDqpm5+2ww5uIE6lkO/6jcOCusfk6LhMHpXXfBLXTZ7Ydyt0j4VoUQv6uGNYbdW+kBA==}
esbuild@0.21.5:
resolution: {integrity: sha512-mg3OPMV4hXywwpoDxu3Qda5xCKQi+vCTZq8S9J/EpkhB2HzKXq4SNFZE3+NK93JYxc8VMSep+lOUSC/RVKaBqw==}
engines: {node: '>=12'}
hasBin: true
estree-walker@3.0.3:
resolution: {integrity: sha512-7RUKfXgSMMkzt6ZuXmqapOurLGPPfgj6l9uRZ7lRGolvk0y2yocc35LdcxKC5PQZdn2DMqioAQ2NoWcrTKmm6g==}
expect-type@1.4.0:
resolution: {integrity: sha512-KfYbmpRm0VbLjEvVa9yGwCi9GI34xvi7A/HXYWQO65CSD2u3MczUJSuwXKFIxlGsgBQizV9q5J9NHj4VG0n+pA==}
engines: {node: '>=12.0.0'}
fsevents@2.3.3:
resolution: {integrity: sha512-5xoDfX+fL7faATnagmWPpbFtwh/R77WmMMqqHGS65C3vvB0YHrgF+B1YmZ3441tMj5n63k0212XNoJwzlhffQw==}
engines: {node: ^8.16.0 || ^10.6.0 || >=11.0.0}
os: [darwin]
loupe@3.2.1:
resolution: {integrity: sha512-CdzqowRJCeLU72bHvWqwRBBlLcMEtIvGrlvef74kMnV2AolS9Y8xUv1I0U/MNAWMhBlKIoyuEgoJ0t/bbwHbLQ==}
magic-string@0.30.21:
resolution: {integrity: sha512-vd2F4YUyEXKGcLHoq+TEyCjxueSeHnFxyyjNp80yg0XV4vUhnDer/lvvlqM/arB5bXQN5K2/3oinyCRyx8T2CQ==}
ms@2.1.3:
resolution: {integrity: sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==}
nanoid@3.3.20:
resolution: {integrity: sha512-uKdg2G3GNCKQn9byYOpxbGqrT2fGO5KRt5J/8b3pok8rT6qxGWF6hxMyJiEYtAf+FVyYuD9hRaDqX5uPFYJ4ZQ==}
engines: {node: ^10 || ^12 || ^13.7 || ^14 || >=15.0.1}
hasBin: true
pathe@1.1.2:
resolution: {integrity: sha512-whLdWMYL2TwI08hn8/ZqAbrVemu0LNaNNJZX73O6qaIdCTfXutsLhMkjdENX0qhsQ9uIimo4/aQOmXkoon2nDQ==}
pathval@2.0.1:
resolution: {integrity: sha512-//nshmD55c46FuFw26xV/xFAaB5HF9Xdap7HJBBnrKdAd6/GxDBaNA1870O79+9ueg61cZLSVc+OaFlfmObYVQ==}
engines: {node: '>= 14.16'}
picocolors@1.1.1:
resolution: {integrity: sha512-xceH2snhtb5M9liqDsmEw56le376mTZkEX/jEb/RxNFyegNul7eNslCXP9FDj/Lcu0X8KEyMceP2ntpaHrDEVA==}
postcss@8.5.29:
resolution: {integrity: sha512-49cGhUbXj8Qenv0iTMxA1cFBzxXoctpC9Ujd77t1WcbJIr6nF/eI7g/8MgxrYldFRuAXvja7xQRwavoW7kgrxQ==}
engines: {node: ^10 || ^12 || >=14}
rollup@4.64.2:
resolution: {integrity: sha512-RxXm0XVVV1hFI3dYe12ruaOgsYQ725lUcUWeMSQqkF9tTxVErY+fy2Ets6vWhItL9MZMSNZGbvZFc7rH/VHp2Q==}
engines: {node: '>=18.0.0', npm: '>=8.0.0'}
hasBin: true
siginfo@2.0.0:
resolution: {integrity: sha512-ybx0WO1/8bSBLEWXZvEd7gMW3Sn3JFlW3TvX1nREbDLRNQNaeNN8WK0meBwPdAaOI7TtRRRJn/Es1zhrrCHu7g==}
source-map-js@1.2.2:
resolution: {integrity: sha512-KGj/8Y43x35aZVDtt+J4mK1hoLGHULMYfSkODJNQjNDC3oW1PqPoxMwo0pLUsWM/UEGzON/NxeHywEfNXNP3Vw==}
engines: {node: '>=0.10.0'}
stackback@0.0.2:
resolution: {integrity: sha512-1XMJE5fQo1jGH6Y/7ebnwPOBEkIEnT4QF32d5R1+VXdXveM0IBMJt8zfaxX1P3QhVwrYe+576+jkANtSS2mBbw==}
std-env@3.10.0:
resolution: {integrity: sha512-5GS12FdOZNliM5mAOxFRg7Ir0pWz8MdpYm6AY6VPkGpbA7ZzmbzNcBJQ0GPvvyWgcY7QAhCgf9Uy89I03faLkg==}
tinybench@2.9.0:
resolution: {integrity: sha512-0+DUvqWMValLmha6lr4kD8iAMK1HzV0/aKnCtWb9v9641TnP/MFb7Pc2bxoxQjTXAErryXVgUOfv2YqNllqGeg==}
tinyexec@0.3.2:
resolution: {integrity: sha512-KQQR9yN7R5+OSwaK0XQoj22pwHoTlgYqmUscPYoknOoWCWfj/5/ABTMRi69FrKU5ffPVh5QcFikpWJI/P1ocHA==}
tinypool@1.1.1:
resolution: {integrity: sha512-Zba82s87IFq9A9XmjiX5uZA/ARWDrB03OHlq+Vw1fSdt0I+4/Kutwy8BP4Y/y/aORMo61FQ0vIb5j44vSo5Pkg==}
engines: {node: ^18.0.0 || >=20.0.0}
tinyrainbow@1.2.0:
resolution: {integrity: sha512-weEDEq7Z5eTHPDh4xjX789+fHfF+P8boiFB+0vbWzpbnbsEr/GRaohi/uMKxg8RZMXnl1ItAi/IUHWMsjDV7kQ==}
engines: {node: '>=14.0.0'}
tinyspy@3.0.2:
resolution: {integrity: sha512-n1cw8k1k0x4pgA2+9XrOkFydTerNcJ1zWCO5Nn9scWHTD+5tp8dghT2x1uduQePZTZgd3Tupf+x9BxJjeJi77Q==}
engines: {node: '>=14.0.0'}
typescript@5.9.3:
resolution: {integrity: sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw==}
engines: {node: '>=14.17'}
hasBin: true
undici-types@6.21.0:
resolution: {integrity: sha512-iwDZqg0QAGrg9Rav5H4n0M64c3mkR59cJ6wQp+7C4nI0gsmExaedaYLNO44eT4AtBBwjbTiGPMlt2Md0T9H9JQ==}
vite-node@2.1.9:
resolution: {integrity: sha512-AM9aQ/IPrW/6ENLQg3AGY4K1N2TGZdR5e4gu/MmmR2xR3Ll1+dib+nook92g4TV3PXVyeyxdWwtaCAiUL0hMxA==}
engines: {node: ^18.0.0 || >=20.0.0}
hasBin: true
vite@5.4.21:
resolution: {integrity: sha512-o5a9xKjbtuhY6Bi5S3+HvbRERmouabWbyUcpXXUA1u+GNUKoROi9byOJ8M0nHbHYHkYICiMlqxkg1KkYmm25Sw==}
engines: {node: ^18.0.0 || >=20.0.0}
hasBin: true
peerDependencies:
'@types/node': ^18.0.0 || >=20.0.0
less: '*'
lightningcss: ^1.21.0
sass: '*'
sass-embedded: '*'
stylus: '*'
sugarss: '*'
terser: ^5.4.0
peerDependenciesMeta:
'@types/node':
optional: true
less:
optional: true
lightningcss:
optional: true
sass:
optional: true
sass-embedded:
optional: true
stylus:
optional: true
sugarss:
optional: true
terser:
optional: true
vitest@2.1.9:
resolution: {integrity: sha512-MSmPM9REYqDGBI8439mA4mWhV5sKmDlBKWIYbA3lRb2PTHACE0mgKwA8yQ2xq9vxDTuk4iPrECBAEW2aoFXY0Q==}
engines: {node: ^18.0.0 || >=20.0.0}
hasBin: true
peerDependencies:
'@edge-runtime/vm': '*'
'@types/node': ^18.0.0 || >=20.0.0
'@vitest/browser': 2.1.9
'@vitest/ui': 2.1.9
happy-dom: '*'
jsdom: '*'
peerDependenciesMeta:
'@edge-runtime/vm':
optional: true
'@types/node':
optional: true
'@vitest/browser':
optional: true
'@vitest/ui':
optional: true
happy-dom:
optional: true
jsdom:
optional: true
why-is-node-running@2.3.0:
resolution: {integrity: sha512-hUrmaWBdVDcxvYqnyh09zunKzROWjbZTiNy8dBEjkS7ehEDQibXJ7XvlmtbwuTclUiIyN+CyXQD4Vmko8fNm8w==}
engines: {node: '>=8'}
hasBin: true
snapshots:
'@esbuild/aix-ppc64@0.21.5':
optional: true
'@esbuild/android-arm64@0.21.5':
optional: true
'@esbuild/android-arm@0.21.5':
optional: true
'@esbuild/android-x64@0.21.5':
optional: true
'@esbuild/darwin-arm64@0.21.5':
optional: true
'@esbuild/darwin-x64@0.21.5':
optional: true
'@esbuild/freebsd-arm64@0.21.5':
optional: true
'@esbuild/freebsd-x64@0.21.5':
optional: true
'@esbuild/linux-arm64@0.21.5':
optional: true
'@esbuild/linux-arm@0.21.5':
optional: true
'@esbuild/linux-ia32@0.21.5':
optional: true
'@esbuild/linux-loong64@0.21.5':
optional: true
'@esbuild/linux-mips64el@0.21.5':
optional: true
'@esbuild/linux-ppc64@0.21.5':
optional: true
'@esbuild/linux-riscv64@0.21.5':
optional: true
'@esbuild/linux-s390x@0.21.5':
optional: true
'@esbuild/linux-x64@0.21.5':
optional: true
'@esbuild/netbsd-x64@0.21.5':
optional: true
'@esbuild/openbsd-x64@0.21.5':
optional: true
'@esbuild/sunos-x64@0.21.5':
optional: true
'@esbuild/win32-arm64@0.21.5':
optional: true
'@esbuild/win32-ia32@0.21.5':
optional: true
'@esbuild/win32-x64@0.21.5':
optional: true
'@jridgewell/sourcemap-codec@1.6.0': {}
'@napi-rs/lzma-linux-x64-gnu@1.5.1':
optional: true
'@rollup/rollup-android-arm-eabi@4.64.2':
optional: true
'@rollup/rollup-android-arm64@4.64.2':
optional: true
'@rollup/rollup-darwin-arm64@4.64.2':
optional: true
'@rollup/rollup-darwin-x64@4.64.2':
optional: true
'@rollup/rollup-freebsd-arm64@4.64.2':
optional: true
'@rollup/rollup-freebsd-x64@4.64.2':
optional: true
'@rollup/rollup-linux-arm-gnueabihf@4.64.2':
optional: true
'@rollup/rollup-linux-arm-musleabihf@4.64.2':
optional: true
'@rollup/rollup-linux-arm64-gnu@4.64.2':
optional: true
'@rollup/rollup-linux-arm64-musl@4.64.2':
optional: true
'@rollup/rollup-linux-loong64-gnu@4.64.2':
optional: true
'@rollup/rollup-linux-loong64-musl@4.64.2':
optional: true
'@rollup/rollup-linux-ppc64-gnu@4.64.2':
optional: true
'@rollup/rollup-linux-ppc64-musl@4.64.2':
optional: true
'@rollup/rollup-linux-riscv64-gnu@4.64.2':
optional: true
'@rollup/rollup-linux-riscv64-musl@4.64.2':
optional: true
'@rollup/rollup-linux-s390x-gnu@4.64.2':
optional: true
'@rollup/rollup-linux-x64-gnu@4.64.2':
optional: true
'@rollup/rollup-linux-x64-musl@4.64.2':
optional: true
'@rollup/rollup-openbsd-x64@4.64.2':
optional: true
'@rollup/rollup-openharmony-arm64@4.64.2':
optional: true
'@rollup/rollup-win32-arm64-msvc@4.64.2':
optional: true
'@rollup/rollup-win32-ia32-msvc@4.64.2':
optional: true
'@rollup/rollup-win32-x64-gnu@4.64.2':
optional: true
'@rollup/rollup-win32-x64-msvc@4.64.2':
optional: true
'@types/estree@1.0.9': {}
'@types/node@20.19.43':
dependencies:
undici-types: 6.21.0
'@vitest/expect@2.1.9':
dependencies:
'@vitest/spy': 2.1.9
'@vitest/utils': 2.1.9
chai: 5.3.3
tinyrainbow: 1.2.0
'@vitest/mocker@2.1.9(vite@5.4.21(@types/node@20.19.43))':
dependencies:
'@vitest/spy': 2.1.9
estree-walker: 3.0.3
magic-string: 0.30.21
optionalDependencies:
vite: 5.4.21(@types/node@20.19.43)
'@vitest/pretty-format@2.1.9':
dependencies:
tinyrainbow: 1.2.0
'@vitest/runner@2.1.9':
dependencies:
'@vitest/utils': 2.1.9
pathe: 1.1.2
'@vitest/snapshot@2.1.9':
dependencies:
'@vitest/pretty-format': 2.1.9
magic-string: 0.30.21
pathe: 1.1.2
'@vitest/spy@2.1.9':
dependencies:
tinyspy: 3.0.2
'@vitest/utils@2.1.9':
dependencies:
'@vitest/pretty-format': 2.1.9
loupe: 3.2.1
tinyrainbow: 1.2.0
assertion-error@2.0.1: {}
cac@6.7.14: {}
chai@5.3.3:
dependencies:
assertion-error: 2.0.1
check-error: 2.1.3
deep-eql: 5.0.2
loupe: 3.2.1
pathval: 2.0.1
check-error@2.1.3: {}
debug@4.4.3:
dependencies:
ms: 2.1.3
deep-eql@5.0.2: {}
es-module-lexer@1.7.0: {}
esbuild@0.21.5:
optionalDependencies:
'@esbuild/aix-ppc64': 0.21.5
'@esbuild/android-arm': 0.21.5
'@esbuild/android-arm64': 0.21.5
'@esbuild/android-x64': 0.21.5
'@esbuild/darwin-arm64': 0.21.5
'@esbuild/darwin-x64': 0.21.5
'@esbuild/freebsd-arm64': 0.21.5
'@esbuild/freebsd-x64': 0.21.5
'@esbuild/linux-arm': 0.21.5
'@esbuild/linux-arm64': 0.21.5
'@esbuild/linux-ia32': 0.21.5
'@esbuild/linux-loong64': 0.21.5
'@esbuild/linux-mips64el': 0.21.5
'@esbuild/linux-ppc64': 0.21.5
'@esbuild/linux-riscv64': 0.21.5
'@esbuild/linux-s390x': 0.21.5
'@esbuild/linux-x64': 0.21.5
'@esbuild/netbsd-x64': 0.21.5
'@esbuild/openbsd-x64': 0.21.5
'@esbuild/sunos-x64': 0.21.5
'@esbuild/win32-arm64': 0.21.5
'@esbuild/win32-ia32': 0.21.5
'@esbuild/win32-x64': 0.21.5
estree-walker@3.0.3:
dependencies:
'@types/estree': 1.0.9
expect-type@1.4.0: {}
fsevents@2.3.3:
optional: true
loupe@3.2.1: {}
magic-string@0.30.21:
dependencies:
'@jridgewell/sourcemap-codec': 1.6.0
ms@2.1.3: {}
nanoid@3.3.20: {}
pathe@1.1.2: {}
pathval@2.0.1: {}
picocolors@1.1.1: {}
postcss@8.5.29:
dependencies:
nanoid: 3.3.20
picocolors: 1.1.1
source-map-js: 1.2.2
rollup@4.64.2:
dependencies:
'@types/estree': 1.0.9
optionalDependencies:
'@napi-rs/lzma-linux-x64-gnu': 1.5.1
'@rollup/rollup-android-arm-eabi': 4.64.2
'@rollup/rollup-android-arm64': 4.64.2
'@rollup/rollup-darwin-arm64': 4.64.2
'@rollup/rollup-darwin-x64': 4.64.2
'@rollup/rollup-freebsd-arm64': 4.64.2
'@rollup/rollup-freebsd-x64': 4.64.2
'@rollup/rollup-linux-arm-gnueabihf': 4.64.2
'@rollup/rollup-linux-arm-musleabihf': 4.64.2
'@rollup/rollup-linux-arm64-gnu': 4.64.2
'@rollup/rollup-linux-arm64-musl': 4.64.2
'@rollup/rollup-linux-loong64-gnu': 4.64.2
'@rollup/rollup-linux-loong64-musl': 4.64.2
'@rollup/rollup-linux-ppc64-gnu': 4.64.2
'@rollup/rollup-linux-ppc64-musl': 4.64.2
'@rollup/rollup-linux-riscv64-gnu': 4.64.2
'@rollup/rollup-linux-riscv64-musl': 4.64.2
'@rollup/rollup-linux-s390x-gnu': 4.64.2
'@rollup/rollup-linux-x64-gnu': 4.64.2
'@rollup/rollup-linux-x64-musl': 4.64.2
'@rollup/rollup-openbsd-x64': 4.64.2
'@rollup/rollup-openharmony-arm64': 4.64.2
'@rollup/rollup-win32-arm64-msvc': 4.64.2
'@rollup/rollup-win32-ia32-msvc': 4.64.2
'@rollup/rollup-win32-x64-gnu': 4.64.2
'@rollup/rollup-win32-x64-msvc': 4.64.2
fsevents: 2.3.3
siginfo@2.0.0: {}
source-map-js@1.2.2: {}
stackback@0.0.2: {}
std-env@3.10.0: {}
tinybench@2.9.0: {}
tinyexec@0.3.2: {}
tinypool@1.1.1: {}
tinyrainbow@1.2.0: {}
tinyspy@3.0.2: {}
typescript@5.9.3: {}
undici-types@6.21.0: {}
vite-node@2.1.9(@types/node@20.19.43):
dependencies:
cac: 6.7.14
debug: 4.4.3
es-module-lexer: 1.7.0
pathe: 1.1.2
vite: 5.4.21(@types/node@20.19.43)
transitivePeerDependencies:
- '@types/node'
- less
- lightningcss
- sass
- sass-embedded
- stylus
- sugarss
- supports-color
- terser
vite@5.4.21(@types/node@20.19.43):
dependencies:
esbuild: 0.21.5
postcss: 8.5.29
rollup: 4.64.2
optionalDependencies:
'@types/node': 20.19.43
fsevents: 2.3.3
vitest@2.1.9(@types/node@20.19.43):
dependencies:
'@vitest/expect': 2.1.9
'@vitest/mocker': 2.1.9(vite@5.4.21(@types/node@20.19.43))
'@vitest/pretty-format': 2.1.9
'@vitest/runner': 2.1.9
'@vitest/snapshot': 2.1.9
'@vitest/spy': 2.1.9
'@vitest/utils': 2.1.9
chai: 5.3.3
debug: 4.4.3
expect-type: 1.4.0
magic-string: 0.30.21
pathe: 1.1.2
std-env: 3.10.0
tinybench: 2.9.0
tinyexec: 0.3.2
tinypool: 1.1.1
tinyrainbow: 1.2.0
vite: 5.4.21(@types/node@20.19.43)
vite-node: 2.1.9(@types/node@20.19.43)
why-is-node-running: 2.3.0
optionalDependencies:
'@types/node': 20.19.43
transitivePeerDependencies:
- less
- lightningcss
- msw
- sass
- sass-embedded
- stylus
- sugarss
- supports-color
- terser
why-is-node-running@2.3.0:
dependencies:
siginfo: 2.0.0
stackback: 0.0.2
upstream/pnpm-lock.yaml on GitHub
upstream/pnpm-workspace.yaml
allowBuilds:
esbuild: true
upstream/pnpm-workspace.yaml on GitHub
upstream/README.md
# ledgerly
Small invoicing API used by the back office: customers, invoices, an outstanding-balance report.
Plain Node 20 + TypeScript, no framework. Data lives in one JSON file (`LEDGERLY_DATA`, default `data/ledgerly.json`).
```bash
pnpm install
pnpm test # vitest
pnpm typecheck # tsc --noEmit
```
## Layout
- `src/domain/` - invoices, customers, money, validation, domain errors. No I/O.
- `src/store/` - the JSON-file store, repositories, and the schema migrations applied on load.
- `src/http/` - a tiny router, request handlers, error-to-status mapping. `createApp()` returns a
`handle(method, path, body)` function that the tests call directly; `src/server.ts` binds it to `node:http`.
## Conventions
- Money is always an integer number of cents. Never floats.
- Domain functions return new objects; they never mutate their input.
- Validation failures throw `ValidationError` (HTTP 400), missing records `NotFoundError` (404),
illegal state transitions `ConflictError` (409).
- The stored file carries a `schemaVersion`. Any change to the stored shape bumps it and adds a
migration in `src/store/migrations.ts`: production files are migrated on load, never by hand.
## Invoice lifecycle
`draft` → (`issue`) → `open` → (`pay`) → `paid`; `draft`/`open` → (`void`) → `void`.
upstream/src/domain/customer.ts
import { ValidationError } from "./errors";
export type Customer = {
id: string;
name: string;
email: string;
createdAt: string;
};
export type NewCustomerInput = { name: string; email: string };
const EMAIL = /^[^\s@]+@[^\s@]+\.[^\s@]+$/;
export function parseNewCustomer(body: unknown): NewCustomerInput {
if (typeof body !== "object" || body === null) {
throw new ValidationError("body must be an object");
}
const { name, email } = body as Record<string, unknown>;
if (typeof name !== "string" || name.trim() === "") {
throw new ValidationError("name is required", "name");
}
if (typeof email !== "string" || !EMAIL.test(email)) {
throw new ValidationError("email is invalid", "email");
}
return { name: name.trim(), email: email.toLowerCase() };
}
upstream/src/domain/customer.ts on GitHub
upstream/src/domain/errors.ts
export class DomainError extends Error {
constructor(message: string) {
super(message);
this.name = new.target.name;
}
}
/** Input that can never be valid (bad shape, out of range). Maps to HTTP 400. */
export class ValidationError extends DomainError {
constructor(
message: string,
readonly field?: string,
) {
super(message);
}
}
/** A record that does not exist. Maps to HTTP 404. */
export class NotFoundError extends DomainError {
constructor(entity: string, id: string) {
super(`${entity} ${id} not found`);
}
}
/** A valid request that the current state forbids (e.g. paying a void invoice). Maps to HTTP 409. */
export class ConflictError extends DomainError {}
upstream/src/domain/errors.ts on GitHub
upstream/src/domain/invoice.ts
import { ConflictError } from "./errors";
import { type Cents, type Currency, sum } from "./money";
export type InvoiceStatus = "draft" | "open" | "paid" | "void";
export type LineItem = {
description: string;
quantity: number;
/** cents */
unitPrice: Cents;
};
export type Invoice = {
id: string;
/** Assigned when the invoice is issued (INV-0001, ...). */
number: string | null;
customerId: string;
currency: Currency;
lines: LineItem[];
/** Flat discount in cents, never above the subtotal. */
discount: Cents;
status: InvoiceStatus;
createdAt: string;
issuedAt: string | null;
dueDate: string | null;
paidAt: string | null;
voidedAt: string | null;
};
export type InvoiceTotals = { subtotal: Cents; discount: Cents; total: Cents };
export function lineTotal(line: LineItem): Cents {
return line.quantity * line.unitPrice;
}
export function computeTotals(invoice: Pick<Invoice, "lines" | "discount">): InvoiceTotals {
const subtotal = sum(invoice.lines.map(lineTotal));
return { subtotal, discount: invoice.discount, total: subtotal - invoice.discount };
}
export function issue(invoice: Invoice, number: string, now: Date): Invoice {
if (invoice.status !== "draft") {
throw new ConflictError(`invoice ${invoice.id} is ${invoice.status}, only drafts can be issued`);
}
return { ...invoice, status: "open", number, issuedAt: now.toISOString() };
}
export function markPaid(invoice: Invoice, now: Date): Invoice {
if (invoice.status !== "open") {
throw new ConflictError(`invoice ${invoice.id} is ${invoice.status}, only open invoices can be paid`);
}
return { ...invoice, status: "paid", paidAt: now.toISOString() };
}
export function voidInvoice(invoice: Invoice, now: Date): Invoice {
if (invoice.status === "paid" || invoice.status === "void") {
throw new ConflictError(`invoice ${invoice.id} is ${invoice.status} and cannot be voided`);
}
return { ...invoice, status: "void", voidedAt: now.toISOString() };
}
export function isOverdue(invoice: Invoice, now: Date): boolean {
return invoice.status === "open" && invoice.dueDate !== null && invoice.dueDate < now.toISOString().slice(0, 10);
}
upstream/src/domain/invoice.ts on GitHub
upstream/src/domain/money.ts
import { ValidationError } from "./errors";
export const CURRENCIES = ["EUR", "USD", "GBP"] as const;
export type Currency = (typeof CURRENCIES)[number];
/** Amounts are integer cents. */
export type Cents = number;
export function isCurrency(value: unknown): value is Currency {
return typeof value === "string" && (CURRENCIES as readonly string[]).includes(value);
}
export function assertCents(value: unknown, field: string, opts: { min?: number } = {}): Cents {
if (typeof value !== "number" || !Number.isInteger(value)) {
throw new ValidationError(`${field} must be an integer number of cents`, field);
}
const min = opts.min ?? 0;
if (value < min) {
throw new ValidationError(`${field} must be >= ${min}`, field);
}
return value;
}
export function formatCents(amount: Cents, currency: Currency): string {
const sign = amount < 0 ? "-" : "";
const abs = Math.abs(amount);
return `${sign}${Math.floor(abs / 100)}.${String(abs % 100).padStart(2, "0")} ${currency}`;
}
export function sum(amounts: Cents[]): Cents {
return amounts.reduce((acc, a) => acc + a, 0);
}
upstream/src/domain/money.ts on GitHub
upstream/src/domain/validation.ts
import { ValidationError } from "./errors";
import { computeTotals, type LineItem } from "./invoice";
import { assertCents, type Currency, isCurrency } from "./money";
export type NewInvoiceInput = {
customerId: string;
currency: Currency;
lines: LineItem[];
discount: number;
dueDate: string | null;
};
const DATE = /^\d{4}-\d{2}-\d{2}$/;
export function parseLineItems(value: unknown): LineItem[] {
if (!Array.isArray(value) || value.length === 0) {
throw new ValidationError("lines must be a non-empty array", "lines");
}
return value.map((raw, i) => {
if (typeof raw !== "object" || raw === null) {
throw new ValidationError(`lines[${i}] must be an object`, `lines[${i}]`);
}
const { description, quantity, unitPrice } = raw as Record<string, unknown>;
if (typeof description !== "string" || description.trim() === "") {
throw new ValidationError(`lines[${i}].description is required`, `lines[${i}].description`);
}
if (typeof quantity !== "number" || !Number.isInteger(quantity) || quantity < 1) {
throw new ValidationError(`lines[${i}].quantity must be a positive integer`, `lines[${i}].quantity`);
}
return { description: description.trim(), quantity, unitPrice: assertCents(unitPrice, `lines[${i}].unitPrice`) };
});
}
export function parseNewInvoice(body: unknown): NewInvoiceInput {
if (typeof body !== "object" || body === null) {
throw new ValidationError("body must be an object");
}
const b = body as Record<string, unknown>;
if (typeof b.customerId !== "string" || b.customerId === "") {
throw new ValidationError("customerId is required", "customerId");
}
if (!isCurrency(b.currency)) {
throw new ValidationError("currency must be one of EUR, USD, GBP", "currency");
}
const lines = parseLineItems(b.lines);
const discount = b.discount === undefined ? 0 : assertCents(b.discount, "discount");
const { subtotal } = computeTotals({ lines, discount: 0 });
if (discount > subtotal) {
throw new ValidationError("discount cannot exceed the subtotal", "discount");
}
let dueDate: string | null = null;
if (b.dueDate !== undefined && b.dueDate !== null) {
if (typeof b.dueDate !== "string" || !DATE.test(b.dueDate)) {
throw new ValidationError("dueDate must be YYYY-MM-DD", "dueDate");
}
dueDate = b.dueDate;
}
return { customerId: b.customerId, currency: b.currency, lines, discount, dueDate };
}
upstream/src/domain/validation.ts on GitHub
upstream/src/http/app.ts
import { openStore, type Store } from "../store/json-store";
import { createContext } from "./context";
import { errorResponse } from "./errors";
import { customerRoutes } from "./handlers/customers";
import { invoiceRoutes } from "./handlers/invoices";
import { reportRoutes } from "./handlers/reports";
import { type Response, Router } from "./router";
export type App = {
handle(method: string, url: string, body?: unknown): Promise<Response>;
};
export function createApp(opts: { store?: Store; now?: () => Date } = {}): App {
const ctx = createContext(opts.store ?? openStore(), opts.now);
const router = new Router();
customerRoutes(router, ctx);
invoiceRoutes(router, ctx);
reportRoutes(router, ctx);
return {
async handle(method, url, body) {
const parsed = new URL(url, "http://local");
const match = router.match(method, parsed.pathname);
if (!match) return { status: 404, body: { error: { code: "not_found", message: `no route ${method} ${parsed.pathname}` } } };
try {
return await match.handler({
method: method as "GET",
path: parsed.pathname,
params: match.params,
query: parsed.searchParams,
body,
});
} catch (err) {
return errorResponse(err);
}
},
};
}
upstream/src/http/app.ts on GitHub
upstream/src/http/context.ts
import type { Store } from "../store/json-store";
import { customerRepo } from "../store/customer-repo";
import { invoiceRepo } from "../store/invoice-repo";
export type AppContext = {
store: Store;
now: () => Date;
customers: ReturnType<typeof customerRepo>;
invoices: ReturnType<typeof invoiceRepo>;
};
export function createContext(store: Store, now: () => Date = () => new Date()): AppContext {
return { store, now, customers: customerRepo(store), invoices: invoiceRepo(store) };
}
upstream/src/http/context.ts on GitHub
upstream/src/http/errors.ts
import { ConflictError, NotFoundError, ValidationError } from "../domain/errors";
import type { Response } from "./router";
export function errorResponse(err: unknown): Response {
if (err instanceof ValidationError) {
return { status: 400, body: { error: { code: "validation_error", message: err.message, field: err.field } } };
}
if (err instanceof NotFoundError) {
return { status: 404, body: { error: { code: "not_found", message: err.message } } };
}
if (err instanceof ConflictError) {
return { status: 409, body: { error: { code: "conflict", message: err.message } } };
}
return { status: 500, body: { error: { code: "internal", message: "internal error" } } };
}
upstream/src/http/errors.ts on GitHub
upstream/src/http/handlers/customers.ts
import { parseNewCustomer } from "../../domain/customer";
import type { Router } from "../router";
import type { AppContext } from "../context";
export function customerRoutes(router: Router, ctx: AppContext): void {
router.on("POST", "/customers", ({ body }) => {
const input = parseNewCustomer(body);
const id = `cus_${ctx.store.nextId("customer")}`;
const customer = ctx.customers.insert({ id, ...input, createdAt: ctx.now().toISOString() });
return { status: 201, body: customer };
});
router.on("GET", "/customers/:id", ({ params }) => ({ status: 200, body: ctx.customers.get(params.id) }));
}
upstream/src/http/handlers/customers.ts on GitHub
upstream/src/http/handlers/invoices.ts
import { ValidationError } from "../../domain/errors";
import { type Invoice, type InvoiceStatus, issue, markPaid, voidInvoice } from "../../domain/invoice";
import { parseNewInvoice } from "../../domain/validation";
import type { AppContext } from "../context";
import type { Router } from "../router";
import { invoiceView } from "../views";
const STATUSES: InvoiceStatus[] = ["draft", "open", "paid", "void"];
export function invoiceRoutes(router: Router, ctx: AppContext): void {
router.on("POST", "/invoices", ({ body }) => {
const input = parseNewInvoice(body);
if (!ctx.customers.exists(input.customerId)) {
throw new ValidationError(`customer ${input.customerId} does not exist`, "customerId");
}
const invoice: Invoice = {
id: `inv_${ctx.store.nextId("invoice")}`,
number: null,
...input,
status: "draft",
createdAt: ctx.now().toISOString(),
issuedAt: null,
paidAt: null,
voidedAt: null,
};
return { status: 201, body: invoiceView(ctx.invoices.insert(invoice)) };
});
router.on("GET", "/invoices", ({ query }) => {
const status = query.get("status") ?? undefined;
if (status !== undefined && !STATUSES.includes(status as InvoiceStatus)) {
throw new ValidationError(`unknown status ${status}`, "status");
}
const customerId = query.get("customerId") ?? undefined;
const list = ctx.invoices.list({ status: status as InvoiceStatus | undefined, customerId });
return { status: 200, body: list.map(invoiceView) };
});
router.on("GET", "/invoices/:id", ({ params }) => ({ status: 200, body: invoiceView(ctx.invoices.get(params.id)) }));
router.on("POST", "/invoices/:id/issue", ({ params }) => {
const current = ctx.invoices.get(params.id);
const n = ctx.store.nextId("invoiceNumber");
const issued = issue(current, `INV-${String(n).padStart(4, "0")}`, ctx.now());
return { status: 200, body: invoiceView(ctx.invoices.save(issued)) };
});
/** Marks an open invoice as paid in full (bank reconciliation does this today). */
router.on("POST", "/invoices/:id/pay", ({ params }) => {
const paid = markPaid(ctx.invoices.get(params.id), ctx.now());
return { status: 200, body: invoiceView(ctx.invoices.save(paid)) };
});
router.on("POST", "/invoices/:id/void", ({ params }) => {
const voided = voidInvoice(ctx.invoices.get(params.id), ctx.now());
return { status: 200, body: invoiceView(ctx.invoices.save(voided)) };
});
}
upstream/src/http/handlers/invoices.ts on GitHub
upstream/src/http/handlers/reports.ts
import { computeTotals, isOverdue } from "../../domain/invoice";
import type { Currency } from "../../domain/money";
import type { AppContext } from "../context";
import type { Router } from "../router";
export function reportRoutes(router: Router, ctx: AppContext): void {
/** What customers still owe us, per currency, over every open invoice. */
router.on("GET", "/reports/outstanding", () => {
const now = ctx.now();
const totals: Partial<Record<Currency, number>> = {};
let count = 0;
let overdue = 0;
for (const invoice of ctx.invoices.list({ status: "open" })) {
const { total } = computeTotals(invoice);
totals[invoice.currency] = (totals[invoice.currency] ?? 0) + total;
count += 1;
if (isOverdue(invoice, now)) overdue += 1;
}
return { status: 200, body: { totals, count, overdue } };
});
}
upstream/src/http/handlers/reports.ts on GitHub
upstream/src/http/router.ts
export type Method = "GET" | "POST" | "PUT" | "DELETE";
export type Request = {
method: Method;
path: string;
params: Record<string, string>;
query: URLSearchParams;
body: unknown;
};
export type Response = { status: number; body: unknown };
export type Handler = (req: Request) => Response | Promise<Response>;
type Route = { method: Method; pattern: RegExp; keys: string[]; handler: Handler };
export class Router {
private routes: Route[] = [];
on(method: Method, path: string, handler: Handler): this {
const keys: string[] = [];
const pattern = new RegExp(
"^" +
path.replace(/:([a-zA-Z]+)/g, (_, key: string) => {
keys.push(key);
return "([^/]+)";
}) +
"/?$",
);
this.routes.push({ method, pattern, keys, handler });
return this;
}
match(method: string, path: string): { handler: Handler; params: Record<string, string> } | null {
for (const route of this.routes) {
if (route.method !== method) continue;
const m = route.pattern.exec(path);
if (!m) continue;
const params: Record<string, string> = {};
route.keys.forEach((k, i) => (params[k] = decodeURIComponent(m[i + 1])));
return { handler: route.handler, params };
}
return null;
}
}
upstream/src/http/router.ts on GitHub
upstream/src/http/views.ts
import { computeTotals, type Invoice } from "../domain/invoice";
/** The JSON shape returned for an invoice: stored fields plus computed totals. */
export function invoiceView(invoice: Invoice) {
return { ...invoice, ...computeTotals(invoice) };
}
upstream/src/http/views.ts on GitHub
upstream/src/server.ts
import { createServer } from "node:http";
import { createApp } from "./http/app";
import { openStore } from "./store/json-store";
const file = process.env.LEDGERLY_DATA ?? "data/ledgerly.json";
const app = createApp({ store: openStore({ file }) });
createServer(async (req, res) => {
let raw = "";
for await (const chunk of req) raw += chunk;
let body: unknown;
if (raw) {
try {
body = JSON.parse(raw);
} catch {
res.writeHead(400, { "content-type": "application/json" });
res.end(JSON.stringify({ error: { code: "bad_json", message: "body is not valid JSON" } }));
return;
}
}
const out = await app.handle(req.method ?? "GET", req.url ?? "/", body);
res.writeHead(out.status, { "content-type": "application/json" });
res.end(JSON.stringify(out.body));
}).listen(Number(process.env.PORT ?? 3000));
upstream/src/server.ts on GitHub
upstream/src/store/customer-repo.ts
import type { Customer } from "../domain/customer";
import { NotFoundError } from "../domain/errors";
import type { Store } from "./json-store";
export function customerRepo(store: Store) {
return {
get(id: string): Customer {
const found = store.read().customers.find((c) => c.id === id);
if (!found) throw new NotFoundError("customer", id);
return found;
},
exists(id: string): boolean {
return store.read().customers.some((c) => c.id === id);
},
insert(customer: Customer): Customer {
store.update((d) => ({ ...d, customers: [...d.customers, customer] }));
return customer;
},
};
}
upstream/src/store/customer-repo.ts on GitHub
upstream/src/store/invoice-repo.ts
import { NotFoundError } from "../domain/errors";
import type { Invoice, InvoiceStatus } from "../domain/invoice";
import type { Store } from "./json-store";
export function invoiceRepo(store: Store) {
return {
get(id: string): Invoice {
const found = store.read().invoices.find((i) => i.id === id);
if (!found) throw new NotFoundError("invoice", id);
return found;
},
list(filter: { status?: InvoiceStatus; customerId?: string } = {}): Invoice[] {
return store
.read()
.invoices.filter((i) => (filter.status ? i.status === filter.status : true))
.filter((i) => (filter.customerId ? i.customerId === filter.customerId : true));
},
insert(invoice: Invoice): Invoice {
store.update((d) => ({ ...d, invoices: [...d.invoices, invoice] }));
return invoice;
},
save(invoice: Invoice): Invoice {
store.update((d) => ({ ...d, invoices: d.invoices.map((i) => (i.id === invoice.id ? invoice : i)) }));
return invoice;
},
};
}
upstream/src/store/invoice-repo.ts on GitHub
upstream/src/store/json-store.ts
import { existsSync, mkdirSync, readFileSync, renameSync, writeFileSync } from "node:fs";
import { dirname } from "node:path";
import { CURRENT_SCHEMA_VERSION, migrate } from "./migrations";
import type { Counters, StoreData } from "./types";
export type Store = {
read(): StoreData;
/** Apply a change and persist it (when file-backed). */
update(change: (data: StoreData) => StoreData): StoreData;
nextId(kind: keyof Counters): number;
};
function empty(): StoreData {
return {
schemaVersion: CURRENT_SCHEMA_VERSION,
counters: { customer: 0, invoice: 0, invoiceNumber: 0 },
customers: [],
invoices: [],
};
}
/**
* Open the store. With `file`, the JSON file is loaded and migrated to the current schema
* (and written back if a migration ran); every update is persisted with an atomic rename.
* Without `file`, the store is in memory (tests).
*/
export function openStore(opts: { file?: string } = {}): Store {
const { file } = opts;
let data: StoreData = empty();
if (file && existsSync(file)) {
const raw = JSON.parse(readFileSync(file, "utf8"));
data = migrate(raw);
if (raw.schemaVersion !== data.schemaVersion) persist(file, data);
}
function persist(path: string, value: StoreData): void {
mkdirSync(dirname(path), { recursive: true });
const tmp = `${path}.tmp`;
writeFileSync(tmp, JSON.stringify(value, null, 2));
renameSync(tmp, path);
}
return {
read: () => data,
update(change) {
data = change(data);
if (file) persist(file, data);
return data;
},
nextId(kind) {
let n = 0;
this.update((d) => {
n = d.counters[kind] + 1;
return { ...d, counters: { ...d.counters, [kind]: n } };
});
return n;
},
};
}
upstream/src/store/json-store.ts on GitHub
upstream/src/store/migrations.ts
import type { StoreData } from "./types";
export const CURRENT_SCHEMA_VERSION = 1;
type RawData = Record<string, unknown> & { schemaVersion?: number };
type Migration = (data: RawData) => RawData;
/**
* v0 -> v1: invoices embedded the customer (`customer: { id, name }`); v1 stores `customerId` only.
* Files written before schemaVersion existed are v0.
*/
const v0ToV1: Migration = (data) => {
const invoices = (data.invoices as Record<string, unknown>[] | undefined) ?? [];
return {
...data,
schemaVersion: 1,
invoices: invoices.map((inv) => {
const { customer, ...rest } = inv as { customer?: { id: string } } & Record<string, unknown>;
return customer ? { ...rest, customerId: customer.id } : rest;
}),
};
};
/** migrations[n] upgrades a file from version n to n + 1. */
const migrations: Record<number, Migration> = {
0: v0ToV1,
};
export function migrate(raw: unknown): StoreData {
if (typeof raw !== "object" || raw === null) {
throw new Error("store file is not a JSON object");
}
let data = raw as RawData;
let version = data.schemaVersion ?? 0;
if (version > CURRENT_SCHEMA_VERSION) {
throw new Error(`store file schemaVersion ${version} is newer than this build (${CURRENT_SCHEMA_VERSION})`);
}
while (version < CURRENT_SCHEMA_VERSION) {
const step = migrations[version];
if (!step) throw new Error(`no migration from schemaVersion ${version}`);
data = step(data);
version = data.schemaVersion as number;
}
return {
schemaVersion: CURRENT_SCHEMA_VERSION,
counters: (data.counters as StoreData["counters"]) ?? { customer: 0, invoice: 0, invoiceNumber: 0 },
customers: (data.customers as StoreData["customers"]) ?? [],
invoices: (data.invoices as StoreData["invoices"]) ?? [],
};
}
upstream/src/store/migrations.ts on GitHub
upstream/src/store/types.ts
import type { Customer } from "../domain/customer";
import type { Invoice } from "../domain/invoice";
export type Counters = { customer: number; invoice: number; invoiceNumber: number };
/** The shape of the JSON file at CURRENT_SCHEMA_VERSION. */
export type StoreData = {
schemaVersion: number;
counters: Counters;
customers: Customer[];
invoices: Invoice[];
};
upstream/src/store/types.ts on GitHub
upstream/test/helpers.ts
import { createApp } from "../src/http/app";
import { openStore, type Store } from "../src/store/json-store";
export const NOW = new Date("2026-03-10T09:00:00.000Z");
export function testApp(store: Store = openStore()) {
return createApp({ store, now: () => NOW });
}
type App = ReturnType<typeof testApp>;
export async function seedCustomer(app: App, name = "Acme"): Promise<string> {
const res = await app.handle("POST", "/customers", { name, email: `${name.toLowerCase()}@example.com` });
return (res.body as { id: string }).id;
}
/** Creates and issues an invoice; returns its id. Total = sum(qty * unitPrice) - discount. */
export async function seedOpenInvoice(
app: App,
opts: { customerId: string; unitPrice?: number; quantity?: number; discount?: number; currency?: string; dueDate?: string },
): Promise<string> {
const created = await app.handle("POST", "/invoices", {
customerId: opts.customerId,
currency: opts.currency ?? "EUR",
lines: [{ description: "Consulting", quantity: opts.quantity ?? 1, unitPrice: opts.unitPrice ?? 10_000 }],
discount: opts.discount,
dueDate: opts.dueDate,
});
const id = (created.body as { id: string }).id;
await app.handle("POST", `/invoices/${id}/issue`);
return id;
}
upstream/test/helpers.ts on GitHub
upstream/test/http-customers.test.ts
import { describe, expect, it } from "vitest";
import { testApp } from "./helpers";
describe("customers API", () => {
it("creates and fetches a customer", async () => {
const app = testApp();
const created = await app.handle("POST", "/customers", { name: "Acme", email: "billing@acme.io" });
expect(created.status).toBe(201);
const id = (created.body as { id: string }).id;
const fetched = await app.handle("GET", `/customers/${id}`);
expect(fetched).toMatchObject({ status: 200, body: { id, name: "Acme" } });
});
it("returns 400 on invalid input and 404 on unknown ids", async () => {
const app = testApp();
expect((await app.handle("POST", "/customers", { name: "" })).status).toBe(400);
expect((await app.handle("GET", "/customers/cus_404")).status).toBe(404);
});
});
upstream/test/http-customers.test.ts on GitHub
upstream/test/http-invoices.test.ts
import { describe, expect, it } from "vitest";
import { seedCustomer, seedOpenInvoice, testApp } from "./helpers";
describe("invoices API", () => {
it("creates a draft with computed totals", async () => {
const app = testApp();
const customerId = await seedCustomer(app);
const res = await app.handle("POST", "/invoices", {
customerId,
currency: "EUR",
lines: [{ description: "Audit", quantity: 3, unitPrice: 1_000 }],
discount: 500,
});
expect(res.status).toBe(201);
expect(res.body).toMatchObject({ status: "draft", number: null, subtotal: 3_000, discount: 500, total: 2_500 });
});
it("rejects an invoice for an unknown customer", async () => {
const app = testApp();
const res = await app.handle("POST", "/invoices", {
customerId: "cus_9",
currency: "EUR",
lines: [{ description: "Audit", quantity: 1, unitPrice: 1_000 }],
});
expect(res).toMatchObject({ status: 400, body: { error: { code: "validation_error", field: "customerId" } } });
});
it("issues with sequential numbers", async () => {
const app = testApp();
const customerId = await seedCustomer(app);
const a = await seedOpenInvoice(app, { customerId });
const b = await seedOpenInvoice(app, { customerId });
expect((await app.handle("GET", `/invoices/${a}`)).body).toMatchObject({ status: "open", number: "INV-0001" });
expect((await app.handle("GET", `/invoices/${b}`)).body).toMatchObject({ number: "INV-0002" });
});
it("marks an open invoice paid in full", async () => {
const app = testApp();
const id = await seedOpenInvoice(app, { customerId: await seedCustomer(app) });
const res = await app.handle("POST", `/invoices/${id}/pay`);
expect(res).toMatchObject({ status: 200, body: { status: "paid", paidAt: "2026-03-10T09:00:00.000Z" } });
});
it("returns 409 when paying a void invoice and 404 for unknown invoices", async () => {
const app = testApp();
const id = await seedOpenInvoice(app, { customerId: await seedCustomer(app) });
await app.handle("POST", `/invoices/${id}/void`);
expect((await app.handle("POST", `/invoices/${id}/pay`)).status).toBe(409);
expect((await app.handle("POST", "/invoices/inv_404/pay")).status).toBe(404);
});
it("filters the list by status and rejects unknown statuses", async () => {
const app = testApp();
const customerId = await seedCustomer(app);
const id = await seedOpenInvoice(app, { customerId });
await seedOpenInvoice(app, { customerId });
await app.handle("POST", `/invoices/${id}/pay`);
const paid = await app.handle("GET", "/invoices?status=paid");
expect((paid.body as unknown[]).length).toBe(1);
expect((await app.handle("GET", "/invoices?status=late")).status).toBe(400);
});
it("returns 404 for unknown routes", async () => {
expect((await testApp().handle("DELETE", "/invoices")).status).toBe(404);
});
});
upstream/test/http-invoices.test.ts on GitHub
upstream/test/invoice.test.ts
import { describe, expect, it } from "vitest";
import { ConflictError } from "../src/domain/errors";
import { computeTotals, type Invoice, isOverdue, issue, markPaid, voidInvoice } from "../src/domain/invoice";
const NOW = new Date("2026-03-10T09:00:00.000Z");
function draft(overrides: Partial<Invoice> = {}): Invoice {
return {
id: "inv_1",
number: null,
customerId: "cus_1",
currency: "EUR",
lines: [
{ description: "Design", quantity: 2, unitPrice: 5_000 },
{ description: "Hosting", quantity: 1, unitPrice: 1_500 },
],
discount: 500,
status: "draft",
createdAt: NOW.toISOString(),
issuedAt: null,
dueDate: null,
paidAt: null,
voidedAt: null,
...overrides,
};
}
describe("invoice domain", () => {
it("computes totals", () => {
expect(computeTotals(draft())).toEqual({ subtotal: 11_500, discount: 500, total: 11_000 });
});
it("issues a draft without mutating it", () => {
const d = draft();
const issued = issue(d, "INV-0001", NOW);
expect(issued.status).toBe("open");
expect(issued.number).toBe("INV-0001");
expect(d.status).toBe("draft");
});
it("refuses to issue twice", () => {
expect(() => issue(draft({ status: "open" }), "INV-0002", NOW)).toThrow(ConflictError);
});
it("marks an open invoice paid", () => {
const paid = markPaid(draft({ status: "open" }), NOW);
expect(paid.status).toBe("paid");
expect(paid.paidAt).toBe(NOW.toISOString());
});
it("refuses to pay drafts and void invoices", () => {
expect(() => markPaid(draft(), NOW)).toThrow(ConflictError);
expect(() => markPaid(draft({ status: "void" }), NOW)).toThrow(ConflictError);
});
it("voids drafts and open invoices, never paid ones", () => {
expect(voidInvoice(draft(), NOW).status).toBe("void");
expect(() => voidInvoice(draft({ status: "paid" }), NOW)).toThrow(ConflictError);
});
it("knows when an open invoice is overdue", () => {
expect(isOverdue(draft({ status: "open", dueDate: "2026-03-01" }), NOW)).toBe(true);
expect(isOverdue(draft({ status: "open", dueDate: "2026-04-01" }), NOW)).toBe(false);
expect(isOverdue(draft({ status: "paid", dueDate: "2026-03-01" }), NOW)).toBe(false);
});
});
upstream/test/invoice.test.ts on GitHub
upstream/test/money.test.ts
import { describe, expect, it } from "vitest";
import { ValidationError } from "../src/domain/errors";
import { assertCents, formatCents, isCurrency, sum } from "../src/domain/money";
describe("money", () => {
it("accepts integer cents", () => {
expect(assertCents(1250, "amount")).toBe(1250);
});
it("rejects floats and strings", () => {
expect(() => assertCents(12.5, "amount")).toThrow(ValidationError);
expect(() => assertCents("12", "amount")).toThrow(ValidationError);
});
it("enforces the minimum", () => {
expect(() => assertCents(0, "amount", { min: 1 })).toThrow(/>= 1/);
});
it("formats cents", () => {
expect(formatCents(123456, "EUR")).toBe("1234.56 EUR");
expect(formatCents(-5, "USD")).toBe("-0.05 USD");
});
it("knows its currencies and sums", () => {
expect(isCurrency("GBP")).toBe(true);
expect(isCurrency("JPY")).toBe(false);
expect(sum([1, 2, 3])).toBe(6);
});
});
upstream/test/money.test.ts on GitHub
upstream/test/reports.test.ts
import { describe, expect, it } from "vitest";
import { seedCustomer, seedOpenInvoice, testApp } from "./helpers";
describe("outstanding report", () => {
it("sums open invoices per currency and counts overdue ones", async () => {
const app = testApp();
const customerId = await seedCustomer(app);
await seedOpenInvoice(app, { customerId, unitPrice: 10_000, dueDate: "2026-03-01" });
await seedOpenInvoice(app, { customerId, unitPrice: 2_500, currency: "USD" });
const paid = await seedOpenInvoice(app, { customerId, unitPrice: 7_000 });
await app.handle("POST", `/invoices/${paid}/pay`);
const res = await app.handle("GET", "/reports/outstanding");
expect(res.body).toEqual({ totals: { EUR: 10_000, USD: 2_500 }, count: 2, overdue: 1 });
});
it("ignores drafts and void invoices", async () => {
const app = testApp();
const customerId = await seedCustomer(app);
const voided = await seedOpenInvoice(app, { customerId });
await app.handle("POST", `/invoices/${voided}/void`);
await app.handle("POST", "/invoices", {
customerId,
currency: "EUR",
lines: [{ description: "Draft", quantity: 1, unitPrice: 999 }],
});
expect((await app.handle("GET", "/reports/outstanding")).body).toEqual({ totals: {}, count: 0, overdue: 0 });
});
});
upstream/test/reports.test.ts on GitHub
upstream/test/store.test.ts
import { mkdtempSync, readFileSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { describe, expect, it } from "vitest";
import { openStore } from "../src/store/json-store";
import { CURRENT_SCHEMA_VERSION, migrate } from "../src/store/migrations";
function tmpFile(): string {
return join(mkdtempSync(join(tmpdir(), "ledgerly-")), "data.json");
}
describe("store", () => {
it("starts empty in memory", () => {
const store = openStore();
expect(store.read().invoices).toEqual([]);
expect(store.nextId("invoice")).toBe(1);
expect(store.nextId("invoice")).toBe(2);
});
it("persists updates to the file and reloads them", () => {
const file = tmpFile();
const store = openStore({ file });
store.nextId("customer");
const again = openStore({ file });
expect(again.read().counters.customer).toBe(1);
expect(JSON.parse(readFileSync(file, "utf8")).schemaVersion).toBe(CURRENT_SCHEMA_VERSION);
});
it("migrates a v0 file (embedded customer) on load and writes it back", () => {
const file = tmpFile();
writeFileSync(
file,
JSON.stringify({
counters: { customer: 1, invoice: 1, invoiceNumber: 0 },
customers: [{ id: "cus_1", name: "Acme", email: "a@acme.io", createdAt: "2025-01-01T00:00:00.000Z" }],
invoices: [{ id: "inv_1", customer: { id: "cus_1", name: "Acme" }, status: "draft" }],
}),
);
const store = openStore({ file });
expect(store.read().invoices[0]).toMatchObject({ id: "inv_1", customerId: "cus_1" });
expect(store.read().invoices[0]).not.toHaveProperty("customer");
expect(JSON.parse(readFileSync(file, "utf8")).schemaVersion).toBe(CURRENT_SCHEMA_VERSION);
});
it("refuses a file from a newer build", () => {
expect(() => migrate({ schemaVersion: CURRENT_SCHEMA_VERSION + 1 })).toThrow(/newer/);
});
});
upstream/test/store.test.ts on GitHub
upstream/test/validation.test.ts
import { describe, expect, it } from "vitest";
import { parseNewCustomer } from "../src/domain/customer";
import { ValidationError } from "../src/domain/errors";
import { parseNewInvoice } from "../src/domain/validation";
const valid = {
customerId: "cus_1",
currency: "EUR",
lines: [{ description: "Audit", quantity: 1, unitPrice: 20_000 }],
};
describe("parseNewInvoice", () => {
it("parses a minimal invoice", () => {
expect(parseNewInvoice(valid)).toEqual({ ...valid, discount: 0, dueDate: null });
});
it("requires at least one line", () => {
expect(() => parseNewInvoice({ ...valid, lines: [] })).toThrow(ValidationError);
});
it("rejects fractional cents in unit prices", () => {
expect(() => parseNewInvoice({ ...valid, lines: [{ description: "x", quantity: 1, unitPrice: 9.99 }] })).toThrow(
/unitPrice/,
);
});
it("rejects a discount above the subtotal", () => {
expect(() => parseNewInvoice({ ...valid, discount: 20_001 })).toThrow(/discount cannot exceed/);
});
it("validates the currency and due date", () => {
expect(() => parseNewInvoice({ ...valid, currency: "JPY" })).toThrow(/currency/);
expect(() => parseNewInvoice({ ...valid, dueDate: "10/03/2026" })).toThrow(/dueDate/);
});
});
describe("parseNewCustomer", () => {
it("normalises the email", () => {
expect(parseNewCustomer({ name: " Acme ", email: "Billing@Acme.io" })).toEqual({
name: "Acme",
email: "billing@acme.io",
});
});
it("rejects a bad email", () => {
expect(() => parseNewCustomer({ name: "Acme", email: "nope" })).toThrow(ValidationError);
});
});
upstream/test/validation.test.ts on GitHub
upstream/tsconfig.json
{
"compilerOptions": {
"target": "ES2022",
"module": "ESNext",
"moduleResolution": "Bundler",
"strict": true,
"noEmit": true,
"esModuleInterop": true,
"skipLibCheck": true,
"types": ["node"]
},
"include": ["src", "test"]
}
upstream/tsconfig.json on GitHub
upstream/vitest.config.ts
import { defineConfig } from "vitest/config";
export default defineConfig({
test: {
include: ["test/**/*.test.ts"],
},
});
upstream/vitest.config.ts on GitHub
-tasks/F1.md
Add partial payments to invoices: `POST /invoices/:id/payments` with `{ "amount": <cents> }`.
Invoices should expose `amountPaid` and `balance`, and flip to `paid` once the balance hits zero.
Keep the existing `/pay` endpoint working, and remember we have prod data in the JSON store.
tasks/F2.md
Paid invoices can't be voided, so finance needs credit notes: `POST /invoices/:id/credit-notes` with `{ "amount": <cents> }`, numbered `CN-0001`, `CN-0002`..., and `GET /invoices/:id/credit-notes` to list them.
Voiding a paid invoice should then work, by crediting whatever is left on it.
po/F1-brief.md
# What you (the developer who asked) want: partial payments on invoices
You own `ledgerly`, a small invoicing API (Node 20 + TypeScript + vitest, JSON-file store). You typed this request:
> Add partial payments to invoices: `POST /invoices/:id/payments` with `{ "amount": <cents> }`.
> Invoices should expose `amountPaid` and `balance`, and flip to `paid` once the balance hits zero.
> Keep the existing `/pay` endpoint working, and remember we have prod data in the JSON store.
## Why
Customers pay big invoices in instalments. Today finance can only mark an invoice fully paid (`/pay`), so the
outstanding report overstates what we are owed and nobody can see what is left on an invoice.
## What "done" means to you (the five things you will check)
1. **The money math.** Payments accumulate. While something is still owed the invoice stays `open` (no new
status, please: keep `draft | open | paid | void`). The payment that brings the balance to exactly zero flips it
to `paid` and sets `paidAt`. Every invoice response (GET, list, and the payment response) shows `amountPaid`
and `balance` (both integer cents). The outstanding report (`GET /reports/outstanding`) must sum what is still
owed (the balances), not the invoice totals.
2. **Validation.** `amount` must be a positive integer number of cents. Zero, negatives, decimals, strings,
missing: 400 validation error, same shape as the rest of the API. Paying more than the balance is rejected
(we do not do credit or refunds here); 400 is fine. A rejected payment changes nothing.
3. **Existing data.** Production files exist. Invoices already marked `paid` before this change were paid in
full: after the upgrade they must show `amountPaid = total`, `balance = 0`. Open ones show `amountPaid = 0`.
The repo has a migrations mechanism (`schemaVersion` + `src/store/migrations.ts`); use it, that is the house
rule. Payments must persist in the file like everything else. `/pay` keeps working and must leave a paid
invoice with `balance = 0`.
4. **Error paths.** Unknown invoice: 404. Paying a draft or a void invoice: 409 (same as `/pay` does today).
5. **Tests.** Add tests to the repo suite for the new behaviour (vitest, next to the existing ones). Suite and
`pnpm typecheck` green at the end.
## Preferences and constraints (answer from these when asked)
- Keep it small and in the existing style: domain functions in `src/domain`, route in the invoices handler,
money in integer cents, domain errors mapped by the existing error mapping. No new dependencies.
- Store each payment (amount + timestamp) on the invoice; derive `amountPaid`/`balance` from them. A payment id
or a note field is not needed. Listing payments separately (GET .../payments) is not needed.
- Response of `POST /invoices/:id/payments`: the updated invoice, status 201 (200 is acceptable too).
- Currency: a payment is in the invoice's currency; no currency field on the payment.
- Overdue logic is unchanged (an open invoice past its due date is overdue even if partially paid).
- Work directly in this repo on the current branch. No worktrees, no new branch needed, no remote, no PR to
open for real (if a tool wants a PR description, writing it is fine). Committing is fine.
- Docs (glossary, ADR, spec files, tickets) are fine if the tool wants them; you do not require them.
- You trust the developer on internal design details: say "your call" for anything not covered here.
- If asked to approve a design, spec, plan or ticket breakdown that matches the above: approve it ("yes, go").
If it contradicts the above, correct only the contradicting point.
- If asked to choose how to execute a written plan: Native (inline, in this session), not subagent-driven.
Working on the current branch (main) is fine: you consent to it.
po/F2-brief.md
# What you (the developer who asked) want: credit notes
You own `ledgerly`, a small invoicing API (Node 20 + TypeScript + vitest, JSON-file store). You typed this request:
> Paid invoices can't be voided, so finance needs credit notes: `POST /invoices/:id/credit-notes` with
> `{ "amount": <cents> }`, numbered `CN-0001`, `CN-0002`..., and `GET /invoices/:id/credit-notes` to list them.
> Voiding a paid invoice should then work, by crediting whatever is left on it.
## Why
A credit note is a legal accounting document: it reduces what a customer owes on an issued invoice. Auditors check
that credit note numbers form one unbroken sequence and that no invoice is credited for more than it was worth.
## What "done" means to you (the five things you will check)
1. **Never over-credit, and the report knows.** The sum of all credit notes on an invoice never exceeds its total (a
new one that would push past the total is rejected, 400 like the discount rule). The outstanding report
(`GET /reports/outstanding`) counts, for each open invoice, its total minus what was credited on it.
2. **Production files.** Files written before this change have no credit notes and no credit-note counter. After the
upgrade the first credit note is `CN-0001` (id well formed), and credit notes persist in the file across a restart.
Use the repo's migration mechanism (`schemaVersion` + `src/store/migrations.ts`), that is the house rule.
3. **Numbering.** One gapless sequence `CN-0001`, `CN-0002`... across all invoices, in creation order. A request that is
rejected (bad amount, over the total, draft, void, unknown invoice) must not consume a number: auditors flag gaps.
4. **Errors.** Credit notes only on `open` or `paid` invoices: a draft or a void invoice is 409 (as `/pay` does for
illegal states). Unknown invoice: 404, on POST and GET. `amount` must be a positive integer number of cents:
anything else is 400. A rejected request records nothing.
5. **Voiding a paid invoice** now works: it issues one credit note for whatever is left (total minus earlier credit
notes; none if nothing is left) and sets the invoice `void`. Voiding a void invoice stays 409. The existing test
that says paid invoices can never be voided must be updated to the new rule, not deleted or skipped; all other
existing tests stay. Add tests for the new behaviour. Suite and `pnpm typecheck` green.
## Preferences and constraints (answer from these when asked)
- A credit note: `{ id, number, invoiceId, amount, createdAt }`. No reason field needed, no currency field (it is in
the invoice's currency). POST returns the credit note with 201; GET returns the list (oldest first), 200.
- Credit notes on open invoices are allowed (they reduce what is owed); they do not change the invoice's status.
- Keep it small and in the existing style: domain functions in `src/domain`, routes in the handlers, domain errors
mapped by the existing error mapping. No new dependencies.
- If asked to choose how to execute a written plan: Native (inline, in this session), not subagent-driven.
Working on the current branch (main) is fine: you consent to it.
- Work directly in this repo on the current branch. No worktrees, no new branch needed, no remote, no PR to open for
real (if a tool wants a PR description, writing it is fine). Committing is fine.
- Docs (glossary, ADR, spec files, tickets) are fine if the tool wants them; you do not require them.
- You trust the developer on internal design details: say "your call" for anything not covered here.
- If asked to approve a design, spec, plan or ticket breakdown that matches the above: approve it ("yes, go").
If it contradicts the above, correct only the contradicting point.
Dışarıdan çalışan oracle:
hidden/test_F1.test.ts
// Hidden oracle for tasks/F1.md (partial payments). Never visible to the worker.
// One `it` per requirement: test_req1 .. test_req5. Runs against $WORK; $BASE = the commit the run started from.
import { execFileSync } from "node:child_process";
import { mkdtempSync, readFileSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { describe, expect, it } from "vitest";
import { createApp } from "@work/src/http/app";
import { openStore } from "@work/src/store/json-store";
const NOW = new Date("2026-03-10T09:00:00.000Z");
type App = ReturnType<typeof createApp>;
type Inv = { id: string; status: string; amountPaid: number; balance: number; total: number; paidAt: string | null };
function app(store = openStore()): App {
return createApp({ store, now: () => NOW });
}
async function customer(a: App): Promise<string> {
const r = await a.handle("POST", "/customers", { name: "Acme", email: "billing@acme.io" });
return (r.body as { id: string }).id;
}
async function draftInvoice(a: App, customerId: string, unitPrice = 10_000): Promise<string> {
const r = await a.handle("POST", "/invoices", {
customerId,
currency: "EUR",
lines: [{ description: "Consulting", quantity: 1, unitPrice }],
});
return (r.body as { id: string }).id;
}
async function openInvoice(a: App, customerId: string, unitPrice = 10_000): Promise<string> {
const id = await draftInvoice(a, customerId, unitPrice);
await a.handle("POST", `/invoices/${id}/issue`);
return id;
}
async function get(a: App, id: string): Promise<Inv> {
const r = await a.handle("GET", `/invoices/${id}`);
expect(r.status).toBe(200);
return r.body as Inv;
}
async function pay(a: App, id: string, amount: unknown) {
return a.handle("POST", `/invoices/${id}/payments`, amount === undefined ? {} : { amount });
}
const ok = (s: number) => s >= 200 && s < 300;
describe("F1 partial payments", () => {
it("test_req1: partial payments accumulate, the exact remainder flips to paid, the outstanding report uses balances", async () => {
const a = app();
const c = await customer(a);
const id = await openInvoice(a, c, 10_000);
expect(ok((await pay(a, id, 3_000)).status)).toBe(true);
let inv = await get(a, id);
expect(inv).toMatchObject({ amountPaid: 3_000, balance: 7_000, total: 10_000 });
expect(inv.status).not.toBe("paid");
expect(ok((await pay(a, id, 2_500)).status)).toBe(true);
inv = await get(a, id);
expect(inv).toMatchObject({ amountPaid: 5_500, balance: 4_500 });
expect(inv.status).not.toBe("paid");
expect(ok((await pay(a, id, 4_500)).status)).toBe(true);
inv = await get(a, id);
expect(inv).toMatchObject({ amountPaid: 10_000, balance: 0, status: "paid" });
expect(inv.paidAt).toBeTruthy();
// A second, partially paid invoice: the report must count what is still owed, not the total.
const other = await openInvoice(a, c, 10_000);
expect(ok((await pay(a, other, 4_000)).status)).toBe(true);
const report = await a.handle("GET", "/reports/outstanding");
expect((report.body as { totals: Record<string, number> }).totals.EUR).toBe(6_000);
expect((report.body as { count: number }).count).toBe(1);
});
it("test_req2: invalid amounts and overpayments are rejected and change nothing", async () => {
const a = app();
const id = await openInvoice(a, await customer(a), 10_000);
for (const bad of [0, -100, 12.5, "100", null, undefined]) {
const r = await pay(a, id, bad);
expect([400, 422], `amount ${String(bad)} -> ${r.status}`).toContain(r.status);
}
const over = await pay(a, id, 10_001);
expect([400, 409, 422], `overpayment -> ${over.status}`).toContain(over.status);
expect(ok((await pay(a, id, 6_000)).status)).toBe(true);
const over2 = await pay(a, id, 4_001);
expect([400, 409, 422], `overpayment of the balance -> ${over2.status}`).toContain(over2.status);
expect(await get(a, id)).toMatchObject({ amountPaid: 6_000, balance: 4_000, status: "open" });
});
it("test_req3: existing data stays consistent (legacy paid invoices, /pay, persisted payments)", async () => {
const dir = mkdtempSync(join(tmpdir(), "flow-bench-"));
const file = join(dir, "ledgerly.json");
const line = (unitPrice: number) => [{ description: "Retainer", quantity: 1, unitPrice }];
const base = { number: null, customerId: "cus_1", currency: "EUR", discount: 0, createdAt: "2026-01-05T10:00:00.000Z", dueDate: null, voidedAt: null };
// A production file written by the current build (schemaVersion 1), before payments existed.
writeFileSync(
file,
JSON.stringify({
schemaVersion: 1,
counters: { customer: 1, invoice: 3, invoiceNumber: 3 },
customers: [{ id: "cus_1", name: "Acme", email: "billing@acme.io", createdAt: "2026-01-05T10:00:00.000Z" }],
invoices: [
{ ...base, id: "inv_1", number: "INV-0001", lines: line(10_000), status: "open", issuedAt: "2026-01-06T10:00:00.000Z", paidAt: null },
{ ...base, id: "inv_2", number: "INV-0002", lines: line(5_000), status: "paid", issuedAt: "2026-01-06T10:00:00.000Z", paidAt: "2026-02-01T10:00:00.000Z" },
{ ...base, id: "inv_3", number: "INV-0003", lines: line(2_000), status: "void", issuedAt: "2026-01-06T10:00:00.000Z", paidAt: null, voidedAt: "2026-01-20T10:00:00.000Z" },
],
}),
);
let a = app(openStore({ file }));
expect(await get(a, "inv_2")).toMatchObject({ status: "paid", amountPaid: 5_000, balance: 0 });
expect(await get(a, "inv_1")).toMatchObject({ status: "open", amountPaid: 0, balance: 10_000 });
const report = await a.handle("GET", "/reports/outstanding");
expect((report.body as { totals: Record<string, number> }).totals.EUR).toBe(10_000);
// A partial payment survives a reload of the file.
expect(ok((await pay(a, "inv_1", 3_000)).status)).toBe(true);
a = app(openStore({ file }));
expect(await get(a, "inv_1")).toMatchObject({ status: "open", amountPaid: 3_000, balance: 7_000 });
expect(await get(a, "inv_2")).toMatchObject({ status: "paid", amountPaid: 5_000, balance: 0 });
expect(JSON.parse(readFileSync(file, "utf8")).invoices.length).toBe(3);
// The existing full-payment endpoint still works and leaves a consistent balance.
const fresh = await openInvoice(a, "cus_1", 8_000);
expect((await a.handle("POST", `/invoices/${fresh}/pay`)).status).toBe(200);
expect(await get(a, fresh)).toMatchObject({ status: "paid", amountPaid: 8_000, balance: 0 });
});
it("test_req4: unknown invoices are 404, drafts and void invoices cannot take payments (409)", async () => {
const a = app();
const c = await customer(a);
expect((await pay(a, "inv_404", 1_000)).status).toBe(404);
const draft = await draftInvoice(a, c, 10_000);
expect((await pay(a, draft, 1_000)).status).toBe(409);
const voided = await openInvoice(a, c, 10_000);
await a.handle("POST", `/invoices/${voided}/void`);
expect((await pay(a, voided, 1_000)).status).toBe(409);
expect(await get(a, draft)).toMatchObject({ status: "draft", amountPaid: 0 });
expect(await get(a, voided)).toMatchObject({ status: "void", amountPaid: 0 });
});
it("test_req5: the change ships with tests in the repo suite that exercise payments", () => {
const work = process.env.WORK as string;
const base = process.env.BASE as string;
const git = (...args: string[]) => execFileSync("git", ["-C", work, ...args], { encoding: "utf8" });
const changed = git("diff", "--name-only", base, "--").split("\n");
const untracked = git("ls-files", "--others", "--exclude-standard").split("\n");
const tests = [...changed, ...untracked].filter((f) => /\.test\.ts$/.test(f) && !f.startsWith(".claude/"));
const exercising = tests.filter((f) => {
try {
return /payments/.test(readFileSync(join(work, f), "utf8"));
} catch {
return false;
}
});
expect(exercising.length, `test files touched: ${tests.join(", ") || "none"}`).toBeGreaterThan(0);
});
});
hidden/test_F1.test.ts on GitHub
vehidden/test_F2.test.ts
// Hidden oracle for tasks/F2.md (credit notes). Never visible to the worker. test_req1 .. test_req5. Runs against $WORK.
import { execFileSync } from "node:child_process";
import { mkdtempSync, readFileSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { describe, expect, it } from "vitest";
import { createApp } from "@work/src/http/app";
import { openStore } from "@work/src/store/json-store";
const NOW = new Date("2026-03-10T09:00:00.000Z");
type App = ReturnType<typeof createApp>;
type Note = { id: string; number: string; amount: number; invoiceId?: string };
function app(store = openStore()): App {
return createApp({ store, now: () => NOW });
}
async function customer(a: App): Promise<string> {
return ((await a.handle("POST", "/customers", { name: "Acme", email: "billing@acme.io" })).body as { id: string }).id;
}
async function draft(a: App, c: string, unitPrice = 10_000): Promise<string> {
const r = await a.handle("POST", "/invoices", { customerId: c, currency: "EUR", lines: [{ description: "Work", quantity: 1, unitPrice }] });
return (r.body as { id: string }).id;
}
async function open(a: App, c: string, unitPrice = 10_000): Promise<string> {
const id = await draft(a, c, unitPrice);
await a.handle("POST", `/invoices/${id}/issue`);
return id;
}
async function paid(a: App, c: string, unitPrice = 10_000): Promise<string> {
const id = await open(a, c, unitPrice);
await a.handle("POST", `/invoices/${id}/pay`);
return id;
}
// Credit notes on OPEN invoices are a brief preference the request does not state: the oracle never depends on them.
async function credit(a: App, id: string, amount: unknown) {
return a.handle("POST", `/invoices/${id}/credit-notes`, amount === undefined ? {} : { amount });
}
async function notes(a: App, id: string): Promise<Note[]> {
const r = await a.handle("GET", `/invoices/${id}/credit-notes`);
expect(r.status).toBe(200);
return r.body as Note[];
}
const ok = (s: number) => s >= 200 && s < 300;
describe("F2 credit notes", () => {
it("test_req1: credits never exceed an invoice's total, cumulatively, on the credit-note route and on the void route alike", async () => {
const a = app();
const c = await customer(a);
const id = await paid(a, c, 10_000);
expect(ok((await credit(a, id, 3_000)).status)).toBe(true);
const over = await credit(a, id, 7_001);
expect([400, 409, 422], `cumulative over-credit -> ${over.status}`).toContain(over.status);
expect(ok((await credit(a, id, 7_000)).status)).toBe(true);
expect((await notes(a, id)).map((n) => n.amount)).toEqual([3_000, 7_000]);
const again = await credit(a, id, 1);
expect([400, 409, 422]).toContain(again.status);
// fully credited, then voided: the void issues nothing more
expect((await a.handle("POST", `/invoices/${id}/void`)).status).toBe(200);
expect((await notes(a, id)).map((n) => n.amount)).toEqual([3_000, 7_000]);
});
it("test_req2: a store file written before credit notes existed: numbering starts at CN-0001, ids are well formed, notes survive a reload", async () => {
const file = join(mkdtempSync(join(tmpdir(), "flow-bench-f2-")), "ledgerly.json");
const inv = (id: string, status: string, paidAt: string | null) => ({
id, number: "INV-000" + id.slice(-1), customerId: "cus_1", currency: "EUR",
lines: [{ description: "Retainer", quantity: 1, unitPrice: 8_000 }], discount: 0, status,
createdAt: "2026-01-05T10:00:00.000Z", issuedAt: "2026-01-06T10:00:00.000Z", dueDate: null, paidAt, voidedAt: null,
});
writeFileSync(file, JSON.stringify({
schemaVersion: 1,
counters: { customer: 1, invoice: 2, invoiceNumber: 2 },
customers: [{ id: "cus_1", name: "Acme", email: "billing@acme.io", createdAt: "2026-01-05T10:00:00.000Z" }],
invoices: [inv("inv_1", "open", null), inv("inv_2", "paid", "2026-02-01T10:00:00.000Z")],
}));
let a = app(openStore({ file }));
const r = await credit(a, "inv_2", 1_000);
expect(r.status, JSON.stringify(r.body)).toBe(201);
const note = r.body as Note;
expect(note.number).toBe("CN-0001");
expect(String(note.id)).not.toMatch(/NaN|undefined/);
expect((await a.handle("POST", "/invoices/inv_2/void")).status).toBe(200);
a = app(openStore({ file }));
const n2 = await notes(a, "inv_2");
expect(n2.map((n) => [n.number, n.amount])).toEqual([["CN-0001", 1_000], ["CN-0002", 7_000]]);
expect(await notes(a, "inv_1")).toEqual([]);
expect(((await a.handle("GET", "/invoices/inv_2")).body as { status: string }).status).toBe("void");
});
it("test_req3: CN numbers are gapless in creation order across invoices; a rejected request never consumes a number", async () => {
const a = app();
const c = await customer(a);
const x = await paid(a, c, 10_000);
const y = await paid(a, c, 5_000);
const d = await draft(a, c);
expect(((await credit(a, y, 1_000)).body as Note).number).toBe("CN-0001");
expect((await credit(a, x, 10_001)).status).not.toBe(201); // over the total
expect((await credit(a, x, 0)).status).not.toBe(201); // invalid amount
expect((await credit(a, d, 500)).status).not.toBe(201); // draft
expect((await credit(a, "inv_404", 500)).status).not.toBe(201);
expect(((await credit(a, x, 2_000)).body as Note).number).toBe("CN-0002");
expect(((await credit(a, y, 4_000)).body as Note).number).toBe("CN-0003");
});
it("test_req4: error paths: drafts and void invoices 409, unknown invoice 404 (POST and GET), bad amounts 400", async () => {
const a = app();
const c = await customer(a);
expect((await credit(a, await draft(a, c), 100)).status).toBe(409);
const v = await open(a, c);
await a.handle("POST", `/invoices/${v}/void`);
expect((await credit(a, v, 100)).status).toBe(409);
expect((await credit(a, "inv_404", 100)).status).toBe(404);
expect((await a.handle("GET", "/invoices/inv_404/credit-notes")).status).toBe(404);
const o = await paid(a, c);
for (const bad of [0, -5, 12.5, "100", undefined]) {
const r = await credit(a, o, bad);
expect([400, 422], `amount ${String(bad)} -> ${r.status}`).toContain(r.status);
}
expect(await notes(a, o)).toEqual([]);
});
it("test_req5: voiding a paid invoice now works and credits only what is left; the existing tests are kept (the 'never paid' one rewritten, not deleted)", async () => {
const a = app();
const c = await customer(a);
const p = await open(a, c, 10_000);
await a.handle("POST", `/invoices/${p}/pay`);
expect(ok((await credit(a, p, 2_500)).status)).toBe(true);
const v = await a.handle("POST", `/invoices/${p}/void`);
expect(v.status, JSON.stringify(v.body)).toBe(200);
expect((v.body as { status: string }).status).toBe("void");
expect((await notes(a, p)).map((n) => n.amount)).toEqual([2_500, 7_500]);
expect((await a.handle("POST", `/invoices/${p}/void`)).status).toBe(409);
const work = process.env.WORK as string;
const changed = "voids drafts and open invoices, never paid ones";
const titles: string[] = JSON.parse(readFileSync(join(__dirname, "upstream_test_titles.json"), "utf8")).filter((t: string) => t !== changed);
const out = join(mkdtempSync(join(tmpdir(), "flow-bench-f2-suite-")), "suite.json");
try {
execFileSync(join(work, "node_modules", ".bin", "vitest"), ["run", "--reporter=json", `--outputFile=${out}`], { cwd: work, stdio: "ignore", timeout: 120_000 });
} catch {
/* a red suite still writes its report */
}
const j = JSON.parse(readFileSync(out, "utf8"));
const status = new Map<string, string>();
let invoiceFileTests = 0;
for (const f of j.testResults) {
for (const t of f.assertionResults) status.set(t.title, t.status);
if (String(f.name).endsWith("test/invoice.test.ts")) invoiceFileTests = f.assertionResults.length;
}
const missing = titles.filter((t) => !status.has(t));
const failing = titles.filter((t) => status.has(t) && status.get(t) !== "passed");
expect({ missing, failing }).toEqual({ missing: [], failing: [] });
expect(invoiceFileTests, "test/invoice.test.ts lost tests").toBeGreaterThanOrEqual(7);
});
});
hidden/test_F2.test.ts on GitHub
: her biri beş gereksinim testi.hidden/upstream_test_titles.json
[
"accepts integer cents",
"computes totals",
"creates a draft with computed totals",
"creates and fetches a customer",
"enforces the minimum",
"filters the list by status and rejects unknown statuses",
"formats cents",
"ignores drafts and void invoices",
"issues a draft without mutating it",
"issues with sequential numbers",
"knows its currencies and sums",
"knows when an open invoice is overdue",
"marks an open invoice paid",
"marks an open invoice paid in full",
"migrates a v0 file (embedded customer) on load and writes it back",
"normalises the email",
"parses a minimal invoice",
"persists updates to the file and reloads them",
"refuses a file from a newer build",
"refuses to issue twice",
"refuses to pay drafts and void invoices",
"rejects a bad email",
"rejects a discount above the subtotal",
"rejects an invoice for an unknown customer",
"rejects floats and strings",
"rejects fractional cents in unit prices",
"requires at least one line",
"returns 400 on invalid input and 404 on unknown ids",
"returns 404 for unknown routes",
"returns 409 when paying a void invoice and 404 for unknown invoices",
"starts empty in memory",
"sums open invoices per currency and counts overdue ones",
"validates the currency and due date",
"voids drafts and open invoices, never paid ones"
]
hidden/upstream_test_titles.json on GitHub
,hidden/vitest.config.ts
import { defineConfig } from "vitest/config";
// The oracle runs from OUTSIDE the work copy: WORK = absolute path of the work copy under test.
const work = process.env.WORK;
if (!work) throw new Error("WORK is not set");
export default defineConfig({
resolve: { alias: { "@work": work } },
server: { fs: { strict: false } },
test: { include: [`test_${process.env.TASK ?? "F1"}.test.ts`], testTimeout: 20_000, fileParallelism: false },
});
hidden/vitest.config.ts on GitHub
,hidden/package.json
{
"name": "flow-bench-hidden",
"private": true,
"type": "module",
"devDependencies": {
"vitest": "2.1.9"
}
}
hidden/pnpm-lock.yaml
lockfileVersion: '9.0'
settings:
autoInstallPeers: true
excludeLinksFromLockfile: false
importers:
.:
devDependencies:
vitest:
specifier: 2.1.9
version: 2.1.9
packages:
'@esbuild/aix-ppc64@0.21.5':
resolution: {integrity: sha512-1SDgH6ZSPTlggy1yI6+Dbkiz8xzpHJEVAlF/AM1tHPLsf5STom9rwtjE4hKAF20FfXXNTFqEYXyJNWh1GiZedQ==}
engines: {node: '>=12'}
cpu: [ppc64]
os: [aix]
'@esbuild/android-arm64@0.21.5':
resolution: {integrity: sha512-c0uX9VAUBQ7dTDCjq+wdyGLowMdtR/GoC2U5IYk/7D1H1JYC0qseD7+11iMP2mRLN9RcCMRcjC4YMclCzGwS/A==}
engines: {node: '>=12'}
cpu: [arm64]
os: [android]
'@esbuild/android-arm@0.21.5':
resolution: {integrity: sha512-vCPvzSjpPHEi1siZdlvAlsPxXl7WbOVUBBAowWug4rJHb68Ox8KualB+1ocNvT5fjv6wpkX6o/iEpbDrf68zcg==}
engines: {node: '>=12'}
cpu: [arm]
os: [android]
'@esbuild/android-x64@0.21.5':
resolution: {integrity: sha512-D7aPRUUNHRBwHxzxRvp856rjUHRFW1SdQATKXH2hqA0kAZb1hKmi02OpYRacl0TxIGz/ZmXWlbZgjwWYaCakTA==}
engines: {node: '>=12'}
cpu: [x64]
os: [android]
'@esbuild/darwin-arm64@0.21.5':
resolution: {integrity: sha512-DwqXqZyuk5AiWWf3UfLiRDJ5EDd49zg6O9wclZ7kUMv2WRFr4HKjXp/5t8JZ11QbQfUS6/cRCKGwYhtNAY88kQ==}
engines: {node: '>=12'}
cpu: [arm64]
os: [darwin]
'@esbuild/darwin-x64@0.21.5':
resolution: {integrity: sha512-se/JjF8NlmKVG4kNIuyWMV/22ZaerB+qaSi5MdrXtd6R08kvs2qCN4C09miupktDitvh8jRFflwGFBQcxZRjbw==}
engines: {node: '>=12'}
cpu: [x64]
os: [darwin]
'@esbuild/freebsd-arm64@0.21.5':
resolution: {integrity: sha512-5JcRxxRDUJLX8JXp/wcBCy3pENnCgBR9bN6JsY4OmhfUtIHe3ZW0mawA7+RDAcMLrMIZaf03NlQiX9DGyB8h4g==}
engines: {node: '>=12'}
cpu: [arm64]
os: [freebsd]
'@esbuild/freebsd-x64@0.21.5':
resolution: {integrity: sha512-J95kNBj1zkbMXtHVH29bBriQygMXqoVQOQYA+ISs0/2l3T9/kj42ow2mpqerRBxDJnmkUDCaQT/dfNXWX/ZZCQ==}
engines: {node: '>=12'}
cpu: [x64]
os: [freebsd]
'@esbuild/linux-arm64@0.21.5':
resolution: {integrity: sha512-ibKvmyYzKsBeX8d8I7MH/TMfWDXBF3db4qM6sy+7re0YXya+K1cem3on9XgdT2EQGMu4hQyZhan7TeQ8XkGp4Q==}
engines: {node: '>=12'}
cpu: [arm64]
os: [linux]
'@esbuild/linux-arm@0.21.5':
resolution: {integrity: sha512-bPb5AHZtbeNGjCKVZ9UGqGwo8EUu4cLq68E95A53KlxAPRmUyYv2D6F0uUI65XisGOL1hBP5mTronbgo+0bFcA==}
engines: {node: '>=12'}
cpu: [arm]
os: [linux]
'@esbuild/linux-ia32@0.21.5':
resolution: {integrity: sha512-YvjXDqLRqPDl2dvRODYmmhz4rPeVKYvppfGYKSNGdyZkA01046pLWyRKKI3ax8fbJoK5QbxblURkwK/MWY18Tg==}
engines: {node: '>=12'}
cpu: [ia32]
os: [linux]
'@esbuild/linux-loong64@0.21.5':
resolution: {integrity: sha512-uHf1BmMG8qEvzdrzAqg2SIG/02+4/DHB6a9Kbya0XDvwDEKCoC8ZRWI5JJvNdUjtciBGFQ5PuBlpEOXQj+JQSg==}
engines: {node: '>=12'}
cpu: [loong64]
os: [linux]
'@esbuild/linux-mips64el@0.21.5':
resolution: {integrity: sha512-IajOmO+KJK23bj52dFSNCMsz1QP1DqM6cwLUv3W1QwyxkyIWecfafnI555fvSGqEKwjMXVLokcV5ygHW5b3Jbg==}
engines: {node: '>=12'}
cpu: [mips64el]
os: [linux]
'@esbuild/linux-ppc64@0.21.5':
resolution: {integrity: sha512-1hHV/Z4OEfMwpLO8rp7CvlhBDnjsC3CttJXIhBi+5Aj5r+MBvy4egg7wCbe//hSsT+RvDAG7s81tAvpL2XAE4w==}
engines: {node: '>=12'}
cpu: [ppc64]
os: [linux]
'@esbuild/linux-riscv64@0.21.5':
resolution: {integrity: sha512-2HdXDMd9GMgTGrPWnJzP2ALSokE/0O5HhTUvWIbD3YdjME8JwvSCnNGBnTThKGEB91OZhzrJ4qIIxk/SBmyDDA==}
engines: {node: '>=12'}
cpu: [riscv64]
os: [linux]
'@esbuild/linux-s390x@0.21.5':
resolution: {integrity: sha512-zus5sxzqBJD3eXxwvjN1yQkRepANgxE9lgOW2qLnmr8ikMTphkjgXu1HR01K4FJg8h1kEEDAqDcZQtbrRnB41A==}
engines: {node: '>=12'}
cpu: [s390x]
os: [linux]
'@esbuild/linux-x64@0.21.5':
resolution: {integrity: sha512-1rYdTpyv03iycF1+BhzrzQJCdOuAOtaqHTWJZCWvijKD2N5Xu0TtVC8/+1faWqcP9iBCWOmjmhoH94dH82BxPQ==}
engines: {node: '>=12'}
cpu: [x64]
os: [linux]
'@esbuild/netbsd-x64@0.21.5':
resolution: {integrity: sha512-Woi2MXzXjMULccIwMnLciyZH4nCIMpWQAs049KEeMvOcNADVxo0UBIQPfSmxB3CWKedngg7sWZdLvLczpe0tLg==}
engines: {node: '>=12'}
cpu: [x64]
os: [netbsd]
'@esbuild/openbsd-x64@0.21.5':
resolution: {integrity: sha512-HLNNw99xsvx12lFBUwoT8EVCsSvRNDVxNpjZ7bPn947b8gJPzeHWyNVhFsaerc0n3TsbOINvRP2byTZ5LKezow==}
engines: {node: '>=12'}
cpu: [x64]
os: [openbsd]
'@esbuild/sunos-x64@0.21.5':
resolution: {integrity: sha512-6+gjmFpfy0BHU5Tpptkuh8+uw3mnrvgs+dSPQXQOv3ekbordwnzTVEb4qnIvQcYXq6gzkyTnoZ9dZG+D4garKg==}
engines: {node: '>=12'}
cpu: [x64]
os: [sunos]
'@esbuild/win32-arm64@0.21.5':
resolution: {integrity: sha512-Z0gOTd75VvXqyq7nsl93zwahcTROgqvuAcYDUr+vOv8uHhNSKROyU961kgtCD1e95IqPKSQKH7tBTslnS3tA8A==}
engines: {node: '>=12'}
cpu: [arm64]
os: [win32]
'@esbuild/win32-ia32@0.21.5':
resolution: {integrity: sha512-SWXFF1CL2RVNMaVs+BBClwtfZSvDgtL//G/smwAc5oVK/UPu2Gu9tIaRgFmYFFKrmg3SyAjSrElf0TiJ1v8fYA==}
engines: {node: '>=12'}
cpu: [ia32]
os: [win32]
'@esbuild/win32-x64@0.21.5':
resolution: {integrity: sha512-tQd/1efJuzPC6rCFwEvLtci/xNFcTZknmXs98FYDfGE4wP9ClFV98nyKrzJKVPMhdDnjzLhdUyMX4PsQAPjwIw==}
engines: {node: '>=12'}
cpu: [x64]
os: [win32]
'@jridgewell/sourcemap-codec@1.6.0':
resolution: {integrity: sha512-T7jf+5zgsZHwNJ4lvQ7/aezbyk0nNX+zJVWpmHA7VYsEx7a7qr5Rg5IbtJFqkgze5Y2sruq1RUY8Q837Od7iFw==}
'@napi-rs/lzma-linux-x64-gnu@1.5.1':
resolution: {integrity: sha512-oTXEIha4SsuXdTA4Iyskj0kpdx2yVXdhd75c2v3xGrHFfVMsbhTPZU/nMPL4sWKo4pBHm3aucLaqGlF696dTyQ==}
engines: {node: ^22.20 || ^24.12 || >=25}
cpu: [x64]
os: [linux]
libc: [glibc]
'@rollup/rollup-android-arm-eabi@4.64.2':
resolution: {integrity: sha512-gjNDUwMswogkw3fNqV00u54uZlgmInuw5EU/1TeqQ6K0bEagoGvqjPdKzy112lKyhe7PkGudZ/kSQHsc9qroIg==}
cpu: [arm]
os: [android]
'@rollup/rollup-android-arm64@4.64.2':
resolution: {integrity: sha512-YkAW8q7p5SCLSaGWaUkmsRnBEb3MGA1pB5nI/8k0lAK5fZsQKRBqkdR1h4UU9p8u3c7ZGdUF2Zcx7mK+E/pUKg==}
cpu: [arm64]
os: [android]
'@rollup/rollup-darwin-arm64@4.64.2':
resolution: {integrity: sha512-hNmuIHv+LnrP/+q/44vsmIMWfoh7X2YXOp5FLtjGuvqN52VdtFpOpnshTc1k28tWv1wE/0D5gLjsozBUowEGbQ==}
cpu: [arm64]
os: [darwin]
'@rollup/rollup-darwin-x64@4.64.2':
resolution: {integrity: sha512-B0t5CoxgF2KL7R5FHAsyN8yVEQCSIkh4XVs8MCDjuBLNjEa8l3zd5tVKaFMv53LfUtHs9p2rnqeavPTOunV+Ww==}
cpu: [x64]
os: [darwin]
'@rollup/rollup-freebsd-arm64@4.64.2':
resolution: {integrity: sha512-QQdYnfwrPmc1DoPlddjFvQPzUHFbStKXsHrG6EBYsWHhBVUCjsCRdcfGGXGAvhDpDtTzKK4vy1UrGFJ45rQaiw==}
cpu: [arm64]
os: [freebsd]
'@rollup/rollup-freebsd-x64@4.64.2':
resolution: {integrity: sha512-1xRxtGWif6BB58VSfMavi0Yfs2M4x6CtRp4o7ScY4W7AGlgvf6GSPKdDNdw7TX3tMiQwZY5W/mvyrOJTq/7AKA==}
cpu: [x64]
os: [freebsd]
'@rollup/rollup-linux-arm-gnueabihf@4.64.2':
resolution: {integrity: sha512-CXFLEt384BUwwOLR9DURWNHl1bQOVLd1ibOee+yKBJm/fZtvrSAOZ0JbxjAxszVMIovYMwFMS+oP+Q1aPcx8Eg==}
cpu: [arm]
os: [linux]
libc: [glibc]
'@rollup/rollup-linux-arm-musleabihf@4.64.2':
resolution: {integrity: sha512-T1dorY0WHm5mMNpMHFl5rYRZ02gLSZcr6J45r66tYb3S3R3DyIRGiGWVdLtF+zOQjratNJaObl9jYumueRiL1A==}
cpu: [arm]
os: [linux]
libc: [musl]
'@rollup/rollup-linux-arm64-gnu@4.64.2':
resolution: {integrity: sha512-ia6bfaCIc0ldp6bvs9oEkFrJVJKNUMoiDF0+LBZsQAZg+vE0znXnzGqH+ew/FYswE30cruAWPTO3dPw7keaVkQ==}
cpu: [arm64]
os: [linux]
libc: [glibc]
'@rollup/rollup-linux-arm64-musl@4.64.2':
resolution: {integrity: sha512-mB/gwfBfXtT0EBpbSssqKCqMFuX8b21Zy2TGoIhehSJEssZ5ksybxymwp1BkEM5Nczh/AAxpmh5q+MkhtundGg==}
cpu: [arm64]
os: [linux]
libc: [musl]
'@rollup/rollup-linux-loong64-gnu@4.64.2':
resolution: {integrity: sha512-VPd0a+XlHoNRI9jIPRHww2UKhDkD6XjbtcKYXZijAnhJS1jLLbiNtVl6a5LyEfNN+eyiyWcJOZXPzBo69oTsew==}
cpu: [loong64]
os: [linux]
libc: [glibc]
'@rollup/rollup-linux-loong64-musl@4.64.2':
resolution: {integrity: sha512-10N1I6oBpERqzGtHKcCLrEpHlbXSBbMK9TURHvqgWbIFZuU3oJROxJ0H2nHfq96TVI0XJHQ9qzolJMFd2tz0LA==}
cpu: [loong64]
os: [linux]
libc: [musl]
'@rollup/rollup-linux-ppc64-gnu@4.64.2':
resolution: {integrity: sha512-dhJrLVZ3wNNhwCmy9/VTg6Pq6J8a8eZ0n15dVRlbSWcDb+8V/1mDfbnvis+vt8W6Pbxzv6gUbTEl4piOT3bbUQ==}
cpu: [ppc64]
os: [linux]
libc: [glibc]
'@rollup/rollup-linux-ppc64-musl@4.64.2':
resolution: {integrity: sha512-tPd2QuwxkTxgju5eARyb3RdgBWWi/O9dn/NFH0VkvGbLzoQeXrH6qMABtW8IODb5lUeBzB3sy2U7qmI4MtQ72g==}
cpu: [ppc64]
os: [linux]
libc: [musl]
'@rollup/rollup-linux-riscv64-gnu@4.64.2':
resolution: {integrity: sha512-njtDdPvSQWI9CDhIPPxsiEMeHayCfu7wwcbSQ7c2kYPhGVviL+WGVhq6Jfj/b+VuhSTkFuQ0z4ZFbFjkWwBFmg==}
cpu: [riscv64]
os: [linux]
libc: [glibc]
'@rollup/rollup-linux-riscv64-musl@4.64.2':
resolution: {integrity: sha512-EbOBo3dqCj9ecEV9Z7zvulB6AieMP2ztnKQYGVOSN0pzIyOu+3JH2YJZnLmbPeVvfn6yP6Qi7NbBM27EzGbj2w==}
cpu: [riscv64]
os: [linux]
libc: [musl]
'@rollup/rollup-linux-s390x-gnu@4.64.2':
resolution: {integrity: sha512-8wLt2Q5n4qmfiE2Gk8V6KNLSf+oJuBX7Kdw5joFDtnZIZvirS0hLzBc79OiJjV71oYDmBUZLNy6pDgvylo4qiQ==}
cpu: [s390x]
os: [linux]
libc: [glibc]
'@rollup/rollup-linux-x64-gnu@4.64.2':
resolution: {integrity: sha512-aIChbkhC4d33bxkVP0LSi2/2lUSGRXE6k4Qa0G7tgpgTOK7kxHTixmx5qC3SKUJ6yUMznHBtQoWaG4QMqLUnhw==}
cpu: [x64]
os: [linux]
libc: [glibc]
'@rollup/rollup-linux-x64-musl@4.64.2':
resolution: {integrity: sha512-gQ+TkVZRfjZVcl/UOAxxfWw+sHjzIXoieNwxEyJlD3tRBmBhR8ZDq11w+QEgQDcpslEB+fhN/OEpaTsDoIIHhQ==}
cpu: [x64]
os: [linux]
libc: [musl]
'@rollup/rollup-openbsd-x64@4.64.2':
resolution: {integrity: sha512-sUaJpMKS2JBCvUk8fYimG7aMq2jHVT4kw/uVHxPolKLrMmkKy0KXirmZyZehdNpL7PffqAlPAWUELQcTiD8ChA==}
cpu: [x64]
os: [openbsd]
'@rollup/rollup-openharmony-arm64@4.64.2':
resolution: {integrity: sha512-j1RdJWb0CrlSuwt12OPjLucPlwfLtA/KMOKVvh5rXSJEtl/WDoIGR3CSykK/jLGTo9RqNfW5Ub6kY5Z5ccYl9w==}
cpu: [arm64]
os: [openharmony]
'@rollup/rollup-win32-arm64-msvc@4.64.2':
resolution: {integrity: sha512-i3r04aHCIY3eggTBLUp/4IxiBjDpAgQaK89Ue2nMJ2MKuIXX5JfoeZjHw+kPjGGNVDlhi8FT8xzuGMJZe3aehg==}
cpu: [arm64]
os: [win32]
'@rollup/rollup-win32-ia32-msvc@4.64.2':
resolution: {integrity: sha512-kCQ15h/uwNMGdN8gjXBZGf+aGE+ODzHfbi90/cAm4+Q89WoMiLODNoj8HoGZmYx87HPxfu5CBNPsZ+ZG4uIAoA==}
cpu: [ia32]
os: [win32]
'@rollup/rollup-win32-x64-gnu@4.64.2':
resolution: {integrity: sha512-yYAuySSs+0X2gSMJPw/Q4jHsHZHyMq1dpAE+XnN790Jg1WuReUAD/+j+e4vuDvS4PXJ4PA4NCzTR4OYtDf18/g==}
cpu: [x64]
os: [win32]
'@rollup/rollup-win32-x64-msvc@4.64.2':
resolution: {integrity: sha512-JY1rQdWOJrKutORqKO1uzrHXV86jFiQzfLFKx2s8S1CjLXL9fC5LOIvk0q7Z7PXJvsByt+sdoE1Q2a9gBr4gLQ==}
cpu: [x64]
os: [win32]
'@types/estree@1.0.9':
resolution: {integrity: sha512-GhdPgy1el4/ImP05X05Uw4cw2/M93BCUmnEvWZNStlCzEKME4Fkk+YpoA5OiHNQmoS7Cafb8Xa3Pya8m1Qrzeg==}
'@vitest/expect@2.1.9':
resolution: {integrity: sha512-UJCIkTBenHeKT1TTlKMJWy1laZewsRIzYighyYiJKZreqtdxSos/S1t+ktRMQWu2CKqaarrkeszJx1cgC5tGZw==}
'@vitest/mocker@2.1.9':
resolution: {integrity: sha512-tVL6uJgoUdi6icpxmdrn5YNo3g3Dxv+IHJBr0GXHaEdTcw3F+cPKnsXFhli6nO+f/6SDKPHEK1UN+k+TQv0Ehg==}
peerDependencies:
msw: ^2.4.9
vite: ^5.0.0
peerDependenciesMeta:
msw:
optional: true
vite:
optional: true
'@vitest/pretty-format@2.1.9':
resolution: {integrity: sha512-KhRIdGV2U9HOUzxfiHmY8IFHTdqtOhIzCpd8WRdJiE7D/HUcZVD0EgQCVjm+Q9gkUXWgBvMmTtZgIG48wq7sOQ==}
'@vitest/runner@2.1.9':
resolution: {integrity: sha512-ZXSSqTFIrzduD63btIfEyOmNcBmQvgOVsPNPe0jYtESiXkhd8u2erDLnMxmGrDCwHCCHE7hxwRDCT3pt0esT4g==}
'@vitest/snapshot@2.1.9':
resolution: {integrity: sha512-oBO82rEjsxLNJincVhLhaxxZdEtV0EFHMK5Kmx5sJ6H9L183dHECjiefOAdnqpIgT5eZwT04PoggUnW88vOBNQ==}
'@vitest/spy@2.1.9':
resolution: {integrity: sha512-E1B35FwzXXTs9FHNK6bDszs7mtydNi5MIfUWpceJ8Xbfb1gBMscAnwLbEu+B44ed6W3XjL9/ehLPHR1fkf1KLQ==}
'@vitest/utils@2.1.9':
resolution: {integrity: sha512-v0psaMSkNJ3A2NMrUEHFRzJtDPFn+/VWZ5WxImB21T9fjucJRmS7xCS3ppEnARb9y11OAzaD+P2Ps+b+BGX5iQ==}
assertion-error@2.0.1:
resolution: {integrity: sha512-Izi8RQcffqCeNVgFigKli1ssklIbpHnCYc6AknXGYoB6grJqyeby7jv12JUQgmTAnIDnbck1uxksT4dzN3PWBA==}
engines: {node: '>=12'}
cac@6.7.14:
resolution: {integrity: sha512-b6Ilus+c3RrdDk+JhLKUAQfzzgLEPy6wcXqS7f/xe1EETvsDP6GORG7SFuOs6cID5YkqchW/LXZbX5bc8j7ZcQ==}
engines: {node: '>=8'}
chai@5.3.3:
resolution: {integrity: sha512-4zNhdJD/iOjSH0A05ea+Ke6MU5mmpQcbQsSOkgdaUMJ9zTlDTD/GYlwohmIE2u0gaxHYiVHEn1Fw9mZ/ktJWgw==}
engines: {node: '>=18'}
check-error@2.1.3:
resolution: {integrity: sha512-PAJdDJusoxnwm1VwW07VWwUN1sl7smmC3OKggvndJFadxxDRyFJBX/ggnu/KE4kQAB7a3Dp8f/YXC1FlUprWmA==}
engines: {node: '>= 16'}
debug@4.4.3:
resolution: {integrity: sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA==}
engines: {node: '>=6.0'}
peerDependencies:
supports-color: '*'
peerDependenciesMeta:
supports-color:
optional: true
deep-eql@5.0.2:
resolution: {integrity: sha512-h5k/5U50IJJFpzfL6nO9jaaumfjO/f2NjK/oYB2Djzm4p9L+3T9qWpZqZ2hAbLPuuYq9wrU08WQyBTL5GbPk5Q==}
engines: {node: '>=6'}
es-module-lexer@1.7.0:
resolution: {integrity: sha512-jEQoCwk8hyb2AZziIOLhDqpm5+2ww5uIE6lkO/6jcOCusfk6LhMHpXXfBLXTZ7Ydyt0j4VoUQv6uGNYbdW+kBA==}
esbuild@0.21.5:
resolution: {integrity: sha512-mg3OPMV4hXywwpoDxu3Qda5xCKQi+vCTZq8S9J/EpkhB2HzKXq4SNFZE3+NK93JYxc8VMSep+lOUSC/RVKaBqw==}
engines: {node: '>=12'}
hasBin: true
estree-walker@3.0.3:
resolution: {integrity: sha512-7RUKfXgSMMkzt6ZuXmqapOurLGPPfgj6l9uRZ7lRGolvk0y2yocc35LdcxKC5PQZdn2DMqioAQ2NoWcrTKmm6g==}
expect-type@1.4.0:
resolution: {integrity: sha512-KfYbmpRm0VbLjEvVa9yGwCi9GI34xvi7A/HXYWQO65CSD2u3MczUJSuwXKFIxlGsgBQizV9q5J9NHj4VG0n+pA==}
engines: {node: '>=12.0.0'}
fsevents@2.3.3:
resolution: {integrity: sha512-5xoDfX+fL7faATnagmWPpbFtwh/R77WmMMqqHGS65C3vvB0YHrgF+B1YmZ3441tMj5n63k0212XNoJwzlhffQw==}
engines: {node: ^8.16.0 || ^10.6.0 || >=11.0.0}
os: [darwin]
loupe@3.2.1:
resolution: {integrity: sha512-CdzqowRJCeLU72bHvWqwRBBlLcMEtIvGrlvef74kMnV2AolS9Y8xUv1I0U/MNAWMhBlKIoyuEgoJ0t/bbwHbLQ==}
magic-string@0.30.21:
resolution: {integrity: sha512-vd2F4YUyEXKGcLHoq+TEyCjxueSeHnFxyyjNp80yg0XV4vUhnDer/lvvlqM/arB5bXQN5K2/3oinyCRyx8T2CQ==}
ms@2.1.3:
resolution: {integrity: sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==}
nanoid@3.3.20:
resolution: {integrity: sha512-uKdg2G3GNCKQn9byYOpxbGqrT2fGO5KRt5J/8b3pok8rT6qxGWF6hxMyJiEYtAf+FVyYuD9hRaDqX5uPFYJ4ZQ==}
engines: {node: ^10 || ^12 || ^13.7 || ^14 || >=15.0.1}
hasBin: true
pathe@1.1.2:
resolution: {integrity: sha512-whLdWMYL2TwI08hn8/ZqAbrVemu0LNaNNJZX73O6qaIdCTfXutsLhMkjdENX0qhsQ9uIimo4/aQOmXkoon2nDQ==}
pathval@2.0.1:
resolution: {integrity: sha512-//nshmD55c46FuFw26xV/xFAaB5HF9Xdap7HJBBnrKdAd6/GxDBaNA1870O79+9ueg61cZLSVc+OaFlfmObYVQ==}
engines: {node: '>= 14.16'}
picocolors@1.1.1:
resolution: {integrity: sha512-xceH2snhtb5M9liqDsmEw56le376mTZkEX/jEb/RxNFyegNul7eNslCXP9FDj/Lcu0X8KEyMceP2ntpaHrDEVA==}
postcss@8.5.29:
resolution: {integrity: sha512-49cGhUbXj8Qenv0iTMxA1cFBzxXoctpC9Ujd77t1WcbJIr6nF/eI7g/8MgxrYldFRuAXvja7xQRwavoW7kgrxQ==}
engines: {node: ^10 || ^12 || >=14}
rollup@4.64.2:
resolution: {integrity: sha512-RxXm0XVVV1hFI3dYe12ruaOgsYQ725lUcUWeMSQqkF9tTxVErY+fy2Ets6vWhItL9MZMSNZGbvZFc7rH/VHp2Q==}
engines: {node: '>=18.0.0', npm: '>=8.0.0'}
hasBin: true
siginfo@2.0.0:
resolution: {integrity: sha512-ybx0WO1/8bSBLEWXZvEd7gMW3Sn3JFlW3TvX1nREbDLRNQNaeNN8WK0meBwPdAaOI7TtRRRJn/Es1zhrrCHu7g==}
source-map-js@1.2.2:
resolution: {integrity: sha512-KGj/8Y43x35aZVDtt+J4mK1hoLGHULMYfSkODJNQjNDC3oW1PqPoxMwo0pLUsWM/UEGzON/NxeHywEfNXNP3Vw==}
engines: {node: '>=0.10.0'}
stackback@0.0.2:
resolution: {integrity: sha512-1XMJE5fQo1jGH6Y/7ebnwPOBEkIEnT4QF32d5R1+VXdXveM0IBMJt8zfaxX1P3QhVwrYe+576+jkANtSS2mBbw==}
std-env@3.10.0:
resolution: {integrity: sha512-5GS12FdOZNliM5mAOxFRg7Ir0pWz8MdpYm6AY6VPkGpbA7ZzmbzNcBJQ0GPvvyWgcY7QAhCgf9Uy89I03faLkg==}
tinybench@2.9.0:
resolution: {integrity: sha512-0+DUvqWMValLmha6lr4kD8iAMK1HzV0/aKnCtWb9v9641TnP/MFb7Pc2bxoxQjTXAErryXVgUOfv2YqNllqGeg==}
tinyexec@0.3.2:
resolution: {integrity: sha512-KQQR9yN7R5+OSwaK0XQoj22pwHoTlgYqmUscPYoknOoWCWfj/5/ABTMRi69FrKU5ffPVh5QcFikpWJI/P1ocHA==}
tinypool@1.1.1:
resolution: {integrity: sha512-Zba82s87IFq9A9XmjiX5uZA/ARWDrB03OHlq+Vw1fSdt0I+4/Kutwy8BP4Y/y/aORMo61FQ0vIb5j44vSo5Pkg==}
engines: {node: ^18.0.0 || >=20.0.0}
tinyrainbow@1.2.0:
resolution: {integrity: sha512-weEDEq7Z5eTHPDh4xjX789+fHfF+P8boiFB+0vbWzpbnbsEr/GRaohi/uMKxg8RZMXnl1ItAi/IUHWMsjDV7kQ==}
engines: {node: '>=14.0.0'}
tinyspy@3.0.2:
resolution: {integrity: sha512-n1cw8k1k0x4pgA2+9XrOkFydTerNcJ1zWCO5Nn9scWHTD+5tp8dghT2x1uduQePZTZgd3Tupf+x9BxJjeJi77Q==}
engines: {node: '>=14.0.0'}
vite-node@2.1.9:
resolution: {integrity: sha512-AM9aQ/IPrW/6ENLQg3AGY4K1N2TGZdR5e4gu/MmmR2xR3Ll1+dib+nook92g4TV3PXVyeyxdWwtaCAiUL0hMxA==}
engines: {node: ^18.0.0 || >=20.0.0}
hasBin: true
vite@5.4.21:
resolution: {integrity: sha512-o5a9xKjbtuhY6Bi5S3+HvbRERmouabWbyUcpXXUA1u+GNUKoROi9byOJ8M0nHbHYHkYICiMlqxkg1KkYmm25Sw==}
engines: {node: ^18.0.0 || >=20.0.0}
hasBin: true
peerDependencies:
'@types/node': ^18.0.0 || >=20.0.0
less: '*'
lightningcss: ^1.21.0
sass: '*'
sass-embedded: '*'
stylus: '*'
sugarss: '*'
terser: ^5.4.0
peerDependenciesMeta:
'@types/node':
optional: true
less:
optional: true
lightningcss:
optional: true
sass:
optional: true
sass-embedded:
optional: true
stylus:
optional: true
sugarss:
optional: true
terser:
optional: true
vitest@2.1.9:
resolution: {integrity: sha512-MSmPM9REYqDGBI8439mA4mWhV5sKmDlBKWIYbA3lRb2PTHACE0mgKwA8yQ2xq9vxDTuk4iPrECBAEW2aoFXY0Q==}
engines: {node: ^18.0.0 || >=20.0.0}
hasBin: true
peerDependencies:
'@edge-runtime/vm': '*'
'@types/node': ^18.0.0 || >=20.0.0
'@vitest/browser': 2.1.9
'@vitest/ui': 2.1.9
happy-dom: '*'
jsdom: '*'
peerDependenciesMeta:
'@edge-runtime/vm':
optional: true
'@types/node':
optional: true
'@vitest/browser':
optional: true
'@vitest/ui':
optional: true
happy-dom:
optional: true
jsdom:
optional: true
why-is-node-running@2.3.0:
resolution: {integrity: sha512-hUrmaWBdVDcxvYqnyh09zunKzROWjbZTiNy8dBEjkS7ehEDQibXJ7XvlmtbwuTclUiIyN+CyXQD4Vmko8fNm8w==}
engines: {node: '>=8'}
hasBin: true
snapshots:
'@esbuild/aix-ppc64@0.21.5':
optional: true
'@esbuild/android-arm64@0.21.5':
optional: true
'@esbuild/android-arm@0.21.5':
optional: true
'@esbuild/android-x64@0.21.5':
optional: true
'@esbuild/darwin-arm64@0.21.5':
optional: true
'@esbuild/darwin-x64@0.21.5':
optional: true
'@esbuild/freebsd-arm64@0.21.5':
optional: true
'@esbuild/freebsd-x64@0.21.5':
optional: true
'@esbuild/linux-arm64@0.21.5':
optional: true
'@esbuild/linux-arm@0.21.5':
optional: true
'@esbuild/linux-ia32@0.21.5':
optional: true
'@esbuild/linux-loong64@0.21.5':
optional: true
'@esbuild/linux-mips64el@0.21.5':
optional: true
'@esbuild/linux-ppc64@0.21.5':
optional: true
'@esbuild/linux-riscv64@0.21.5':
optional: true
'@esbuild/linux-s390x@0.21.5':
optional: true
'@esbuild/linux-x64@0.21.5':
optional: true
'@esbuild/netbsd-x64@0.21.5':
optional: true
'@esbuild/openbsd-x64@0.21.5':
optional: true
'@esbuild/sunos-x64@0.21.5':
optional: true
'@esbuild/win32-arm64@0.21.5':
optional: true
'@esbuild/win32-ia32@0.21.5':
optional: true
'@esbuild/win32-x64@0.21.5':
optional: true
'@jridgewell/sourcemap-codec@1.6.0': {}
'@napi-rs/lzma-linux-x64-gnu@1.5.1':
optional: true
'@rollup/rollup-android-arm-eabi@4.64.2':
optional: true
'@rollup/rollup-android-arm64@4.64.2':
optional: true
'@rollup/rollup-darwin-arm64@4.64.2':
optional: true
'@rollup/rollup-darwin-x64@4.64.2':
optional: true
'@rollup/rollup-freebsd-arm64@4.64.2':
optional: true
'@rollup/rollup-freebsd-x64@4.64.2':
optional: true
'@rollup/rollup-linux-arm-gnueabihf@4.64.2':
optional: true
'@rollup/rollup-linux-arm-musleabihf@4.64.2':
optional: true
'@rollup/rollup-linux-arm64-gnu@4.64.2':
optional: true
'@rollup/rollup-linux-arm64-musl@4.64.2':
optional: true
'@rollup/rollup-linux-loong64-gnu@4.64.2':
optional: true
'@rollup/rollup-linux-loong64-musl@4.64.2':
optional: true
'@rollup/rollup-linux-ppc64-gnu@4.64.2':
optional: true
'@rollup/rollup-linux-ppc64-musl@4.64.2':
optional: true
'@rollup/rollup-linux-riscv64-gnu@4.64.2':
optional: true
'@rollup/rollup-linux-riscv64-musl@4.64.2':
optional: true
'@rollup/rollup-linux-s390x-gnu@4.64.2':
optional: true
'@rollup/rollup-linux-x64-gnu@4.64.2':
optional: true
'@rollup/rollup-linux-x64-musl@4.64.2':
optional: true
'@rollup/rollup-openbsd-x64@4.64.2':
optional: true
'@rollup/rollup-openharmony-arm64@4.64.2':
optional: true
'@rollup/rollup-win32-arm64-msvc@4.64.2':
optional: true
'@rollup/rollup-win32-ia32-msvc@4.64.2':
optional: true
'@rollup/rollup-win32-x64-gnu@4.64.2':
optional: true
'@rollup/rollup-win32-x64-msvc@4.64.2':
optional: true
'@types/estree@1.0.9': {}
'@vitest/expect@2.1.9':
dependencies:
'@vitest/spy': 2.1.9
'@vitest/utils': 2.1.9
chai: 5.3.3
tinyrainbow: 1.2.0
'@vitest/mocker@2.1.9(vite@5.4.21)':
dependencies:
'@vitest/spy': 2.1.9
estree-walker: 3.0.3
magic-string: 0.30.21
optionalDependencies:
vite: 5.4.21
'@vitest/pretty-format@2.1.9':
dependencies:
tinyrainbow: 1.2.0
'@vitest/runner@2.1.9':
dependencies:
'@vitest/utils': 2.1.9
pathe: 1.1.2
'@vitest/snapshot@2.1.9':
dependencies:
'@vitest/pretty-format': 2.1.9
magic-string: 0.30.21
pathe: 1.1.2
'@vitest/spy@2.1.9':
dependencies:
tinyspy: 3.0.2
'@vitest/utils@2.1.9':
dependencies:
'@vitest/pretty-format': 2.1.9
loupe: 3.2.1
tinyrainbow: 1.2.0
assertion-error@2.0.1: {}
cac@6.7.14: {}
chai@5.3.3:
dependencies:
assertion-error: 2.0.1
check-error: 2.1.3
deep-eql: 5.0.2
loupe: 3.2.1
pathval: 2.0.1
check-error@2.1.3: {}
debug@4.4.3:
dependencies:
ms: 2.1.3
deep-eql@5.0.2: {}
es-module-lexer@1.7.0: {}
esbuild@0.21.5:
optionalDependencies:
'@esbuild/aix-ppc64': 0.21.5
'@esbuild/android-arm': 0.21.5
'@esbuild/android-arm64': 0.21.5
'@esbuild/android-x64': 0.21.5
'@esbuild/darwin-arm64': 0.21.5
'@esbuild/darwin-x64': 0.21.5
'@esbuild/freebsd-arm64': 0.21.5
'@esbuild/freebsd-x64': 0.21.5
'@esbuild/linux-arm': 0.21.5
'@esbuild/linux-arm64': 0.21.5
'@esbuild/linux-ia32': 0.21.5
'@esbuild/linux-loong64': 0.21.5
'@esbuild/linux-mips64el': 0.21.5
'@esbuild/linux-ppc64': 0.21.5
'@esbuild/linux-riscv64': 0.21.5
'@esbuild/linux-s390x': 0.21.5
'@esbuild/linux-x64': 0.21.5
'@esbuild/netbsd-x64': 0.21.5
'@esbuild/openbsd-x64': 0.21.5
'@esbuild/sunos-x64': 0.21.5
'@esbuild/win32-arm64': 0.21.5
'@esbuild/win32-ia32': 0.21.5
'@esbuild/win32-x64': 0.21.5
estree-walker@3.0.3:
dependencies:
'@types/estree': 1.0.9
expect-type@1.4.0: {}
fsevents@2.3.3:
optional: true
loupe@3.2.1: {}
magic-string@0.30.21:
dependencies:
'@jridgewell/sourcemap-codec': 1.6.0
ms@2.1.3: {}
nanoid@3.3.20: {}
pathe@1.1.2: {}
pathval@2.0.1: {}
picocolors@1.1.1: {}
postcss@8.5.29:
dependencies:
nanoid: 3.3.20
picocolors: 1.1.1
source-map-js: 1.2.2
rollup@4.64.2:
dependencies:
'@types/estree': 1.0.9
optionalDependencies:
'@napi-rs/lzma-linux-x64-gnu': 1.5.1
'@rollup/rollup-android-arm-eabi': 4.64.2
'@rollup/rollup-android-arm64': 4.64.2
'@rollup/rollup-darwin-arm64': 4.64.2
'@rollup/rollup-darwin-x64': 4.64.2
'@rollup/rollup-freebsd-arm64': 4.64.2
'@rollup/rollup-freebsd-x64': 4.64.2
'@rollup/rollup-linux-arm-gnueabihf': 4.64.2
'@rollup/rollup-linux-arm-musleabihf': 4.64.2
'@rollup/rollup-linux-arm64-gnu': 4.64.2
'@rollup/rollup-linux-arm64-musl': 4.64.2
'@rollup/rollup-linux-loong64-gnu': 4.64.2
'@rollup/rollup-linux-loong64-musl': 4.64.2
'@rollup/rollup-linux-ppc64-gnu': 4.64.2
'@rollup/rollup-linux-ppc64-musl': 4.64.2
'@rollup/rollup-linux-riscv64-gnu': 4.64.2
'@rollup/rollup-linux-riscv64-musl': 4.64.2
'@rollup/rollup-linux-s390x-gnu': 4.64.2
'@rollup/rollup-linux-x64-gnu': 4.64.2
'@rollup/rollup-linux-x64-musl': 4.64.2
'@rollup/rollup-openbsd-x64': 4.64.2
'@rollup/rollup-openharmony-arm64': 4.64.2
'@rollup/rollup-win32-arm64-msvc': 4.64.2
'@rollup/rollup-win32-ia32-msvc': 4.64.2
'@rollup/rollup-win32-x64-gnu': 4.64.2
'@rollup/rollup-win32-x64-msvc': 4.64.2
fsevents: 2.3.3
siginfo@2.0.0: {}
source-map-js@1.2.2: {}
stackback@0.0.2: {}
std-env@3.10.0: {}
tinybench@2.9.0: {}
tinyexec@0.3.2: {}
tinypool@1.1.1: {}
tinyrainbow@1.2.0: {}
tinyspy@3.0.2: {}
vite-node@2.1.9:
dependencies:
cac: 6.7.14
debug: 4.4.3
es-module-lexer: 1.7.0
pathe: 1.1.2
vite: 5.4.21
transitivePeerDependencies:
- '@types/node'
- less
- lightningcss
- sass
- sass-embedded
- stylus
- sugarss
- supports-color
- terser
vite@5.4.21:
dependencies:
esbuild: 0.21.5
postcss: 8.5.29
rollup: 4.64.2
optionalDependencies:
fsevents: 2.3.3
vitest@2.1.9:
dependencies:
'@vitest/expect': 2.1.9
'@vitest/mocker': 2.1.9(vite@5.4.21)
'@vitest/pretty-format': 2.1.9
'@vitest/runner': 2.1.9
'@vitest/snapshot': 2.1.9
'@vitest/spy': 2.1.9
'@vitest/utils': 2.1.9
chai: 5.3.3
debug: 4.4.3
expect-type: 1.4.0
magic-string: 0.30.21
pathe: 1.1.2
std-env: 3.10.0
tinybench: 2.9.0
tinyexec: 0.3.2
tinypool: 1.1.1
tinyrainbow: 1.2.0
vite: 5.4.21
vite-node: 2.1.9
why-is-node-running: 2.3.0
transitivePeerDependencies:
- less
- lightningcss
- msw
- sass
- sass-embedded
- stylus
- sugarss
- supports-color
- terser
why-is-node-running@2.3.0:
dependencies:
siginfo: 2.0.0
stackback: 0.0.2
hidden/pnpm-lock.yaml on GitHub
vehidden/pnpm-workspace.yaml
allowBuilds:
esbuild: true
hidden/pnpm-workspace.yaml on GitHub
kurulumu. -reference/F1.patch
diff --git a/src/domain/invoice.ts b/src/domain/invoice.ts
index 5b69fc9..0b9b69f 100644
--- a/src/domain/invoice.ts
+++ b/src/domain/invoice.ts
@@ -1,4 +1,4 @@
-import { ConflictError } from "./errors";
+import { ConflictError, ValidationError } from "./errors";
import { type Cents, type Currency, sum } from "./money";
export type InvoiceStatus = "draft" | "open" | "paid" | "void";
@@ -10,6 +10,12 @@ export type LineItem = {
unitPrice: Cents;
};
+export type Payment = {
+ /** cents, > 0 */
+ amount: Cents;
+ receivedAt: string;
+};
+
export type Invoice = {
id: string;
/** Assigned when the invoice is issued (INV-0001, ...). */
@@ -25,6 +31,8 @@ export type Invoice = {
dueDate: string | null;
paidAt: string | null;
voidedAt: string | null;
+ /** Every payment received, oldest first. A paid invoice's payments add up to its total. */
+ payments: Payment[];
};
export type InvoiceTotals = { subtotal: Cents; discount: Cents; total: Cents };
@@ -38,6 +46,14 @@ export function computeTotals(invoice: Pick<Invoice, "lines" | "discount">): Inv
return { subtotal, discount: invoice.discount, total: subtotal - invoice.discount };
}
+export function amountPaid(invoice: Pick<Invoice, "payments">): Cents {
+ return sum(invoice.payments.map((p) => p.amount));
+}
+
+export function balance(invoice: Pick<Invoice, "lines" | "discount" | "payments">): Cents {
+ return computeTotals(invoice).total - amountPaid(invoice);
+}
+
export function issue(invoice: Invoice, number: string, now: Date): Invoice {
if (invoice.status !== "draft") {
throw new ConflictError(`invoice ${invoice.id} is ${invoice.status}, only drafts can be issued`);
@@ -45,11 +61,28 @@ export function issue(invoice: Invoice, number: string, now: Date): Invoice {
return { ...invoice, status: "open", number, issuedAt: now.toISOString() };
}
+/** Pays whatever is left on an open invoice in one go. */
export function markPaid(invoice: Invoice, now: Date): Invoice {
if (invoice.status !== "open") {
throw new ConflictError(`invoice ${invoice.id} is ${invoice.status}, only open invoices can be paid`);
}
- return { ...invoice, status: "paid", paidAt: now.toISOString() };
+ const remaining = balance(invoice);
+ const payments = remaining > 0 ? [...invoice.payments, { amount: remaining, receivedAt: now.toISOString() }] : invoice.payments;
+ return { ...invoice, payments, status: "paid", paidAt: now.toISOString() };
+}
+
+/** Records a partial (or final) payment; the invoice flips to paid when its balance reaches zero. */
+export function recordPayment(invoice: Invoice, amount: Cents, now: Date): Invoice {
+ if (invoice.status !== "open") {
+ throw new ConflictError(`invoice ${invoice.id} is ${invoice.status}, only open invoices can take payments`);
+ }
+ const remaining = balance(invoice);
+ if (amount > remaining) {
+ throw new ValidationError(`amount ${amount} exceeds the balance of ${remaining}`, "amount");
+ }
+ const payments = [...invoice.payments, { amount, receivedAt: now.toISOString() }];
+ const settled = amount === remaining;
+ return { ...invoice, payments, status: settled ? "paid" : "open", paidAt: settled ? now.toISOString() : null };
}
export function voidInvoice(invoice: Invoice, now: Date): Invoice {
diff --git a/src/domain/validation.ts b/src/domain/validation.ts
index 08dfa92..f2af418 100644
--- a/src/domain/validation.ts
+++ b/src/domain/validation.ts
@@ -10,6 +10,13 @@ export type NewInvoiceInput = {
dueDate: string | null;
};
+export function parsePayment(body: unknown): { amount: number } {
+ if (typeof body !== "object" || body === null) {
+ throw new ValidationError("body must be an object");
+ }
+ return { amount: assertCents((body as Record<string, unknown>).amount, "amount", { min: 1 }) };
+}
+
const DATE = /^\d{4}-\d{2}-\d{2}$/;
export function parseLineItems(value: unknown): LineItem[] {
diff --git a/src/http/handlers/invoices.ts b/src/http/handlers/invoices.ts
index 09959e1..ac2768e 100644
--- a/src/http/handlers/invoices.ts
+++ b/src/http/handlers/invoices.ts
@@ -1,6 +1,6 @@
import { ValidationError } from "../../domain/errors";
-import { type Invoice, type InvoiceStatus, issue, markPaid, voidInvoice } from "../../domain/invoice";
-import { parseNewInvoice } from "../../domain/validation";
+import { type Invoice, type InvoiceStatus, issue, markPaid, recordPayment, voidInvoice } from "../../domain/invoice";
+import { parseNewInvoice, parsePayment } from "../../domain/validation";
import type { AppContext } from "../context";
import type { Router } from "../router";
import { invoiceView } from "../views";
@@ -22,6 +22,7 @@ export function invoiceRoutes(router: Router, ctx: AppContext): void {
issuedAt: null,
paidAt: null,
voidedAt: null,
+ payments: [],
};
return { status: 201, body: invoiceView(ctx.invoices.insert(invoice)) };
});
@@ -51,6 +52,14 @@ export function invoiceRoutes(router: Router, ctx: AppContext): void {
return { status: 200, body: invoiceView(ctx.invoices.save(paid)) };
});
+ /** Records a partial payment; the last one (balance reaches zero) marks the invoice paid. */
+ router.on("POST", "/invoices/:id/payments", ({ params, body }) => {
+ const current = ctx.invoices.get(params.id);
+ const { amount } = parsePayment(body);
+ const updated = recordPayment(current, amount, ctx.now());
+ return { status: 201, body: invoiceView(ctx.invoices.save(updated)) };
+ });
+
router.on("POST", "/invoices/:id/void", ({ params }) => {
const voided = voidInvoice(ctx.invoices.get(params.id), ctx.now());
return { status: 200, body: invoiceView(ctx.invoices.save(voided)) };
diff --git a/src/http/handlers/reports.ts b/src/http/handlers/reports.ts
index 19262c9..656969e 100644
--- a/src/http/handlers/reports.ts
+++ b/src/http/handlers/reports.ts
@@ -1,18 +1,17 @@
-import { computeTotals, isOverdue } from "../../domain/invoice";
+import { balance, isOverdue } from "../../domain/invoice";
import type { Currency } from "../../domain/money";
import type { AppContext } from "../context";
import type { Router } from "../router";
export function reportRoutes(router: Router, ctx: AppContext): void {
- /** What customers still owe us, per currency, over every open invoice. */
+ /** What customers still owe us (balances, net of partial payments), per currency, over every open invoice. */
router.on("GET", "/reports/outstanding", () => {
const now = ctx.now();
const totals: Partial<Record<Currency, number>> = {};
let count = 0;
let overdue = 0;
for (const invoice of ctx.invoices.list({ status: "open" })) {
- const { total } = computeTotals(invoice);
- totals[invoice.currency] = (totals[invoice.currency] ?? 0) + total;
+ totals[invoice.currency] = (totals[invoice.currency] ?? 0) + balance(invoice);
count += 1;
if (isOverdue(invoice, now)) overdue += 1;
}
diff --git a/src/http/views.ts b/src/http/views.ts
index 856944c..5b11bff 100644
--- a/src/http/views.ts
+++ b/src/http/views.ts
@@ -1,6 +1,6 @@
-import { computeTotals, type Invoice } from "../domain/invoice";
+import { amountPaid, balance, computeTotals, type Invoice } from "../domain/invoice";
/** The JSON shape returned for an invoice: stored fields plus computed totals. */
export function invoiceView(invoice: Invoice) {
- return { ...invoice, ...computeTotals(invoice) };
+ return { ...invoice, ...computeTotals(invoice), amountPaid: amountPaid(invoice), balance: balance(invoice) };
}
diff --git a/src/store/migrations.ts b/src/store/migrations.ts
index c8b73a9..1eab4df 100644
--- a/src/store/migrations.ts
+++ b/src/store/migrations.ts
@@ -1,6 +1,7 @@
+import { computeTotals, type Invoice } from "../domain/invoice";
import type { StoreData } from "./types";
-export const CURRENT_SCHEMA_VERSION = 1;
+export const CURRENT_SCHEMA_VERSION = 2;
type RawData = Record<string, unknown> & { schemaVersion?: number };
type Migration = (data: RawData) => RawData;
@@ -21,9 +22,31 @@ const v0ToV1: Migration = (data) => {
};
};
+/**
+ * v1 -> v2: invoices carry `payments`. Invoices paid before payments existed were paid in full:
+ * they get one payment of their total at `paidAt`, so their balance is zero.
+ */
+const v1ToV2: Migration = (data) => {
+ const invoices = (data.invoices as Record<string, unknown>[] | undefined) ?? [];
+ return {
+ ...data,
+ schemaVersion: 2,
+ invoices: invoices.map((inv) => {
+ if (Array.isArray(inv.payments)) return inv;
+ const invoice = inv as unknown as Invoice;
+ const payments =
+ invoice.status === "paid"
+ ? [{ amount: computeTotals(invoice).total, receivedAt: invoice.paidAt ?? invoice.issuedAt ?? invoice.createdAt }]
+ : [];
+ return { ...inv, payments };
+ }),
+ };
+};
+
/** migrations[n] upgrades a file from version n to n + 1. */
const migrations: Record<number, Migration> = {
0: v0ToV1,
+ 1: v1ToV2,
};
export function migrate(raw: unknown): StoreData {
diff --git a/test/invoice.test.ts b/test/invoice.test.ts
index 85202f2..e043b5d 100644
--- a/test/invoice.test.ts
+++ b/test/invoice.test.ts
@@ -21,6 +21,7 @@ function draft(overrides: Partial<Invoice> = {}): Invoice {
dueDate: null,
paidAt: null,
voidedAt: null,
+ payments: [],
...overrides,
};
}
diff --git a/test/payments.test.ts b/test/payments.test.ts
new file mode 100644
index 0000000..baf9da7
--- /dev/null
+++ b/test/payments.test.ts
@@ -0,0 +1,53 @@
+import { describe, expect, it } from "vitest";
+import { openStore } from "../src/store/json-store";
+import { seedCustomer, seedOpenInvoice, testApp } from "./helpers";
+
+describe("partial payments", () => {
+ it("accumulates payments and flips to paid at a zero balance", async () => {
+ const app = testApp();
+ const id = await seedOpenInvoice(app, { customerId: await seedCustomer(app), unitPrice: 10_000 });
+ expect((await app.handle("POST", `/invoices/${id}/payments`, { amount: 4_000 })).body).toMatchObject({
+ status: "open",
+ amountPaid: 4_000,
+ balance: 6_000,
+ });
+ const last = await app.handle("POST", `/invoices/${id}/payments`, { amount: 6_000 });
+ expect(last).toMatchObject({ status: 201, body: { status: "paid", amountPaid: 10_000, balance: 0 } });
+ });
+
+ it("rejects bad amounts and overpayments with 400", async () => {
+ const app = testApp();
+ const id = await seedOpenInvoice(app, { customerId: await seedCustomer(app), unitPrice: 10_000 });
+ for (const amount of [0, -1, 1.5, "10", undefined]) {
+ expect((await app.handle("POST", `/invoices/${id}/payments`, { amount })).status).toBe(400);
+ }
+ expect((await app.handle("POST", `/invoices/${id}/payments`, { amount: 10_001 })).status).toBe(400);
+ });
+
+ it("refuses payments on drafts, void and unknown invoices", async () => {
+ const app = testApp();
+ const customerId = await seedCustomer(app);
+ const draft = await app.handle("POST", "/invoices", {
+ customerId,
+ currency: "EUR",
+ lines: [{ description: "x", quantity: 1, unitPrice: 100 }],
+ });
+ expect((await app.handle("POST", `/invoices/${(draft.body as { id: string }).id}/payments`, { amount: 10 })).status).toBe(409);
+ const voided = await seedOpenInvoice(app, { customerId });
+ await app.handle("POST", `/invoices/${voided}/void`);
+ expect((await app.handle("POST", `/invoices/${voided}/payments`, { amount: 10 })).status).toBe(409);
+ expect((await app.handle("POST", "/invoices/inv_404/payments", { amount: 10 })).status).toBe(404);
+ });
+
+ it("keeps /pay consistent and reports balances, not totals", async () => {
+ const app = testApp(openStore());
+ const customerId = await seedCustomer(app);
+ const a = await seedOpenInvoice(app, { customerId, unitPrice: 10_000 });
+ const b = await seedOpenInvoice(app, { customerId, unitPrice: 5_000 });
+ await app.handle("POST", `/invoices/${a}/payments`, { amount: 2_500 });
+ expect((await app.handle("GET", "/reports/outstanding")).body).toMatchObject({ totals: { EUR: 12_500 } });
+ expect((await app.handle("POST", `/invoices/${a}/pay`)).body).toMatchObject({ status: "paid", amountPaid: 10_000, balance: 0 });
+ expect((await app.handle("GET", "/reports/outstanding")).body).toMatchObject({ totals: { EUR: 5_000 }, count: 1 });
+ expect(b).toBeTruthy();
+ });
+});
diff --git a/test/store.test.ts b/test/store.test.ts
index a1bf345..3201e2a 100644
--- a/test/store.test.ts
+++ b/test/store.test.ts
@@ -42,6 +42,26 @@ describe("store", () => {
expect(JSON.parse(readFileSync(file, "utf8")).schemaVersion).toBe(CURRENT_SCHEMA_VERSION);
});
+ it("migrates a v1 file: invoices paid before payments existed get one payment of their total", () => {
+ const file = tmpFile();
+ const inv = { customerId: "cus_1", currency: "EUR", lines: [{ description: "x", quantity: 2, unitPrice: 1_500 }], discount: 0 };
+ writeFileSync(
+ file,
+ JSON.stringify({
+ schemaVersion: 1,
+ counters: { customer: 1, invoice: 2, invoiceNumber: 2 },
+ customers: [],
+ invoices: [
+ { ...inv, id: "inv_1", status: "paid", paidAt: "2026-02-01T00:00:00.000Z" },
+ { ...inv, id: "inv_2", status: "open", paidAt: null },
+ ],
+ }),
+ );
+ const [paid, open] = openStore({ file }).read().invoices;
+ expect(paid.payments).toEqual([{ amount: 3_000, receivedAt: "2026-02-01T00:00:00.000Z" }]);
+ expect(open.payments).toEqual([]);
+ });
+
it("refuses a file from a newer build", () => {
expect(() => migrate({ schemaVersion: CURRENT_SCHEMA_VERSION + 1 })).toThrow(/newer/);
});
reference/F2.patch
diff --git a/src/domain/credit-note.ts b/src/domain/credit-note.ts
new file mode 100644
index 0000000..7504c53
--- /dev/null
+++ b/src/domain/credit-note.ts
@@ -0,0 +1,44 @@
+import { ConflictError, ValidationError } from "./errors";
+import { computeTotals, type Invoice } from "./invoice";
+import { assertCents, type Cents, sum } from "./money";
+
+/** A numbered accounting document that reduces what is owed on an issued invoice. Never edited or deleted. */
+export type CreditNote = {
+ id: string;
+ /** CN-0001, ... gapless, in the order credit notes are created. */
+ number: string;
+ invoiceId: string;
+ amount: Cents;
+ createdAt: string;
+};
+
+export function creditedAmount(notes: Pick<CreditNote, "amount">[]): Cents {
+ return sum(notes.map((n) => n.amount));
+}
+
+/** What can still be credited on an invoice: its total minus every credit note already issued against it. */
+export function creditable(invoice: Invoice, notes: Pick<CreditNote, "amount">[]): Cents {
+ return computeTotals(invoice).total - creditedAmount(notes);
+}
+
+export function parseCreditAmount(body: unknown): Cents {
+ if (typeof body !== "object" || body === null) {
+ throw new ValidationError("body must be an object");
+ }
+ return assertCents((body as Record<string, unknown>).amount, "amount", { min: 1 });
+}
+
+/** Throws unless `amount` can be credited on `invoice` now. Call it BEFORE taking a number. */
+export function assertCreditable(invoice: Invoice, notes: Pick<CreditNote, "amount">[], amount: Cents): void {
+ if (invoice.status !== "open" && invoice.status !== "paid") {
+ throw new ConflictError(`invoice ${invoice.id} is ${invoice.status}: only open or paid invoices can be credited`);
+ }
+ const left = creditable(invoice, notes);
+ if (amount > left) {
+ throw new ValidationError(`amount ${amount} exceeds what is left to credit (${left})`, "amount");
+ }
+}
+
+export function creditNoteNumber(n: number): string {
+ return `CN-${String(n).padStart(4, "0")}`;
+}
diff --git a/src/domain/invoice.ts b/src/domain/invoice.ts
index 5b69fc9..2ee6fa4 100644
--- a/src/domain/invoice.ts
+++ b/src/domain/invoice.ts
@@ -52,8 +52,9 @@ export function markPaid(invoice: Invoice, now: Date): Invoice {
return { ...invoice, status: "paid", paidAt: now.toISOString() };
}
+/** Voids a draft, open or paid invoice. A paid one must be fully credited first (the handler issues that credit note). */
export function voidInvoice(invoice: Invoice, now: Date): Invoice {
- if (invoice.status === "paid" || invoice.status === "void") {
+ if (invoice.status === "void") {
throw new ConflictError(`invoice ${invoice.id} is ${invoice.status} and cannot be voided`);
}
return { ...invoice, status: "void", voidedAt: now.toISOString() };
diff --git a/src/http/context.ts b/src/http/context.ts
index c0a89d3..003b7af 100644
--- a/src/http/context.ts
+++ b/src/http/context.ts
@@ -1,4 +1,5 @@
import type { Store } from "../store/json-store";
+import { creditNoteRepo } from "../store/credit-note-repo";
import { customerRepo } from "../store/customer-repo";
import { invoiceRepo } from "../store/invoice-repo";
@@ -7,8 +8,9 @@ export type AppContext = {
now: () => Date;
customers: ReturnType<typeof customerRepo>;
invoices: ReturnType<typeof invoiceRepo>;
+ creditNotes: ReturnType<typeof creditNoteRepo>;
};
export function createContext(store: Store, now: () => Date = () => new Date()): AppContext {
- return { store, now, customers: customerRepo(store), invoices: invoiceRepo(store) };
+ return { store, now, customers: customerRepo(store), invoices: invoiceRepo(store), creditNotes: creditNoteRepo(store) };
}
diff --git a/src/http/handlers/invoices.ts b/src/http/handlers/invoices.ts
index 09959e1..469a402 100644
--- a/src/http/handlers/invoices.ts
+++ b/src/http/handlers/invoices.ts
@@ -1,3 +1,4 @@
+import { assertCreditable, type CreditNote, creditNoteNumber, creditable, parseCreditAmount } from "../../domain/credit-note";
import { ValidationError } from "../../domain/errors";
import { type Invoice, type InvoiceStatus, issue, markPaid, voidInvoice } from "../../domain/invoice";
import { parseNewInvoice } from "../../domain/validation";
@@ -51,8 +52,37 @@ export function invoiceRoutes(router: Router, ctx: AppContext): void {
return { status: 200, body: invoiceView(ctx.invoices.save(paid)) };
});
+ /** Validation first, number last: a rejected credit note never consumes a CN number (numbers stay gapless). */
+ function credit(invoice: Invoice, amount: number): CreditNote {
+ assertCreditable(invoice, ctx.creditNotes.forInvoice(invoice.id), amount);
+ const n = ctx.store.nextId("creditNote");
+ return ctx.creditNotes.insert({
+ id: `cn_${n}`,
+ number: creditNoteNumber(n),
+ invoiceId: invoice.id,
+ amount,
+ createdAt: ctx.now().toISOString(),
+ });
+ }
+
+ router.on("POST", "/invoices/:id/credit-notes", ({ params, body }) => {
+ const invoice = ctx.invoices.get(params.id);
+ return { status: 201, body: credit(invoice, parseCreditAmount(body)) };
+ });
+
+ router.on("GET", "/invoices/:id/credit-notes", ({ params }) => {
+ ctx.invoices.get(params.id);
+ return { status: 200, body: ctx.creditNotes.forInvoice(params.id) };
+ });
+
+ /** Voiding a paid invoice credits whatever is left on it first. */
router.on("POST", "/invoices/:id/void", ({ params }) => {
- const voided = voidInvoice(ctx.invoices.get(params.id), ctx.now());
+ const invoice = ctx.invoices.get(params.id);
+ const voided = voidInvoice(invoice, ctx.now());
+ if (invoice.status === "paid") {
+ const left = creditable(invoice, ctx.creditNotes.forInvoice(invoice.id));
+ if (left > 0) credit(invoice, left);
+ }
return { status: 200, body: invoiceView(ctx.invoices.save(voided)) };
});
}
diff --git a/src/http/handlers/reports.ts b/src/http/handlers/reports.ts
index 19262c9..f980c9b 100644
--- a/src/http/handlers/reports.ts
+++ b/src/http/handlers/reports.ts
@@ -1,4 +1,5 @@
-import { computeTotals, isOverdue } from "../../domain/invoice";
+import { creditable } from "../../domain/credit-note";
+import { isOverdue } from "../../domain/invoice";
import type { Currency } from "../../domain/money";
import type { AppContext } from "../context";
import type { Router } from "../router";
@@ -11,8 +12,8 @@ export function reportRoutes(router: Router, ctx: AppContext): void {
let count = 0;
let overdue = 0;
for (const invoice of ctx.invoices.list({ status: "open" })) {
- const { total } = computeTotals(invoice);
- totals[invoice.currency] = (totals[invoice.currency] ?? 0) + total;
+ const owed = creditable(invoice, ctx.creditNotes.forInvoice(invoice.id));
+ totals[invoice.currency] = (totals[invoice.currency] ?? 0) + owed;
count += 1;
if (isOverdue(invoice, now)) overdue += 1;
}
diff --git a/src/store/credit-note-repo.ts b/src/store/credit-note-repo.ts
new file mode 100644
index 0000000..eac6632
--- /dev/null
+++ b/src/store/credit-note-repo.ts
@@ -0,0 +1,17 @@
+import type { CreditNote } from "../domain/credit-note";
+import type { Store } from "./json-store";
+
+export function creditNoteRepo(store: Store) {
+ return {
+ forInvoice(invoiceId: string): CreditNote[] {
+ return store.read().creditNotes.filter((n) => n.invoiceId === invoiceId);
+ },
+ all(): CreditNote[] {
+ return store.read().creditNotes;
+ },
+ insert(note: CreditNote): CreditNote {
+ store.update((d) => ({ ...d, creditNotes: [...d.creditNotes, note] }));
+ return note;
+ },
+ };
+}
diff --git a/src/store/json-store.ts b/src/store/json-store.ts
index b404c51..99205b7 100644
--- a/src/store/json-store.ts
+++ b/src/store/json-store.ts
@@ -13,9 +13,10 @@ export type Store = {
function empty(): StoreData {
return {
schemaVersion: CURRENT_SCHEMA_VERSION,
- counters: { customer: 0, invoice: 0, invoiceNumber: 0 },
+ counters: { customer: 0, invoice: 0, invoiceNumber: 0, creditNote: 0 },
customers: [],
invoices: [],
+ creditNotes: [],
};
}
diff --git a/src/store/migrations.ts b/src/store/migrations.ts
index c8b73a9..eb7abfb 100644
--- a/src/store/migrations.ts
+++ b/src/store/migrations.ts
@@ -1,6 +1,6 @@
import type { StoreData } from "./types";
-export const CURRENT_SCHEMA_VERSION = 1;
+export const CURRENT_SCHEMA_VERSION = 2;
type RawData = Record<string, unknown> & { schemaVersion?: number };
type Migration = (data: RawData) => RawData;
@@ -21,9 +21,18 @@ const v0ToV1: Migration = (data) => {
};
};
+/** v1 -> v2: credit notes (a collection and their own counter). */
+const v1ToV2: Migration = (data) => ({
+ ...data,
+ schemaVersion: 2,
+ counters: { ...(data.counters as Record<string, number> | undefined), creditNote: 0 },
+ creditNotes: [],
+});
+
/** migrations[n] upgrades a file from version n to n + 1. */
const migrations: Record<number, Migration> = {
0: v0ToV1,
+ 1: v1ToV2,
};
export function migrate(raw: unknown): StoreData {
@@ -43,8 +52,9 @@ export function migrate(raw: unknown): StoreData {
}
return {
schemaVersion: CURRENT_SCHEMA_VERSION,
- counters: (data.counters as StoreData["counters"]) ?? { customer: 0, invoice: 0, invoiceNumber: 0 },
+ counters: { customer: 0, invoice: 0, invoiceNumber: 0, creditNote: 0, ...(data.counters as Partial<StoreData["counters"]>) },
customers: (data.customers as StoreData["customers"]) ?? [],
invoices: (data.invoices as StoreData["invoices"]) ?? [],
+ creditNotes: (data.creditNotes as StoreData["creditNotes"]) ?? [],
};
}
diff --git a/src/store/types.ts b/src/store/types.ts
index 222b191..a9d7277 100644
--- a/src/store/types.ts
+++ b/src/store/types.ts
@@ -1,7 +1,8 @@
+import type { CreditNote } from "../domain/credit-note";
import type { Customer } from "../domain/customer";
import type { Invoice } from "../domain/invoice";
-export type Counters = { customer: number; invoice: number; invoiceNumber: number };
+export type Counters = { customer: number; invoice: number; invoiceNumber: number; creditNote: number };
/** The shape of the JSON file at CURRENT_SCHEMA_VERSION. */
export type StoreData = {
@@ -9,4 +10,5 @@ export type StoreData = {
counters: Counters;
customers: Customer[];
invoices: Invoice[];
+ creditNotes: CreditNote[];
};
diff --git a/test/credit-notes.test.ts b/test/credit-notes.test.ts
new file mode 100644
index 0000000..240f820
--- /dev/null
+++ b/test/credit-notes.test.ts
@@ -0,0 +1,52 @@
+import { mkdtempSync, writeFileSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+import { describe, expect, it } from "vitest";
+import { openStore } from "../src/store/json-store";
+import { seedCustomer, seedOpenInvoice, testApp } from "./helpers";
+
+describe("credit notes", () => {
+ it("numbers credit notes gaplessly; rejected ones take no number", async () => {
+ const app = testApp();
+ const id = await seedOpenInvoice(app, { customerId: await seedCustomer(app), unitPrice: 10_000 });
+ expect((await app.handle("POST", `/invoices/${id}/credit-notes`, { amount: 3_000 })).body).toMatchObject({ number: "CN-0001" });
+ expect((await app.handle("POST", `/invoices/${id}/credit-notes`, { amount: 7_001 })).status).toBe(400);
+ expect((await app.handle("POST", `/invoices/${id}/credit-notes`, { amount: 0 })).status).toBe(400);
+ expect((await app.handle("POST", `/invoices/${id}/credit-notes`, { amount: 7_000 })).body).toMatchObject({ number: "CN-0002" });
+ expect(((await app.handle("GET", `/invoices/${id}/credit-notes`)).body as unknown[]).length).toBe(2);
+ });
+
+ it("refuses drafts and void invoices (409) and unknown ones (404)", async () => {
+ const app = testApp();
+ const customerId = await seedCustomer(app);
+ const draft = await app.handle("POST", "/invoices", { customerId, currency: "EUR", lines: [{ description: "x", quantity: 1, unitPrice: 100 }] });
+ expect((await app.handle("POST", `/invoices/${(draft.body as { id: string }).id}/credit-notes`, { amount: 10 })).status).toBe(409);
+ expect((await app.handle("POST", "/invoices/inv_404/credit-notes", { amount: 10 })).status).toBe(404);
+ });
+
+ it("voids a paid invoice by crediting what is left", async () => {
+ const app = testApp();
+ const id = await seedOpenInvoice(app, { customerId: await seedCustomer(app), unitPrice: 10_000 });
+ await app.handle("POST", `/invoices/${id}/pay`);
+ await app.handle("POST", `/invoices/${id}/credit-notes`, { amount: 2_500 });
+ expect((await app.handle("POST", `/invoices/${id}/void`)).body).toMatchObject({ status: "void" });
+ const notes = (await app.handle("GET", `/invoices/${id}/credit-notes`)).body as { amount: number }[];
+ expect(notes.map((n) => n.amount)).toEqual([2_500, 7_500]);
+ });
+
+ it("nets credit notes out of the outstanding report", async () => {
+ const app = testApp();
+ const id = await seedOpenInvoice(app, { customerId: await seedCustomer(app), unitPrice: 10_000 });
+ await app.handle("POST", `/invoices/${id}/credit-notes`, { amount: 4_000 });
+ expect((await app.handle("GET", "/reports/outstanding")).body).toMatchObject({ totals: { EUR: 6_000 }, count: 1 });
+ });
+
+ it("works on a store file written before credit notes existed", async () => {
+ const file = join(mkdtempSync(join(tmpdir(), "ledgerly-")), "data.json");
+ writeFileSync(file, JSON.stringify({ schemaVersion: 1, counters: { customer: 0, invoice: 0, invoiceNumber: 0 }, customers: [], invoices: [] }));
+ const app = testApp(openStore({ file }));
+ const id = await seedOpenInvoice(app, { customerId: await seedCustomer(app) });
+ expect((await app.handle("POST", `/invoices/${id}/credit-notes`, { amount: 100 })).body).toMatchObject({ id: "cn_1", number: "CN-0001" });
+ expect(openStore({ file }).read().creditNotes.length).toBe(1);
+ });
+});
diff --git a/test/invoice.test.ts b/test/invoice.test.ts
index 85202f2..4219226 100644
--- a/test/invoice.test.ts
+++ b/test/invoice.test.ts
@@ -54,8 +54,11 @@ describe("invoice domain", () => {
});
it("voids drafts and open invoices, never paid ones", () => {
+ // Meaning changed with credit notes: a paid invoice can now be voided (the handler credits what is left first);
+ // only an already void invoice is refused.
expect(voidInvoice(draft(), NOW).status).toBe("void");
- expect(() => voidInvoice(draft({ status: "paid" }), NOW)).toThrow(ConflictError);
+ expect(voidInvoice(draft({ status: "paid" }), NOW).status).toBe("void");
+ expect(() => voidInvoice(draft({ status: "void" }), NOW)).toThrow(ConflictError);
});
it("knows when an open invoice is overdue", () => {
Runner ve driver:
run.sh
#!/usr/bin/env bash
# One run: fresh work copy + pack fixtures, the driver (worker turns + PO), then finalize.
# Usage: [TASK=F1|F2] run.sh <cfg> <rep> cfg: bare|sp|spd|mp|both
# RUN_TAG=x -> runs/<name>-x (smoke runs; analyze.py ignores tagged runs)
# The work copy lives OUTSIDE the repo tree ($FLOW_BENCH_WORK, default /private/tmp/flow-bench/<name>): Claude Code
# loads every CLAUDE.md of the parent directories, so a work copy under this repo read the repo's CLAUDE.md (and the
# instructions it pulls in) in every session. After finalize the copy (without node_modules) is archived to work/<name>.
set -uo pipefail
CFG="$1"; REP="${2:-1}"
HERE="$(cd "$(dirname "$0")" && pwd)"
. "$HERE/env.sh"
TASK="${TASK:-F1}"; export TASK
PFX=""; [ "$TASK" != "F1" ] && PFX="$(echo "$TASK" | tr "A-Z" "a-z")-"
NAME="${PFX}${CFG}-r${REP}${RUN_TAG:+-$RUN_TAG}"
WROOT="${FLOW_BENCH_WORK:-/private/tmp/flow-bench}"
OUT="$HERE/runs/$NAME"; W="$WROOT/$NAME"
rm -rf "$OUT"; mkdir -p "$OUT"
"$HERE/prep.sh" "$CFG" "$W" > "$OUT/prep.txt" 2>&1 || { echo "$NAME: prep failed"; cat "$OUT/prep.txt"; exit 2; }
BASE=$(tail -1 "$OUT/prep.txt")
cp "$W/.claude/settings.json" "$OUT/settings.json"
printf '{"task":"%s","cfg":"%s","rep":%s,"tag":"%s","work":"%s","name":"%s","base":"%s","started":"%s","claude_version":"%s"}\n' \
"$TASK" "$CFG" "$REP" "${RUN_TAG:-}" "$W" "$NAME" "$BASE" "$(date '+%F %T')" "$(claude --version | head -1)" > "$OUT/meta.json"
python3 "$HERE/lib/driver.py" "$CFG" "$OUT" "$W" "$BASE" > "$OUT/driver.out" 2> "$OUT/driver.err"
python3 "$HERE/lib/finalize.py" "$OUT"
rm -rf "$HERE/work/$NAME"; mkdir -p "$HERE/work"
rsync -a --exclude node_modules "$W/" "$HERE/work/$NAME/"
prep.sh
#!/usr/bin/env bash
# Fresh work copy of upstream (node_modules included) + the pack fixtures of <cfg>.
# Usage: prep.sh <cfg> <work dir> Prints the base commit (the oracle's BASE) on its last line.
set -euo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"
CFG="$1"; W="$2"
SP=$(python3 -c "import json;print(int(json.load(open('$HERE/configs.json'))['$CFG']['superpowers']))")
MP=$(python3 -c "import json;print(int(json.load(open('$HERE/configs.json'))['$CFG']['mattpocock']))")
rm -rf "$W"; mkdir -p "$(dirname "$W")"
cp -R "$HERE/upstream" "$W"
cd "$W"
# .claude/ never shows in git status or in any diff (hook config, installed skills).
echo ".claude/" >> .git/info/exclude
mkdir -p .claude/skills
if [ "$MP" = 1 ]; then
cp -R "$HERE/packs/mattpocock-skills/.claude/skills/." .claude/skills/
# One-time per-repo precondition of the pack (/setup-matt-pocock-skills), answered once with the defaults it
# recommends for a repo with no remote: local-markdown tracker, single-context domain docs, no triage labels
# (triage is installed, so Section B would ask; default labels kept). Committed so it never shows in the diff.
S="$HERE/packs/mattpocock-skills/.claude/skills/setup-matt-pocock-skills"
mkdir -p docs/agents
cp "$S/issue-tracker-local.md" docs/agents/issue-tracker.md
cp "$S/domain.md" docs/agents/domain.md
cp "$S/triage-labels.md" docs/agents/triage-labels.md
cat > CLAUDE.md <<'MD'
## Agent skills
### Issue tracker
Issues and specs live as local markdown files under `.scratch/`. See `docs/agents/issue-tracker.md`.
### Triage labels
Default labels (`needs-triage`, `needs-info`, `ready-for-agent`, `ready-for-human`, `wontfix`). See `docs/agents/triage-labels.md`.
### Domain docs
Single-context: one `GLOSSARY.md` + `docs/adr/` at the repo root. See `docs/agents/domain.md`.
MD
git add CLAUDE.md docs/agents
GIT_AUTHOR_DATE="2026-03-09T18:00:00" GIT_COMMITTER_DATE="2026-03-09T18:00:00" \
git -c user.name="Dana Whitfield" -c user.email="dana@ledgerly.dev" commit -q -m "chore: configure agent skills (local tracker)"
fi
if [ "$SP" = 1 ]; then
# The plugin root as Claude Code would hold it, so the hook script finds ../skills/using-superpowers.
mkdir -p .claude/plugins
cp -R "$HERE/packs/superpowers" .claude/plugins/superpowers
cp -R "$HERE/packs/superpowers/skills/." .claude/skills/
fi
python3 - "$HERE" "$SP" "$CFG" > .claude/settings.json <<'PY'
import json, sys
bench, sp, cfg = sys.argv[1], sys.argv[2] == "1", sys.argv[3]
# Oracle, reference, PO brief, other runs: denied by path (Read/Grep/Glob) and by command text (Bash).
deny = ["Read(/%s/hidden/**)" % bench, "Read(/%s/reference/**)" % bench, "Read(/%s/runs/**)" % bench,
"Read(/%s/po/**)" % bench, "Read(/%s/validation/**)" % bench,
"Bash(*hidden*)", "Bash(*reference/F*)", "Bash(*-brief.md*)", "Bash(*/runs/*)", "Bash(*../*)", "Bash(find *)"]
if cfg == "spd":
# executing-plans runs its sibling scripts as ../subagent-driven-development/scripts/...: a "../" deny would block the
# pack's own loop. The oracle stays protected by the Read denies and Bash(*hidden*).
deny.remove("Bash(*../*)")
settings = {"permissions": {"deny": deny}}
if sp:
# = packs/superpowers/hooks/hooks.json, with ${CLAUDE_PLUGIN_ROOT} provided the way the plugin loader provides it
# (the script also picks its JSON output shape from CLAUDE_PLUGIN_ROOT being set).
root = '$CLAUDE_PROJECT_DIR/.claude/plugins/superpowers'
hooks = json.load(open("%s/packs/superpowers/hooks/hooks.json" % bench))["hooks"]
for groups in hooks.values():
for g in groups:
for h in g["hooks"]:
h["command"] = 'CLAUDE_PLUGIN_ROOT="%s" %s' % (root, h["command"].replace("${CLAUDE_PLUGIN_ROOT}", root))
settings["hooks"] = hooks
print(json.dumps(settings, indent=2))
PY
[ "$(ls .claude/skills | wc -l)" -eq 0 ] && rmdir .claude/skills
test -z "$(git status --porcelain)" || { echo "work copy not clean"; git status --porcelain; exit 1; }
git rev-parse HEAD
env.sh
# Sourced by run.sh, gate.sh and validate.sh.
# 1. Scrub every CLAUDE* variable inherited from a launching Claude Code session
# (CLAUDE_EFFORT, CLAUDE_CODE_SUBAGENT_MODEL, CLAUDE_CODE_SESSION_ID, CLAUDECODE,
# CLAUDE_CODE_CHILD_SESSION, ...): they would silently change the worker's effort,
# subagent model and session bookkeeping.
for _v in $(env | sed -n 's/^\(CLAUDE[A-Za-z0-9_]*\)=.*/\1/p'); do unset "$_v"; done
unset _v
# 2. Auth: the user's default config dir (~/.claude, OAuth). See README traps (--bare cannot auth).
unset CLAUDE_CONFIG_DIR
# 3. Fresh work copies at the same path must not share auto-memory across reps.
export CLAUDE_CODE_DISABLE_AUTO_MEMORY=1
# 4. No bytecode in the shared hidden/ dir when 3 runs execute hidden tests in parallel.
export PYTHONDONTWRITEBYTECODE=1
# 5. The bench venv first: `python -m pytest` inside the work copy uses it.
GATE_BENCH="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
export GATE_BENCH
export PATH="$GATE_BENCH/.venv/bin:$PATH"
configs.json
{
"bare": {
"superpowers": false,
"mattpocock": false,
"flow": "bare",
"desc": "no skills, no hooks (project settings carry only the permission deny-list)"
},
"sp": {
"superpowers": true,
"mattpocock": false,
"flow": "sp",
"desc": "HOOK-ROUTED: obra/superpowers v6.4.2: skills in .claude/skills, SessionStart hook in .claude/settings.json as hooks/hooks.json wires it; turn 1 = the bare task, the hook decides"
},
"mp": {
"superpowers": false,
"mattpocock": true,
"flow": "mp",
"desc": "mattpocock/skills v1.3.1 via npx skills add (27 promoted skills, copied), setup-matt-pocock-skills output committed (local tracker)"
},
"both": {
"superpowers": true,
"mattpocock": true,
"flow": "mp",
"desc": "sp + mp in one project; driven with the mp slash-command flow (the stacking question: does the Superpowers hook hijack it?)"
},
"spd": {
"superpowers": true,
"mattpocock": false,
"flow": "spd",
"desc": "Superpowers, DRIVEN: same project install as sp (hook included); the driver types /brainstorming <task> -> /writing-plans -> /executing-plans (Native/inline); the pack's own scripts allowed in Bash"
}
}
lib/driver.py
"""One run of the flow bench: drives a headless Sonnet 5.5 session turn by turn, routes questions to the PO.
driver.py <cfg> <run dir> <work dir> <base sha>
Every worker turn: `claude -p <prompt> [--resume <sid>] --output-format stream-json --verbose --include-hook-events
--include-partial-messages ...` with cwd = the work copy. Per turn it writes turn-NN.stream.jsonl (raw stdout),
turn-NN.times (receive time of every stdout line), turn-NN.stderr, and appends one record to turns.jsonl.
Caps (SPEC): 12 driver turns, --max-turns 60 per claude turn, 40 min wall per run (PO time included).
"""
import json
import os
import subprocess
import sys
import threading
import time
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import po # noqa: E402
from accounting import account_turn # noqa: E402
BENCH = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
MODEL = "claude-sonnet-5-5"
MAX_DRIVER_TURNS = int(os.environ.get("BENCH_MAX_DRIVER_TURNS", "12"))
MAX_TURNS = os.environ.get("BENCH_MAX_TURNS", "60")
WALL_CAP = float(os.environ.get("BENCH_WALL_CAP_S", str(40 * 60)))
# SPEC allow-list. AskUserQuestion is removed (headless: questions come as plain text); plan mode and worktree
# tools are removed too (a headless session cannot leave plan mode, and a worktree would move the work out of
# the scored copy). Read-only shell commands are auto-approved by Claude Code whatever the list says.
ALLOWED = ",".join([
"Read", "Edit", "Write", "MultiEdit", "Glob", "Grep", "Skill", "TodoWrite", "Task", "Agent",
"Bash(node:*)", "Bash(npm:*)", "Bash(pnpm:*)", "Bash(npx vitest:*)", "Bash(npx tsc:*)", "Bash(tsc:*)",
"Bash(git status:*)", "Bash(git diff:*)", "Bash(git log:*)", "Bash(git add:*)", "Bash(git commit:*)",
"Bash(git checkout -b:*)", "Bash(ls:*)", "Bash(cat:*)", "Bash(mkdir:*)", "Bash(head:*)", "Bash(tail:*)",
"Bash(grep:*)", "Bash(wc:*)", "Bash(gh:*)",
])
# spd only: the scripts executing-plans / subagent-driven-development call (task-start, task-done, sdd-workspace,
# task-brief, review-package), wherever the model addresses them from.
SPD_EXTRA = ",".join("Bash(*scripts/%s*)" % x for x in ("task-start", "task-done", "sdd-workspace", "task-brief", "review-package"))
DISALLOWED = "AskUserQuestion,EnterPlanMode,ExitPlanMode,EnterWorktree,ExitWorktree,WebFetch,WebSearch"
PRE_IMPLEMENT = ("/grill-with-docs", "/to-spec", "/to-tickets")
SPD_PRE_IMPLEMENT = ("/brainstorming", "/writing-plans")
DONE_SUFFIX = "\n\nWhen you are done, stop and summarise what you changed."
def worker_env():
env = {k: v for k, v in os.environ.items() if not k.startswith("CLAUDE")}
env["CLAUDE_CODE_DISABLE_AUTO_MEMORY"] = "1"
return env
def run_claude(prompt, sid, work, out_prefix, timeout, allowed=ALLOWED):
cmd = ["claude", "-p", prompt]
if sid:
cmd += ["--resume", sid]
cmd += ["--model", MODEL, "--output-format", "stream-json", "--verbose", "--include-hook-events",
"--include-partial-messages", "--max-turns", MAX_TURNS, "--permission-mode", "acceptEdits",
"--allowedTools", allowed, "--disallowedTools", DISALLOWED,
"--setting-sources", "project", "--strict-mcp-config"]
t0 = time.time()
fo = open(out_prefix + ".stream.jsonl", "w")
ft = open(out_prefix + ".times", "w")
fe = open(out_prefix + ".stderr", "w")
p = subprocess.Popen(cmd, cwd=work, env=worker_env(), stdin=subprocess.DEVNULL, stdout=subprocess.PIPE,
stderr=fe, text=True, bufsize=1)
killed = {"v": False}
def watchdog():
try:
p.wait(timeout=timeout)
except subprocess.TimeoutExpired:
killed["v"] = True
p.kill()
th = threading.Thread(target=watchdog, daemon=True)
th.start()
for line in p.stdout:
fo.write(line)
ft.write("%.3f\n" % (time.time() - t0))
p.wait()
th.join(timeout=5)
for f in (fo, ft, fe):
f.close()
return {"exit_code": p.returncode, "wall_s": round(time.time() - t0, 2), "killed_wall_cap": killed["v"], "t0": t0}
def scratch_issues(work):
root = os.path.join(work, ".scratch")
if not os.path.isdir(root):
return None
for d in sorted(os.listdir(root)):
if os.path.isdir(os.path.join(root, d, "issues")):
return ".scratch/%s/issues/" % d
return None
def main():
cfg, run_dir, work, base = sys.argv[1:5]
conf = json.load(open(os.path.join(BENCH, "configs.json")))[cfg]
flow = conf["flow"]
task_id = os.environ.get("TASK", "F1")
task = open(os.path.join(BENCH, "tasks", task_id + ".md")).read().strip()
allowed = ALLOWED + ("," + SPD_EXTRA if flow == "spd" else "")
spd_stages = ["/brainstorming", "/writing-plans", "/executing-plans"]
spd_i = 0
t_run = time.time()
sid = None
prev = None # cumulative modelUsage / cost of the session so far
stage_in_force = "(none)"
caps = []
turns_log = open(os.path.join(run_dir, "turns.jsonl"), "a")
# mp flow: the slash commands in SPEC order; the implement stage is built when it is reached.
mp_stages = ["/grill-with-docs", "/to-spec", "/to-tickets", "/implement", "/code-review", "/pr"]
mp_i = 0
def mp_prompt(i):
s = mp_stages[i]
if s == "/grill-with-docs":
return "/grill-with-docs " + task
if s == "/implement":
issues = scratch_issues(work)
return "/implement " + ("the tickets in %s, all of them, blockers first" % issues if issues
else "the spec and tickets above")
if s == "/code-review":
return "/code-review since %s" % base
return s
if flow == "bare":
prompt, stage = task + DONE_SUFFIX, "bare"
elif flow == "spd":
prompt, stage = "/brainstorming " + task, spd_stages[0]
stage_in_force = "brainstorming"
elif flow == "sp":
prompt, stage = task, None
else:
prompt, stage = mp_prompt(0), mp_stages[0]
source = "driver"
end_reason = None
for n in range(1, MAX_DRIVER_TURNS + 1):
remaining = WALL_CAP - (time.time() - t_run)
if remaining <= 5:
caps.append("wall_40min")
end_reason = "cap_wall"
break
prefix = os.path.join(run_dir, "turn-%02d" % n)
r = run_claude(prompt, sid, work, prefix, remaining, allowed)
seg_mode = flow in ("sp", "spd") # stage split by skill in force inside the turn
acc = account_turn(prefix + ".stream.jsonl", prefix + ".times", prev,
None if seg_mode else stage, stage_in_force)
if acc.get("session_id"):
sid = acc["session_id"]
if acc.get("cumulative"):
prev = acc["cumulative"]
if seg_mode:
stage_in_force = acc["stage_at_end"]
rec = {"n": n, "flow": flow, "task": task_id, "stage": acc["stage_at_end"] if flow == "sp" else stage,
"prompt": prompt,
"prompt_source": source, **{k: v for k, v in r.items() if k != "t0"}, **{k: v for k, v in acc.items() if k != "cumulative"}}
if r["killed_wall_cap"]:
caps.append("wall_40min")
if acc.get("subtype") == "error_max_turns":
caps.append("max_turns_60@turn%d" % n)
last = acc.get("result_text") or acc.get("last_assistant_text") or ""
open(prefix + ".last.txt", "w").write(last)
if r["killed_wall_cap"]:
rec["po"] = None
turns_log.write(json.dumps(rec) + "\n")
turns_log.flush()
end_reason = "cap_wall"
break
po_stage = stage if flow != "sp" else acc["stage_at_end"]
ans = po.ask(run_dir, n, flow, po_stage, last, task_id)
rec["po"] = ans
turns_log.write(json.dumps(rec) + "\n")
turns_log.flush()
# Next prompt
if flow in ("bare", "sp"):
if ans["kind"] == "done":
end_reason = "done"
break
prompt = ans["reply"] if ans["kind"] == "question" else (
"go on" if flow == "bare" else "go on, proceed with the plan")
source = "po" if ans["kind"] == "question" else "driver"
elif flow == "spd":
if ans["kind"] == "question":
prompt, source = ans["reply"], "po"
if stage in SPD_PRE_IMPLEMENT and "implementing yet" not in prompt:
prompt += "\n\nDon't start implementing yet."
elif ans["kind"] == "waiting" or acc.get("subtype") == "error_max_turns":
prompt, source = "go on", "driver"
else:
spd_i += 1
if spd_i >= len(spd_stages):
end_reason = "done"
break
stage, prompt, source = spd_stages[spd_i], spd_stages[spd_i], "driver"
stage_in_force = stage.lstrip("/") # a typed slash command expands inline: no Skill event
else:
if ans["kind"] == "question":
prompt, source = ans["reply"], "po"
# A user following the Main Flow builds only at /implement. Without this line the worker took the
# answers to its last grilling questions as a go and coded inside /grill-with-docs (both-r1, twice).
if stage in PRE_IMPLEMENT and "implementing yet" not in prompt:
prompt += "\n\nDon't start implementing yet."
elif ans["kind"] == "waiting" or acc.get("subtype") == "error_max_turns":
prompt, source = "go on", "driver" # stage not finished: stay in it
else:
mp_i += 1
if mp_i >= len(mp_stages):
end_reason = "done"
break
stage, prompt, source = mp_stages[mp_i], mp_prompt(mp_i), "driver"
if n == MAX_DRIVER_TURNS:
caps.append("driver_turns_12")
end_reason = "cap_driver_turns"
meta = {"end_reason": end_reason, "caps": sorted(set(caps)), "session_id": sid,
"run_wall_s": round(time.time() - t_run, 1), "mp_stage_reached": mp_stages[min(mp_i, 5)] if flow == "mp" else None,
"spd_stage_reached": spd_stages[min(spd_i, 2)] if flow == "spd" else None, "task": task_id}
json.dump(meta, open(os.path.join(run_dir, "driver.json"), "w"), indent=2)
print(json.dumps(meta))
if __name__ == "__main__":
main()
lib/po.py
"""Product-owner simulator: a headless Sonnet 5.5 with NO tools that read po/F1-brief.md and answers the worker.
po.py <run dir> <turn n> <flow> <stage> <last message file> -> prints one JSON line, writes po-<n>.json
Output: {"kind": "question" | "waiting" | "done", "reply": str, "cost_usd", "wall_s", "error", "heuristic_question"}
- question: the worker asks the user something (a question, options, an approval request) -> reply answers it
- waiting: it stopped without asking (announced a next step, paused between phases) -> driver sends "go on"
- done: it reports the whole change finished -> driver stops (bare/sp)
Isolation (same as the previous benches' reviewers): --setting-sources "", disableAllHooks, --strict-mcp-config,
--tools "", CLAUDE_CODE_DISABLE_CLAUDE_MDS=1, CLAUDE_CODE_DISABLE_AUTO_MEMORY=1, scrubbed CLAUDE* env,
cwd = an empty temp dir (never the work copy).
"""
import json
import os
import re
import subprocess
import sys
import tempfile
import time
BENCH = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
MODEL = "claude-sonnet-5-5"
SCHEMA = {
"type": "object",
"properties": {
"kind": {"type": "string", "enum": ["question", "waiting", "done"]},
"reply": {"type": "string"},
},
"required": ["kind", "reply"],
"additionalProperties": False,
}
FLOW_NOTE = {
"bare": "The assistant was asked to implement the request and stop with a summary when done.",
"sp": "The assistant runs a process (brainstorm -> design -> written plan -> execution -> review -> finish). "
"It often stops to ask for approval of a design, a spec or a plan, or to ask how to finish the branch.",
"spd": "You are driving the assistant through the Superpowers stages by slash command, one stage at a time "
"(brainstorming -> writing-plans -> executing-plans); the current stage is shown below and YOU send the "
"next command once a stage is over. RULE FOR THIS FLOW: when the stage's output is delivered (a design "
"approved and written, a plan saved) and the assistant only asks you to approve it or whether to proceed, "
"answer what is open (approve it; if asked for an execution approach, choose Native / inline in this "
"session; consent to working on the current branch) and classify it kind=question only if something "
"beyond approval is open, otherwise kind=done. Never tell it to start implementing yourself before the "
"executing-plans stage. During executing-plans, approve its rulings and ask it to finish.",
"mp": "You are driving the assistant through slash commands, one stage at a time (grill -> spec -> tickets -> "
"implement -> code review -> PR); the current stage is shown below and YOU send the next command once "
"a stage is over. Interviews (grilling) come in rounds of numbered questions with a recommended answer "
"each. RULE FOR THIS FLOW: when the stage's output is delivered and the assistant only asks whether to "
"proceed, start building, move on, or confirms its own summary ('Ready to write the spec?', 'Shall I "
"start?', 'Does this capture it?' after a full recap), that is kind=done with an empty reply: the next "
"slash command will follow. Never tell it to start implementing or to move to another phase yourself; "
"answer only the open design questions. Before the /implement stage, if the assistant offers to build "
"next, end your reply with: \"Don't start implementing yet.\"",
}
def heuristic_question(text):
"""SPEC's driver rule: a turn that ends on a question mark or an explicit options list."""
tail = (text or "").strip()[-600:]
if tail.rstrip("*_` \n").endswith("?"):
return True
lines = [l.strip() for l in tail.splitlines() if l.strip()]
opts = [l for l in lines[-8:] if re.match(r"^(\(?[A-Da-d1-4][\).:]|[-*] \*\*Option)", l)]
return len(opts) >= 2 or "?" in "\n".join(lines[-3:])
def ask(run_dir, n, flow, stage, last_text, task="F1"):
brief = open(os.path.join(BENCH, "po", "%s-brief.md" % task)).read()
prompt = (
"You play the developer who asked a coding assistant for a change in your repo. You are busy: you answer "
"tersely (1 to 6 short sentences or a short numbered list), the way a dev types in a chat. You never write "
"code, never mention how you will check the work, never invent requirements beyond your brief. You answer "
"ONLY what the assistant asked: never volunteer a requirement, an edge case or a constraint it did not ask "
"about (a question it asked is fair game, even a broad one like 'anything else I should know?').\n\n"
"=== YOUR BRIEF (private, the assistant has not seen it) ===\n" + brief + "\n=== END BRIEF ===\n\n"
+ FLOW_NOTE.get(flow, "") + "\nCurrent stage: " + stage + "\n\n"
"=== THE ASSISTANT'S LAST MESSAGE (it is now waiting for you) ===\n" + (last_text or "(empty message)")[-12000:]
+ "\n=== END MESSAGE ===\n\n"
"Classify the message and write your reply:\n"
"- kind=question when it asks you anything or waits for a decision or an approval (numbered questions, "
"options, 'does this look right?', 'shall I proceed?'). Reply by answering EVERY open question in one go, "
"from your brief; where your brief is silent, accept the assistant's recommendation or say 'your call'. "
"Approve designs, specs, plans and ticket breakdowns that match your brief; correct only what contradicts it. "
"If it asks how to finish a branch (merge, PR, keep), say: keep it on the current branch, no PR, done. "
"If a code review it ran lists real defects, tell it to fix them.\n"
"- kind=waiting when it stopped without asking anything but the work is not finished (it announced a next "
"step, or paused between phases). Reply: 'go on'.\n"
"- kind=done when it reports the change finished (implemented, tests run) or, for a slash-command stage, "
"when that stage's output is delivered and nothing is asked. Reply: ''.\n"
)
env = {k: v for k, v in os.environ.items() if not k.startswith("CLAUDE")}
env.update({"CLAUDE_CODE_DISABLE_CLAUDE_MDS": "1", "CLAUDE_CODE_DISABLE_AUTO_MEMORY": "1"})
cmd = ["claude", "-p", prompt, "--model", MODEL, "--output-format", "json", "--json-schema", json.dumps(SCHEMA),
"--tools", "", "--max-turns", "4", "--permission-mode", "dontAsk", "--setting-sources", "",
"--settings", '{"disableAllHooks":true}', "--strict-mcp-config"]
out = {"kind": None, "reply": "", "cost_usd": 0.0, "wall_s": 0.0, "error": None,
"heuristic_question": heuristic_question(last_text)}
t0 = time.time()
for attempt in range(2):
try:
with tempfile.TemporaryDirectory(prefix="po-") as cwd:
p = subprocess.run(cmd, cwd=cwd, env=env, stdin=subprocess.DEVNULL, capture_output=True,
text=True, timeout=300)
res = json.loads(p.stdout.strip().splitlines()[-1])
out["cost_usd"] += res.get("total_cost_usd") or 0.0
out["models"] = sorted((res.get("modelUsage") or {}).keys())
so = res.get("structured_output")
if isinstance(so, str):
so = json.loads(so)
if not so or so.get("kind") not in ("question", "waiting", "done"):
raise ValueError("no structured_output: %s" % (res.get("result") or "")[:300])
out["kind"], out["reply"], out["error"] = so["kind"], so["reply"].strip(), None
break
except Exception as e: # noqa: BLE001
out["error"] = "%s: %s" % (type(e).__name__, e)
time.sleep(3)
if out["kind"] is None: # PO failed twice: fall back on the SPEC heuristic
out["kind"] = "question" if out["heuristic_question"] else "waiting"
out["reply"] = "Your call on all of these; go on." if out["kind"] == "question" else "go on"
out["fallback"] = True
if out["kind"] == "waiting" and not out["reply"]:
out["reply"] = "go on"
out["wall_s"] = round(time.time() - t0, 2)
json.dump(dict(out, n=n, stage=stage, prompt_chars=len(prompt)), open(os.path.join(run_dir, "po-%02d.json" % n), "w"),
indent=2)
return out
if __name__ == "__main__":
rd, n, flow, stage, f = sys.argv[1:6]
print(json.dumps(ask(rd, int(n), flow, stage, open(f).read(), os.environ.get("TASK", "F1"))))
lib/accounting.py
"""Per-turn and per-stage accounting from one claude -p stream-json (with --include-partial-messages).
Sources of truth (checked on 2.1.294, see README traps):
- result.total_cost_usd and result.modelUsage are CUMULATIVE over a resumed session and include subagents:
per-turn USD and tokens = delta against the previous turn's result.
- result.usage covers the main thread of this invocation only (no subagents): not used for totals.
- assistant events carry a usage SNAPSHOT taken when the message started (output_tokens partial);
stream_event message_start/message_delta (main thread only) carry the exact per-API-call usage.
- Subagent API calls emit only assistant snapshots (parent_tool_use_id = the Task/Agent tool_use id).
Stage split inside a turn (sp: the skill in force): main-thread calls are priced exactly from their own usage
(Sonnet 5.5 list prices, verified against total_cost_usd); the subagent remainder (turn delta - main calls) is
split over the Task calls that spawned them, weighted by their snapshot tokens. Wall per stage = the receive-time
gaps of the stdout lines, attributed to the stage in force.
"""
import json
from collections import Counter, defaultdict
# USD per token, claude-sonnet-5-5 list prices (derived from two probe turns: exact match to total_cost_usd).
PRICE = {"in": 2e-6, "out": 10e-6, "cr": 0.2e-6, "cw1h": 4e-6, "cw5m": 2.5e-6}
SP_INJECT_MARK = "You have superpowers"
EDIT_TOOLS = {"Edit", "Write", "MultiEdit", "NotebookEdit"}
DESIGN_STAGES = {"(none)", "using-superpowers", "brainstorming", "writing-plans"}
NO_SKILL_IMPL = "implementation (no skill)"
def _tok(u):
"""Normalise an API usage dict to in/out/cr/cw (+ cw split)."""
u = u or {}
cc = u.get("cache_creation") or {}
cw = u.get("cache_creation_input_tokens") or 0
cw1h = cc.get("ephemeral_1h_input_tokens")
cw5m = cc.get("ephemeral_5m_input_tokens")
if cw1h is None and cw5m is None:
cw1h, cw5m = cw, 0
return {"in": u.get("input_tokens") or 0, "out": u.get("output_tokens") or 0,
"cr": u.get("cache_read_input_tokens") or 0, "cw": cw, "cw1h": cw1h or 0, "cw5m": cw5m or 0}
def _usd(t):
return t["in"] * PRICE["in"] + t["out"] * PRICE["out"] + t["cr"] * PRICE["cr"] + \
t["cw1h"] * PRICE["cw1h"] + t["cw5m"] * PRICE["cw5m"]
def _mu_tokens(mu):
out = {}
for m, v in (mu or {}).items():
out[m] = {"in": v.get("inputTokens") or 0, "out": v.get("outputTokens") or 0,
"cr": v.get("cacheReadInputTokens") or 0, "cw": v.get("cacheCreationInputTokens") or 0,
"usd": v.get("costUSD") or 0.0}
return out
def _sub(a, b):
keys = set(a) | set(b)
return {m: {k: (a.get(m, {}).get(k, 0) - b.get(m, {}).get(k, 0)) for k in ("in", "out", "cr", "cw", "usd")}
for m in keys}
def _sum_models(per_model):
t = {k: 0 for k in ("in", "out", "cr", "cw", "usd")}
for v in per_model.values():
for k in t:
t[k] += v.get(k, 0)
return t
def account_turn(stream_path, times_path, prev, fixed_stage, stage_in_force):
lines, times = [], []
try:
raw = open(stream_path, errors="replace").read().splitlines()
tms = [float(x) for x in open(times_path).read().split()]
except FileNotFoundError:
raw, tms = [], []
for i, l in enumerate(raw):
l = l.strip()
if l.startswith("{"):
try:
lines.append(json.loads(l))
times.append(tms[i] if i < len(tms) else (times[-1] if times else 0.0))
except Exception:
pass
stage = fixed_stage or stage_in_force
out = {"session_id": None, "subtype": None, "is_error": None, "num_turns": None, "result_text": None,
"last_assistant_text": "", "stage_at_start": stage}
tools_main, tools_sub = Counter(), Counter()
skills, hooks, files = [], [], set()
skill_ids = {}
task_stage = {} # Task/Agent tool_use id -> stage when spawned
main_calls = {} # msg id -> {"stage", "tok"}
sub_snap = {} # msg id -> (parent id, tokens)
seg_wall = defaultdict(float)
order = [stage]
last_t = 0.0
result = None
denials = []
for o, t in zip(lines, times):
seg_wall[stage] += max(0.0, t - last_t)
last_t = t
typ = o.get("type")
if typ == "system" and o.get("subtype") == "init":
out["session_id"] = o.get("session_id")
out["init_model"] = o.get("model")
out["init_skills"] = o.get("skills")
out["init_plugins"] = [p.get("name") for p in o.get("plugins") or []]
elif typ == "system" and str(o.get("subtype", "")).startswith("hook"):
if o.get("subtype") == "hook_response":
outp = o.get("output") or ""
hooks.append({"event": o.get("hook_event"), "name": o.get("hook_name"),
"superpowers_injected": SP_INJECT_MARK in outp, "output_chars": len(outp),
"exit_code": o.get("exit_code"), "outcome": o.get("outcome")})
elif typ == "stream_event" and not o.get("parent_tool_use_id"):
ev = o.get("event") or {}
if ev.get("type") == "message_start":
m = ev.get("message") or {}
main_calls[m.get("id")] = {"stage": stage, "tok": _tok(m.get("usage")), "model": m.get("model")}
main_calls["__last__"] = m.get("id")
elif ev.get("type") == "message_delta":
mid = main_calls.get("__last__")
if mid in main_calls:
u = ev.get("usage") or {}
tk = main_calls[mid]["tok"]
tk["out"] = u.get("output_tokens", tk["out"]) or tk["out"]
elif typ == "assistant":
m = o.get("message") or {}
parent = o.get("parent_tool_use_id")
if parent:
sub_snap[m.get("id")] = (parent, _tok(m.get("usage")))
for c in m.get("content") or []:
if c.get("type") == "text" and not parent:
out["last_assistant_text"] = c.get("text") or out["last_assistant_text"]
if c.get("type") != "tool_use":
continue
name = c.get("name")
inp = c.get("input") or {}
(tools_sub if parent else tools_main)[name] += 1
if name in EDIT_TOOLS and inp.get("file_path"):
files.add(inp["file_path"])
# sp: code written while only a design skill is in force = implementation with no skill
if (not parent and not fixed_stage and stage in DESIGN_STAGES
and ("/src/" in inp["file_path"] or "/test/" in inp["file_path"])):
stage = NO_SKILL_IMPL
order.append(stage)
if name in ("Task", "Agent") and not parent:
task_stage[c.get("id")] = stage
if name == "Skill":
sk = inp.get("skill") or inp.get("command") or "?"
rec = {"skill": sk, "t": round(t, 2), "thread": "sub" if parent else "main", "ok": None,
"args": (inp.get("args") or "")[:200]}
skills.append(rec)
skill_ids[c.get("id")] = rec
if not parent and not fixed_stage:
stage = sk.split(":")[-1]
order.append(stage)
elif typ == "user":
for c in (o.get("message") or {}).get("content") or []:
if isinstance(c, dict) and c.get("type") == "tool_result" and c.get("tool_use_id") in skill_ids:
txt = c.get("content")
txt = txt if isinstance(txt, str) else json.dumps(txt)
skill_ids[c["tool_use_id"]]["ok"] = not c.get("is_error") and "Launching skill" in txt
skill_ids[c["tool_use_id"]]["result"] = txt[:200]
elif typ == "result":
result = o
main_calls.pop("__last__", None)
if result:
out.update({"subtype": result.get("subtype"), "is_error": result.get("is_error"),
"num_turns": result.get("num_turns"), "result_text": result.get("result"),
"terminal_reason": result.get("terminal_reason"), "stop_reason": result.get("stop_reason")})
denials = result.get("permission_denials") or []
cum = {"cost": result.get("total_cost_usd") or 0.0, "models": _mu_tokens(result.get("modelUsage"))}
pm = prev["models"] if prev else {}
delta_models = _sub(cum["models"], pm)
delta_cost = cum["cost"] - (prev["cost"] if prev else 0.0)
out["cumulative"] = cum
out["estimated"] = False
else: # killed or crashed: best effort from the stream itself
delta_models = {}
for c in main_calls.values():
d = delta_models.setdefault(c.get("model") or "?", {"in": 0, "out": 0, "cr": 0, "cw": 0, "usd": 0.0})
for k in ("in", "out", "cr", "cw"):
d[k] += c["tok"][k]
d["usd"] += _usd(c["tok"])
for _, (parent, tk) in sub_snap.items():
d = delta_models.setdefault("?sub", {"in": 0, "out": 0, "cr": 0, "cw": 0, "usd": 0.0})
for k in ("in", "out", "cr", "cw"):
d[k] += tk[k]
d["usd"] += _usd(tk)
delta_cost = sum(v["usd"] for v in delta_models.values())
out["cumulative"] = None
out["estimated"] = True
tot = _sum_models(delta_models)
out["usd"] = round(delta_cost, 6)
out["tokens"] = {k: tot[k] for k in ("in", "out", "cr", "cw")}
out["tokens"]["total"] = sum(out["tokens"].values())
out["models"] = {m: v for m, v in delta_models.items() if any(v[k] for k in ("in", "out", "cr", "cw"))}
# Stage segments
segs = defaultdict(lambda: {"in": 0, "out": 0, "cr": 0, "cw": 0, "usd": 0.0, "api_calls": 0, "wall_s": 0.0})
main_tok = {"in": 0, "out": 0, "cr": 0, "cw": 0}
main_usd = 0.0
for c in main_calls.values():
s = segs[c["stage"]]
for k in ("in", "out", "cr", "cw"):
s[k] += c["tok"][k]
main_tok[k] += c["tok"][k]
u = _usd(c["tok"])
s["usd"] += u
s["api_calls"] += 1
main_usd += u
rem = {k: max(0, out["tokens"][k] - main_tok[k]) for k in ("in", "out", "cr", "cw")}
rem_usd = max(0.0, delta_cost - main_usd) if result else 0.0
weights = defaultdict(float)
for _, (parent, tk) in sub_snap.items():
# subagent spend is its own segment, labelled with the stage that dispatched it
weights[task_stage.get(parent, stage) + " [subagent]"] += tk["in"] + tk["cr"] + tk["cw"] + tk["out"]
wsum = sum(weights.values())
if result and (wsum > 0 or rem_usd > 0):
if wsum == 0:
weights = {stage + " [subagent]": 1.0}
wsum = 1.0
for st, w in weights.items():
s = segs[st]
f = w / wsum
for k in ("in", "out", "cr", "cw"):
s[k] += int(round(rem[k] * f))
s["usd"] += rem_usd * f
for st, w in seg_wall.items():
segs[st]["wall_s"] += w
seen = []
for st in order:
if st not in seen:
seen.append(st)
for st in segs:
if st not in seen:
seen.append(st)
out["segments"] = [dict(stage=st, **{k: (round(v, 6) if isinstance(v, float) else v) for k, v in segs[st].items()},
total=segs[st]["in"] + segs[st]["out"] + segs[st]["cr"] + segs[st]["cw"])
for st in seen if st in segs]
out["pricing_check"] = {"main_calls_usd": round(main_usd, 6), "turn_usd": round(delta_cost, 6),
"subagent_usd": round(rem_usd, 6), "main_calls": len(main_calls),
"subagent_msgs": len(sub_snap)}
out["stage_at_end"] = stage
out["tools_main"] = dict(tools_main)
out["tools_sub"] = dict(tools_sub)
out["tool_calls"] = sum(tools_main.values()) + sum(tools_sub.values())
out["skills"] = skills
out["hooks"] = hooks
out["files_touched"] = sorted(files)
out["permission_denials"] = [{"tool": d.get("tool_name"), "input": json.dumps(d.get("tool_input"))[:200]}
for d in denials]
out["stream_lines"] = len(lines)
return out
lib/reaccount.py
"""Recompute the accounting fields of runs/<run>/turns.jsonl from the saved turn streams (after a change to
lib/accounting.py), keeping what the driver recorded (prompt, source, wall, exit code, PO answer). Then run finalize.py.
reaccount.py <run dir>
"""
import json
import os
import sys
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from accounting import account_turn # noqa: E402
KEEP = ("n", "flow", "prompt", "prompt_source", "exit_code", "wall_s", "killed_wall_cap", "po")
run = os.path.abspath(sys.argv[1])
turns = [json.loads(l) for l in open(os.path.join(run, "turns.jsonl"))]
prev, stage_in_force, out = None, "(none)", []
prev_slash = None
for t in turns:
prefix = os.path.join(run, "turn-%02d" % t["n"])
if t["flow"] == "spd" and t["stage"] != prev_slash: # a typed slash command starts its skill (no Skill event)
stage_in_force, prev_slash = t["stage"].lstrip("/"), t["stage"]
sp = t["flow"] in ("sp", "spd")
acc = account_turn(prefix + ".stream.jsonl", prefix + ".times", prev, None if sp else t["stage"], stage_in_force)
if acc.get("cumulative"):
prev = acc["cumulative"]
if sp:
stage_in_force = acc["stage_at_end"]
rec = {k: t[k] for k in KEEP if k in t}
rec["stage"] = acc["stage_at_end"] if t["flow"] == "sp" else t["stage"]
rec.update({k: v for k, v in acc.items() if k != "cumulative"})
out.append(rec)
with open(os.path.join(run, "turns.jsonl"), "w") as f:
for r in out:
f.write(json.dumps(r) + "\n")
print("%s: %d turns reaccounted" % (os.path.basename(run), len(out)))
lib/finalize.py
"""After the driver: final diff, suite, typecheck, hidden oracle, per-stage table -> score.json, stages.json.
finalize.py <run dir> (re-runnable; reads meta.json, turns.jsonl, driver.json)
"""
import json
import os
import re
import shutil
import subprocess
import sys
from collections import OrderedDict
BENCH = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
HIDDEN = os.path.join(BENCH, "hidden")
MODEL = "claude-sonnet-5-5"
UPSTREAM_TESTS = 34
def sh(args, cwd, env=None, timeout=600):
try:
p = subprocess.run(args, cwd=cwd, env=env, stdin=subprocess.DEVNULL, stdout=subprocess.PIPE,
stderr=subprocess.STDOUT, timeout=timeout)
return p.returncode, p.stdout.decode("utf-8", "replace")
except subprocess.TimeoutExpired as e:
return 124, (e.stdout or b"").decode("utf-8", "replace") + "\n[timeout]"
def clean_env(extra=None):
env = {k: v for k, v in os.environ.items() if not k.startswith("CLAUDE")}
env.update(extra or {})
return env
def work_diff(work, base):
"""Working tree vs the base commit (commits made by the worker included) + untracked files. Index untouched."""
_, d = sh(["git", "diff", base, "--no-color", "--", "."], work)
_, others = sh(["git", "ls-files", "--others", "--exclude-standard"], work)
for f in [l for l in others.splitlines() if l.strip()]:
_, nd = sh(["git", "diff", "--no-color", "--no-index", "--", "/dev/null", f], work)
d += nd
return d
def numstat(work, base):
rows = []
_, ns = sh(["git", "diff", base, "--numstat", "--", "."], work)
for l in ns.splitlines():
p = l.split("\t")
if len(p) == 3:
rows.append((int(p[0]) if p[0] != "-" else 0, int(p[1]) if p[1] != "-" else 0, p[2]))
_, others = sh(["git", "ls-files", "--others", "--exclude-standard"], work)
for f in [l for l in others.splitlines() if l.strip()]:
try:
rows.append((sum(1 for _ in open(os.path.join(work, f), errors="replace")), 0, f))
except Exception:
pass
return rows
def run_hidden(work, base, task="F1"):
rep = os.path.join(os.path.dirname(work), ".hidden-%s.json" % os.path.basename(work))
rc, txt = sh([os.path.join(HIDDEN, "node_modules", ".bin", "vitest"), "run", "--reporter=json",
"--outputFile=" + rep], HIDDEN, env=clean_env({"WORK": work, "BASE": base, "TASK": task}), timeout=300)
reqs = OrderedDict(("test_req%d" % i, False) for i in range(1, 6))
fails = {}
try:
j = json.load(open(rep))
for tr in j.get("testResults", []):
for a in tr.get("assertionResults", []):
m = re.match(r"(test_req\d)", a.get("title", ""))
if m:
reqs[m.group(1)] = a.get("status") == "passed"
if a.get("status") != "passed":
fails[m.group(1)] = (" ".join(a.get("failureMessages") or []))[:600]
if tr.get("status") == "failed" and not tr.get("assertionResults"):
fails["collect"] = (tr.get("message") or "")[:800]
os.remove(rep)
except Exception as e: # noqa: BLE001
fails["runner"] = "%s: %s" % (type(e).__name__, txt[-800:])
return reqs, fails, txt
def main():
out = os.path.abspath(sys.argv[1])
meta = json.load(open(os.path.join(out, "meta.json")))
work, base, cfg = meta["work"], meta["base"], meta["cfg"]
if not os.path.isdir(work): # tmp work copy gone: rescore the archived copy
work = os.path.join(BENCH, "work", meta["name"])
drv = json.load(open(os.path.join(out, "driver.json"))) if os.path.exists(os.path.join(out, "driver.json")) else {}
turns = [json.loads(l) for l in open(os.path.join(out, "turns.jsonl"))] if os.path.exists(
os.path.join(out, "turns.jsonl")) else []
have_nm = os.path.isdir(os.path.join(work, "node_modules"))
if not have_nm: # rescoring after cleanup: restore deps from upstream
shutil.copytree(os.path.join(BENCH, "upstream", "node_modules"), os.path.join(work, "node_modules"), symlinks=True)
diff = work_diff(work, base)
open(os.path.join(out, "post.diff"), "w").write(diff)
rows = numstat(work, base)
open(os.path.join(out, "numstat.txt"), "w").write("".join("%d\t%d\t%s\n" % r for r in rows))
code = [r for r in rows if r[2].startswith(("src/", "test/"))]
tests_files = [r for r in rows if r[2].endswith(".test.ts")]
rc_s, suite = sh([os.path.join(work, "node_modules", ".bin", "vitest"), "run"], work, env=clean_env(), timeout=300)
open(os.path.join(out, "suite.txt"), "w").write(suite)
clean = re.sub(r"\x1b\[[0-9;]*m", "", suite)
m = re.search(r"Tests\s+(?:(\d+) failed \| )?(\d+) passed(?: \| \d+ \w+)? \((\d+)\)", clean)
s_fail = int(m.group(1) or 0) if m else None
s_pass = int(m.group(2)) if m else 0
s_total = int(m.group(3)) if m else 0
if m is None:
mf = re.search(r"Tests\s+(\d+) failed \((\d+)\)", clean)
s_fail, s_total = (int(mf.group(1)), int(mf.group(2))) if mf else (None, 0)
rc_t, tc = sh([os.path.join(work, "node_modules", ".bin", "tsc"), "--noEmit"], work, env=clean_env(), timeout=300)
open(os.path.join(out, "typecheck.txt"), "w").write(tc)
task = meta.get("task", "F1")
reqs, fails, htxt = run_hidden(work, base, task)
open(os.path.join(out, "hidden.txt"), "w").write(htxt + "\n" + json.dumps(fails, indent=2))
_, commits = sh(["git", "log", "--oneline", "%s..HEAD" % base], work)
_, branch = sh(["git", "rev-parse", "--abbrev-ref", "HEAD"], work)
_, branches = sh(["git", "branch", "--format=%(refname:short)"], work)
added_its = 0
for l in diff.splitlines():
if re.match(r"^\+\s*(it|test)(\.each\([^)]*\))?\s*\(", l):
added_its += 1
# Per-stage table (sum of turn segments; for mp/bare/both a segment = the turn's stage)
stages = OrderedDict()
for t in turns:
for sgm in t.get("segments") or []:
st = sgm["stage"] if cfg in ("sp", "spd") else t["stage"]
s = stages.setdefault(st, {"turns": set(), "in": 0, "out": 0, "cr": 0, "cw": 0, "total": 0, "usd": 0.0,
"wall_s": 0.0, "api_calls": 0})
s["turns"].add(t["n"])
for k in ("in", "out", "cr", "cw", "total", "api_calls"):
s[k] += sgm.get(k, 0)
s["usd"] += sgm.get("usd", 0.0)
s["wall_s"] += sgm.get("wall_s", 0.0)
if cfg not in ("sp", "spd") and not t.get("segments"):
stages.setdefault(t["stage"], {"turns": {t["n"]}, "in": 0, "out": 0, "cr": 0, "cw": 0, "total": 0,
"usd": 0.0, "wall_s": t.get("wall_s", 0), "api_calls": 0})
# Turn-level wall/usd are exact; mp/bare stage wall = turn wall (segments only cover stdout gaps)
if cfg not in ("sp", "spd"):
for st, s in stages.items():
s["wall_s"] = sum(t.get("wall_s", 0) for t in turns if t["stage"] == st)
s["usd"] = sum(t.get("usd", 0) for t in turns if t["stage"] == st)
for s in stages.values():
s["turns"] = sorted(s["turns"])
s["usd"] = round(s["usd"], 4)
s["wall_s"] = round(s["wall_s"], 1)
json.dump(stages, open(os.path.join(out, "stages.json"), "w"), indent=2)
tok = {k: sum(t.get("tokens", {}).get(k, 0) for t in turns) for k in ("in", "out", "cr", "cw", "total")}
models = sorted({m for t in turns for m in (t.get("models") or {})})
skills = [dict(s, turn=t["n"]) for t in turns for s in t.get("skills") or []]
hooks = [dict(h, turn=t["n"]) for t in turns for h in t.get("hooks") or []]
po_cost = sum((t.get("po") or {}).get("cost_usd", 0) for t in turns)
files = sorted({f.replace(work + "/", "") for t in turns for f in t.get("files_touched") or []})
score = {
"run": meta["name"], "task": meta.get("task", "F1"), "cfg": cfg, "rep": meta["rep"], "tag": meta.get("tag"),
"hidden_pass": sum(reqs.values()), "hidden_reqs": reqs, "hidden_fail_msgs": fails,
"suite_green": rc_s == 0 and (s_fail or 0) == 0 and s_total > 0, "suite_pass": s_pass, "suite_total": s_total,
"tests_added_suite": max(0, s_total - UPSTREAM_TESTS), "tests_added_diff": added_its,
"typecheck_green": rc_t == 0,
"diff_files": len(rows), "diff_added": sum(r[0] for r in rows), "diff_removed": sum(r[1] for r in rows),
"code_diff_added": sum(r[0] for r in code), "code_diff_removed": sum(r[1] for r in code),
"code_files": len(code), "test_files_touched": len(tests_files),
"non_code_files": sorted(r[2] for r in rows if not r[2].startswith(("src/", "test/"))),
"files_touched_by_tools": files,
"commits": len([l for l in commits.splitlines() if l.strip()]), "branch_at_end": branch.strip(),
"branches": branches.split(),
"driver_turns": len(turns), "claude_turns_total": sum(t.get("num_turns") or 0 for t in turns),
"tool_calls": sum(t.get("tool_calls", 0) for t in turns),
"tokens": tok, "usd": round(sum(t.get("usd", 0) for t in turns), 4),
"usd_po": round(po_cost, 4),
"wall_s": round(sum(t.get("wall_s", 0) for t in turns), 1), "run_wall_s": drv.get("run_wall_s"),
"end_reason": drv.get("end_reason"), "caps": drv.get("caps", []),
"questions_routed_to_po": sum(1 for t in turns if (t.get("po") or {}).get("kind") == "question"),
"skills_invoked": [s["skill"] for s in skills],
"skills_failed": [s["skill"] for s in skills if s.get("ok") is False],
"skill_events": skills,
"hook_events": hooks,
"superpowers_injected": any(h.get("superpowers_injected") for h in hooks),
"init_plugins": (turns[0].get("init_plugins") if turns else None),
"models": models, "model_ok": bool(models) and set(models) <= {MODEL},
"permission_denials": sum(len(t.get("permission_denials") or []) for t in turns),
"estimated_turns": [t["n"] for t in turns if t.get("estimated")],
"spd_slash_stage_per_turn": [t["stage"] for t in turns] if cfg == "spd" else None,
"per_turn": [{"n": t["n"], "stage": t["stage"], "source": t.get("prompt_source"), "usd": t.get("usd"),
"tokens": t.get("tokens", {}).get("total"), "wall_s": t.get("wall_s"),
"num_turns": t.get("num_turns"), "subtype": t.get("subtype"),
"skills": [s["skill"] for s in t.get("skills") or []],
"po_kind": (t.get("po") or {}).get("kind")} for t in turns],
"stages": stages,
}
json.dump(score, open(os.path.join(out, "score.json"), "w"), indent=2)
if os.environ.get("KEEP_NODE_MODULES") != "1":
shutil.rmtree(os.path.join(work, "node_modules"), ignore_errors=True)
print("%s: hidden %d/5 suite %s (%d/%d) tsc %s diff +%d/-%d (code +%d/-%d) tokens %d usd %.2f wall %ss turns %d "
"end=%s caps=%s skills=%s" % (
meta["name"], score["hidden_pass"], "green" if score["suite_green"] else "RED", s_pass, s_total,
"ok" if score["typecheck_green"] else "FAIL", score["diff_added"], score["diff_removed"],
score["code_diff_added"], score["code_diff_removed"], tok["total"], score["usd"], score["wall_s"],
len(turns), score["end_reason"], ",".join(score["caps"]) or "-",
",".join(OrderedDict.fromkeys(score["skills_invoked"])) or "-"))
if __name__ == "__main__":
main()
analyze.py
#!/usr/bin/env python3
"""runs/*/score.json (untagged runs only) -> results.json + results.md."""
import json
import os
import statistics as st
from collections import OrderedDict, defaultdict
HERE = os.path.dirname(os.path.abspath(__file__))
CFG_ORDER = ["bare", "sp", "spd", "mp", "both"]
CFG_LABEL = {"bare": "bare", "sp": "sp (hook-routed)", "spd": "spd (Superpowers driven)", "mp": "mp (Main Flow)", "both": "both (sp+mp, mp flow)"}
TASKS = ["F1", "F2"]
SPD_ORDER = ["/brainstorming", "/writing-plans", "/executing-plans"]
MP_ORDER = ["/grill-with-docs", "/to-spec", "/to-tickets", "/implement", "/code-review", "/pr"]
# Superpowers phases (the raw skill-in-force stages are kept too)
SP_PHASE = {
"(none)": "before any skill", "using-superpowers": "before any skill",
"brainstorming": "brainstorming", "writing-plans": "writing-plans",
"executing-plans": "execution", "subagent-driven-development": "execution",
"test-driven-development": "execution", "systematic-debugging": "execution",
"verification-before-completion": "verification", "dispatching-parallel-agents": "execution",
"requesting-code-review": "review", "receiving-code-review": "review",
"finishing-a-development-branch": "finishing", "using-git-worktrees": "execution",
"implementation (no skill)": "implementation (no skill)",
}
PHASE_ORDER = ["before any skill", "brainstorming", "writing-plans", "implementation (no skill)", "execution", "verification", "review", "finishing", "subagents (reviewers, explorers)", "other"]
def mean(xs):
xs = [x for x in xs if x is not None]
return round(st.mean(xs), 3) if xs else None
def med(xs):
xs = [x for x in xs if x is not None]
return round(st.median(xs), 3) if xs else None
def load():
runs = []
for d in sorted(os.listdir(os.path.join(HERE, "runs"))):
p = os.path.join(HERE, "runs", d, "score.json")
if not os.path.exists(p):
continue
s = json.load(open(p))
if s.get("tag"):
continue
runs.append(s)
return runs
def stage_agg(runs, key_fn, order):
agg = defaultdict(lambda: defaultdict(list))
for s in runs:
per = defaultdict(lambda: {"total": 0, "usd": 0.0, "wall_s": 0.0, "out": 0})
for name, v in s["stages"].items():
k = key_fn(name)
for f in ("total", "usd", "wall_s", "out"):
per[k][f] += v.get(f, 0)
for k, v in per.items():
if not (v["total"] or v["usd"] or v["wall_s"]):
continue # empty subagent placeholder segment
for f in v:
agg[k][f].append(v[f])
names = [n for n in order if n in agg] + sorted(n for n in agg if n not in order)
out = OrderedDict()
for n in names:
a = agg[n]
out[n] = {"runs_with_stage": len(a["total"]), "tokens_mean": mean(a["total"]), "tokens_median": med(a["total"]),
"out_tokens_mean": mean(a["out"]), "usd_mean": mean(a["usd"]), "usd_median": med(a["usd"]),
"wall_s_mean": mean(a["wall_s"]), "wall_s_median": med(a["wall_s"])}
return out
def turn_stage_agg(runs, order):
"""Per slash-command stage from the per-turn records (exact turn USD / tokens / wall)."""
agg = defaultdict(lambda: defaultdict(list))
for s in runs:
per = defaultdict(lambda: {"total": 0, "usd": 0.0, "wall_s": 0.0, "out": 0})
for t in s["per_turn"]:
per[t["stage"]]["total"] += t.get("tokens") or 0
per[t["stage"]]["usd"] += t.get("usd") or 0
per[t["stage"]]["wall_s"] += t.get("wall_s") or 0
for k, v in per.items():
for f in v:
agg[k][f].append(v[f])
out = OrderedDict()
for n in [n for n in order if n in agg] + sorted(n for n in agg if n not in order):
a = agg[n]
out[n] = {"runs_with_stage": len(a["total"]), "tokens_mean": mean(a["total"]), "tokens_median": med(a["total"]),
"out_tokens_mean": None, "usd_mean": mean(a["usd"]), "usd_median": med(a["usd"]),
"wall_s_mean": mean(a["wall_s"]), "wall_s_median": med(a["wall_s"])}
return out
def main():
runs = load()
for s in runs:
s.setdefault("task", "F1")
by = OrderedDict(("%s/%s" % (t, c), [s for s in runs if s["cfg"] == c and s["task"] == t]) for t in TASKS for c in CFG_ORDER)
res = {"configs": OrderedDict(), "stages": OrderedDict(), "runs": []}
for key, rs in by.items():
if not rs:
continue
task, c = key.split("/")
m = lambda f: [f(s) for s in rs] # noqa: E731
res["configs"][key] = {
"task": task, "cfg": c, "label": CFG_LABEL[c],
"n": len(rs),
"hidden_mean": mean(m(lambda s: s["hidden_pass"])), "hidden_median": med(m(lambda s: s["hidden_pass"])),
"hidden_per_run": m(lambda s: s["hidden_pass"]),
"req_pass_rate": {r: sum(1 for s in rs if s["hidden_reqs"].get(r)) for r in ["test_req%d" % i for i in range(1, 6)]},
"suite_green": sum(1 for s in rs if s["suite_green"]), "typecheck_green": sum(1 for s in rs if s["typecheck_green"]),
"tests_added_mean": mean(m(lambda s: s["tests_added_suite"])),
"code_diff_mean": mean(m(lambda s: s["code_diff_added"] + s["code_diff_removed"])),
"code_diff_median": med(m(lambda s: s["code_diff_added"] + s["code_diff_removed"])),
"all_diff_mean": mean(m(lambda s: s["diff_added"] + s["diff_removed"])),
"non_code_files_mean": mean(m(lambda s: len(s["non_code_files"]))),
"tokens_mean": mean(m(lambda s: s["tokens"]["total"])), "tokens_median": med(m(lambda s: s["tokens"]["total"])),
"out_tokens_mean": mean(m(lambda s: s["tokens"]["out"])),
"usd_mean": mean(m(lambda s: s["usd"])), "usd_median": med(m(lambda s: s["usd"])),
"usd_po_mean": mean(m(lambda s: s["usd_po"])),
"wall_s_mean": mean(m(lambda s: s["wall_s"])), "wall_s_median": med(m(lambda s: s["wall_s"])),
"driver_turns_mean": mean(m(lambda s: s["driver_turns"])),
"questions_mean": mean(m(lambda s: s["questions_routed_to_po"])),
"caps": {s["run"]: s["caps"] for s in rs if s["caps"]},
"end_reasons": m(lambda s: s["end_reason"]),
"models": sorted({x for s in rs for x in s["models"]}),
"superpowers_injected": m(lambda s: s["superpowers_injected"]),
"skills_per_run": {s["run"]: list(OrderedDict.fromkeys(s["skills_invoked"])) for s in rs},
"skills_failed": {s["run"]: s["skills_failed"] for s in rs if s["skills_failed"]},
}
if c in ("sp", "spd"):
res["stages"]["%s/%s_skill" % (task, c)] = stage_agg(rs, lambda n: n, list(SP_PHASE))
res["stages"]["%s/%s_phase" % (task, c)] = stage_agg(rs, lambda n: "subagents (reviewers, explorers)" if n.endswith("[subagent]") else SP_PHASE.get(n, "other"), PHASE_ORDER)
if c == "spd":
res["stages"]["%s/spd_slash" % task] = turn_stage_agg(rs, SPD_ORDER)
if c in ("mp", "both"):
res["stages"]["%s/%s" % (task, c)] = stage_agg(rs, lambda n: n, MP_ORDER)
for s in runs:
res["runs"].append({k: s[k] for k in ("run", "task", "cfg", "hidden_pass", "hidden_reqs", "suite_green", "typecheck_green",
"code_diff_added", "code_diff_removed", "diff_added", "diff_removed",
"tests_added_suite", "tokens", "usd", "usd_po", "wall_s", "driver_turns",
"questions_routed_to_po", "end_reason", "caps", "skills_invoked",
"superpowers_injected", "commits", "models", "stages")})
json.dump(res, open(os.path.join(HERE, "results.json"), "w"), indent=2)
L = ["# flow-bench results", "",
"Untagged runs only (smoke/validation excluded). Tokens = input + output + cache read + cache write, all models, "
"subagents included. USD = list-price equivalent from `total_cost_usd` deltas (plan quota, not a bill). "
"Wall = sum of worker turn walls (PO time excluded). Diff = lines added+removed vs the base commit, src/ + test/.", "",
"## Per config (mean / median)", "",
"| task / config | n | hidden /5 (per run) | suite green | tsc green | code diff +/- | all files diff | tests added | tokens | USD | wall s | driver turns | questions | caps hit |",
"|---|---|---|---|---|---|---|---|---|---|---|---|---|---|"]
for c, v in res["configs"].items():
L.append("| %s | %d | %s / %s (%s) | %d/%d | %d/%d | %s / %s | %s | %s | %s / %s | %s / %s | %s / %s | %s | %s | %s |" % (
"%s / %s" % (v["task"], v["label"]), v["n"], v["hidden_mean"], v["hidden_median"], ",".join(map(str, v["hidden_per_run"])), v["suite_green"], v["n"],
v["typecheck_green"], v["n"], v["code_diff_mean"], v["code_diff_median"], v["all_diff_mean"], v["tests_added_mean"],
fmt(v["tokens_mean"]), fmt(v["tokens_median"]), v["usd_mean"], v["usd_median"], v["wall_s_mean"], v["wall_s_median"],
v["driver_turns_mean"], v["questions_mean"], "; ".join("%s: %s" % (k, ",".join(x)) for k, x in v["caps"].items()) or "none"))
L += ["", "Requirement pass counts (runs passing each hidden req):", ""]
for c, v in res["configs"].items():
L.append("- **%s %s**: %s" % (v["task"], v["label"], ", ".join("%s %d/%d" % (k.replace("test_", ""), n, v["n"]) for k, n in v["req_pass_rate"].items())))
titles = []
for t in TASKS:
titles += [("%s/spd_slash" % t, "%s spd (Superpowers driven) per slash-command stage (turn level, exact)" % t),
("%s/spd_skill" % t, "%s spd per skill in force (within-turn split)" % t),
("%s/sp_phase" % t, "%s sp (hook-routed) per phase" % t),
("%s/sp_skill" % t, "%s sp (hook-routed) per skill in force" % t),
("%s/mp" % t, "%s mp per slash-command stage" % t), ("%s/both" % t, "%s both per slash-command stage" % t)]
for key, title in titles:
if key not in res["stages"]:
continue
L += ["", "## %s (mean over the runs that had the stage)" % title, "",
"| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |",
"|---|---|---|---|---|---|---|---|---|"]
for n, v in res["stages"][key].items():
L.append("| %s | %d | %s | %s | %s | %s | %s | %s | %s |" % (n, v["runs_with_stage"], fmt(v["tokens_mean"]),
fmt(v["tokens_median"]), fmt(v["out_tokens_mean"]), v["usd_mean"], v["usd_median"], v["wall_s_mean"], v["wall_s_median"]))
L += ["", "## Per run", "",
"| run | task | hidden | reqs failed | suite | tsc | code +/- | tests added | tokens | USD | wall s | turns | end | skills (first use order) |",
"|---|---|---|---|---|---|---|---|---|---|---|---|---|---|"]
for s in sorted(runs, key=lambda s: (s["task"], CFG_ORDER.index(s["cfg"]), s["run"])):
L.append("| %s | %s | %d/5 | %s | %s | %s | +%d/-%d | %d | %s | %.2f | %.0f | %d | %s%s | %s |" % (
s["run"], s["task"], s["hidden_pass"], ",".join(k.replace("test_", "") for k, v in s["hidden_reqs"].items() if not v) or "-",
"green" if s["suite_green"] else "RED", "ok" if s["typecheck_green"] else "FAIL", s["code_diff_added"],
s["code_diff_removed"], s["tests_added_suite"], fmt(s["tokens"]["total"]), s["usd"], s["wall_s"], s["driver_turns"],
s["end_reason"], (" (" + ",".join(s["caps"]) + ")") if s["caps"] else "",
" > ".join(OrderedDict.fromkeys(x.split(":")[-1] for x in s["skills_invoked"])) or "-"))
open(os.path.join(HERE, "results.md"), "w").write("\n".join(L) + "\n")
print("\n".join(L))
def fmt(x):
if x is None:
return "-"
return "{:,.0f}".format(x)
if __name__ == "__main__":
main()
results.json
{
"configs": {
"F1/bare": {
"task": "F1",
"cfg": "bare",
"label": "bare",
"n": 3,
"hidden_mean": 5,
"hidden_median": 5,
"hidden_per_run": [
5,
5,
5
],
"req_pass_rate": {
"test_req1": 3,
"test_req2": 3,
"test_req3": 3,
"test_req4": 3,
"test_req5": 3
},
"suite_green": 3,
"typecheck_green": 3,
"tests_added_mean": 4.667,
"code_diff_mean": 141.667,
"code_diff_median": 137,
"all_diff_mean": 142.333,
"non_code_files_mean": 0.333,
"tokens_mean": 519113.667,
"tokens_median": 361842,
"out_tokens_mean": 12001,
"usd_mean": 0.343,
"usd_median": 0.286,
"usd_po_mean": 0.021,
"wall_s_mean": 84.5,
"wall_s_median": 71.7,
"driver_turns_mean": 1.333,
"questions_mean": 0.333,
"caps": {},
"end_reasons": [
"done",
"done",
"done"
],
"models": [
"claude-sonnet-5-5"
],
"superpowers_injected": [
false,
false,
false
],
"skills_per_run": {
"bare-r1": [],
"bare-r2": [],
"bare-r3": []
},
"skills_failed": {}
},
"F1/sp": {
"task": "F1",
"cfg": "sp",
"label": "sp (hook-routed)",
"n": 3,
"hidden_mean": 5,
"hidden_median": 5,
"hidden_per_run": [
5,
5,
5
],
"req_pass_rate": {
"test_req1": 3,
"test_req2": 3,
"test_req3": 3,
"test_req4": 3,
"test_req5": 3
},
"suite_green": 3,
"typecheck_green": 3,
"tests_added_mean": 12.333,
"code_diff_mean": 235,
"code_diff_median": 227,
"all_diff_mean": 237,
"non_code_files_mean": 0.667,
"tokens_mean": 724600.667,
"tokens_median": 765108,
"out_tokens_mean": 16869.333,
"usd_mean": 0.502,
"usd_median": 0.525,
"usd_po_mean": 0.041,
"wall_s_mean": 118.1,
"wall_s_median": 119.6,
"driver_turns_mean": 2.667,
"questions_mean": 1.667,
"caps": {},
"end_reasons": [
"done",
"done",
"done"
],
"models": [
"claude-sonnet-5-5"
],
"superpowers_injected": [
true,
true,
true
],
"skills_per_run": {
"sp-r1": [
"brainstorming"
],
"sp-r2": [
"brainstorming",
"test-driven-development"
],
"sp-r3": [
"brainstorming"
]
},
"skills_failed": {}
},
"F1/spd": {
"task": "F1",
"cfg": "spd",
"label": "spd (Superpowers driven)",
"n": 3,
"hidden_mean": 5,
"hidden_median": 5,
"hidden_per_run": [
5,
5,
5
],
"req_pass_rate": {
"test_req1": 3,
"test_req2": 3,
"test_req3": 3,
"test_req4": 3,
"test_req5": 3
},
"suite_green": 3,
"typecheck_green": 3,
"tests_added_mean": 27.667,
"code_diff_mean": 386,
"code_diff_median": 371,
"all_diff_mean": 582,
"non_code_files_mean": 2,
"tokens_mean": 4479876.333,
"tokens_median": 3887015,
"out_tokens_mean": 61684.667,
"usd_mean": 3.738,
"usd_median": 3.412,
"usd_po_mean": 0.116,
"wall_s_mean": 607.367,
"wall_s_median": 600.8,
"driver_turns_mean": 7,
"questions_mean": 4,
"caps": {},
"end_reasons": [
"done",
"done",
"done"
],
"models": [
"claude-fable-5-1",
"claude-sonnet-5-5"
],
"superpowers_injected": [
true,
true,
true
],
"skills_per_run": {
"spd-r1": [
"test-driven-development"
],
"spd-r2": [
"test-driven-development"
],
"spd-r3": [
"test-driven-development",
"finishing-a-development-branch"
]
},
"skills_failed": {}
},
"F1/mp": {
"task": "F1",
"cfg": "mp",
"label": "mp (Main Flow)",
"n": 3,
"hidden_mean": 5,
"hidden_median": 5,
"hidden_per_run": [
5,
5,
5
],
"req_pass_rate": {
"test_req1": 3,
"test_req2": 3,
"test_req3": 3,
"test_req4": 3,
"test_req5": 3
},
"suite_green": 3,
"typecheck_green": 3,
"tests_added_mean": 27.667,
"code_diff_mean": 372,
"code_diff_median": 369,
"all_diff_mean": 578,
"non_code_files_mean": 7.667,
"tokens_mean": 4476103.667,
"tokens_median": 4286993,
"out_tokens_mean": 56959,
"usd_mean": 2.043,
"usd_median": 2.102,
"usd_po_mean": 0.205,
"wall_s_mean": 439.033,
"wall_s_median": 472.4,
"driver_turns_mean": 12,
"questions_mean": 7,
"caps": {
"mp-r1": [
"driver_turns_12"
],
"mp-r3": [
"driver_turns_12"
]
},
"end_reasons": [
"cap_driver_turns",
"done",
"cap_driver_turns"
],
"models": [
"claude-sonnet-5-5"
],
"superpowers_injected": [
false,
false,
false
],
"skills_per_run": {
"mp-r1": [
"grilling",
"domain-modeling",
"tdd",
"code-review"
],
"mp-r2": [
"grilling",
"domain-modeling",
"tdd",
"code-review"
],
"mp-r3": [
"grilling",
"domain-modeling",
"tdd",
"code-review"
]
},
"skills_failed": {}
},
"F1/both": {
"task": "F1",
"cfg": "both",
"label": "both (sp+mp, mp flow)",
"n": 1,
"hidden_mean": 5,
"hidden_median": 5,
"hidden_per_run": [
5
],
"req_pass_rate": {
"test_req1": 1,
"test_req2": 1,
"test_req3": 1,
"test_req4": 1,
"test_req5": 1
},
"suite_green": 1,
"typecheck_green": 1,
"tests_added_mean": 22,
"code_diff_mean": 374,
"code_diff_median": 374,
"all_diff_mean": 561,
"non_code_files_mean": 8,
"tokens_mean": 5754354,
"tokens_median": 5754354,
"out_tokens_mean": 57632,
"usd_mean": 2.326,
"usd_median": 2.326,
"usd_po_mean": 0.218,
"wall_s_mean": 448.2,
"wall_s_median": 448.2,
"driver_turns_mean": 12,
"questions_mean": 6,
"caps": {},
"end_reasons": [
"done"
],
"models": [
"claude-sonnet-5-5"
],
"superpowers_injected": [
true
],
"skills_per_run": {
"both-r1": [
"grilling",
"domain-modeling",
"tdd",
"code-review"
]
},
"skills_failed": {}
},
"F2/bare": {
"task": "F2",
"cfg": "bare",
"label": "bare",
"n": 2,
"hidden_mean": 4.5,
"hidden_median": 4.5,
"hidden_per_run": [
4,
5
],
"req_pass_rate": {
"test_req1": 2,
"test_req2": 2,
"test_req3": 1,
"test_req4": 2,
"test_req5": 2
},
"suite_green": 2,
"typecheck_green": 2,
"tests_added_mean": 5,
"code_diff_mean": 206.5,
"code_diff_median": 206.5,
"all_diff_mean": 209.5,
"non_code_files_mean": 1,
"tokens_mean": 651322,
"tokens_median": 651322.0,
"out_tokens_mean": 15475,
"usd_mean": 0.413,
"usd_median": 0.413,
"usd_po_mean": 0.031,
"wall_s_mean": 103.15,
"wall_s_median": 103.15,
"driver_turns_mean": 2,
"questions_mean": 0.5,
"caps": {},
"end_reasons": [
"done",
"done"
],
"models": [
"claude-sonnet-5-5"
],
"superpowers_injected": [
false,
false
],
"skills_per_run": {
"f2-bare-r1": [],
"f2-bare-r2": []
},
"skills_failed": {}
},
"F2/spd": {
"task": "F2",
"cfg": "spd",
"label": "spd (Superpowers driven)",
"n": 2,
"hidden_mean": 5,
"hidden_median": 5.0,
"hidden_per_run": [
5,
5
],
"req_pass_rate": {
"test_req1": 2,
"test_req2": 2,
"test_req3": 2,
"test_req4": 2,
"test_req5": 2
},
"suite_green": 2,
"typecheck_green": 2,
"tests_added_mean": 33,
"code_diff_mean": 488.5,
"code_diff_median": 488.5,
"all_diff_mean": 750.5,
"non_code_files_mean": 2.5,
"tokens_mean": 4490365,
"tokens_median": 4490365.0,
"out_tokens_mean": 56644.5,
"usd_mean": 3.213,
"usd_median": 3.213,
"usd_po_mean": 0.101,
"wall_s_mean": 511.7,
"wall_s_median": 511.7,
"driver_turns_mean": 6.5,
"questions_mean": 3.5,
"caps": {},
"end_reasons": [
"done",
"done"
],
"models": [
"claude-fable-5-1",
"claude-sonnet-5-5"
],
"superpowers_injected": [
true,
true
],
"skills_per_run": {
"f2-spd-r1": [
"writing-plans",
"test-driven-development"
],
"f2-spd-r2": [
"test-driven-development"
]
},
"skills_failed": {}
},
"F2/mp": {
"task": "F2",
"cfg": "mp",
"label": "mp (Main Flow)",
"n": 2,
"hidden_mean": 5,
"hidden_median": 5.0,
"hidden_per_run": [
5,
5
],
"req_pass_rate": {
"test_req1": 2,
"test_req2": 2,
"test_req3": 2,
"test_req4": 2,
"test_req5": 2
},
"suite_green": 2,
"typecheck_green": 2,
"tests_added_mean": 31,
"code_diff_mean": 469,
"code_diff_median": 469.0,
"all_diff_mean": 658,
"non_code_files_mean": 6.5,
"tokens_mean": 4520638.5,
"tokens_median": 4520638.5,
"out_tokens_mean": 57142,
"usd_mean": 2.067,
"usd_median": 2.067,
"usd_po_mean": 0.203,
"wall_s_mean": 421.2,
"wall_s_median": 421.2,
"driver_turns_mean": 12,
"questions_mean": 6.5,
"caps": {
"f2-mp-r1": [
"driver_turns_12"
]
},
"end_reasons": [
"cap_driver_turns",
"done"
],
"models": [
"claude-sonnet-5-5"
],
"superpowers_injected": [
false,
false
],
"skills_per_run": {
"f2-mp-r1": [
"grilling",
"domain-modeling",
"tdd",
"code-review"
],
"f2-mp-r2": [
"grilling",
"domain-modeling",
"tdd",
"code-review"
]
},
"skills_failed": {}
}
},
"stages": {
"F1/sp_skill": {
"(none)": {
"runs_with_stage": 3,
"tokens_mean": 19549,
"tokens_median": 19538,
"out_tokens_mean": 238,
"usd_mean": 0.03,
"usd_median": 0.03,
"wall_s_mean": 5.333,
"wall_s_median": 5.4
},
"brainstorming": {
"runs_with_stage": 3,
"tokens_mean": 249260,
"tokens_median": 247900,
"out_tokens_mean": 8037.333,
"usd_mean": 0.219,
"usd_median": 0.244,
"wall_s_mean": 40.533,
"wall_s_median": 40.1
},
"test-driven-development": {
"runs_with_stage": 1,
"tokens_mean": 595318,
"tokens_median": 595318,
"out_tokens_mean": 15622,
"usd_mean": 0.405,
"usd_median": 0.405,
"wall_s_mean": 93.0,
"wall_s_median": 93.0
},
"implementation (no skill)": {
"runs_with_stage": 2,
"tokens_mean": 386028.5,
"tokens_median": 386028.5,
"out_tokens_mean": 5080,
"usd_mean": 0.176,
"usd_median": 0.176,
"wall_s_mean": 60.3,
"wall_s_median": 60.3
}
},
"F1/sp_phase": {
"before any skill": {
"runs_with_stage": 3,
"tokens_mean": 19549,
"tokens_median": 19538,
"out_tokens_mean": 238,
"usd_mean": 0.03,
"usd_median": 0.03,
"wall_s_mean": 5.333,
"wall_s_median": 5.4
},
"brainstorming": {
"runs_with_stage": 3,
"tokens_mean": 249260,
"tokens_median": 247900,
"out_tokens_mean": 8037.333,
"usd_mean": 0.219,
"usd_median": 0.244,
"wall_s_mean": 40.533,
"wall_s_median": 40.1
},
"implementation (no skill)": {
"runs_with_stage": 2,
"tokens_mean": 386028.5,
"tokens_median": 386028.5,
"out_tokens_mean": 5080,
"usd_mean": 0.176,
"usd_median": 0.176,
"wall_s_mean": 60.3,
"wall_s_median": 60.3
},
"execution": {
"runs_with_stage": 1,
"tokens_mean": 595318,
"tokens_median": 595318,
"out_tokens_mean": 15622,
"usd_mean": 0.405,
"usd_median": 0.405,
"wall_s_mean": 93.0,
"wall_s_median": 93.0
}
},
"F1/spd_skill": {
"brainstorming": {
"runs_with_stage": 3,
"tokens_mean": 182021.667,
"tokens_median": 160834,
"out_tokens_mean": 4160.667,
"usd_mean": 0.166,
"usd_median": 0.156,
"wall_s_mean": 38.8,
"wall_s_median": 33.6
},
"writing-plans": {
"runs_with_stage": 3,
"tokens_mean": 321354.667,
"tokens_median": 298267,
"out_tokens_mean": 11827,
"usd_mean": 0.266,
"usd_median": 0.274,
"wall_s_mean": 86.733,
"wall_s_median": 95.1
},
"executing-plans": {
"runs_with_stage": 3,
"tokens_mean": 66938,
"tokens_median": 68739,
"out_tokens_mean": 505,
"usd_mean": 0.047,
"usd_median": 0.047,
"wall_s_mean": 4.6,
"wall_s_median": 4.6
},
"test-driven-development": {
"runs_with_stage": 3,
"tokens_mean": 2927098.333,
"tokens_median": 2854967,
"out_tokens_mean": 22508.333,
"usd_mean": 0.974,
"usd_median": 0.936,
"wall_s_mean": 457.033,
"wall_s_median": 448.9
},
"finishing-a-development-branch": {
"runs_with_stage": 1,
"tokens_mean": 976505,
"tokens_median": 976505,
"out_tokens_mean": 5297,
"usd_mean": 0.282,
"usd_median": 0.282,
"wall_s_mean": 50.7,
"wall_s_median": 50.7
},
"test-driven-development [subagent]": {
"runs_with_stage": 3,
"tokens_mean": 656962,
"tokens_median": 565811,
"out_tokens_mean": 20918,
"usd_mean": 2.192,
"usd_median": 2.047,
"wall_s_mean": 0.0,
"wall_s_median": 0.0
}
},
"F1/spd_phase": {
"brainstorming": {
"runs_with_stage": 3,
"tokens_mean": 182021.667,
"tokens_median": 160834,
"out_tokens_mean": 4160.667,
"usd_mean": 0.166,
"usd_median": 0.156,
"wall_s_mean": 38.8,
"wall_s_median": 33.6
},
"writing-plans": {
"runs_with_stage": 3,
"tokens_mean": 321354.667,
"tokens_median": 298267,
"out_tokens_mean": 11827,
"usd_mean": 0.266,
"usd_median": 0.274,
"wall_s_mean": 86.733,
"wall_s_median": 95.1
},
"execution": {
"runs_with_stage": 3,
"tokens_mean": 2994036.333,
"tokens_median": 2916459,
"out_tokens_mean": 23013.333,
"usd_mean": 1.021,
"usd_median": 0.981,
"wall_s_mean": 461.633,
"wall_s_median": 453.5
},
"finishing": {
"runs_with_stage": 1,
"tokens_mean": 976505,
"tokens_median": 976505,
"out_tokens_mean": 5297,
"usd_mean": 0.282,
"usd_median": 0.282,
"wall_s_mean": 50.7,
"wall_s_median": 50.7
},
"subagents (reviewers, explorers)": {
"runs_with_stage": 3,
"tokens_mean": 656962,
"tokens_median": 565811,
"out_tokens_mean": 20918,
"usd_mean": 2.192,
"usd_median": 2.047,
"wall_s_mean": 0.0,
"wall_s_median": 0.0
}
},
"F1/spd_slash": {
"/brainstorming": {
"runs_with_stage": 3,
"tokens_mean": 182021.667,
"tokens_median": 160834,
"out_tokens_mean": null,
"usd_mean": 0.166,
"usd_median": 0.156,
"wall_s_mean": 40.283,
"wall_s_median": 35.44
},
"/writing-plans": {
"runs_with_stage": 3,
"tokens_mean": 321354.667,
"tokens_median": 298267,
"out_tokens_mean": null,
"usd_mean": 0.266,
"usd_median": 0.274,
"wall_s_mean": 88.133,
"wall_s_median": 95.82
},
"/executing-plans": {
"runs_with_stage": 3,
"tokens_mean": 3976500,
"tokens_median": 3427914,
"out_tokens_mean": null,
"usd_mean": 3.307,
"usd_median": 3.028,
"wall_s_mean": 478.95,
"wall_s_median": 453.89
}
},
"F1/mp": {
"/grill-with-docs": {
"runs_with_stage": 3,
"tokens_mean": 316507.333,
"tokens_median": 348862,
"out_tokens_mean": 5999,
"usd_mean": 0.204,
"usd_median": 0.215,
"wall_s_mean": 59.933,
"wall_s_median": 59.9
},
"/to-spec": {
"runs_with_stage": 3,
"tokens_mean": 163344,
"tokens_median": 167578,
"out_tokens_mean": 5134.667,
"usd_mean": 0.118,
"usd_median": 0.113,
"wall_s_mean": 42.1,
"wall_s_median": 40.8
},
"/to-tickets": {
"runs_with_stage": 3,
"tokens_mean": 257955.333,
"tokens_median": 268062,
"out_tokens_mean": 5867.333,
"usd_mean": 0.143,
"usd_median": 0.151,
"wall_s_mean": 44.4,
"wall_s_median": 47.8
},
"/implement": {
"runs_with_stage": 3,
"tokens_mean": 2655564.667,
"tokens_median": 2613062,
"out_tokens_mean": 30214.667,
"usd_mean": 1.125,
"usd_median": 1.195,
"wall_s_mean": 224.567,
"wall_s_median": 245.4
},
"/code-review": {
"runs_with_stage": 3,
"tokens_mean": 1047585.667,
"tokens_median": 1107555,
"out_tokens_mean": 9293.667,
"usd_mean": 0.439,
"usd_median": 0.445,
"wall_s_mean": 64.2,
"wall_s_median": 65.8
},
"/pr": {
"runs_with_stage": 1,
"tokens_mean": 105440,
"tokens_median": 105440,
"out_tokens_mean": 1349,
"usd_mean": 0.041,
"usd_median": 0.041,
"wall_s_mean": 11.6,
"wall_s_median": 11.6
}
},
"F1/both": {
"/grill-with-docs": {
"runs_with_stage": 1,
"tokens_mean": 344391,
"tokens_median": 344391,
"out_tokens_mean": 6193,
"usd_mean": 0.225,
"usd_median": 0.225,
"wall_s_mean": 61.8,
"wall_s_median": 61.8
},
"/to-spec": {
"runs_with_stage": 1,
"tokens_mean": 217572,
"tokens_median": 217572,
"out_tokens_mean": 4835,
"usd_mean": 0.123,
"usd_median": 0.123,
"wall_s_mean": 42.9,
"wall_s_median": 42.9
},
"/to-tickets": {
"runs_with_stage": 1,
"tokens_mean": 382953,
"tokens_median": 382953,
"out_tokens_mean": 7115,
"usd_mean": 0.196,
"usd_median": 0.196,
"wall_s_mean": 49.4,
"wall_s_median": 49.4
},
"/implement": {
"runs_with_stage": 1,
"tokens_mean": 3070579,
"tokens_median": 3070579,
"out_tokens_mean": 24618,
"usd_mean": 1.112,
"usd_median": 1.112,
"wall_s_mean": 194.2,
"wall_s_median": 194.2
},
"/code-review": {
"runs_with_stage": 1,
"tokens_mean": 1615265,
"tokens_median": 1615265,
"out_tokens_mean": 13410,
"usd_mean": 0.625,
"usd_median": 0.625,
"wall_s_mean": 86.8,
"wall_s_median": 86.8
},
"/pr": {
"runs_with_stage": 1,
"tokens_mean": 123594,
"tokens_median": 123594,
"out_tokens_mean": 1461,
"usd_mean": 0.045,
"usd_median": 0.045,
"wall_s_mean": 13.1,
"wall_s_median": 13.1
}
},
"F2/spd_skill": {
"brainstorming": {
"runs_with_stage": 2,
"tokens_mean": 253890,
"tokens_median": 253890.0,
"out_tokens_mean": 6038,
"usd_mean": 0.207,
"usd_median": 0.207,
"wall_s_mean": 53.05,
"wall_s_median": 53.05
},
"writing-plans": {
"runs_with_stage": 2,
"tokens_mean": 397486.5,
"tokens_median": 397486.5,
"out_tokens_mean": 14059.5,
"usd_mean": 0.307,
"usd_median": 0.307,
"wall_s_mean": 90.75,
"wall_s_median": 90.75
},
"executing-plans": {
"runs_with_stage": 2,
"tokens_mean": 69973,
"tokens_median": 69973.0,
"out_tokens_mean": 286.5,
"usd_mean": 0.046,
"usd_median": 0.046,
"wall_s_mean": 3.5,
"wall_s_median": 3.5
},
"test-driven-development": {
"runs_with_stage": 2,
"tokens_mean": 3304676.5,
"tokens_median": 3304676.5,
"out_tokens_mean": 23243.5,
"usd_mean": 1.052,
"usd_median": 1.052,
"wall_s_mean": 361.25,
"wall_s_median": 361.25
},
"test-driven-development [subagent]": {
"runs_with_stage": 2,
"tokens_mean": 464339,
"tokens_median": 464339.0,
"out_tokens_mean": 13017,
"usd_mean": 1.602,
"usd_median": 1.602,
"wall_s_mean": 0.0,
"wall_s_median": 0.0
}
},
"F2/spd_phase": {
"brainstorming": {
"runs_with_stage": 2,
"tokens_mean": 253890,
"tokens_median": 253890.0,
"out_tokens_mean": 6038,
"usd_mean": 0.207,
"usd_median": 0.207,
"wall_s_mean": 53.05,
"wall_s_median": 53.05
},
"writing-plans": {
"runs_with_stage": 2,
"tokens_mean": 397486.5,
"tokens_median": 397486.5,
"out_tokens_mean": 14059.5,
"usd_mean": 0.307,
"usd_median": 0.307,
"wall_s_mean": 90.75,
"wall_s_median": 90.75
},
"execution": {
"runs_with_stage": 2,
"tokens_mean": 3374649.5,
"tokens_median": 3374649.5,
"out_tokens_mean": 23530,
"usd_mean": 1.098,
"usd_median": 1.098,
"wall_s_mean": 364.75,
"wall_s_median": 364.75
},
"subagents (reviewers, explorers)": {
"runs_with_stage": 2,
"tokens_mean": 464339,
"tokens_median": 464339.0,
"out_tokens_mean": 13017,
"usd_mean": 1.602,
"usd_median": 1.602,
"wall_s_mean": 0.0,
"wall_s_median": 0.0
}
},
"F2/spd_slash": {
"/brainstorming": {
"runs_with_stage": 2,
"tokens_mean": 485364,
"tokens_median": 485364.0,
"out_tokens_mean": null,
"usd_mean": 0.354,
"usd_median": 0.354,
"wall_s_mean": 96.815,
"wall_s_median": 96.815
},
"/writing-plans": {
"runs_with_stage": 2,
"tokens_mean": 166012.5,
"tokens_median": 166012.5,
"out_tokens_mean": null,
"usd_mean": 0.159,
"usd_median": 0.159,
"wall_s_mean": 49.755,
"wall_s_median": 49.755
},
"/executing-plans": {
"runs_with_stage": 2,
"tokens_mean": 3838988.5,
"tokens_median": 3838988.5,
"out_tokens_mean": null,
"usd_mean": 2.7,
"usd_median": 2.7,
"wall_s_mean": 365.1,
"wall_s_median": 365.1
}
},
"F2/mp": {
"/grill-with-docs": {
"runs_with_stage": 2,
"tokens_mean": 232965,
"tokens_median": 232965.0,
"out_tokens_mean": 5946,
"usd_mean": 0.182,
"usd_median": 0.182,
"wall_s_mean": 56.7,
"wall_s_median": 56.7
},
"/to-spec": {
"runs_with_stage": 2,
"tokens_mean": 180084.5,
"tokens_median": 180084.5,
"out_tokens_mean": 4452,
"usd_mean": 0.115,
"usd_median": 0.115,
"wall_s_mean": 37.6,
"wall_s_median": 37.6
},
"/to-tickets": {
"runs_with_stage": 2,
"tokens_mean": 253688.5,
"tokens_median": 253688.5,
"out_tokens_mean": 5152.5,
"usd_mean": 0.131,
"usd_median": 0.131,
"wall_s_mean": 38.5,
"wall_s_median": 38.5
},
"/implement": {
"runs_with_stage": 2,
"tokens_mean": 2689071,
"tokens_median": 2689071.0,
"out_tokens_mean": 31452,
"usd_mean": 1.159,
"usd_median": 1.159,
"wall_s_mean": 218.35,
"wall_s_median": 218.35
},
"/code-review": {
"runs_with_stage": 2,
"tokens_mean": 1107122.5,
"tokens_median": 1107122.5,
"out_tokens_mean": 9453.5,
"usd_mean": 0.458,
"usd_median": 0.458,
"wall_s_mean": 63.85,
"wall_s_median": 63.85
},
"/pr": {
"runs_with_stage": 1,
"tokens_mean": 115414,
"tokens_median": 115414,
"out_tokens_mean": 1372,
"usd_mean": 0.043,
"usd_median": 0.043,
"wall_s_mean": 12.4,
"wall_s_median": 12.4
}
}
},
"runs": [
{
"run": "bare-r1",
"task": "F1",
"cfg": "bare",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 149,
"code_diff_removed": 11,
"diff_added": 150,
"diff_removed": 12,
"tests_added_suite": 6,
"tokens": {
"in": 42,
"out": 16265,
"cr": 786548,
"cw": 37896,
"total": 840751
},
"usd": 0.4716,
"usd_po": 0.0321,
"wall_s": 118.8,
"driver_turns": 2,
"questions_routed_to_po": 1,
"end_reason": "done",
"caps": [],
"skills_invoked": [],
"superpowers_injected": false,
"commits": 0,
"models": [
"claude-sonnet-5-5"
],
"stages": {
"bare": {
"turns": [
1,
2
],
"in": 42,
"out": 16265,
"cr": 786548,
"cw": 37896,
"total": 840751,
"usd": 0.4716,
"wall_s": 118.8,
"api_calls": 21
}
}
},
{
"run": "bare-r2",
"task": "F1",
"cfg": "bare",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 118,
"code_diff_removed": 10,
"diff_added": 118,
"diff_removed": 10,
"tests_added_suite": 4,
"tokens": {
"in": 22,
"out": 9513,
"cr": 316959,
"cw": 28254,
"total": 354748
},
"usd": 0.2716,
"usd_po": 0.016,
"wall_s": 63.0,
"driver_turns": 1,
"questions_routed_to_po": 0,
"end_reason": "done",
"caps": [],
"skills_invoked": [],
"superpowers_injected": false,
"commits": 0,
"models": [
"claude-sonnet-5-5"
],
"stages": {
"bare": {
"turns": [
1
],
"in": 22,
"out": 9513,
"cr": 316959,
"cw": 28254,
"total": 354748,
"usd": 0.2716,
"wall_s": 63.0,
"api_calls": 11
}
}
},
{
"run": "bare-r3",
"task": "F1",
"cfg": "bare",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 125,
"code_diff_removed": 12,
"diff_added": 125,
"diff_removed": 12,
"tests_added_suite": 4,
"tokens": {
"in": 22,
"out": 10225,
"cr": 321705,
"cw": 29890,
"total": 361842
},
"usd": 0.2862,
"usd_po": 0.0148,
"wall_s": 71.7,
"driver_turns": 1,
"questions_routed_to_po": 0,
"end_reason": "done",
"caps": [],
"skills_invoked": [],
"superpowers_injected": false,
"commits": 0,
"models": [
"claude-sonnet-5-5"
],
"stages": {
"bare": {
"turns": [
1
],
"in": 22,
"out": 10225,
"cr": 321705,
"cw": 29890,
"total": 361842,
"usd": 0.2862,
"wall_s": 71.7,
"api_calls": 11
}
}
},
{
"run": "both-r1",
"task": "F1",
"cfg": "both",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 356,
"code_diff_removed": 18,
"diff_added": 543,
"diff_removed": 18,
"tests_added_suite": 22,
"tokens": {
"in": 192,
"out": 57632,
"cr": 5502518,
"cw": 194012,
"total": 5754354
},
"usd": 2.3259,
"usd_po": 0.2185,
"wall_s": 448.2,
"driver_turns": 12,
"questions_routed_to_po": 6,
"end_reason": "done",
"caps": [],
"skills_invoked": [
"grilling",
"domain-modeling",
"tdd",
"code-review"
],
"superpowers_injected": true,
"commits": 7,
"models": [
"claude-sonnet-5-5"
],
"stages": {
"/grill-with-docs": {
"turns": [
1,
2,
3,
4
],
"in": 22,
"out": 6193,
"cr": 313031,
"cw": 25145,
"total": 344391,
"usd": 0.2252,
"wall_s": 61.8,
"api_calls": 11
},
"/to-spec": {
"turns": [
5,
6
],
"in": 12,
"out": 4835,
"cr": 204355,
"cw": 8370,
"total": 217572,
"usd": 0.1227,
"wall_s": 42.9,
"api_calls": 5
},
"/to-tickets": {
"turns": [
7,
8
],
"in": 16,
"out": 7115,
"cr": 362815,
"cw": 13007,
"total": 382953,
"usd": 0.1958,
"wall_s": 49.4,
"api_calls": 7
},
"/implement": {
"turns": [
9
],
"in": 92,
"out": 24618,
"cr": 2960003,
"cw": 85866,
"total": 3070579,
"usd": 1.1123,
"wall_s": 194.2,
"api_calls": 36
},
"/code-review": {
"turns": [
10,
11
],
"in": 46,
"out": 13410,
"cr": 1541663,
"cw": 60146,
"total": 1615265,
"usd": 0.6253,
"wall_s": 86.8,
"api_calls": 13
},
"/pr": {
"turns": [
12
],
"in": 4,
"out": 1461,
"cr": 120651,
"cw": 1478,
"total": 123594,
"usd": 0.0447,
"wall_s": 13.1,
"api_calls": 1
}
}
},
{
"run": "f2-bare-r1",
"task": "F2",
"cfg": "bare",
"hidden_pass": 4,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": false,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 204,
"code_diff_removed": 10,
"diff_added": 208,
"diff_removed": 10,
"tests_added_suite": 5,
"tokens": {
"in": 28,
"out": 15066,
"cr": 444890,
"cw": 34247,
"total": 494231
},
"usd": 0.3767,
"usd_po": 0.0167,
"wall_s": 91.6,
"driver_turns": 1,
"questions_routed_to_po": 0,
"end_reason": "done",
"caps": [],
"skills_invoked": [],
"superpowers_injected": false,
"commits": 0,
"models": [
"claude-sonnet-5-5"
],
"stages": {
"bare": {
"turns": [
1
],
"in": 28,
"out": 15066,
"cr": 444890,
"cw": 34247,
"total": 494231,
"usd": 0.3767,
"wall_s": 91.6,
"api_calls": 14
}
}
},
{
"run": "f2-bare-r2",
"task": "F2",
"cfg": "bare",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 189,
"code_diff_removed": 10,
"diff_added": 191,
"diff_removed": 10,
"tests_added_suite": 5,
"tokens": {
"in": 42,
"out": 15884,
"cr": 757517,
"cw": 34970,
"total": 808413
},
"usd": 0.4503,
"usd_po": 0.0453,
"wall_s": 114.7,
"driver_turns": 3,
"questions_routed_to_po": 1,
"end_reason": "done",
"caps": [],
"skills_invoked": [],
"superpowers_injected": false,
"commits": 1,
"models": [
"claude-sonnet-5-5"
],
"stages": {
"bare": {
"turns": [
1,
2,
3
],
"in": 42,
"out": 15884,
"cr": 757517,
"cw": 34970,
"total": 808413,
"usd": 0.4503,
"wall_s": 114.7,
"api_calls": 21
}
}
},
{
"run": "f2-mp-r1",
"task": "F2",
"cfg": "mp",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 460,
"code_diff_removed": 23,
"diff_added": 666,
"diff_removed": 25,
"tests_added_suite": 31,
"tokens": {
"in": 172,
"out": 60658,
"cr": 4559790,
"cw": 202896,
"total": 4823516
},
"usd": 2.1824,
"usd_po": 0.1989,
"wall_s": 443.0,
"driver_turns": 12,
"questions_routed_to_po": 7,
"end_reason": "cap_driver_turns",
"caps": [
"driver_turns_12"
],
"skills_invoked": [
"grilling",
"domain-modeling",
"tdd",
"code-review"
],
"superpowers_injected": false,
"commits": 7,
"models": [
"claude-sonnet-5-5"
],
"stages": {
"/grill-with-docs": {
"turns": [
1,
2,
3
],
"in": 18,
"out": 6627,
"cr": 229372,
"cw": 22326,
"total": 258343,
"usd": 0.2015,
"wall_s": 62.8,
"api_calls": 9
},
"/to-spec": {
"turns": [
4,
5
],
"in": 12,
"out": 5218,
"cr": 195137,
"cw": 10113,
"total": 210480,
"usd": 0.1317,
"wall_s": 45.4,
"api_calls": 5
},
"/to-tickets": {
"turns": [
6,
7,
8
],
"in": 16,
"out": 7288,
"cr": 356825,
"cw": 10505,
"total": 374634,
"usd": 0.1863,
"wall_s": 53.0,
"api_calls": 7
},
"/implement": {
"turns": [
9,
10
],
"in": 92,
"out": 34442,
"cr": 2978947,
"cw": 104921,
"total": 3118402,
"usd": 1.2795,
"wall_s": 238.8,
"api_calls": 36
},
"/code-review": {
"turns": [
11,
12
],
"in": 34,
"out": 7083,
"cr": 799509,
"cw": 55031,
"total": 861657,
"usd": 0.3835,
"wall_s": 43.1,
"api_calls": 6
}
}
},
{
"run": "f2-mp-r2",
"task": "F2",
"cfg": "mp",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 440,
"code_diff_removed": 15,
"diff_added": 607,
"diff_removed": 18,
"tests_added_suite": 31,
"tokens": {
"in": 152,
"out": 53626,
"cr": 3976936,
"cw": 187047,
"total": 4217761
},
"usd": 1.9511,
"usd_po": 0.2073,
"wall_s": 399.4,
"driver_turns": 12,
"questions_routed_to_po": 6,
"end_reason": "done",
"caps": [],
"skills_invoked": [
"grilling",
"domain-modeling",
"tdd",
"code-review"
],
"superpowers_injected": false,
"commits": 2,
"models": [
"claude-sonnet-5-5"
],
"stages": {
"/grill-with-docs": {
"turns": [
1,
2,
3
],
"in": 16,
"out": 5265,
"cr": 184055,
"cw": 18251,
"total": 207587,
"usd": 0.1625,
"wall_s": 50.6,
"api_calls": 8
},
"/to-spec": {
"turns": [
4,
5
],
"in": 10,
"out": 3686,
"cr": 137547,
"cw": 8446,
"total": 149689,
"usd": 0.0982,
"wall_s": 29.8,
"api_calls": 4
},
"/to-tickets": {
"turns": [
6,
7
],
"in": 8,
"out": 3017,
"cr": 124377,
"cw": 5341,
"total": 132743,
"usd": 0.0764,
"wall_s": 24.0,
"api_calls": 3
},
"/implement": {
"turns": [
8
],
"in": 74,
"out": 28462,
"cr": 2129926,
"cw": 101278,
"total": 2259740,
"usd": 1.0391,
"wall_s": 197.9,
"api_calls": 27
},
"/code-review": {
"turns": [
9,
10,
11
],
"in": 43,
"out": 11824,
"cr": 1288603,
"cw": 52118,
"total": 1352588,
"usd": 0.5323,
"wall_s": 84.6,
"api_calls": 11
},
"/pr": {
"turns": [
12
],
"in": 1,
"out": 1372,
"cr": 112428,
"cw": 1613,
"total": 115414,
"usd": 0.0427,
"wall_s": 12.4,
"api_calls": 1
}
}
},
{
"run": "f2-spd-r1",
"task": "F2",
"cfg": "spd",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 546,
"code_diff_removed": 14,
"diff_added": 856,
"diff_removed": 17,
"tests_added_suite": 46,
"tokens": {
"in": 380,
"out": 56881,
"cr": 5074733,
"cw": 178622,
"total": 5310616
},
"usd": 3.4244,
"usd_po": 0.09,
"wall_s": 506.3,
"driver_turns": 6,
"questions_routed_to_po": 3,
"end_reason": "done",
"caps": [],
"skills_invoked": [
"writing-plans",
"test-driven-development"
],
"superpowers_injected": true,
"commits": 8,
"models": [
"claude-fable-5-1",
"claude-sonnet-5-5"
],
"stages": {
"brainstorming": {
"turns": [
1,
2,
3
],
"in": 18,
"out": 8103,
"cr": 280105,
"cw": 28324,
"total": 316550,
"usd": 0.2504,
"wall_s": 64.3,
"api_calls": 9
},
"writing-plans": {
"turns": [
3,
4,
5
],
"in": 20,
"out": 13028,
"cr": 491569,
"cw": 24216,
"total": 528833,
"usd": 0.3255,
"wall_s": 89.2,
"api_calls": 9
},
"writing-plans [subagent]": {
"turns": [
4
],
"in": 0,
"out": 0,
"cr": 0,
"cw": 0,
"total": 0,
"usd": 0.0,
"wall_s": 0.0,
"api_calls": 0
},
"executing-plans": {
"turns": [
6
],
"in": 4,
"out": 291,
"cr": 65551,
"cw": 7617,
"total": 73463,
"usd": 0.0465,
"wall_s": 4.0,
"api_calls": 1
},
"test-driven-development": {
"turns": [
6
],
"in": 80,
"out": 24030,
"cr": 3786616,
"cw": 42748,
"total": 3853474,
"usd": 1.1688,
"wall_s": 345.9,
"api_calls": 40
},
"test-driven-development [subagent]": {
"turns": [
6
],
"in": 258,
"out": 11429,
"cr": 450892,
"cw": 75717,
"total": 538296,
"usd": 1.6332,
"wall_s": 0.0,
"api_calls": 0
}
}
},
{
"run": "f2-spd-r2",
"task": "F2",
"cfg": "spd",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 401,
"code_diff_removed": 16,
"diff_added": 611,
"diff_removed": 17,
"tests_added_suite": 20,
"tokens": {
"in": 316,
"out": 56408,
"cr": 3456411,
"cw": 156979,
"total": 3670114
},
"usd": 3.0022,
"usd_po": 0.1126,
"wall_s": 517.1,
"driver_turns": 7,
"questions_routed_to_po": 4,
"end_reason": "done",
"caps": [],
"skills_invoked": [
"test-driven-development"
],
"superpowers_injected": true,
"commits": 6,
"models": [
"claude-fable-5-1",
"claude-sonnet-5-5"
],
"stages": {
"brainstorming": {
"turns": [
1,
2,
3,
4
],
"in": 12,
"out": 3973,
"cr": 164538,
"cw": 22707,
"total": 191230,
"usd": 0.1635,
"wall_s": 41.8,
"api_calls": 6
},
"brainstorming [subagent]": {
"turns": [
1,
3
],
"in": 0,
"out": 0,
"cr": 0,
"cw": 0,
"total": 0,
"usd": 0.0,
"wall_s": 0.0,
"api_calls": 0
},
"writing-plans": {
"turns": [
5,
6
],
"in": 12,
"out": 15091,
"cr": 228188,
"cw": 22849,
"total": 266140,
"usd": 0.288,
"wall_s": 92.3,
"api_calls": 5
},
"writing-plans [subagent]": {
"turns": [
6
],
"in": 0,
"out": 0,
"cr": 0,
"cw": 0,
"total": 0,
"usd": 0.0,
"wall_s": 0.0,
"api_calls": 0
},
"executing-plans": {
"turns": [
7
],
"in": 4,
"out": 282,
"cr": 58567,
"cw": 7630,
"total": 66483,
"usd": 0.0451,
"wall_s": 3.0,
"api_calls": 1
},
"test-driven-development": {
"turns": [
7
],
"in": 62,
"out": 22457,
"cr": 2690340,
"cw": 43020,
"total": 2755879,
"usd": 0.9348,
"wall_s": 376.6,
"api_calls": 31
},
"test-driven-development [subagent]": {
"turns": [
7
],
"in": 226,
"out": 14605,
"cr": 314778,
"cw": 60773,
"total": 390382,
"usd": 1.5709,
"wall_s": 0.0,
"api_calls": 0
}
}
},
{
"run": "mp-r1",
"task": "F1",
"cfg": "mp",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 349,
"code_diff_removed": 20,
"diff_added": 524,
"diff_removed": 20,
"tests_added_suite": 29,
"tokens": {
"in": 181,
"out": 60902,
"cr": 4842442,
"cw": 194978,
"total": 5098503
},
"usd": 2.2217,
"usd_po": 0.2058,
"wall_s": 472.4,
"driver_turns": 12,
"questions_routed_to_po": 7,
"end_reason": "cap_driver_turns",
"caps": [
"driver_turns_12"
],
"skills_invoked": [
"grilling",
"domain-modeling",
"tdd",
"code-review"
],
"superpowers_injected": false,
"commits": 7,
"models": [
"claude-sonnet-5-5"
],
"stages": {
"/grill-with-docs": {
"turns": [
1,
2,
3
],
"in": 24,
"out": 6223,
"cr": 320351,
"cw": 22264,
"total": 348862,
"usd": 0.2154,
"wall_s": 59.9,
"api_calls": 12
},
"/to-spec": {
"turns": [
4,
5
],
"in": 10,
"out": 4516,
"cr": 153800,
"cw": 9252,
"total": 167578,
"usd": 0.1129,
"wall_s": 37.6,
"api_calls": 4
},
"/to-tickets": {
"turns": [
6,
7,
8
],
"in": 16,
"out": 7382,
"cr": 349965,
"cw": 10865,
"total": 368228,
"usd": 0.1873,
"wall_s": 55.8,
"api_calls": 7
},
"/implement": {
"turns": [
9,
10
],
"in": 92,
"out": 32290,
"cr": 2890397,
"cw": 100256,
"total": 3023035,
"usd": 1.226,
"wall_s": 251.2,
"api_calls": 35
},
"/code-review": {
"turns": [
11,
12
],
"in": 39,
"out": 10491,
"cr": 1127929,
"cw": 52341,
"total": 1190800,
"usd": 0.4801,
"wall_s": 67.9,
"api_calls": 9
}
}
},
{
"run": "mp-r2",
"task": "F1",
"cfg": "mp",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 314,
"code_diff_removed": 15,
"diff_added": 527,
"diff_removed": 15,
"tests_added_suite": 25,
"tokens": {
"in": 154,
"out": 48104,
"cr": 3826019,
"cw": 168538,
"total": 4042815
},
"usd": 1.8045,
"usd_po": 0.1988,
"wall_s": 365.0,
"driver_turns": 12,
"questions_routed_to_po": 6,
"end_reason": "done",
"caps": [],
"skills_invoked": [
"grilling",
"domain-modeling",
"tdd",
"code-review"
],
"superpowers_injected": false,
"commits": 7,
"models": [
"claude-sonnet-5-5"
],
"stages": {
"/grill-with-docs": {
"turns": [
1,
2,
3
],
"in": 16,
"out": 4727,
"cr": 187718,
"cw": 18567,
"total": 211028,
"usd": 0.1591,
"wall_s": 47.0,
"api_calls": 8
},
"/to-spec": {
"turns": [
4,
5
],
"in": 10,
"out": 4965,
"cr": 136863,
"cw": 8781,
"total": 150619,
"usd": 0.1122,
"wall_s": 40.8,
"api_calls": 4
},
"/to-tickets": {
"turns": [
6,
7
],
"in": 8,
"out": 4035,
"cr": 126982,
"cw": 6551,
"total": 137576,
"usd": 0.092,
"wall_s": 29.6,
"api_calls": 3
},
"/implement": {
"turns": [
8,
9
],
"in": 78,
"out": 24215,
"cr": 2224544,
"cw": 81760,
"total": 2330597,
"usd": 0.9551,
"wall_s": 177.1,
"api_calls": 31
},
"/code-review": {
"turns": [
10,
11
],
"in": 38,
"out": 8813,
"cr": 1047606,
"cw": 51098,
"total": 1107555,
"usd": 0.4451,
"wall_s": 58.9,
"api_calls": 10
},
"/pr": {
"turns": [
12
],
"in": 4,
"out": 1349,
"cr": 102306,
"cw": 1781,
"total": 105440,
"usd": 0.0411,
"wall_s": 11.6,
"api_calls": 1
}
}
},
{
"run": "mp-r3",
"task": "F1",
"cfg": "mp",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 395,
"code_diff_removed": 23,
"diff_added": 625,
"diff_removed": 23,
"tests_added_suite": 29,
"tokens": {
"in": 167,
"out": 61871,
"cr": 4014893,
"cw": 210062,
"total": 4286993
},
"usd": 2.1024,
"usd_po": 0.2105,
"wall_s": 479.7,
"driver_turns": 12,
"questions_routed_to_po": 8,
"end_reason": "cap_driver_turns",
"caps": [
"driver_turns_12"
],
"skills_invoked": [
"grilling",
"domain-modeling",
"tdd",
"code-review"
],
"superpowers_injected": false,
"commits": 5,
"models": [
"claude-sonnet-5-5"
],
"stages": {
"/grill-with-docs": {
"turns": [
1,
2,
3,
4
],
"in": 26,
"out": 7047,
"cr": 358800,
"cw": 23759,
"total": 389632,
"usd": 0.2373,
"wall_s": 72.9,
"api_calls": 13
},
"/to-spec": {
"turns": [
5,
6
],
"in": 10,
"out": 5923,
"cr": 156583,
"cw": 9319,
"total": 171835,
"usd": 0.1278,
"wall_s": 47.9,
"api_calls": 4
},
"/to-tickets": {
"turns": [
7,
8,
9
],
"in": 9,
"out": 6185,
"cr": 252224,
"cw": 9644,
"total": 268062,
"usd": 0.1509,
"wall_s": 47.8,
"api_calls": 5
},
"/implement": {
"turns": [
10,
11
],
"in": 82,
"out": 34139,
"cr": 2465960,
"cw": 112881,
"total": 2613062,
"usd": 1.1946,
"wall_s": 245.4,
"api_calls": 27
},
"/code-review": {
"turns": [
12
],
"in": 40,
"out": 8577,
"cr": 781326,
"cw": 54459,
"total": 844402,
"usd": 0.3917,
"wall_s": 65.8,
"api_calls": 5
}
}
},
{
"run": "sp-r1",
"task": "F1",
"cfg": "sp",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 182,
"code_diff_removed": 15,
"diff_added": 183,
"diff_removed": 16,
"tests_added_suite": 10,
"tokens": {
"in": 28,
"out": 14522,
"cr": 553287,
"cw": 45597,
"total": 613434
},
"usd": 0.4383,
"usd_po": 0.0424,
"wall_s": 99.4,
"driver_turns": 3,
"questions_routed_to_po": 2,
"end_reason": "done",
"caps": [],
"skills_invoked": [
"brainstorming"
],
"superpowers_injected": true,
"commits": 1,
"models": [
"claude-sonnet-5-5"
],
"stages": {
"(none)": {
"turns": [
1
],
"in": 2,
"out": 209,
"cr": 13011,
"cw": 6298,
"total": 19520,
"usd": 0.0299,
"wall_s": 5.4,
"api_calls": 1
},
"brainstorming": {
"turns": [
1,
2
],
"in": 14,
"out": 9666,
"cr": 212052,
"cw": 26168,
"total": 247900,
"usd": 0.2438,
"wall_s": 40.1,
"api_calls": 7
},
"implementation (no skill)": {
"turns": [
2,
3
],
"in": 12,
"out": 4647,
"cr": 328224,
"cw": 13131,
"total": 346014,
"usd": 0.1647,
"wall_s": 52.8,
"api_calls": 6
},
"implementation (no skill) [subagent]": {
"turns": [
2
],
"in": 0,
"out": 0,
"cr": 0,
"cw": 0,
"total": 0,
"usd": 0.0,
"wall_s": 0.0,
"api_calls": 0
}
}
},
{
"run": "sp-r2",
"task": "F1",
"cfg": "sp",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 212,
"code_diff_removed": 15,
"diff_added": 216,
"diff_removed": 15,
"tests_added_suite": 13,
"tokens": {
"in": 32,
"out": 18160,
"cr": 691441,
"cw": 55475,
"total": 765108
},
"usd": 0.5419,
"usd_po": 0.0305,
"wall_s": 119.6,
"driver_turns": 2,
"questions_routed_to_po": 1,
"end_reason": "done",
"caps": [],
"skills_invoked": [
"brainstorming",
"test-driven-development"
],
"superpowers_injected": true,
"commits": 0,
"models": [
"claude-sonnet-5-5"
],
"stages": {
"(none)": {
"turns": [
1
],
"in": 2,
"out": 227,
"cr": 13011,
"cw": 6298,
"total": 19538,
"usd": 0.0301,
"wall_s": 5.7,
"api_calls": 1
},
"brainstorming": {
"turns": [
1,
2
],
"in": 10,
"out": 2311,
"cr": 133731,
"cw": 14200,
"total": 150252,
"usd": 0.1067,
"wall_s": 20.4,
"api_calls": 5
},
"test-driven-development": {
"turns": [
2
],
"in": 20,
"out": 15622,
"cr": 544699,
"cw": 34977,
"total": 595318,
"usd": 0.4051,
"wall_s": 93.0,
"api_calls": 10
}
}
},
{
"run": "sp-r3",
"task": "F1",
"cfg": "sp",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 267,
"code_diff_removed": 14,
"diff_added": 267,
"diff_removed": 14,
"tests_added_suite": 14,
"tokens": {
"in": 34,
"out": 17926,
"cr": 727283,
"cw": 50017,
"total": 795260
},
"usd": 0.5249,
"usd_po": 0.0489,
"wall_s": 135.3,
"driver_turns": 3,
"questions_routed_to_po": 2,
"end_reason": "done",
"caps": [],
"skills_invoked": [
"brainstorming"
],
"superpowers_injected": true,
"commits": 1,
"models": [
"claude-sonnet-5-5"
],
"stages": {
"(none)": {
"turns": [
1
],
"in": 2,
"out": 278,
"cr": 13011,
"cw": 6298,
"total": 19589,
"usd": 0.0306,
"wall_s": 4.9,
"api_calls": 1
},
"brainstorming": {
"turns": [
1,
2
],
"in": 18,
"out": 12135,
"cr": 306378,
"cw": 31097,
"total": 349628,
"usd": 0.307,
"wall_s": 61.1,
"api_calls": 9
},
"implementation (no skill)": {
"turns": [
2,
3
],
"in": 14,
"out": 5513,
"cr": 407894,
"cw": 12622,
"total": 426043,
"usd": 0.1872,
"wall_s": 67.8,
"api_calls": 7
},
"implementation (no skill) [subagent]": {
"turns": [
2
],
"in": 0,
"out": 0,
"cr": 0,
"cw": 0,
"total": 0,
"usd": 0.0,
"wall_s": 0.0,
"api_calls": 0
}
}
},
{
"run": "spd-r1",
"task": "F1",
"cfg": "spd",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 350,
"code_diff_removed": 21,
"diff_added": 516,
"diff_removed": 21,
"tests_added_suite": 21,
"tokens": {
"in": 340,
"out": 59243,
"cr": 3428307,
"cw": 179396,
"total": 3667286
},
"usd": 3.4118,
"usd_po": 0.1084,
"wall_s": 600.8,
"driver_turns": 6,
"questions_routed_to_po": 3,
"end_reason": "done",
"caps": [],
"skills_invoked": [
"test-driven-development"
],
"superpowers_injected": true,
"commits": 4,
"models": [
"claude-fable-5-1",
"claude-sonnet-5-5"
],
"stages": {
"brainstorming": {
"turns": [
1,
2,
3
],
"in": 14,
"out": 5191,
"cr": 198912,
"cw": 25310,
"total": 229427,
"usd": 0.193,
"wall_s": 49.8,
"api_calls": 7
},
"brainstorming [subagent]": {
"turns": [
2
],
"in": 0,
"out": 0,
"cr": 0,
"cw": 0,
"total": 0,
"usd": 0.0,
"wall_s": 0.0,
"api_calls": 0
},
"writing-plans": {
"turns": [
4,
5
],
"in": 12,
"out": 13010,
"cr": 243336,
"cw": 23856,
"total": 280214,
"usd": 0.2742,
"wall_s": 95.1,
"api_calls": 5
},
"executing-plans": {
"turns": [
6
],
"in": 4,
"out": 614,
"cr": 62177,
"cw": 7788,
"total": 70583,
"usd": 0.0497,
"wall_s": 4.6,
"api_calls": 1
},
"test-driven-development": {
"turns": [
6
],
"in": 52,
"out": 22476,
"cr": 2455319,
"cw": 43404,
"total": 2521251,
"usd": 0.8895,
"wall_s": 448.9,
"api_calls": 26
},
"test-driven-development [subagent]": {
"turns": [
6
],
"in": 258,
"out": 17952,
"cr": 468563,
"cw": 79038,
"total": 565811,
"usd": 2.0053,
"wall_s": 0.0,
"api_calls": 0
}
}
},
{
"run": "spd-r2",
"task": "F1",
"cfg": "spd",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 317,
"code_diff_removed": 16,
"diff_added": 527,
"diff_removed": 16,
"tests_added_suite": 36,
"tokens": {
"in": 354,
"out": 53270,
"cr": 3666473,
"cw": 166918,
"total": 3887015
},
"usd": 3.3947,
"usd_po": 0.1118,
"wall_s": 520.0,
"driver_turns": 7,
"questions_routed_to_po": 4,
"end_reason": "done",
"caps": [],
"skills_invoked": [
"test-driven-development"
],
"superpowers_injected": true,
"commits": 5,
"models": [
"claude-fable-5-1",
"claude-sonnet-5-5"
],
"stages": {
"brainstorming": {
"turns": [
1,
2,
3
],
"in": 10,
"out": 3724,
"cr": 134220,
"cw": 22880,
"total": 160834,
"usd": 0.1556,
"wall_s": 33.6,
"api_calls": 5
},
"writing-plans": {
"turns": [
4,
5,
6
],
"in": 14,
"out": 8546,
"cr": 271872,
"cw": 17835,
"total": 298267,
"usd": 0.2112,
"wall_s": 63.9,
"api_calls": 6
},
"writing-plans [subagent]": {
"turns": [
6
],
"in": 0,
"out": 0,
"cr": 0,
"cw": 0,
"total": 0,
"usd": 0.0,
"wall_s": 0.0,
"api_calls": 0
},
"executing-plans": {
"turns": [
7
],
"in": 4,
"out": 385,
"cr": 53726,
"cw": 7377,
"total": 61492,
"usd": 0.0441,
"wall_s": 3.7,
"api_calls": 1
},
"test-driven-development": {
"turns": [
7
],
"in": 68,
"out": 21023,
"cr": 2791978,
"cw": 41898,
"total": 2854967,
"usd": 0.9364,
"wall_s": 415.1,
"api_calls": 34
},
"test-driven-development [subagent]": {
"turns": [
7
],
"in": 258,
"out": 19592,
"cr": 414677,
"cw": 76928,
"total": 511455,
"usd": 2.0474,
"wall_s": 0.0,
"api_calls": 0
}
}
},
{
"run": "spd-r3",
"task": "F1",
"cfg": "spd",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 434,
"code_diff_removed": 20,
"diff_added": 646,
"diff_removed": 20,
"tests_added_suite": 26,
"tokens": {
"in": 504,
"out": 72541,
"cr": 5615970,
"cw": 196313,
"total": 5885328
},
"usd": 4.4087,
"usd_po": 0.1286,
"wall_s": 701.3,
"driver_turns": 8,
"questions_routed_to_po": 5,
"end_reason": "done",
"caps": [],
"skills_invoked": [
"test-driven-development",
"finishing-a-development-branch"
],
"superpowers_injected": true,
"commits": 6,
"models": [
"claude-fable-5-1",
"claude-sonnet-5-5"
],
"stages": {
"brainstorming": {
"turns": [
1,
2,
3
],
"in": 10,
"out": 3567,
"cr": 130512,
"cw": 21715,
"total": 155804,
"usd": 0.1487,
"wall_s": 33.0,
"api_calls": 5
},
"writing-plans": {
"turns": [
4,
5,
6
],
"in": 16,
"out": 13925,
"cr": 345667,
"cw": 25975,
"total": 385583,
"usd": 0.3123,
"wall_s": 101.2,
"api_calls": 7
},
"writing-plans [subagent]": {
"turns": [
5,
6
],
"in": 0,
"out": 0,
"cr": 0,
"cw": 0,
"total": 0,
"usd": 0.0,
"wall_s": 0.0,
"api_calls": 0
},
"executing-plans": {
"turns": [
7
],
"in": 4,
"out": 516,
"cr": 60701,
"cw": 7518,
"total": 68739,
"usd": 0.0474,
"wall_s": 5.5,
"api_calls": 1
},
"test-driven-development": {
"turns": [
7
],
"in": 72,
"out": 24026,
"cr": 3334129,
"cw": 46850,
"total": 3405077,
"usd": 1.0946,
"wall_s": 507.1,
"api_calls": 36
},
"finishing-a-development-branch": {
"turns": [
7,
8
],
"in": 16,
"out": 5297,
"cr": 962018,
"cw": 9174,
"total": 976505,
"usd": 0.2821,
"wall_s": 50.7,
"api_calls": 8
},
"test-driven-development [subagent]": {
"turns": [
7
],
"in": 386,
"out": 25210,
"cr": 782943,
"cw": 85081,
"total": 893620,
"usd": 2.5236,
"wall_s": 0.0,
"api_calls": 0
},
"finishing-a-development-branch [subagent]": {
"turns": [
8
],
"in": 0,
"out": 0,
"cr": 0,
"cw": 0,
"total": 0,
"usd": 0.0,
"wall_s": 0.0,
"api_calls": 0
}
}
}
]
}
results.md
# flow-bench results
Untagged runs only (smoke/validation excluded). Tokens = input + output + cache read + cache write, all models, subagents included. USD = list-price equivalent from `total_cost_usd` deltas (plan quota, not a bill). Wall = sum of worker turn walls (PO time excluded). Diff = lines added+removed vs the base commit, src/ + test/.
## Per config (mean / median)
| task / config | n | hidden /5 (per run) | suite green | tsc green | code diff +/- | all files diff | tests added | tokens | USD | wall s | driver turns | questions | caps hit |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| F1 / bare | 3 | 5 / 5 (5,5,5) | 3/3 | 3/3 | 141.667 / 137 | 142.333 | 4.667 | 519,114 / 361,842 | 0.343 / 0.286 | 84.5 / 71.7 | 1.333 | 0.333 | none |
| F1 / sp (hook-routed) | 3 | 5 / 5 (5,5,5) | 3/3 | 3/3 | 235 / 227 | 237 | 12.333 | 724,601 / 765,108 | 0.502 / 0.525 | 118.1 / 119.6 | 2.667 | 1.667 | none |
| F1 / spd (Superpowers driven) | 3 | 5 / 5 (5,5,5) | 3/3 | 3/3 | 386 / 371 | 582 | 27.667 | 4,479,876 / 3,887,015 | 3.738 / 3.412 | 607.367 / 600.8 | 7 | 4 | none |
| F1 / mp (Main Flow) | 3 | 5 / 5 (5,5,5) | 3/3 | 3/3 | 372 / 369 | 578 | 27.667 | 4,476,104 / 4,286,993 | 2.043 / 2.102 | 439.033 / 472.4 | 12 | 7 | mp-r1: driver_turns_12; mp-r3: driver_turns_12 |
| F1 / both (sp+mp, mp flow) | 1 | 5 / 5 (5) | 1/1 | 1/1 | 374 / 374 | 561 | 22 | 5,754,354 / 5,754,354 | 2.326 / 2.326 | 448.2 / 448.2 | 12 | 6 | none |
| F2 / bare | 2 | 4.5 / 4.5 (4,5) | 2/2 | 2/2 | 206.5 / 206.5 | 209.5 | 5 | 651,322 / 651,322 | 0.413 / 0.413 | 103.15 / 103.15 | 2 | 0.5 | none |
| F2 / spd (Superpowers driven) | 2 | 5 / 5.0 (5,5) | 2/2 | 2/2 | 488.5 / 488.5 | 750.5 | 33 | 4,490,365 / 4,490,365 | 3.213 / 3.213 | 511.7 / 511.7 | 6.5 | 3.5 | none |
| F2 / mp (Main Flow) | 2 | 5 / 5.0 (5,5) | 2/2 | 2/2 | 469 / 469.0 | 658 | 31 | 4,520,638 / 4,520,638 | 2.067 / 2.067 | 421.2 / 421.2 | 12 | 6.5 | f2-mp-r1: driver_turns_12 |
Requirement pass counts (runs passing each hidden req):
- **F1 bare**: req1 3/3, req2 3/3, req3 3/3, req4 3/3, req5 3/3
- **F1 sp (hook-routed)**: req1 3/3, req2 3/3, req3 3/3, req4 3/3, req5 3/3
- **F1 spd (Superpowers driven)**: req1 3/3, req2 3/3, req3 3/3, req4 3/3, req5 3/3
- **F1 mp (Main Flow)**: req1 3/3, req2 3/3, req3 3/3, req4 3/3, req5 3/3
- **F1 both (sp+mp, mp flow)**: req1 1/1, req2 1/1, req3 1/1, req4 1/1, req5 1/1
- **F2 bare**: req1 2/2, req2 2/2, req3 1/2, req4 2/2, req5 2/2
- **F2 spd (Superpowers driven)**: req1 2/2, req2 2/2, req3 2/2, req4 2/2, req5 2/2
- **F2 mp (Main Flow)**: req1 2/2, req2 2/2, req3 2/2, req4 2/2, req5 2/2
## F1 spd (Superpowers driven) per slash-command stage (turn level, exact) (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| /brainstorming | 3 | 182,022 | 160,834 | - | 0.166 | 0.156 | 40.283 | 35.44 |
| /writing-plans | 3 | 321,355 | 298,267 | - | 0.266 | 0.274 | 88.133 | 95.82 |
| /executing-plans | 3 | 3,976,500 | 3,427,914 | - | 3.307 | 3.028 | 478.95 | 453.89 |
## F1 spd per skill in force (within-turn split) (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| brainstorming | 3 | 182,022 | 160,834 | 4,161 | 0.166 | 0.156 | 38.8 | 33.6 |
| writing-plans | 3 | 321,355 | 298,267 | 11,827 | 0.266 | 0.274 | 86.733 | 95.1 |
| executing-plans | 3 | 66,938 | 68,739 | 505 | 0.047 | 0.047 | 4.6 | 4.6 |
| test-driven-development | 3 | 2,927,098 | 2,854,967 | 22,508 | 0.974 | 0.936 | 457.033 | 448.9 |
| finishing-a-development-branch | 1 | 976,505 | 976,505 | 5,297 | 0.282 | 0.282 | 50.7 | 50.7 |
| test-driven-development [subagent] | 3 | 656,962 | 565,811 | 20,918 | 2.192 | 2.047 | 0.0 | 0.0 |
## F1 sp (hook-routed) per phase (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| before any skill | 3 | 19,549 | 19,538 | 238 | 0.03 | 0.03 | 5.333 | 5.4 |
| brainstorming | 3 | 249,260 | 247,900 | 8,037 | 0.219 | 0.244 | 40.533 | 40.1 |
| implementation (no skill) | 2 | 386,028 | 386,028 | 5,080 | 0.176 | 0.176 | 60.3 | 60.3 |
| execution | 1 | 595,318 | 595,318 | 15,622 | 0.405 | 0.405 | 93.0 | 93.0 |
## F1 sp (hook-routed) per skill in force (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| (none) | 3 | 19,549 | 19,538 | 238 | 0.03 | 0.03 | 5.333 | 5.4 |
| brainstorming | 3 | 249,260 | 247,900 | 8,037 | 0.219 | 0.244 | 40.533 | 40.1 |
| test-driven-development | 1 | 595,318 | 595,318 | 15,622 | 0.405 | 0.405 | 93.0 | 93.0 |
| implementation (no skill) | 2 | 386,028 | 386,028 | 5,080 | 0.176 | 0.176 | 60.3 | 60.3 |
## F1 mp per slash-command stage (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| /grill-with-docs | 3 | 316,507 | 348,862 | 5,999 | 0.204 | 0.215 | 59.933 | 59.9 |
| /to-spec | 3 | 163,344 | 167,578 | 5,135 | 0.118 | 0.113 | 42.1 | 40.8 |
| /to-tickets | 3 | 257,955 | 268,062 | 5,867 | 0.143 | 0.151 | 44.4 | 47.8 |
| /implement | 3 | 2,655,565 | 2,613,062 | 30,215 | 1.125 | 1.195 | 224.567 | 245.4 |
| /code-review | 3 | 1,047,586 | 1,107,555 | 9,294 | 0.439 | 0.445 | 64.2 | 65.8 |
| /pr | 1 | 105,440 | 105,440 | 1,349 | 0.041 | 0.041 | 11.6 | 11.6 |
## F1 both per slash-command stage (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| /grill-with-docs | 1 | 344,391 | 344,391 | 6,193 | 0.225 | 0.225 | 61.8 | 61.8 |
| /to-spec | 1 | 217,572 | 217,572 | 4,835 | 0.123 | 0.123 | 42.9 | 42.9 |
| /to-tickets | 1 | 382,953 | 382,953 | 7,115 | 0.196 | 0.196 | 49.4 | 49.4 |
| /implement | 1 | 3,070,579 | 3,070,579 | 24,618 | 1.112 | 1.112 | 194.2 | 194.2 |
| /code-review | 1 | 1,615,265 | 1,615,265 | 13,410 | 0.625 | 0.625 | 86.8 | 86.8 |
| /pr | 1 | 123,594 | 123,594 | 1,461 | 0.045 | 0.045 | 13.1 | 13.1 |
## F2 spd (Superpowers driven) per slash-command stage (turn level, exact) (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| /brainstorming | 2 | 485,364 | 485,364 | - | 0.354 | 0.354 | 96.815 | 96.815 |
| /writing-plans | 2 | 166,012 | 166,012 | - | 0.159 | 0.159 | 49.755 | 49.755 |
| /executing-plans | 2 | 3,838,988 | 3,838,988 | - | 2.7 | 2.7 | 365.1 | 365.1 |
## F2 spd per skill in force (within-turn split) (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| brainstorming | 2 | 253,890 | 253,890 | 6,038 | 0.207 | 0.207 | 53.05 | 53.05 |
| writing-plans | 2 | 397,486 | 397,486 | 14,060 | 0.307 | 0.307 | 90.75 | 90.75 |
| executing-plans | 2 | 69,973 | 69,973 | 286 | 0.046 | 0.046 | 3.5 | 3.5 |
| test-driven-development | 2 | 3,304,676 | 3,304,676 | 23,244 | 1.052 | 1.052 | 361.25 | 361.25 |
| test-driven-development [subagent] | 2 | 464,339 | 464,339 | 13,017 | 1.602 | 1.602 | 0.0 | 0.0 |
## F2 mp per slash-command stage (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| /grill-with-docs | 2 | 232,965 | 232,965 | 5,946 | 0.182 | 0.182 | 56.7 | 56.7 |
| /to-spec | 2 | 180,084 | 180,084 | 4,452 | 0.115 | 0.115 | 37.6 | 37.6 |
| /to-tickets | 2 | 253,688 | 253,688 | 5,152 | 0.131 | 0.131 | 38.5 | 38.5 |
| /implement | 2 | 2,689,071 | 2,689,071 | 31,452 | 1.159 | 1.159 | 218.35 | 218.35 |
| /code-review | 2 | 1,107,122 | 1,107,122 | 9,454 | 0.458 | 0.458 | 63.85 | 63.85 |
| /pr | 1 | 115,414 | 115,414 | 1,372 | 0.043 | 0.043 | 12.4 | 12.4 |
## Per run
| run | task | hidden | reqs failed | suite | tsc | code +/- | tests added | tokens | USD | wall s | turns | end | skills (first use order) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bare-r1 | F1 | 5/5 | - | green | ok | +149/-11 | 6 | 840,751 | 0.47 | 119 | 2 | done | - |
| bare-r2 | F1 | 5/5 | - | green | ok | +118/-10 | 4 | 354,748 | 0.27 | 63 | 1 | done | - |
| bare-r3 | F1 | 5/5 | - | green | ok | +125/-12 | 4 | 361,842 | 0.29 | 72 | 1 | done | - |
| sp-r1 | F1 | 5/5 | - | green | ok | +182/-15 | 10 | 613,434 | 0.44 | 99 | 3 | done | brainstorming |
| sp-r2 | F1 | 5/5 | - | green | ok | +212/-15 | 13 | 765,108 | 0.54 | 120 | 2 | done | brainstorming > test-driven-development |
| sp-r3 | F1 | 5/5 | - | green | ok | +267/-14 | 14 | 795,260 | 0.52 | 135 | 3 | done | brainstorming |
| spd-r1 | F1 | 5/5 | - | green | ok | +350/-21 | 21 | 3,667,286 | 3.41 | 601 | 6 | done | test-driven-development |
| spd-r2 | F1 | 5/5 | - | green | ok | +317/-16 | 36 | 3,887,015 | 3.39 | 520 | 7 | done | test-driven-development |
| spd-r3 | F1 | 5/5 | - | green | ok | +434/-20 | 26 | 5,885,328 | 4.41 | 701 | 8 | done | test-driven-development > finishing-a-development-branch |
| mp-r1 | F1 | 5/5 | - | green | ok | +349/-20 | 29 | 5,098,503 | 2.22 | 472 | 12 | cap_driver_turns (driver_turns_12) | grilling > domain-modeling > tdd > code-review |
| mp-r2 | F1 | 5/5 | - | green | ok | +314/-15 | 25 | 4,042,815 | 1.80 | 365 | 12 | done | grilling > domain-modeling > tdd > code-review |
| mp-r3 | F1 | 5/5 | - | green | ok | +395/-23 | 29 | 4,286,993 | 2.10 | 480 | 12 | cap_driver_turns (driver_turns_12) | grilling > domain-modeling > tdd > code-review |
| both-r1 | F1 | 5/5 | - | green | ok | +356/-18 | 22 | 5,754,354 | 2.33 | 448 | 12 | done | grilling > domain-modeling > tdd > code-review |
| f2-bare-r1 | F2 | 4/5 | req3 | green | ok | +204/-10 | 5 | 494,231 | 0.38 | 92 | 1 | done | - |
| f2-bare-r2 | F2 | 5/5 | - | green | ok | +189/-10 | 5 | 808,413 | 0.45 | 115 | 3 | done | - |
| f2-spd-r1 | F2 | 5/5 | - | green | ok | +546/-14 | 46 | 5,310,616 | 3.42 | 506 | 6 | done | writing-plans > test-driven-development |
| f2-spd-r2 | F2 | 5/5 | - | green | ok | +401/-16 | 20 | 3,670,114 | 3.00 | 517 | 7 | done | test-driven-development |
| f2-mp-r1 | F2 | 5/5 | - | green | ok | +460/-23 | 31 | 4,823,516 | 2.18 | 443 | 12 | cap_driver_turns (driver_turns_12) | grilling > domain-modeling > tdd > code-review |
| f2-mp-r2 | F2 | 5/5 | - | green | ok | +440/-15 | 31 | 4,217,761 | 1.95 | 399 | 12 | done | grilling > domain-modeling > tdd > code-review |
matrix.sh
#!/usr/bin/env bash
# Run matrix.txt ("<cfg> <rep>" lines), MATRIX_JOBS (default 2) at a time, each run in its own work copy.
# Appends to matrix.log. Resumable: SKIP_DONE=1. Usage: nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 &
set -uo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"; cd "$HERE"
LIST="${1:-matrix.txt}"; JOBS="${MATRIX_JOBS:-2}"
mkdir "$HERE/.matrix.lock" 2>/dev/null || { echo "another matrix.sh is running (remove .matrix.lock if stale)" >&2; exit 1; }
trap 'rmdir "$HERE/.matrix.lock"' EXIT
echo "=== $(date '+%F %T') start $LIST jobs=$JOBS pid=$$" >> matrix.log
grep -vE '^[[:space:]]*(#|$)' "$LIST" | xargs -P "$JOBS" -L 1 ./matrix-one.sh
echo "=== $(date '+%F %T') done $LIST" >> matrix.log
matrix-one.sh
#!/usr/bin/env bash
# One matrix cell (called by matrix.sh through xargs). Args: <cfg> <rep>
HERE="$(cd "$(dirname "$0")" && pwd)"; cd "$HERE"
CFG="$1"; REP="$2"; export TASK="${3:-F1}"
PFX=""; [ "$TASK" != "F1" ] && PFX="$(echo "$TASK" | tr "A-Z" "a-z")-"
if [ "${SKIP_DONE:-0}" = 1 ] && [ -f "runs/${PFX}${CFG}-r${REP}/score.json" ]; then
echo "$(date '+%F %T') skip ${PFX}${CFG}-r${REP}" >> matrix.log; exit 0
fi
echo "$(date '+%F %T') start ${PFX}${CFG}-r${REP}" >> matrix.log
line=$(./run.sh "$CFG" "$REP" < /dev/null 2>&1 | tail -1)
echo "$(date '+%F %T') $line" >> matrix.log
echo "$line"
matrix.txt
# <cfg> <rep>, run 2 at a time in this order (rep 1 of every config first)
bare 1
sp 1
mp 1
both 1
sp 2
mp 2
bare 2
mp 3
sp 3
bare 3
matrix2.txt
# <cfg> <rep> <task>; 2 lanes; long runs interleaved with short ones
spd 1 F1
mp 1 F1
bare 1 F1
sp 1 F1
spd 1 F2
mp 1 F2
bare 1 F2
spd 2 F1
mp 2 F1
bare 2 F1
sp 2 F1
spd 2 F2
mp 2 F2
bare 2 F2
both 1 F1
spd 3 F1
mp 3 F1
bare 3 F1
sp 3 F1
matrix-mp2.txt
mp 1
both 1
mp 2
mp 3
validate.sh
#!/usr/bin/env bash
# The 5 validation steps of SPEC.md, appended to validation.log.
# Usage: ./validate.sh [steps] default "1 2 3 4 5"; steps 3-5 (smoke runs, RUN_TAG=smoke) run in parallel.
set -uo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"; cd "$HERE"
. "$HERE/env.sh"
STEPS="${*:-1 2 3 4 5}"
LOG="$HERE/validation.log"
log(){ echo "$(date '+%F %T') $*" | tee -a "$LOG"; }
static_run(){ # <name> <patch or ""> [task] : prep a bare copy, optionally apply a patch, score it with finalize.py
local name="$1" patch="$2" task="${3:-F1}" out="$HERE/runs/$1" w="$HERE/work/$1"
rm -rf "$out"; mkdir -p "$out"
local base; base=$("$HERE/prep.sh" bare "$w" | tail -1)
if [ -n "$patch" ]; then ( cd "$w" && git apply "$patch" ) || { log "FAIL $name: git apply"; return 1; }; fi
printf '{"task":"%s","cfg":"bare","rep":0,"tag":"validation","work":"%s","name":"%s","base":"%s"}\n' "$task" "$w" "$name" "$base" > "$out/meta.json"
: > "$out/turns.jsonl"; echo '{"end_reason":"static"}' > "$out/driver.json"
python3 "$HERE/lib/finalize.py" "$out"
}
check(){ # <run dir> <python expr over s (score.json)> <label>
python3 - "$1" "$2" "$3" <<'PY'
import json, sys
s = json.load(open(sys.argv[1] + "/score.json"))
ok = eval(sys.argv[2], {"s": s})
print(("PASS " if ok else "FAIL ") + sys.argv[3])
sys.exit(0 if ok else 1)
PY
}
log "=== validate.sh steps: $STEPS (claude $(claude --version | head -1))"
for st in $STEPS; do case $st in
1) log "step 1: reference/F1.patch -> hidden 5/5, suite green, typecheck green"
static_run val1-reference "$HERE/reference/F1.patch" | tee -a "$LOG"
check runs/val1-reference 's["hidden_pass"]==5 and s["suite_green"] and s["typecheck_green"]' "step 1" | tee -a "$LOG" ;;
2) log "step 2: upstream unchanged -> hidden 0/5 (honest number recorded), suite green"
static_run val2-upstream "" | tee -a "$LOG"
check runs/val2-upstream 's["hidden_pass"]==0 and s["suite_green"] and s["suite_total"]==34' "step 2" | tee -a "$LOG" ;;
f2-1) log "F2 step 1: reference/F2.patch -> hidden 5/5, suite green, typecheck green"
static_run val1-f2-reference "$HERE/reference/F2.patch" F2 | tee -a "$LOG"
check runs/val1-f2-reference 's["hidden_pass"]==5 and s["suite_green"] and s["typecheck_green"]' "F2 step 1" | tee -a "$LOG" ;;
f2-2) log "F2 step 2: upstream unchanged -> hidden 0/5, suite green"
static_run val2-f2-upstream "" F2 | tee -a "$LOG"
check runs/val2-f2-upstream 's["hidden_pass"]==0 and s["suite_green"]' "F2 step 2" | tee -a "$LOG" ;;
esac; done
pids=""
for st in $STEPS; do case $st in
3) log "step 3: bare smoke (RUN_TAG=smoke) started"; ( RUN_TAG=smoke ./run.sh bare 1 > runs/.smoke-bare.out 2>&1 ) & pids="$pids $!" ;;
4) log "step 4: sp smoke (RUN_TAG=smoke) started"; ( RUN_TAG=smoke ./run.sh sp 1 > runs/.smoke-sp.out 2>&1 ) & pids="$pids $!" ;;
5) log "step 5: mp smoke (RUN_TAG=smoke) started"; ( RUN_TAG=smoke ./run.sh mp 1 > runs/.smoke-mp.out 2>&1 ) & pids="$pids $!" ;;
f2-3) log "F2 step 3: bare smoke on F2 (RUN_TAG=smoke): bare must score < 5"; ( TASK=F2 RUN_TAG=smoke ./run.sh bare 1 > runs/.smoke-f2-bare.out 2>&1 ) & pids="$pids $!" ;;
spd) log "spd smoke on F1 (RUN_TAG=smoke)"; ( RUN_TAG=smoke ./run.sh spd 1 > runs/.smoke-spd.out 2>&1 ) & pids="$pids $!" ;;
esac; done
[ -n "$pids" ] && wait $pids
for st in $STEPS; do case $st in
3) tail -1 runs/.smoke-bare.out | tee -a "$LOG"
check runs/bare-r1-smoke 's["end_reason"] in ("done","cap_driver_turns") and len(s["per_turn"])>=1 and all(t["po_kind"] for t in s["per_turn"])' "step 3a: bare ran end to end, the PO classified every turn" | tee -a "$LOG"
check runs/bare-r1-smoke 'not s["superpowers_injected"] and not s["skill_events"] and "superpowers" not in (s["init_plugins"] or [])' "step 3b: isolation: no Superpowers hook output, no Skill events, user plugin not loaded" | tee -a "$LOG" ;;
4) tail -1 runs/.smoke-sp.out | tee -a "$LOG"
check runs/sp-r1-smoke 's["superpowers_injected"] and any(h["event"]=="SessionStart" for h in s["hook_events"])' "step 4a: SessionStart hook fired and injected using-superpowers" | tee -a "$LOG"
check runs/sp-r1-smoke '"brainstorming" in [x.split(":")[-1] for x in s["skills_invoked"]]' "step 4b: brainstorming invoked (Skill event)" | tee -a "$LOG" ;;
5) tail -1 runs/.smoke-mp.out | tee -a "$LOG"
check runs/mp-r1-smoke 's["per_turn"][0]["stage"]=="/grill-with-docs" and {"grilling","domain-modeling"} <= set(s["per_turn"][0]["skills"])' "step 5: /grill-with-docs resolved (turn 1 loads grilling + domain-modeling, as its SKILL.md instructs)" | tee -a "$LOG" ;;
f2-3) tail -1 runs/.smoke-f2-bare.out | tee -a "$LOG"
check runs/f2-bare-r1-smoke 's["hidden_pass"] < 5' "F2 step 3: bare scores < 5 on F2" | tee -a "$LOG" ;;
spd) tail -1 runs/.smoke-spd.out | tee -a "$LOG"
check runs/spd-r1-smoke '{"/brainstorming","/writing-plans","/executing-plans"} <= set(s["spd_slash_stage_per_turn"] or [])' "spd: the three slash stages were all reached" | tee -a "$LOG" ;;
esac; done
log "=== validate.sh done"
İki paketi kendi özelliğinizde çalıştırmak için README'yi izleyin:
- Repo'nuzu
upstream/yerine koyun, isteğinizitasks/altına yazın ve product owner'ın brief'inipo/altına düz kelimelerle yazın, asla test olarak değil. - Bir senior'ın bekleyeceği her gereksinim için bir test içeren bir gizli test dosyası ve bir referans patch yazın, sonra oracle'ı
./validate.sh 1 2ile kontrol edin. - Paketleri
prep.sh'in gösterdiği gibi kurun: Superpowers'ı.claude/skillsiçine, SessionStart hook'unu.claude/settings.jsoniçine kopyalayarak; Pocock'un paketininpx skills addile, sonra/setup-matt-pocock-skills'i bir kez materialize edin. - Bir hücreyi
RUN_TAG=x ./run.sh <cfg> <rep>ile çalıştırın. Etiketli çalıştırmalar analizde yok sayılır, bu yüzden etiketi smoke çalıştırmaları için kullanın. - Matrix'i
nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 &ile çalıştırın,SKIP_DONE=1ile devam edin, sonra konfigürasyon ve aşama tablolarını almak içinpython3 analyze.pyçalıştırın.
README'deki üç tuzağı başlamadan önce bilmeye değer. Work copy'yi CLAUDE.md olan hiçbir dizin ağacının dışında çalıştırın, çünkü Claude Code her üst dizinin CLAUDE.md'sini yüklüyor ve ilk turumuz bu yüzden çöpe gitti. Abonelik girişinde asla --bare kullanmayın. Ve Superpowers hook'u yalnızca ilk turda tetikleniyor, yani headless bir çalıştırma bir kez yönlendiriyor; yönlendirmenin kendisi çalıştırmadan çalıştırmaya değişti, bu yüzden hangi skill'i seçtiğini sayın.
Pazartesi bunu yapın
- Hangi Pocock sürümüne sahip olduğunuzu kontrol edin. Claude plugin'ini kurduysanız 1.2.3'e sabitli ve implement-spec, pr ya da retro yok; v1.3.1 istiyorsanız installer ile yeniden kurun.
- Superpowers kullanıyorsanız executing-plans reviewer talimatını açın ve istediği modele bakın. Daha ucuz birini sabitleyin, sonra ölçün.
- İstek bulanık olduğunda /brainstorming'i kendiniz yazın. Burada hook'la yönlendirilen 3 çalıştırmanın 3'ünde brainstorming kendiliğinden çalıştı, ama yönlendirme garanti değil.
- Bir sonraki bulanık isteğinizde grill-with-docs, to-spec ve to-tickets'ı deneyin ve kendi repo'nuzda neye mal olduklarını okuyun.
- İki paketi test etmeden router olarak üst üste kurmayın. Tek üst üste çalıştırmamızda hiçbir şey bozulmadı, ama bu tek çalıştırma.
- Bench'i geçen ay gönderdiğiniz bir özelliğe kopyalayın, kendi gizli testlerinizle, ve bare, Superpowers ve Main Flow'u birer kez çalıştırın.
- Toplamı değil, aşama başına tabloyu okuyun. Başka bir şeyi değiştirmeden önce faturanızın çoğunu oluşturan tek aşamayı bulun.
- v1.3'ü benimserseniz CONTEXT.md'yi GLOSSARY.md olarak yeniden adlandırın, çünkü skill'ler yalnızca yeni adlara bakıyor.
Ölçümler
Kaynak:
results.md
# flow-bench results
Untagged runs only (smoke/validation excluded). Tokens = input + output + cache read + cache write, all models, subagents included. USD = list-price equivalent from `total_cost_usd` deltas (plan quota, not a bill). Wall = sum of worker turn walls (PO time excluded). Diff = lines added+removed vs the base commit, src/ + test/.
## Per config (mean / median)
| task / config | n | hidden /5 (per run) | suite green | tsc green | code diff +/- | all files diff | tests added | tokens | USD | wall s | driver turns | questions | caps hit |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| F1 / bare | 3 | 5 / 5 (5,5,5) | 3/3 | 3/3 | 141.667 / 137 | 142.333 | 4.667 | 519,114 / 361,842 | 0.343 / 0.286 | 84.5 / 71.7 | 1.333 | 0.333 | none |
| F1 / sp (hook-routed) | 3 | 5 / 5 (5,5,5) | 3/3 | 3/3 | 235 / 227 | 237 | 12.333 | 724,601 / 765,108 | 0.502 / 0.525 | 118.1 / 119.6 | 2.667 | 1.667 | none |
| F1 / spd (Superpowers driven) | 3 | 5 / 5 (5,5,5) | 3/3 | 3/3 | 386 / 371 | 582 | 27.667 | 4,479,876 / 3,887,015 | 3.738 / 3.412 | 607.367 / 600.8 | 7 | 4 | none |
| F1 / mp (Main Flow) | 3 | 5 / 5 (5,5,5) | 3/3 | 3/3 | 372 / 369 | 578 | 27.667 | 4,476,104 / 4,286,993 | 2.043 / 2.102 | 439.033 / 472.4 | 12 | 7 | mp-r1: driver_turns_12; mp-r3: driver_turns_12 |
| F1 / both (sp+mp, mp flow) | 1 | 5 / 5 (5) | 1/1 | 1/1 | 374 / 374 | 561 | 22 | 5,754,354 / 5,754,354 | 2.326 / 2.326 | 448.2 / 448.2 | 12 | 6 | none |
| F2 / bare | 2 | 4.5 / 4.5 (4,5) | 2/2 | 2/2 | 206.5 / 206.5 | 209.5 | 5 | 651,322 / 651,322 | 0.413 / 0.413 | 103.15 / 103.15 | 2 | 0.5 | none |
| F2 / spd (Superpowers driven) | 2 | 5 / 5.0 (5,5) | 2/2 | 2/2 | 488.5 / 488.5 | 750.5 | 33 | 4,490,365 / 4,490,365 | 3.213 / 3.213 | 511.7 / 511.7 | 6.5 | 3.5 | none |
| F2 / mp (Main Flow) | 2 | 5 / 5.0 (5,5) | 2/2 | 2/2 | 469 / 469.0 | 658 | 31 | 4,520,638 / 4,520,638 | 2.067 / 2.067 | 421.2 / 421.2 | 12 | 6.5 | f2-mp-r1: driver_turns_12 |
Requirement pass counts (runs passing each hidden req):
- **F1 bare**: req1 3/3, req2 3/3, req3 3/3, req4 3/3, req5 3/3
- **F1 sp (hook-routed)**: req1 3/3, req2 3/3, req3 3/3, req4 3/3, req5 3/3
- **F1 spd (Superpowers driven)**: req1 3/3, req2 3/3, req3 3/3, req4 3/3, req5 3/3
- **F1 mp (Main Flow)**: req1 3/3, req2 3/3, req3 3/3, req4 3/3, req5 3/3
- **F1 both (sp+mp, mp flow)**: req1 1/1, req2 1/1, req3 1/1, req4 1/1, req5 1/1
- **F2 bare**: req1 2/2, req2 2/2, req3 1/2, req4 2/2, req5 2/2
- **F2 spd (Superpowers driven)**: req1 2/2, req2 2/2, req3 2/2, req4 2/2, req5 2/2
- **F2 mp (Main Flow)**: req1 2/2, req2 2/2, req3 2/2, req4 2/2, req5 2/2
## F1 spd (Superpowers driven) per slash-command stage (turn level, exact) (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| /brainstorming | 3 | 182,022 | 160,834 | - | 0.166 | 0.156 | 40.283 | 35.44 |
| /writing-plans | 3 | 321,355 | 298,267 | - | 0.266 | 0.274 | 88.133 | 95.82 |
| /executing-plans | 3 | 3,976,500 | 3,427,914 | - | 3.307 | 3.028 | 478.95 | 453.89 |
## F1 spd per skill in force (within-turn split) (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| brainstorming | 3 | 182,022 | 160,834 | 4,161 | 0.166 | 0.156 | 38.8 | 33.6 |
| writing-plans | 3 | 321,355 | 298,267 | 11,827 | 0.266 | 0.274 | 86.733 | 95.1 |
| executing-plans | 3 | 66,938 | 68,739 | 505 | 0.047 | 0.047 | 4.6 | 4.6 |
| test-driven-development | 3 | 2,927,098 | 2,854,967 | 22,508 | 0.974 | 0.936 | 457.033 | 448.9 |
| finishing-a-development-branch | 1 | 976,505 | 976,505 | 5,297 | 0.282 | 0.282 | 50.7 | 50.7 |
| test-driven-development [subagent] | 3 | 656,962 | 565,811 | 20,918 | 2.192 | 2.047 | 0.0 | 0.0 |
## F1 sp (hook-routed) per phase (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| before any skill | 3 | 19,549 | 19,538 | 238 | 0.03 | 0.03 | 5.333 | 5.4 |
| brainstorming | 3 | 249,260 | 247,900 | 8,037 | 0.219 | 0.244 | 40.533 | 40.1 |
| implementation (no skill) | 2 | 386,028 | 386,028 | 5,080 | 0.176 | 0.176 | 60.3 | 60.3 |
| execution | 1 | 595,318 | 595,318 | 15,622 | 0.405 | 0.405 | 93.0 | 93.0 |
## F1 sp (hook-routed) per skill in force (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| (none) | 3 | 19,549 | 19,538 | 238 | 0.03 | 0.03 | 5.333 | 5.4 |
| brainstorming | 3 | 249,260 | 247,900 | 8,037 | 0.219 | 0.244 | 40.533 | 40.1 |
| test-driven-development | 1 | 595,318 | 595,318 | 15,622 | 0.405 | 0.405 | 93.0 | 93.0 |
| implementation (no skill) | 2 | 386,028 | 386,028 | 5,080 | 0.176 | 0.176 | 60.3 | 60.3 |
## F1 mp per slash-command stage (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| /grill-with-docs | 3 | 316,507 | 348,862 | 5,999 | 0.204 | 0.215 | 59.933 | 59.9 |
| /to-spec | 3 | 163,344 | 167,578 | 5,135 | 0.118 | 0.113 | 42.1 | 40.8 |
| /to-tickets | 3 | 257,955 | 268,062 | 5,867 | 0.143 | 0.151 | 44.4 | 47.8 |
| /implement | 3 | 2,655,565 | 2,613,062 | 30,215 | 1.125 | 1.195 | 224.567 | 245.4 |
| /code-review | 3 | 1,047,586 | 1,107,555 | 9,294 | 0.439 | 0.445 | 64.2 | 65.8 |
| /pr | 1 | 105,440 | 105,440 | 1,349 | 0.041 | 0.041 | 11.6 | 11.6 |
## F1 both per slash-command stage (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| /grill-with-docs | 1 | 344,391 | 344,391 | 6,193 | 0.225 | 0.225 | 61.8 | 61.8 |
| /to-spec | 1 | 217,572 | 217,572 | 4,835 | 0.123 | 0.123 | 42.9 | 42.9 |
| /to-tickets | 1 | 382,953 | 382,953 | 7,115 | 0.196 | 0.196 | 49.4 | 49.4 |
| /implement | 1 | 3,070,579 | 3,070,579 | 24,618 | 1.112 | 1.112 | 194.2 | 194.2 |
| /code-review | 1 | 1,615,265 | 1,615,265 | 13,410 | 0.625 | 0.625 | 86.8 | 86.8 |
| /pr | 1 | 123,594 | 123,594 | 1,461 | 0.045 | 0.045 | 13.1 | 13.1 |
## F2 spd (Superpowers driven) per slash-command stage (turn level, exact) (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| /brainstorming | 2 | 485,364 | 485,364 | - | 0.354 | 0.354 | 96.815 | 96.815 |
| /writing-plans | 2 | 166,012 | 166,012 | - | 0.159 | 0.159 | 49.755 | 49.755 |
| /executing-plans | 2 | 3,838,988 | 3,838,988 | - | 2.7 | 2.7 | 365.1 | 365.1 |
## F2 spd per skill in force (within-turn split) (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| brainstorming | 2 | 253,890 | 253,890 | 6,038 | 0.207 | 0.207 | 53.05 | 53.05 |
| writing-plans | 2 | 397,486 | 397,486 | 14,060 | 0.307 | 0.307 | 90.75 | 90.75 |
| executing-plans | 2 | 69,973 | 69,973 | 286 | 0.046 | 0.046 | 3.5 | 3.5 |
| test-driven-development | 2 | 3,304,676 | 3,304,676 | 23,244 | 1.052 | 1.052 | 361.25 | 361.25 |
| test-driven-development [subagent] | 2 | 464,339 | 464,339 | 13,017 | 1.602 | 1.602 | 0.0 | 0.0 |
## F2 mp per slash-command stage (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| /grill-with-docs | 2 | 232,965 | 232,965 | 5,946 | 0.182 | 0.182 | 56.7 | 56.7 |
| /to-spec | 2 | 180,084 | 180,084 | 4,452 | 0.115 | 0.115 | 37.6 | 37.6 |
| /to-tickets | 2 | 253,688 | 253,688 | 5,152 | 0.131 | 0.131 | 38.5 | 38.5 |
| /implement | 2 | 2,689,071 | 2,689,071 | 31,452 | 1.159 | 1.159 | 218.35 | 218.35 |
| /code-review | 2 | 1,107,122 | 1,107,122 | 9,454 | 0.458 | 0.458 | 63.85 | 63.85 |
| /pr | 1 | 115,414 | 115,414 | 1,372 | 0.043 | 0.043 | 12.4 | 12.4 |
## Per run
| run | task | hidden | reqs failed | suite | tsc | code +/- | tests added | tokens | USD | wall s | turns | end | skills (first use order) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bare-r1 | F1 | 5/5 | - | green | ok | +149/-11 | 6 | 840,751 | 0.47 | 119 | 2 | done | - |
| bare-r2 | F1 | 5/5 | - | green | ok | +118/-10 | 4 | 354,748 | 0.27 | 63 | 1 | done | - |
| bare-r3 | F1 | 5/5 | - | green | ok | +125/-12 | 4 | 361,842 | 0.29 | 72 | 1 | done | - |
| sp-r1 | F1 | 5/5 | - | green | ok | +182/-15 | 10 | 613,434 | 0.44 | 99 | 3 | done | brainstorming |
| sp-r2 | F1 | 5/5 | - | green | ok | +212/-15 | 13 | 765,108 | 0.54 | 120 | 2 | done | brainstorming > test-driven-development |
| sp-r3 | F1 | 5/5 | - | green | ok | +267/-14 | 14 | 795,260 | 0.52 | 135 | 3 | done | brainstorming |
| spd-r1 | F1 | 5/5 | - | green | ok | +350/-21 | 21 | 3,667,286 | 3.41 | 601 | 6 | done | test-driven-development |
| spd-r2 | F1 | 5/5 | - | green | ok | +317/-16 | 36 | 3,887,015 | 3.39 | 520 | 7 | done | test-driven-development |
| spd-r3 | F1 | 5/5 | - | green | ok | +434/-20 | 26 | 5,885,328 | 4.41 | 701 | 8 | done | test-driven-development > finishing-a-development-branch |
| mp-r1 | F1 | 5/5 | - | green | ok | +349/-20 | 29 | 5,098,503 | 2.22 | 472 | 12 | cap_driver_turns (driver_turns_12) | grilling > domain-modeling > tdd > code-review |
| mp-r2 | F1 | 5/5 | - | green | ok | +314/-15 | 25 | 4,042,815 | 1.80 | 365 | 12 | done | grilling > domain-modeling > tdd > code-review |
| mp-r3 | F1 | 5/5 | - | green | ok | +395/-23 | 29 | 4,286,993 | 2.10 | 480 | 12 | cap_driver_turns (driver_turns_12) | grilling > domain-modeling > tdd > code-review |
| both-r1 | F1 | 5/5 | - | green | ok | +356/-18 | 22 | 5,754,354 | 2.33 | 448 | 12 | done | grilling > domain-modeling > tdd > code-review |
| f2-bare-r1 | F2 | 4/5 | req3 | green | ok | +204/-10 | 5 | 494,231 | 0.38 | 92 | 1 | done | - |
| f2-bare-r2 | F2 | 5/5 | - | green | ok | +189/-10 | 5 | 808,413 | 0.45 | 115 | 3 | done | - |
| f2-spd-r1 | F2 | 5/5 | - | green | ok | +546/-14 | 46 | 5,310,616 | 3.42 | 506 | 6 | done | writing-plans > test-driven-development |
| f2-spd-r2 | F2 | 5/5 | - | green | ok | +401/-16 | 20 | 3,670,114 | 3.00 | 517 | 7 | done | test-driven-development |
| f2-mp-r1 | F2 | 5/5 | - | green | ok | +460/-23 | 31 | 4,823,516 | 2.18 | 443 | 12 | cap_driver_turns (driver_turns_12) | grilling > domain-modeling > tdd > code-review |
| f2-mp-r2 | F2 | 5/5 | - | green | ok | +440/-15 | 31 | 4,217,761 | 1.95 | 399 | 12 | done | grilling > domain-modeling > tdd > code-review |
| Görev ve konfigürasyon | Çalıştırma | Gizli /5 (çalıştırma başına) | Ort. USD | Ort. duvar s | Ort. token | Ort. kod diff +/- |
|---|---|---|---|---|---|---|
| F1 bare | 3 | 5 (5,5,5) | 0.343 | 84.5 | 519,114 | 141.667 |
| F1 sp (hook-routed) | 3 | 5 (5,5,5) | 0.502 | 118.1 | 724,601 | 235 |
| F1 spd (Superpowers driven) | 3 | 5 (5,5,5) | 3.738 | 607.367 | 4,479,876 | 386 |
| F1 mp (Main Flow) | 3 | 5 (5,5,5) | 2.043 | 439.033 | 4,476,104 | 372 |
| F1 both (sp+mp, mp flow) | 1 | 5 (5) | 2.326 | 448.2 | 5,754,354 | 374 |
| F2 bare | 2 | 4.5 (4,5) | 0.413 | 103.15 | 651,322 | 206.5 |
| F2 spd (Superpowers driven) | 2 | 5 (5,5) | 3.213 | 511.7 | 4,490,365 | 488.5 |
| F2 mp (Main Flow) | 2 | 5 (5,5) | 2.067 | 421.2 | 4,520,638 | 469 |
Superpowers driven, F1, aşama başına (3 çalıştırmanın ortalaması):
| Aşama | Token | USD | Duvar s |
|---|---|---|---|
| brainstorming | 182,022 | 0.166 | 40.283 |
| writing-plans | 321,355 | 0.266 | 88.133 |
| executing-plans | 3,976,500 | 3.307 | 478.95 |
| bunun içinde reviewer subagent | 656,962 | 2.192 | n/a |
Reviewer satırı, executing-plans içindeki subagent kalanı ve sonuç dosyasında "test-driven-development [subagent]" olarak etiketli. executing-plans içindeki oturum içi build'in (test-driven-development) ortalaması $0.974 oldu.
Main Flow, F1, aşama başına (aşamayı içeren çalıştırmaların ortalaması; /pr 3 çalıştırmanın 1'inde çalıştı):
| Aşama | Token | USD | Duvar s |
|---|---|---|---|
| grill-with-docs | 316,507 | 0.204 | 59.933 |
| to-spec | 163,344 | 0.118 | 42.1 |
| to-tickets | 257,955 | 0.143 | 44.4 |
| implement | 2,655,565 | 1.125 | 224.567 |
| code-review | 1,047,586 | 0.439 | 64.2 |
| pr | 105,440 | 0.041 | 11.6 |
Hook'la yönlendirilen Superpowers, F1: herhangi bir skill'den önce $0.03, brainstorming $0.219, skill'siz implementasyon $0.176 (2 çalıştırma), test-driven-development ile yürütme $0.405 (1 çalıştırma). Brainstorming 3 çalıştırmanın hepsinde, writing-plans hiçbirinde çalıştı. Sürülen döngü, F1 ortalamalarında hook'la yönlendirilenin 7.4 katına mal oldu ($3.738'e karşı $0.502). Üç Main Flow çalıştırması (mp-r1, mp-r3 ve f2-mp-r1) /pr'dan önce 12 driver turu sınırına takıldı ve öyle kaydedildi.
FINDINGS.md
# flow-bench findings (2026-10-09, round 2: isolated work copies, Sonnet 5.5 headless, 19 counted runs)
**F1 (partial payments): quality did not separate.** All 13 F1 runs hidden 5/5, suite and tsc green.
Cost per run: bare $0.27 (bare-r2) to $0.47 (bare-r1); sp, hook-routed, $0.44 (sp-r1) to $0.54 (sp-r2); mp $1.80
(mp-r2) to $2.22 (mp-r1); spd, Superpowers driven, $3.39 (spd-r2) to $4.41 (spd-r3).
Wall: mp 365 to 480 s (mp-r2, mp-r3), spd 520 to 701 s (spd-r2, spd-r3).
**Where the money goes.** mp-r2 (complete flow): grill $0.16, spec $0.11, tickets $0.09, implement $0.96, code-review
$0.45, pr $0.04 (F1 means: implement $1.13, code-review $0.44). spd: brainstorming $0.17, writing-plans $0.27,
executing-plans $3.31 (F1 means). Inside executing-plans the build itself (TDD in session) costs $0.97 on average; the
rest is the one whole-branch reviewer the skill dispatches "on the most capable model", which ran on claude-fable-5-1
in all five spd runs: $2.01 of spd-r1's $3.41, $2.52 of spd-r3's $4.41, $1.57 of f2-spd-r2's $3.00.
**Hook vs driven.** Left to its hook, Superpowers ran brainstorming in sp-r1, sp-r2 and sp-r3 and then coded; writing-
plans ran 0 times in 3. Driven by slash commands, the full loop ran in every spd run and cost 7.4 times more (F1 means: spd $3.74, sp $0.50). The cost is the final review, not the planning.
**F2 (credit notes) separated, barely.** f2-bare-r1 shipped one missing requirement, R3: credit-note numbers are not
gapless. It took the number before validating, like the existing `/issue` route, so rejected requests burned numbers
(CN-0004 where CN-0002 was due; the F2 bare smoke failed the same way). f2-bare-r2, f2-spd-r1, f2-spd-r2, f2-mp-r1 and
f2-mp-r2 scored 5/5. Costs: bare $0.38/$0.45, mp $2.18/$1.95, spd $3.42/$3.00 (r1/r2). Caveat: the first F2 oracle
also required credit notes on open invoices; both bare runs allowed paid invoices only, which the request's wording
permits, so that check was removed and the F2 runs rescored.
**Stacking broke nothing.** both-r1: hook injected, no Superpowers skill invoked, $2.33, inside the mp range.
**Caps.** mp-r1, mp-r3 and f2-mp-r1 hit the 12-driver-turn cap before /pr. spd never needed more than 8 turns.
**Caveats.** Round-1 runs read this repo's CLAUDE.md (parent-directory loading), so they were discarded and rerun outside the repo. A simulated PO answers each round in one message. The SessionStart hook fires on turn 1 only.
/implement covers every ticket in one turn. 5 to 11 compound shell lines per spd run were denied. PO cost is excluded.
Daha fazlası için
- Debugging bench'i. r/claudeskills'teki bir yazı, tek bir repo'nun geçmişinden 15 bug'ı, her biri üç çalıştırmayla denedi. Üç varyant da 15'ten 9'unu düzeltti, ekstra skill'ler bug başına yaklaşık 3 dakika ekledi ve Pocock'un debugging skill'i, başka bir testi bozmadan, 41 ve 48'e karşı medyan 32 satırla en küçük değişiklikleri yaptı s8. Yalnızca debugging skill'lerini test ediyor, tam akışları değil ve yazı 41 ile 48'in hangi varyanta ait olduğunu söylemiyor. Bench harness'ı ile kendi bug geçmişinizde yeniden çalıştırın.
- Geçiş yapanların blogları. Orkestrasyonun vergiye dönüştüğü tezini s9 ve Superpowers'ın hook'larını silip grill-with-docs'u tutan geçiş hikayesini s10 okuyun, sonra iddialarını yukarıdaki aşama başına tabloyla karşılaştırın.
- Diğer paketler. Pstack ve Osmani'nin paketi, çalıştırmadığımız iki skill seti. Aynı harness,
.claude/skillsiçine kurulan her paketi alır;configs.json
{
"bare": {
"superpowers": false,
"mattpocock": false,
"flow": "bare",
"desc": "no skills, no hooks (project settings carry only the permission deny-list)"
},
"sp": {
"superpowers": true,
"mattpocock": false,
"flow": "sp",
"desc": "HOOK-ROUTED: obra/superpowers v6.4.2: skills in .claude/skills, SessionStart hook in .claude/settings.json as hooks/hooks.json wires it; turn 1 = the bare task, the hook decides"
},
"mp": {
"superpowers": false,
"mattpocock": true,
"flow": "mp",
"desc": "mattpocock/skills v1.3.1 via npx skills add (27 promoted skills, copied), setup-matt-pocock-skills output committed (local tracker)"
},
"both": {
"superpowers": true,
"mattpocock": true,
"flow": "mp",
"desc": "sp + mp in one project; driven with the mp slash-command flow (the stacking question: does the Superpowers hook hijack it?)"
},
"spd": {
"superpowers": true,
"mattpocock": false,
"flow": "spd",
"desc": "Superpowers, DRIVEN: same project install as sp (hook included); the driver types /brainstorming <task> -> /writing-plans -> /executing-plans (Native/inline); the pack's own scripts allowed in Bash"
}
}
prep.sh
#!/usr/bin/env bash
# Fresh work copy of upstream (node_modules included) + the pack fixtures of <cfg>.
# Usage: prep.sh <cfg> <work dir> Prints the base commit (the oracle's BASE) on its last line.
set -euo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"
CFG="$1"; W="$2"
SP=$(python3 -c "import json;print(int(json.load(open('$HERE/configs.json'))['$CFG']['superpowers']))")
MP=$(python3 -c "import json;print(int(json.load(open('$HERE/configs.json'))['$CFG']['mattpocock']))")
rm -rf "$W"; mkdir -p "$(dirname "$W")"
cp -R "$HERE/upstream" "$W"
cd "$W"
# .claude/ never shows in git status or in any diff (hook config, installed skills).
echo ".claude/" >> .git/info/exclude
mkdir -p .claude/skills
if [ "$MP" = 1 ]; then
cp -R "$HERE/packs/mattpocock-skills/.claude/skills/." .claude/skills/
# One-time per-repo precondition of the pack (/setup-matt-pocock-skills), answered once with the defaults it
# recommends for a repo with no remote: local-markdown tracker, single-context domain docs, no triage labels
# (triage is installed, so Section B would ask; default labels kept). Committed so it never shows in the diff.
S="$HERE/packs/mattpocock-skills/.claude/skills/setup-matt-pocock-skills"
mkdir -p docs/agents
cp "$S/issue-tracker-local.md" docs/agents/issue-tracker.md
cp "$S/domain.md" docs/agents/domain.md
cp "$S/triage-labels.md" docs/agents/triage-labels.md
cat > CLAUDE.md <<'MD'
## Agent skills
### Issue tracker
Issues and specs live as local markdown files under `.scratch/`. See `docs/agents/issue-tracker.md`.
### Triage labels
Default labels (`needs-triage`, `needs-info`, `ready-for-agent`, `ready-for-human`, `wontfix`). See `docs/agents/triage-labels.md`.
### Domain docs
Single-context: one `GLOSSARY.md` + `docs/adr/` at the repo root. See `docs/agents/domain.md`.
MD
git add CLAUDE.md docs/agents
GIT_AUTHOR_DATE="2026-03-09T18:00:00" GIT_COMMITTER_DATE="2026-03-09T18:00:00" \
git -c user.name="Dana Whitfield" -c user.email="dana@ledgerly.dev" commit -q -m "chore: configure agent skills (local tracker)"
fi
if [ "$SP" = 1 ]; then
# The plugin root as Claude Code would hold it, so the hook script finds ../skills/using-superpowers.
mkdir -p .claude/plugins
cp -R "$HERE/packs/superpowers" .claude/plugins/superpowers
cp -R "$HERE/packs/superpowers/skills/." .claude/skills/
fi
python3 - "$HERE" "$SP" "$CFG" > .claude/settings.json <<'PY'
import json, sys
bench, sp, cfg = sys.argv[1], sys.argv[2] == "1", sys.argv[3]
# Oracle, reference, PO brief, other runs: denied by path (Read/Grep/Glob) and by command text (Bash).
deny = ["Read(/%s/hidden/**)" % bench, "Read(/%s/reference/**)" % bench, "Read(/%s/runs/**)" % bench,
"Read(/%s/po/**)" % bench, "Read(/%s/validation/**)" % bench,
"Bash(*hidden*)", "Bash(*reference/F*)", "Bash(*-brief.md*)", "Bash(*/runs/*)", "Bash(*../*)", "Bash(find *)"]
if cfg == "spd":
# executing-plans runs its sibling scripts as ../subagent-driven-development/scripts/...: a "../" deny would block the
# pack's own loop. The oracle stays protected by the Read denies and Bash(*hidden*).
deny.remove("Bash(*../*)")
settings = {"permissions": {"deny": deny}}
if sp:
# = packs/superpowers/hooks/hooks.json, with ${CLAUDE_PLUGIN_ROOT} provided the way the plugin loader provides it
# (the script also picks its JSON output shape from CLAUDE_PLUGIN_ROOT being set).
root = '$CLAUDE_PROJECT_DIR/.claude/plugins/superpowers'
hooks = json.load(open("%s/packs/superpowers/hooks/hooks.json" % bench))["hooks"]
for groups in hooks.values():
for g in groups:
for h in g["hooks"]:
h["command"] = 'CLAUDE_PLUGIN_ROOT="%s" %s' % (root, h["command"].replace("${CLAUDE_PLUGIN_ROOT}", root))
settings["hooks"] = hooks
print(json.dumps(settings, indent=2))
PY
[ "$(ls .claude/skills | wc -l)" -eq 0 ] && rmdir .claude/skills
test -z "$(git status --porcelain)" || { echo "work copy not clean"; git status --porcelain; exit 1; }
git rev-parse HEAD
SSS
İki paketten biri daha iyi kod mu gönderdi?
Özellik 1'de hayır: 13 çalıştırmanın hepsi 5 üzerinden 5 aldı, suite ve typecheck yeşil. Özellik 2'de bir bare çalıştırma bir gereksinimi kaçırdı, boşluksuz credit note numarasını, çünkü mevcut issue route'unun yaptığı gibi isteği doğrulamadan önce numarayı aldı. İki döngü de kodu yazmadan önce numara dizisi için bir test yazdı. İkinci bare çalıştırma doğrusunu yaptı.
Superpowers döngüsü burada neden Main Flow'dan bu kadar pahalı?
Tek aşama. executing-plans ortalama $3.307 tuttu ve içindeki bir reviewer subagent $2.192 tuttu. Reviewer, sürülen beş çalıştırmanın hepsinde claude-fable-5-1 üzerinde çalıştı ve skill en yetenekli modeli istiyor. Planlama aşamaları $0.166 ve $0.266 tuttu. Daha ucuz bir reviewer ile yeniden çalıştırmadık, bu yüzden o değişiklik için ölçülmüş bir rakamımız yok.
İki paketi birlikte kurmak bir şeyi bozar mı?
Tek üst üste çalıştırmamızda hayır: hook enjekte edildi, hiçbir Superpowers skill'i çağrılmadı, slash komutları kazandı ve çalıştırma 5 üzerinden 5 ile $2.326 tuttu. Pocock'un kullanıcı tarafından çağrılan skill'lerinde modelin onları kendi başına yüklemesini engelleyen bir bayrak var ve iki paket hiçbir skill dizin adını paylaşmıyor. Bu tek bir çalıştırma, o yüzden kendi kurulumunuzda test edin.
Pocock'un skill'leri için neden Claude plugin'ini kullanmayayım?
Çünkü marketplace onu 1.2.3'te sabitliyor, upstream ise v1.3.1'de s32. Installer, istediğiniz release'i çekiyor.
Bu, Main Flow'un kalitesi için adil bir test mi?
Her paketin neye mal olduğu ve küçük bir repoda beş gereksinimi gönderip göndermediği için adil bir test. Main Flow'un artefaktları, bir spec ve ticket'lar, burada skorlanmadı ve daha büyük bir özellik ya da bir ekip onlara daha fazla değer verebilir.