AIDiveTL;DR와 결론
- 순정 Claude Code 세션, Superpowers, Matt Pocock의 Main Flow가 지어낸 repo 하나에서 같은 기능을 만들었습니다. headless Sonnet 5.5, 집계한 실행 19번. token, 시간, 정가 기준 비용은 실행 단위가 아니라 stage별로 읽었습니다.
- Stage 지도: Superpowers를 hook 라우팅 그대로 유지합니다. 요청이 모호할 때 Pocock의 전반부 grill-with-docs, to-spec, to-tickets를 빌려 씁니다. executing-plans 리뷰어의 모델 줄을 바꿉니다. 그 한 stage가 두 pack 사이 격차의 거의 전부이기 때문입니다.
- 기능 1(부분 결제), 구성당 3회 평균: bare $0.343에 84.5 s, hook에 맡긴 Superpowers $0.502에 118.1 s, 전체 루프로 돌린 Superpowers $3.738에 607.367 s, Main Flow $2.043에 439.033 s. 두 pack을 겹친 실행 1회는 $2.326. 이 실행 모두 숨겨진 테스트 5개 중 5개를 통과했습니다.
- 직접 돌린 Superpowers 루프에서 executing-plans는 전체 $3.738 중 $3.307였고, 그중 $2.192는 skill이 가장 강력한 모델로 보내는 리뷰어 subagent 하나였습니다. Brainstorming은 $0.166, writing-plans는 $0.266.
- 기능 2(크레딧 노트), 더 어려움: bare는 $0.413에 두 번 중 한 번 실패, Superpowers 직접 구동 $3.213, Main Flow $2.067이며 뒤의 둘은 5개 중 5개. 19번의 실행에서 놓친 유일한 요구사항은 공백 없는 크레딧 노트 번호였고, bare 실행이 냈습니다.
- 한계: 지어낸 repo 하나, worker 모델 하나, 기능 두 개, 각본대로 답하는 프로덕트 오너, 칸마다 작은 n. 여기서 pack을 가른 것은 품질이 아니라 비용이었습니다.
출처가 말하는 것
실행 당시의 버전: Superpowers는 v6.4.2(2026-09-25 릴리스) s20, mattpocock/skills는 v1.3.1(2026-10-04 릴리스) s26입니다. Superpowers는 skill 15개를 싣고, Pocock의 repo는 배포 skill 27개를 나열합니다 s1 s3. 공유 4개, Superpowers 전용 10개, Pocock 전용 12개라는 이전 중복 집계는 Superpowers 6.0.3 대 Pocock 1.0.1, 즉 14개 대 17개 skill 시점의 것이라 낡았습니다 s6.
두 pack은 정반대에 건 내기입니다. Superpowers는 startup, clear, compact 때 EXTREMELY_IMPORTANT 태그로 감싼 using-superpowers skill을 주입하는 SessionStart hook을 연결합니다 s23. 그 문구: "If you think there is even a 1% chance a skill might apply to what you are doing, you ABSOLUTELY MUST invoke the skill." s22 또한 확인 질문을 포함해 어떤 응답이나 행동 전에도 관련 skill을 호출하라고 말합니다 s22. Pocock의 pack에는 세션 hook이 없습니다. "There is no session hook; nothing gets injected at startup." s6 repo 트리에도 hooks 파일이 없습니다 s3.
skill 파일은 짧습니다. grill-with-docs는 35단어이고, 하는 일은 grilling과 domain-modeling을 위해 Skill 도구를 두 번 호출하는 것뿐입니다 s27. implement는 103단어이고 disable-model-invocation이 붙어 있으며 code-review 호출로 끝납니다 s28. Matt이 발표한 63%라는 수치는 실행이 아니라 skill 설명의 token 비용 감소입니다 s34 s35.
Main Flow는 라우터 skill이 펼쳐 놓은 대로 pull request step을 거쳐 루프를 닫는 retro로 끝납니다 s29. 개요 페이지는 Main Flow에 skill 6개를 나열하고 /pr은 reference skill로 분류합니다 s1. v1.3 노트에서 Matt은 이렇게 씁니다. "The agent loop is worse than a deterministic loop. But it is a good way to start with AFK work" s2. 거친 부분도 적어 둡니다. "worktrees do not remove collisions, they move them to merge time" s2. v1.3은 CONTEXT.md를 GLOSSARY.md로, CONTEXT-MAP.md를 GLOSSARY-MAP.md로 바꿉니다 s2 s26.
설치 경로. README는 plugin 명령어와 설치 프로그램 명령어, claude plugin install mattpocock-skills@claude-plugins-official와 npx skills@latest add mattpocock/skills -a <agent>를 주고, agent당 하나만 고르라고 합니다. 둘 다 설치하면 모든 skill이 두 번 생기기 때문입니다 s3. 마켓플레이스 사본은 뒤처집니다. 공식 Claude plugin 마켓플레이스는 mattpocock-skills를 1.2.3에 고정해 두었고, 버전을 올려 달라는 요청은 아직 열려 있습니다 s32. 우리가 확인했을 때 upstream main은 그 고정본보다 71 commit 앞서 있었고, 고정본에는 implement-spec, pr, retro가 없습니다 s31 s32. 같은 뒤처짐을 겪은 사용자 보고가 Pocock의 repo에 열려 있고 s31, 2026-10-08에 Matt이 X에서 캐시된 버전을 올려 달라고 요청했습니다. "All my users are stuck on v1.2." s33 같은 확인에서 마켓플레이스는 Superpowers를 v6.4.2보다 패치 하나 뒤인 6.4.1에 고정해 두었습니다 s20.
Superpowers 쪽에서는 6.4.1이 executing-plans를 네이티브 인라인 모드, "the cheapest way to run a plan"으로 다시 만들었고 s21, 6.4.2는 writing-plans를 더 가볍게 했습니다 s20. 사람들이 인용하는 열린 이슈로는 skill 설치 후 느려진다는 것 s24, 그리고 Codex의 subagent 구동 실행이 5시간 예산을 통째로 소비했다는 보고가 있는데, 이는 6.4 릴리스가 아니라 Superpowers 5.0.7과 Codex에서 보고된 것입니다 s25. /implement의 token 비용에 대한 Pocock 쪽 이슈는 사용자 토론이고, Matt이 skill의 결함이 아니라 토론으로 닫았습니다 s30.
측정 없이 사람들이 하는 말. 갈아탄 한 사람은 Pocock의 Q&A가 Superpowers의 brainstorming보다 "with no worse results"로 token을 덜 썼다고 합니다 s10. 다른 사람은 의무적 orchestration이 안전망이기를 그치고 세금이 된다고 씁니다 s9. Hacker News 글타래는 Superpowers를 아예 쓰지 않을 때와의 벤치마크를 요구하고, 옆에서는 eval로 만들어졌다는 옹호가 나옵니다 s13. grilling에 대한 글타래는 짧은 구현 앞에 "5 hours wayfinding and spec reading"이 있었다고 보고하고, 한 댓글은 더 새로운 모델에서 grilling이 "feels like a hinderance"라고 합니다 s7. pack 두 개를 라우터로 겹쳐 쓰면 슬래시 명령어와 세션 hook이 충돌한다는 글도 있습니다 s11. 어느 것도 stage별 token 수치를 보고하지 않으며, 이 벤치가 메우는 것이 바로 그 빈틈입니다.
벤치
벤치는 ledgerly라는, 지어낸 작은 TypeScript 청구서 API입니다. domain 계층, 스키마 마이그레이션이 있는 JSON 파일 store, HTTP 계층, 통과하는 테스트 34개, commit 4개로 되어 있습니다. 기능 요청은 각각 개발자가 치는 그대로 세 줄입니다. 시니어 엔지니어는 요청에 적혀 있지 않은 다섯 가지를 기대하고, 그 다섯이 worker가 보지 못하는 숨겨진 테스트입니다. worker는 headless Sonnet 5.5입니다. 두 번째 headless 모델이 프로덕트 오너를 맡는데, 도구가 없고, 비공개 브리프를 읽고, worker의 질문에 짧게 답합니다. 모든 turn은 token, 정가 기준 비용, 소요 시간, 도구 호출, skill 호출과 함께 기록됩니다. 비용은 CLI의 total_cost_usd이며, 정가 환산값이지 청구액이 아닙니다. 누구의 실제 코드에서도 돌리지 않았으니, 자기 repo용 템플릿으로 보세요.
README.md
# flow-bench: Superpowers loop vs Matt Pocock's Main Flow, one feature, one repo
Built 2026-10-09 from `SPEC.md` (the fixed decisions are implemented as written; deviations and their reasons are
listed under "Decisions taken where SPEC left a choice"). Worker = headless `claude-sonnet-5-5` (`claude -p`, OAuth
from `~/.claude`), Claude Code 2.1.294.
## Commands (from this dir)
```bash
./validate.sh # SPEC validation steps 1-5 -> validation.log (3-5 are smoke runs, RUN_TAG=smoke)
./validate.sh 1 2 # only the static steps (reference patch, untouched upstream)
RUN_TAG=x ./run.sh <cfg> <rep> # one run -> runs/<cfg>-r<rep>-x/, work copy work/<same>/ ; tagged runs are ignored by analyze.py
nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 & # the 10-run matrix, 2 lanes (MATRIX_JOBS=2)
SKIP_DONE=1 nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 & # resume
python3 analyze.py # results.json + results.md
python3 lib/finalize.py runs/<run> # rescore one run (restores node_modules into the work copy if it was removed)
python3 lib/reaccount.py runs/<run> # recompute turns.jsonl accounting from the saved streams (then finalize)
```
## Layout
| path | what |
|---|---|
| `upstream/` | **ledgerly**, the invented repo: Node 20 + TS + vitest + pnpm, 18 source files (domain / store / http layers), 34 tests in 7 files, 4 commits (HEAD 980d542), `test` + `typecheck` scripts, `node_modules` installed (pnpm 11) |
| `tasks/F1.md` | the task, verbatim (3 lines, the way a dev types it) |
| `po/F1-brief.md` | the product owner's full intent: the 5 requirements in plain words, preferences, constraints (never the tests) |
| `hidden/` | `test_F1.test.ts` (the oracle: `test_req1..5`), its own `vitest.config.ts` (alias `@work` -> `$WORK`) and `node_modules` (vitest 2.1.9). Runs from OUTSIDE the work copy |
| `reference/F1.patch` | reference solution: hidden 5/5, suite 39/39, tsc green |
| `configs.json` | `bare`, `sp`, `mp`, `both` |
| `packs/` | `superpowers/` (plugin root of obra/superpowers v6.4.2), `mattpocock-skills/.claude/skills/` (27 skills of mattpocock/skills v1.3.1 installed by `npx skills add`), `VERSIONS.md`, `mattpocock-install.log`; `*-src/` = the clones (gitignored) |
| `env.sh` | copied from an earlier bench of the same kind: scrubs inherited `CLAUDE*` vars, `CLAUDE_CODE_DISABLE_AUTO_MEMORY=1` |
| `prep.sh <cfg> <work>` | fresh work copy + the pack fixtures of the config + `.claude/settings.json`; prints the base commit |
| `run.sh <cfg> <rep>` | prep -> `lib/driver.py` -> `lib/finalize.py` |
| `lib/driver.py` | the turn loop (flows below), caps, `turns.jsonl` |
| `lib/accounting.py` | per-turn and per-stage tokens / USD / wall / tools / Skill events / hook events from one turn's stream |
| `lib/po.py` | the product-owner simulator |
| `lib/finalize.py` | diff vs base, suite, typecheck, hidden oracle, stages -> `score.json`, `stages.json` |
| `analyze.py` | `results.json`, `results.md` |
| `validate.sh`, `validation.log`, `validation/routing-probes/` | pre-matrix checks (+ 6 one-turn probes: plugin wiring vs project wiring) |
| `matrix.txt`, `matrix.sh`, `matrix-one.sh`, `matrix.log` | 10 runs, 2 at a time |
### The feature and the oracle
`tasks/F1.md`: partial payments, `POST /invoices/:id/payments` with an amount in cents, `amountPaid`/`balance` on
invoices, flip to `paid` at zero balance, keep `/pay` working, "we have prod data in the JSON store".
The 5 hidden requirements (one `it` each) and the codebase precedent that makes each one a senior's expectation:
| req | checks | precedent in the repo |
|---|---|---|
| R1 edge cases | payments accumulate; status stays non-`paid` until the exact remainder; `paidAt` set; `/reports/outstanding` sums balances, not totals | the report sums `total` of open invoices |
| R2 validation | 0, negative, 12.5, "100", null, missing -> 400/422; overpayment (of the total, then of the balance) -> 400/409/422; state unchanged | `assertCents`, "discount cannot exceed the subtotal" |
| R3 migration of existing data | a schemaVersion-1 file: legacy `paid` invoice shows amountPaid=total, balance 0; open shows 0/total; payments persist across a reload; `/pay` leaves balance 0 | `schemaVersion` + `src/store/migrations.ts` (v0->v1), README house rule |
| R4 error path | unknown id 404; draft 409; void 409; nothing recorded | `markPaid` throws `ConflictError` on draft/void |
| R5 a test | a `*.test.ts` changed or added since the base commit mentions `payments` | 34 existing tests |
Lenient where the brief would be a guess (status codes for overpayment accept 400/409/422, payment response 2xx,
response body never inspected: state is read back with GET).
## Configs and how each pack is wired (`prep.sh`)
- Every config: `cp -R upstream`, `.claude/` in `.git/info/exclude`, `.claude/settings.json` with a `permissions.deny`
list (Read of `hidden/ reference/ runs/ po/ validation/` of this bench; Bash commands containing `hidden`,
`reference/F1`, `F1-brief`, `/runs/`, `../`, and `find`). Bare has nothing else: no skills, no hooks.
- `sp`: the plugin root is copied to `.claude/plugins/superpowers/` (the hook script reads
`../skills/using-superpowers/SKILL.md` relative to itself), the 15 skills to `.claude/skills/<name>/`, and
`hooks/hooks.json` is reproduced in `.claude/settings.json` verbatim (`SessionStart`, matcher
`startup|clear|compact`, `"shell": "bash"`, `"async": false`) with the command prefixed by
`CLAUDE_PLUGIN_ROOT="$CLAUDE_PROJECT_DIR/.claude/plugins/superpowers"`: Claude Code sets that variable for plugin hooks,
and `hooks/session-start` uses it to pick the `hookSpecificOutput.additionalContext` output shape. Superpowers ships no
other hook (`hooks-cursor.json` is Cursor's).
- `mp`: the 27 skills to `.claude/skills/`. Pocock ships **no hook** (verified: no settings or hooks file in the install;
`git-guardrails-claude-code`, a skill that writes hooks, is outside the promoted set and was trimmed). The pack's
per-repo precondition `/setup-matt-pocock-skills` is materialised once: `docs/agents/issue-tracker.md` (local
markdown under `.scratch/`), `docs/agents/domain.md`, `docs/agents/triage-labels.md` from the skill's own seed
templates, and a `CLAUDE.md` "## Agent skills" block, committed as "chore: configure agent skills (local tracker)"
so the diff only shows the feature. Base commit for mp/both = that commit.
- `both`: sp + mp in the same `.claude/` (no skill-name collision between the two packs), driven with the mp flow.
## Flows (`lib/driver.py`)
Every worker turn: `claude -p <prompt> [--resume <sid>] --model claude-sonnet-5-5 --output-format stream-json --verbose
--include-hook-events --include-partial-messages --max-turns 60 --permission-mode acceptEdits --allowedTools <SPEC list>
--disallowedTools AskUserQuestion,EnterPlanMode,ExitPlanMode,EnterWorktree,ExitWorktree,WebFetch,WebSearch
--setting-sources project --strict-mcp-config`, cwd = work copy, stdin /dev/null, scrubbed env.
Allow-list: `Read,Edit,Write,MultiEdit,Glob,Grep,Skill,TodoWrite,Task,Agent`, Bash `node npm pnpm npx vitest npx tsc tsc
git status|diff|log|add|commit|checkout -b ls cat mkdir head tail grep wc gh`.
- `bare`: turn 1 = task + "When you are done, stop and summarise what you changed."
- `sp`: turn 1 = the task alone (the SessionStart hook is supposed to route it).
- `mp` / `both`: `/grill-with-docs <task>` -> `/to-spec` -> `/to-tickets` -> `/implement the tickets in
.scratch/<slug>/issues/, all of them, blockers first` -> `/code-review since <base sha>` -> `/pr`.
- After every turn the PO reads the last assistant message and classifies it: `question` (its reply is the next
prompt), `waiting` (driver sends "go on" / "go on, proceed with the plan"; mp: stays in the stage), `done` (bare/sp:
stop; mp: next slash command; after `/pr`: stop). If the PO call fails twice the SPEC heuristic (ends on "?" or an
options list) decides.
- Caps: 12 driver turns, 60 `--max-turns` per claude turn (a turn that hits it gets "go on"), 40 min wall per run
(the claude process is killed at the deadline). Caps hit are recorded in `driver.json` / `score.json`, never discarded.
### PO simulator (`lib/po.py`)
Headless `claude-sonnet-5-5`, `--tools ""`, `--setting-sources ""`, `--settings '{"disableAllHooks":true}'`,
`--strict-mcp-config`, `CLAUDE_CODE_DISABLE_CLAUDE_MDS=1`, `CLAUDE_CODE_DISABLE_AUTO_MEMORY=1`, cwd = an empty temp dir,
`--json-schema {kind, reply}`. Prompt = role + `po/F1-brief.md` + flow note + stage + the last message. Rules: terse,
answer every open question at once, approve what matches the brief, correct only contradictions, "your call" where
the brief is silent, **answer only what was asked** (first smoke: the PO volunteered the migration requirement nobody
had asked about; the rule was added before any counted run), how to finish a branch: "keep it on the current branch,
no PR". mp rule: a stage whose output is delivered and that only asks "shall I proceed / start?" is `done` (first mp
smoke: the PO's "yes, go" after the grilling recap made the worker start implementing inside `/grill-with-docs`; that
smoke was killed and rerun). PO cost is recorded apart (`usd_po`), never in the worker's USD.
## Accounting (the per-turn / per-stage numbers)
Checked on 2.1.294 with probes before writing `lib/accounting.py`:
- `result.total_cost_usd` and `result.modelUsage` are **cumulative over a resumed session** and **include subagents**:
per-turn USD and tokens = delta against the previous turn's result line.
- `result.usage` = main thread of the invocation only (a Task subagent's tokens are missing from it): not used.
- `assistant` events carry the usage snapshot of the message START (`output_tokens` partial, e.g. 3 for a 783-token
message): useless for output. `--include-partial-messages` adds `stream_event` `message_start` / `message_delta`
with the exact per-API-call usage, main thread only.
- Sonnet 5.5 list prices derived from two probe turns and matched to the cent: $2/M input, $10/M output, $0.20/M cache
read, $4/M cache write (1 h TTL, what Claude Code uses here); 5-min writes priced $2.5/M (unseen). In every turn without
subagents, the per-call priced sum equals the `total_cost_usd` delta exactly (`pricing_check` in `turns.jsonl`).
- Stage = slash command of the turn (mp/both, exact) or the skill in force (sp: switches at each main-thread `Skill`
tool_use; turn 1 starts at `(none)`). Inside a turn, main-thread calls are attributed exactly; the subagent remainder
(turn delta minus main calls) goes to the stage that spawned the Task, weighted by the subagents' snapshot tokens.
Wall per stage = receive-time gaps of the stdout lines (`turn-NN.times`) attributed to the stage in force.
- `analyze.py` also groups sp skills into phases (brainstorming / writing-plans / execution incl. TDD /
verification / review / finishing).
- Tokens reported = input + output + cache read + cache write. Cache reads dominate (re-reading the context each call).
### Per run (`runs/<cfg>-r<rep>/`)
`meta.json`, `prep.txt`, `settings.json` (copy of the work copy's), per turn `turn-NN.stream.jsonl` (raw stdout, every
line), `turn-NN.times`, `turn-NN.stderr`, `turn-NN.last.txt`, `po-NN.json`; `turns.jsonl` (one accounting record per
turn: prompt, source, wall, subtype, USD, tokens per model, segments, tools, Skill events, hook events, files, denials,
PO answer), `driver.json` (end reason, caps), `post.diff`, `numstat.txt`, `suite.txt`, `typecheck.txt`, `hidden.txt`,
`stages.json`, `score.json`. Work copies stay in `work/` without `node_modules` (49 MB each; finalize removes it,
`KEEP_NODE_MODULES=1` keeps it).
## Decisions taken where SPEC left a choice
- **Implement stage = `/implement`, not `/implement-spec`.** v1.3.1 ships both. `/implement-spec` requires implementer
subagents "each in its own worktree on its own branch" plus merger subagents: `git worktree`, `git merge`, `git branch`
are outside the SPEC Bash allow-list, so it could not run as designed. `/implement` is sent once with the ticket
directory ("all of them, blockers first"), since one session cannot `/clear` between tickets as `ask-matt` suggests.
- `/code-review` gets `since <base sha>` (the skill reviews "the diff since a fixed point"); Pocock's `code-review`
replaces Claude Code's bundled `code-review` skill of the same name in the session (init lists it once, the Skill
event loads `.claude/skills/code-review`).
- `/retro` not run (optional in SPEC).
- `both` uses the mp flow: the stacking question is whether the Superpowers SessionStart routing hijacks the slash
commands.
- Reps / order: `matrix.txt` (rep 1 of every config first), 2 lanes.
## Traps met
- **Superpowers routing is stochastic headless.** On turn 1 the hook always fires, but the model picks
`test-driven-development` instead of `brainstorming` part of the time. Six one-turn probes on the F1 prompt
(`validation/routing-probes/`): real plugin wiring (`--plugin-dir packs/superpowers`, namespaced
`superpowers:*`) 1/3 brainstorming, 2/3 TDD; project wiring (this bench) 2/3 brainstorming, 1/3 TDD. So the project
wiring is not the cause; the first sp smoke went TDD-only (one turn, hidden 5/5). A second sp smoke (`smoke2`) was run
to see the brainstorming path end to end.
- **The SessionStart hook fires only on turn 1.** Its matcher is `startup|clear|compact`; every later turn is
`--resume` (source `resume`), so the injection is not repeated; it stays in the transcript.
- **User plugins do not leak** with `--setting-sources project`: the init event lists only Claude Code's built-in
plugins (`cc-plugin-agents-md`, `cc-plugin-telemetry`, `cc-plugin-plugin-authoring`), never the user's Superpowers
6.4.1; bare smoke: no hook event, no Skill event. Built-in skills (`verify`, `debug`, `simplify`, `code-review`, ...)
are still listed in every config.
- **`--bare` is never used** (cannot authenticate on this OAuth account; see the previous bench).
- **Read-only shell commands bypass the allow-list.** Claude Code auto-approves read-only commands (`git ls-files`,
`sed -n`, pipes of read-only commands) whatever `--allowedTools` says; `python3` edits are denied (the bare smoke had 4
`permission_denials`, all `python3` heredoc edits; the worker fell back to Edit). `cat > /tmp/x <<EOF` matches
`Bash(cat:*)`: the allow-list is a prefix match, not a sandbox.
- **pnpm 11** refuses to install without approving esbuild's build script: `pnpm-workspace.yaml` with
`allowBuilds: {esbuild: true}` (pnpm 11 no longer reads the `pnpm` field of package.json). No `packages:` key, so it
is not a monorepo signal for Pocock's setup.
- **`npx skills add` ignores `--skill a,b` and `--skill "a b"`** ("No matching skills found"); repeated `--skill x`
flags installed every skill of the repo (37). Trimmed to the plugin's 27 (`packs/VERSIONS.md`). Pinning works with
the `https://github.com/mattpocock/skills/tree/v1.3.1` source (main is past the release).
- **mattpocock skills are mostly user-invoked** (`disable-model-invocation: true` on grill-with-docs, to-spec,
to-tickets, implement, implement-spec, ...): reachable as slash commands only; `/grill-with-docs` itself is one line
("Call the Skill tool twice, for grilling and domain-modeling"), so the proof that it resolved is those two Skill
events on turn 1.
- **The model mistypes the bench path.** Sonnet 5.5 regularly writes absolute paths with
`superpowers-vs-matt-pocock-skins-...` (or `skips`) instead of `skills`: 0 to 10 tool calls per run (0 to 7% of calls;
e.g. mp-r3-po1 10/145, mp-r2 0/124, sp-r2 3/48, bare-r1 0/28), in every config. Edits on the wrong path fail and are
retried (a small, roughly proportional token tax); **Writes succeed outside the work copy** because `Write` in
`--allowedTools` is not path-limited: the runs left stray files under a mistyped copy of the bench path (a spec
copy, an `x`) and three 1-byte placeholder files, deleted after the matrix
(nothing else in the repo was touched; `git status` checked). A future bench should put work copies on a short
path without the word `skills` and deny `Write(//**)` outside the work copy.
- **Shared plan meter.** Workers and PO bill the user's plan quota; USD = the CLI's list-price equivalent.
- **rtk** rewrites interactive `diff`/`grep` in the launching session only; scripts call binaries directly.
## Validation (2026-10-09, `validation.log`)
1. `reference/F1.patch` -> hidden 5/5, suite 39/39 green, tsc green: PASS.
2. upstream unchanged -> hidden 0/5, suite 34/34 green: PASS.
3. `bare-r1-smoke` -> 2 driver turns (the PO answered one closing question), hidden 5/5, $0.52, 125 s; isolation: no
hook event, no Skill event, only built-in plugins in init: PASS.
4. `sp-r1-smoke` -> hook fired and injected `using-superpowers` (PASS) but the model went straight to
`test-driven-development` (brainstorming FAIL in that run). Retry `sp-r1-smoke2`: hook fired, `brainstorming` on
turn 1, PO approved the design, then `test-driven-development`: PASS. Routing probes: see Traps.
5. `mp-r1-smoke` -> turn 1 loads `grilling` + `domain-modeling` (what `/grill-with-docs` says to do): PASS. The run then
used all 12 driver turns (grill 4, to-spec 2, to-tickets 2, implement 2, code-review 2) and never reached `/pr`.
Fixes made during validation, before any counted run: the PO volunteering unasked requirements; the PO telling the
worker "yes, go" at the end of the grilling (the worker started implementing inside `/grill-with-docs`; that first mp
smoke was killed at turn 4 and rerun).
## Matrix launch
`nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 &` at 2026-10-09 10:41:33, 2 lanes, order in `matrix.txt`.
Done 11:06:29 (10 runs, 25 min).
### Reruns of the mp-flow runs (harness fixes, `matrix.log`)
- `both-r1` (first): the PO answered the grilling recap's branch question with an approval and the worker coded inside
`/grill-with-docs` (turn 4, 12 src/test files). Kept as `runs/both-r1-pofault` (tag `pofault`, ignored by analyze).
PO rule added (mp flow: before `/implement`, end with "Don't start implementing yet." when the worker offers to build).
- `both-r1` (second): the PO did not apply that rule and the worker coded inside `/grill-with-docs` again (turn 3, after
"say so and I'll start with failing tests"). The rule was made deterministic in `lib/driver.py`: in `/grill-with-docs`,
`/to-spec`, `/to-tickets`, the driver appends "Don't start implementing yet." to every PO answer. Since that changes
the conditions of all mp-flow runs, mp-r1..r3 and both-r1 were moved to `runs/*-po1` (tag `po1`; mp-r1..r3-po1 never
coded early, hidden 5/5 each, $2.22 to $2.57) and rerun with `matrix-mp2.txt` (11:18 to 11:40). The counted mp/both
runs are the reruns.
- sp stage labels: brainstorming's SKILL.md ends at the approved design, but in sp-r1 and sp-r3 the model then coded with
no new Skill call, so "skill in force" would bill the whole build to brainstorming. `lib/accounting.py` now switches the
stage to `implementation (no skill)` at the first main-thread Edit/Write under `src/` or `test/` while only a design
skill (`(none)`, `using-superpowers`, `brainstorming`, `writing-plans`) is in force; the sp runs were re-accounted from
their saved streams with `lib/reaccount.py` (tokens, USD and wall totals unchanged, only the split).
## Results
`results.md` / `results.json` (`python3 analyze.py`), reading in `FINDINGS.md`.
---
# Round 2 (2026-10-09 afternoon): `spd`, feature F2, isolation fix, full rerun
## Isolation leak found and fixed (all round-1 runs superseded)
The first `spd` smoke said "the most capable model your CLAUDE.md permits for subagents": the workers were reading
CLAUDE.md files. `--setting-sources project` does keep `~/.claude/CLAUDE.md` out (probe in an empty dir outside the
repo: "NONE"), but Claude Code also loads **every CLAUDE.md of the parent directories**, and the work copies lived under
the host repository, so every round-1 session carried that repository's CLAUDE.md and what it imports
(a probe in a round-1 work copy showed the host rules present; the round-1 mp code
reviews even cited them). Fix: `run.sh` builds the work copy in
`/private/tmp/flow-bench/<run>` (outside any CLAUDE.md tree) and archives it to `work/<run>` (without node_modules)
after finalize; probe in an mp copy there: only the pack's own "## Agent skills" CLAUDE.md block appears.
Every round-1 run was renamed `runs/*-leak` (tag `leak`, ignored by analyze) and the whole matrix rerun
(`matrix2.txt`). The short path also removes the "pocock-skins" typo surface (see Traps).
## `spd`: Superpowers, driven
Same project install as `sp` (skills in `.claude/skills/`, SessionStart hook). Installed in the project, the skills
resolve **without the plugin namespace**: init `slash_commands` lists `brainstorming`, `writing-plans`,
`executing-plans` (with `--plugin-dir` they are `superpowers:brainstorming`, ...; see `validation/routing-probes/`),
so the driver types `/brainstorming <task>` -> `/writing-plans` -> `/executing-plans`. writing-plans ends by asking
"Subagent-driven or Native?": the PO answers Native (added to both briefs: "If asked to choose how to execute a written
plan: Native (inline, in this session) ... working on main is fine: you consent"). Before `/executing-plans` the driver
appends "Don't start implementing yet." to PO answers (same rule as mp). A stage whose output is delivered and that
only asks for approval/go is `done` -> next command; after `/executing-plans` the run ends when the PO says done
(executing-plans itself runs the final whole-branch review and finishing-a-development-branch).
- Allow-list: the SPEC list + `Bash(*scripts/task-start*)`, `task-done`, `sdd-workspace`, `task-brief`,
`review-package` (executing-plans drives its loop with these pack scripts), and the `Bash(*../*)` deny is dropped for
spd (the skill calls `../subagent-driven-development/scripts/...`); the oracle stays behind the Read denies and
`Bash(*hidden*)`.
- Stages: per slash command (turn level, exact) AND per skill in force inside the turn (a typed slash command starts
its skill; later Skill events switch). Subagent spend is now its own segment `<dispatching stage> [subagent]`
(round 1 folded it into the dispatching stage); executing-plans' final reviewer is dispatched "on the most capable
available model": it ran on `claude-fable-5-1` in the spd smoke ($2.19 of $3.88).
- `sp` is unchanged and labelled **hook-routed** in results.
## F2: credit notes (`tasks/F2.md`, `po/F2-brief.md`, `hidden/test_F2.test.ts`, `reference/F2.patch`)
Request (2 lines): credit notes `POST/GET /invoices/:id/credit-notes`, numbered `CN-0001`..., and "voiding a paid
invoice should then work, by crediting whatever is left on it". Implicit requirements (hidden, one `it` each):
| req | category | checks | why a careless build misses it |
|---|---|---|---|
| R1 | cross-cutting invariant, 2 layers | credits never exceed the total cumulatively (domain) and `/reports/outstanding` owes total minus credits (report) | the report is not in the request |
| R2 | stored-data compat | a schemaVersion-1 file (no `creditNotes`, no credit-note counter): first note `CN-0001`, id not `NaN`, notes survive a reload; a legacy paid invoice can be voided | `migrate()` rebuilds only known keys and casts `counters`: a new counter is `undefined + 1` |
| R3 | ordering rule | one gapless CN sequence across invoices; rejected requests (over-credit, bad amount, draft, unknown) consume no number | the existing `/issue` route takes `nextId()` before validating: copying it burns numbers |
| R4 | error path, specific status | draft/void 409, unknown 404 on POST and GET, bad amounts 400, nothing recorded | |
| R5 | existing test changes meaning | void of a paid invoice: 200, credits only the remainder, void again 409; the 33 other upstream tests present and green and `test/invoice.test.ts` keeps >= 7 tests ("voids drafts and open invoices, never paid ones" must be rewritten, not deleted) | |
Validation (`validation.log`): reference 5/5, suite 39/39, tsc green; upstream 0/5, suite 34/34; bare smoke (isolated)
**4/5** (R3 failed: `CN-0004` where `CN-0002` was due, rejected requests burned numbers). A first F2 (payment terms,
`validation/f2-v1/`) was dropped: its bare smoke scored 5/5.
`hidden/upstream_test_titles.json` = the 34 upstream test titles (for R5). `hidden/vitest.config.ts` picks
`test_${TASK}.test.ts`. Run names: F1 `<cfg>-r<rep>`, F2 `f2-<cfg>-r<rep>`; `TASK=F2 ./run.sh <cfg> <rep>`.
## Matrix round 2 (`matrix2.txt`, 2 lanes, started 12:09:53)
F1: bare x3, sp x3, spd x3, mp x3, both x1. F2: bare x2, spd x2, mp x2. Same caps (12 driver turns, 60 max-turns,
40 min).
Done 13:0x (19 runs). Then `analyze.py` (results.md / results.json carry F1 and F2, five configs).
### After the matrix
- **F2 oracle v3.** Both F2 bare runs allowed credit notes on `paid` invoices only (409 on open ones), a reading the
request's framing ("paid invoices can't be voided, so finance needs credit notes") permits; oracle v2 ran R1-R3 on
open invoices, so that one choice failed three requirements. R1-R4 now run on paid invoices; credit notes on open
invoices and the report netting are out of the oracle (the brief still prefers them). v2 kept in
`validation/test_F2.v2-open-invoices.test.ts`; reference 5/5 and upstream 0/5 re-checked; the 7 F2 runs rescored
with `lib/finalize.py` (worker runs untouched). With v3: f2-bare-r1 4/5 (R3, numbers burned), every other F2 run 5/5.
- **spd permission denials:** 5 to 11 per run, mostly compound shell lines (`W=.superpowers/sdd/...; mkdir ...; cat ...`)
that match no allow rule; the pack scripts themselves ran (task-start/task-done in every spd run).
- **"pocock-skins" path typos** (Traps) concern round 1 only: round-2 work copies live at `/private/tmp/flow-bench/<run>`,
a path without the word `skills`; no stray file was found in the repo after round 2.
- Zero-cost `[subagent]` placeholder segments are skipped by analyze.py.
SPEC.md
# flow-bench - Superpowers loop vs Matt Pocock's Main Flow, same feature, one repo
Question the video must answer: on one small invented repo, for one bounded brownfield feature,
what does each pack cost PER STAGE (tokens, wall-clock, USD) and what does it deliver (hidden
acceptance tests passed, repo suite green, diff size, bugs shipped), against a bare Claude Code run?
Secondary question: does stacking both packs in one project break (hook / slash conflicts)?
## Fixed decisions (2026-10-09)
- Worker model: `claude-sonnet-5-5`, headless (`claude -p`), OAuth from `~/.claude` (see traps below).
- The repo `upstream/`: INVENTED, built here, a small TypeScript project (about 12-20 source files,
vitest, pnpm or npm, a green suite of ~25-40 tests): a plausible brownfield service, e.g. an
invoicing API or a booking CLI with a store layer, a domain layer and an HTTP or CLI layer. It is a
git repo with a clean history (3-6 commits), a short README and a `package.json` with
`test` and `typecheck` scripts. No framework exotica; plain Node 20 + TS + vitest.
- ONE feature task `tasks/F1.md`: a terse developer request (headline ask + one or two constraints,
2-4 lines, the way a dev types it), whose honest completion needs 5 requirements that a senior
would expect (edge cases, a validation, a migration of existing data, an error path, a test).
`hidden/test_F1.test.ts` is the oracle (5 requirement tests, never visible to the worker, run
from OUTSIDE the work copy against it). `reference/F1.patch` passes all 5 and the repo suite.
- Configs (`configs.json`):
- `bare`: no skills, no hooks.
- `sp`: obra/superpowers installed in the PROJECT (not the user's plugin): clone the repo at the
latest release tag (record it), expose its skills under `<work>/.claude/skills/` and its
SessionStart hook + any hooks it ships under `<work>/.claude/settings.json` exactly the way the
plugin wires them (read its `hooks/` and `plugin.json`; reproduce, don't improvise). Verify in a
smoke run that the `using-superpowers` text actually appears in the session (stream-json shows
the hook output) and that `brainstorming` is invoked.
- `mp`: mattpocock/skills installed in the project via the official installer
(`npx skills add mattpocock/skills` or the README's exact command, latest release, record the
version); verify the Main Flow slash commands resolve (`/grill-with-docs`, `/to-spec`,
`/to-tickets`, `/implement` or `/implement-spec`, `/code-review`, `/pr`, `/retro`; use the names
the installed version really has).
- `both`: `sp` + `mp` together (one or two runs only: the stacking question).
- Flows (the driver `run.sh <cfg> <rep>` / `lib/driver.py`):
- `bare`: one session, the task prompt, "when done, stop and summarise".
- `sp`: one session; turn 1 = the task prompt (the hook is supposed to route into brainstorming).
Then the Superpowers loop as it unfolds (brainstorming → writing-plans → executing-plans →
review); the driver only answers questions and says "go on / proceed with the plan" when the
assistant ends a turn waiting. Record which skills were invoked and when (Skill tool events).
- `mp`: one session, one stage per driver turn, in this order: `/grill-with-docs <task>` →
`/to-spec` → `/to-tickets` → the implement stage (`/implement` or `/implement-spec`, whichever
v1.3 ships for the loop over tickets) → `/code-review` → `/pr`. (`/retro` optional, one run.)
The driver answers questions between turns.
- Every turn is `claude -p … --resume <session_id> --output-format stream-json --verbose
--include-hook-events`; the first turn creates the session. Per TURN: tokens in/out/cache
read/cache write, USD (`result.total_cost_usd` deltas), wall-clock, tool calls, Skill
invocations, files touched. Per STAGE: the sum of its turns. A "stage" for `sp` is the skill in
force (from the Skill events), for `mp` the slash command of the turn.
- **Product-owner simulator** (`lib/po.py`): both packs interrogate the user. A headless
`claude-sonnet-5-5` with NO tools, `--setting-sources ""`, `CLAUDE_CODE_DISABLE_CLAUDE_MDS=1`,
reads `po/F1-brief.md` (the full intent, the 5 requirements in plain words, preferences and
constraints; never the tests) and answers the worker's questions tersely, as a busy dev would.
When the worker ends a turn without a question, the PO replies "go on" (or the next slash
command for `mp`). Hard caps: 12 driver turns per run, 60 `--max-turns` per claude turn, 40 min
wall per run. A run that hits a cap is recorded as such, not discarded.
- Reps: 3 per config for `bare`, `sp`, `mp`; 1 for `both`. 10 runs. Parallel 2.
- Scoring (`lib/finalize.py`): hidden 0-5, repo suite pass/fail, typecheck pass/fail, `git diff
--stat` lines added/removed, files touched, tests added, plus the per-stage table. `analyze.py`
→ `results.json` + `results.md` (per config: mean/median per metric; per stage: tokens, USD, wall).
- Validation BEFORE the matrix (`validate.sh`, logged): (1) `reference/F1.patch` → hidden 5/5 and
suite green; (2) upstream unchanged → hidden 0/5 (or the honest number, recorded) and suite green;
(3) one `bare` smoke run end to end with the PO loop exercised; (4) one `sp` smoke proving the
hook fired and brainstorming ran; (5) one `mp` smoke proving `/grill-with-docs` resolved. A
smoke run is tagged (`RUN_TAG=smoke`) and ignored by analyze.py.
## Traps already known (from a previous bench of the same kind)
- `claude -p --bare` cannot authenticate on an OAuth (Max plan) account: never use `--bare`. Use
`--setting-sources project` for workers so the user's own plugins (Superpowers 6.4.1 is installed
at user scope on this machine) never leak into a `bare` or `mp` run. VERIFY this isolation in the
smoke: a `bare` run must show no Superpowers hook output and no Skill events.
- Scrub inherited `CLAUDE*` env vars (copy `env.sh` from the previous bench), set
`CLAUDE_CODE_DISABLE_AUTO_MEMORY=1`.
- Fresh work copy per run at a unique path; `.claude/` excluded via `.git/info/exclude` so diffs
stay about the feature.
- A worker may try to read the hidden tests: the hidden dir lives outside the work copy and the
allow-list has no `find`/`ls` outside cwd.
- Allowed tools: Read,Edit,Write,MultiEdit,Glob,Grep,Skill,TodoWrite,Task,AskUserQuestion is NOT
available headless (questions come as plain text; the driver detects a turn that ends on a
question mark or an explicit "?"/options list and routes it to the PO); Bash limited to
node/npm/pnpm/npx vitest/tsc/git status|diff|log|add|commit|checkout -b/ls/cat/mkdir/head/tail/
grep/wc. Pocock's `/pr` wants `gh`: allow `Bash(gh:*)` but with no remote the PR step records
its output text as the artefact (no network push).
## Deliverables
`README.md` (how to run, layout, every trap met), `SPEC.md` (this), `upstream/`, `tasks/F1.md`,
`po/F1-brief.md`, `hidden/`, `reference/`, `configs.json`, `env.sh`, `prep.sh`, `run.sh`,
`validate.sh`, `matrix.sh`, `lib/{driver.py,po.py,finalize.py}`, `analyze.py`, `validation.log`,
`runs/<cfg>-r<rep>/` (stream.jsonl per turn, stages.json, score.json), `results.json`,
`results.md`, and `FINDINGS.md`: the honest reading in 300 words, every number with its run name.
FINDINGS.md
# flow-bench findings (2026-10-09, round 2: isolated work copies, Sonnet 5.5 headless, 19 counted runs)
**F1 (partial payments): quality did not separate.** All 13 F1 runs hidden 5/5, suite and tsc green.
Cost per run: bare $0.27 (bare-r2) to $0.47 (bare-r1); sp, hook-routed, $0.44 (sp-r1) to $0.54 (sp-r2); mp $1.80
(mp-r2) to $2.22 (mp-r1); spd, Superpowers driven, $3.39 (spd-r2) to $4.41 (spd-r3).
Wall: mp 365 to 480 s (mp-r2, mp-r3), spd 520 to 701 s (spd-r2, spd-r3).
**Where the money goes.** mp-r2 (complete flow): grill $0.16, spec $0.11, tickets $0.09, implement $0.96, code-review
$0.45, pr $0.04 (F1 means: implement $1.13, code-review $0.44). spd: brainstorming $0.17, writing-plans $0.27,
executing-plans $3.31 (F1 means). Inside executing-plans the build itself (TDD in session) costs $0.97 on average; the
rest is the one whole-branch reviewer the skill dispatches "on the most capable model", which ran on claude-fable-5-1
in all five spd runs: $2.01 of spd-r1's $3.41, $2.52 of spd-r3's $4.41, $1.57 of f2-spd-r2's $3.00.
**Hook vs driven.** Left to its hook, Superpowers ran brainstorming in sp-r1, sp-r2 and sp-r3 and then coded; writing-
plans ran 0 times in 3. Driven by slash commands, the full loop ran in every spd run and cost 7.4 times more (F1 means: spd $3.74, sp $0.50). The cost is the final review, not the planning.
**F2 (credit notes) separated, barely.** f2-bare-r1 shipped one missing requirement, R3: credit-note numbers are not
gapless. It took the number before validating, like the existing `/issue` route, so rejected requests burned numbers
(CN-0004 where CN-0002 was due; the F2 bare smoke failed the same way). f2-bare-r2, f2-spd-r1, f2-spd-r2, f2-mp-r1 and
f2-mp-r2 scored 5/5. Costs: bare $0.38/$0.45, mp $2.18/$1.95, spd $3.42/$3.00 (r1/r2). Caveat: the first F2 oracle
also required credit notes on open invoices; both bare runs allowed paid invoices only, which the request's wording
permits, so that check was removed and the F2 runs rescored.
**Stacking broke nothing.** both-r1: hook injected, no Superpowers skill invoked, $2.33, inside the mp range.
**Caps.** mp-r1, mp-r3 and f2-mp-r1 hit the 12-driver-turn cap before /pr. spd never needed more than 8 turns.
**Caveats.** Round-1 runs read this repo's CLAUDE.md (parent-directory loading), so they were discarded and rerun outside the repo. A simulated PO answers each round in one message. The SessionStart hook fires on turn 1 only.
/implement covers every ticket in one turn. 5 to 11 compound shell lines per spd run were denied. PO cost is excluded.
repo와 과제:
upstream/package.json
{
"name": "ledgerly",
"version": "0.4.0",
"private": true,
"description": "Small invoicing API: customers, invoices, reports. JSON-file store.",
"type": "module",
"engines": {
"node": ">=20"
},
"scripts": {
"test": "vitest run",
"typecheck": "tsc --noEmit"
},
"devDependencies": {
"@types/node": "^20.14.0",
"typescript": "^5.6.0",
"vitest": "^2.1.0"
}
}
upstream/package.json on GitHub
upstream/pnpm-lock.yaml
lockfileVersion: '9.0'
settings:
autoInstallPeers: true
excludeLinksFromLockfile: false
importers:
.:
devDependencies:
'@types/node':
specifier: ^20.14.0
version: 20.19.43
typescript:
specifier: ^5.6.0
version: 5.9.3
vitest:
specifier: ^2.1.0
version: 2.1.9(@types/node@20.19.43)
packages:
'@esbuild/aix-ppc64@0.21.5':
resolution: {integrity: sha512-1SDgH6ZSPTlggy1yI6+Dbkiz8xzpHJEVAlF/AM1tHPLsf5STom9rwtjE4hKAF20FfXXNTFqEYXyJNWh1GiZedQ==}
engines: {node: '>=12'}
cpu: [ppc64]
os: [aix]
'@esbuild/android-arm64@0.21.5':
resolution: {integrity: sha512-c0uX9VAUBQ7dTDCjq+wdyGLowMdtR/GoC2U5IYk/7D1H1JYC0qseD7+11iMP2mRLN9RcCMRcjC4YMclCzGwS/A==}
engines: {node: '>=12'}
cpu: [arm64]
os: [android]
'@esbuild/android-arm@0.21.5':
resolution: {integrity: sha512-vCPvzSjpPHEi1siZdlvAlsPxXl7WbOVUBBAowWug4rJHb68Ox8KualB+1ocNvT5fjv6wpkX6o/iEpbDrf68zcg==}
engines: {node: '>=12'}
cpu: [arm]
os: [android]
'@esbuild/android-x64@0.21.5':
resolution: {integrity: sha512-D7aPRUUNHRBwHxzxRvp856rjUHRFW1SdQATKXH2hqA0kAZb1hKmi02OpYRacl0TxIGz/ZmXWlbZgjwWYaCakTA==}
engines: {node: '>=12'}
cpu: [x64]
os: [android]
'@esbuild/darwin-arm64@0.21.5':
resolution: {integrity: sha512-DwqXqZyuk5AiWWf3UfLiRDJ5EDd49zg6O9wclZ7kUMv2WRFr4HKjXp/5t8JZ11QbQfUS6/cRCKGwYhtNAY88kQ==}
engines: {node: '>=12'}
cpu: [arm64]
os: [darwin]
'@esbuild/darwin-x64@0.21.5':
resolution: {integrity: sha512-se/JjF8NlmKVG4kNIuyWMV/22ZaerB+qaSi5MdrXtd6R08kvs2qCN4C09miupktDitvh8jRFflwGFBQcxZRjbw==}
engines: {node: '>=12'}
cpu: [x64]
os: [darwin]
'@esbuild/freebsd-arm64@0.21.5':
resolution: {integrity: sha512-5JcRxxRDUJLX8JXp/wcBCy3pENnCgBR9bN6JsY4OmhfUtIHe3ZW0mawA7+RDAcMLrMIZaf03NlQiX9DGyB8h4g==}
engines: {node: '>=12'}
cpu: [arm64]
os: [freebsd]
'@esbuild/freebsd-x64@0.21.5':
resolution: {integrity: sha512-J95kNBj1zkbMXtHVH29bBriQygMXqoVQOQYA+ISs0/2l3T9/kj42ow2mpqerRBxDJnmkUDCaQT/dfNXWX/ZZCQ==}
engines: {node: '>=12'}
cpu: [x64]
os: [freebsd]
'@esbuild/linux-arm64@0.21.5':
resolution: {integrity: sha512-ibKvmyYzKsBeX8d8I7MH/TMfWDXBF3db4qM6sy+7re0YXya+K1cem3on9XgdT2EQGMu4hQyZhan7TeQ8XkGp4Q==}
engines: {node: '>=12'}
cpu: [arm64]
os: [linux]
'@esbuild/linux-arm@0.21.5':
resolution: {integrity: sha512-bPb5AHZtbeNGjCKVZ9UGqGwo8EUu4cLq68E95A53KlxAPRmUyYv2D6F0uUI65XisGOL1hBP5mTronbgo+0bFcA==}
engines: {node: '>=12'}
cpu: [arm]
os: [linux]
'@esbuild/linux-ia32@0.21.5':
resolution: {integrity: sha512-YvjXDqLRqPDl2dvRODYmmhz4rPeVKYvppfGYKSNGdyZkA01046pLWyRKKI3ax8fbJoK5QbxblURkwK/MWY18Tg==}
engines: {node: '>=12'}
cpu: [ia32]
os: [linux]
'@esbuild/linux-loong64@0.21.5':
resolution: {integrity: sha512-uHf1BmMG8qEvzdrzAqg2SIG/02+4/DHB6a9Kbya0XDvwDEKCoC8ZRWI5JJvNdUjtciBGFQ5PuBlpEOXQj+JQSg==}
engines: {node: '>=12'}
cpu: [loong64]
os: [linux]
'@esbuild/linux-mips64el@0.21.5':
resolution: {integrity: sha512-IajOmO+KJK23bj52dFSNCMsz1QP1DqM6cwLUv3W1QwyxkyIWecfafnI555fvSGqEKwjMXVLokcV5ygHW5b3Jbg==}
engines: {node: '>=12'}
cpu: [mips64el]
os: [linux]
'@esbuild/linux-ppc64@0.21.5':
resolution: {integrity: sha512-1hHV/Z4OEfMwpLO8rp7CvlhBDnjsC3CttJXIhBi+5Aj5r+MBvy4egg7wCbe//hSsT+RvDAG7s81tAvpL2XAE4w==}
engines: {node: '>=12'}
cpu: [ppc64]
os: [linux]
'@esbuild/linux-riscv64@0.21.5':
resolution: {integrity: sha512-2HdXDMd9GMgTGrPWnJzP2ALSokE/0O5HhTUvWIbD3YdjME8JwvSCnNGBnTThKGEB91OZhzrJ4qIIxk/SBmyDDA==}
engines: {node: '>=12'}
cpu: [riscv64]
os: [linux]
'@esbuild/linux-s390x@0.21.5':
resolution: {integrity: sha512-zus5sxzqBJD3eXxwvjN1yQkRepANgxE9lgOW2qLnmr8ikMTphkjgXu1HR01K4FJg8h1kEEDAqDcZQtbrRnB41A==}
engines: {node: '>=12'}
cpu: [s390x]
os: [linux]
'@esbuild/linux-x64@0.21.5':
resolution: {integrity: sha512-1rYdTpyv03iycF1+BhzrzQJCdOuAOtaqHTWJZCWvijKD2N5Xu0TtVC8/+1faWqcP9iBCWOmjmhoH94dH82BxPQ==}
engines: {node: '>=12'}
cpu: [x64]
os: [linux]
'@esbuild/netbsd-x64@0.21.5':
resolution: {integrity: sha512-Woi2MXzXjMULccIwMnLciyZH4nCIMpWQAs049KEeMvOcNADVxo0UBIQPfSmxB3CWKedngg7sWZdLvLczpe0tLg==}
engines: {node: '>=12'}
cpu: [x64]
os: [netbsd]
'@esbuild/openbsd-x64@0.21.5':
resolution: {integrity: sha512-HLNNw99xsvx12lFBUwoT8EVCsSvRNDVxNpjZ7bPn947b8gJPzeHWyNVhFsaerc0n3TsbOINvRP2byTZ5LKezow==}
engines: {node: '>=12'}
cpu: [x64]
os: [openbsd]
'@esbuild/sunos-x64@0.21.5':
resolution: {integrity: sha512-6+gjmFpfy0BHU5Tpptkuh8+uw3mnrvgs+dSPQXQOv3ekbordwnzTVEb4qnIvQcYXq6gzkyTnoZ9dZG+D4garKg==}
engines: {node: '>=12'}
cpu: [x64]
os: [sunos]
'@esbuild/win32-arm64@0.21.5':
resolution: {integrity: sha512-Z0gOTd75VvXqyq7nsl93zwahcTROgqvuAcYDUr+vOv8uHhNSKROyU961kgtCD1e95IqPKSQKH7tBTslnS3tA8A==}
engines: {node: '>=12'}
cpu: [arm64]
os: [win32]
'@esbuild/win32-ia32@0.21.5':
resolution: {integrity: sha512-SWXFF1CL2RVNMaVs+BBClwtfZSvDgtL//G/smwAc5oVK/UPu2Gu9tIaRgFmYFFKrmg3SyAjSrElf0TiJ1v8fYA==}
engines: {node: '>=12'}
cpu: [ia32]
os: [win32]
'@esbuild/win32-x64@0.21.5':
resolution: {integrity: sha512-tQd/1efJuzPC6rCFwEvLtci/xNFcTZknmXs98FYDfGE4wP9ClFV98nyKrzJKVPMhdDnjzLhdUyMX4PsQAPjwIw==}
engines: {node: '>=12'}
cpu: [x64]
os: [win32]
'@jridgewell/sourcemap-codec@1.6.0':
resolution: {integrity: sha512-T7jf+5zgsZHwNJ4lvQ7/aezbyk0nNX+zJVWpmHA7VYsEx7a7qr5Rg5IbtJFqkgze5Y2sruq1RUY8Q837Od7iFw==}
'@napi-rs/lzma-linux-x64-gnu@1.5.1':
resolution: {integrity: sha512-oTXEIha4SsuXdTA4Iyskj0kpdx2yVXdhd75c2v3xGrHFfVMsbhTPZU/nMPL4sWKo4pBHm3aucLaqGlF696dTyQ==}
engines: {node: ^22.20 || ^24.12 || >=25}
cpu: [x64]
os: [linux]
libc: [glibc]
'@rollup/rollup-android-arm-eabi@4.64.2':
resolution: {integrity: sha512-gjNDUwMswogkw3fNqV00u54uZlgmInuw5EU/1TeqQ6K0bEagoGvqjPdKzy112lKyhe7PkGudZ/kSQHsc9qroIg==}
cpu: [arm]
os: [android]
'@rollup/rollup-android-arm64@4.64.2':
resolution: {integrity: sha512-YkAW8q7p5SCLSaGWaUkmsRnBEb3MGA1pB5nI/8k0lAK5fZsQKRBqkdR1h4UU9p8u3c7ZGdUF2Zcx7mK+E/pUKg==}
cpu: [arm64]
os: [android]
'@rollup/rollup-darwin-arm64@4.64.2':
resolution: {integrity: sha512-hNmuIHv+LnrP/+q/44vsmIMWfoh7X2YXOp5FLtjGuvqN52VdtFpOpnshTc1k28tWv1wE/0D5gLjsozBUowEGbQ==}
cpu: [arm64]
os: [darwin]
'@rollup/rollup-darwin-x64@4.64.2':
resolution: {integrity: sha512-B0t5CoxgF2KL7R5FHAsyN8yVEQCSIkh4XVs8MCDjuBLNjEa8l3zd5tVKaFMv53LfUtHs9p2rnqeavPTOunV+Ww==}
cpu: [x64]
os: [darwin]
'@rollup/rollup-freebsd-arm64@4.64.2':
resolution: {integrity: sha512-QQdYnfwrPmc1DoPlddjFvQPzUHFbStKXsHrG6EBYsWHhBVUCjsCRdcfGGXGAvhDpDtTzKK4vy1UrGFJ45rQaiw==}
cpu: [arm64]
os: [freebsd]
'@rollup/rollup-freebsd-x64@4.64.2':
resolution: {integrity: sha512-1xRxtGWif6BB58VSfMavi0Yfs2M4x6CtRp4o7ScY4W7AGlgvf6GSPKdDNdw7TX3tMiQwZY5W/mvyrOJTq/7AKA==}
cpu: [x64]
os: [freebsd]
'@rollup/rollup-linux-arm-gnueabihf@4.64.2':
resolution: {integrity: sha512-CXFLEt384BUwwOLR9DURWNHl1bQOVLd1ibOee+yKBJm/fZtvrSAOZ0JbxjAxszVMIovYMwFMS+oP+Q1aPcx8Eg==}
cpu: [arm]
os: [linux]
libc: [glibc]
'@rollup/rollup-linux-arm-musleabihf@4.64.2':
resolution: {integrity: sha512-T1dorY0WHm5mMNpMHFl5rYRZ02gLSZcr6J45r66tYb3S3R3DyIRGiGWVdLtF+zOQjratNJaObl9jYumueRiL1A==}
cpu: [arm]
os: [linux]
libc: [musl]
'@rollup/rollup-linux-arm64-gnu@4.64.2':
resolution: {integrity: sha512-ia6bfaCIc0ldp6bvs9oEkFrJVJKNUMoiDF0+LBZsQAZg+vE0znXnzGqH+ew/FYswE30cruAWPTO3dPw7keaVkQ==}
cpu: [arm64]
os: [linux]
libc: [glibc]
'@rollup/rollup-linux-arm64-musl@4.64.2':
resolution: {integrity: sha512-mB/gwfBfXtT0EBpbSssqKCqMFuX8b21Zy2TGoIhehSJEssZ5ksybxymwp1BkEM5Nczh/AAxpmh5q+MkhtundGg==}
cpu: [arm64]
os: [linux]
libc: [musl]
'@rollup/rollup-linux-loong64-gnu@4.64.2':
resolution: {integrity: sha512-VPd0a+XlHoNRI9jIPRHww2UKhDkD6XjbtcKYXZijAnhJS1jLLbiNtVl6a5LyEfNN+eyiyWcJOZXPzBo69oTsew==}
cpu: [loong64]
os: [linux]
libc: [glibc]
'@rollup/rollup-linux-loong64-musl@4.64.2':
resolution: {integrity: sha512-10N1I6oBpERqzGtHKcCLrEpHlbXSBbMK9TURHvqgWbIFZuU3oJROxJ0H2nHfq96TVI0XJHQ9qzolJMFd2tz0LA==}
cpu: [loong64]
os: [linux]
libc: [musl]
'@rollup/rollup-linux-ppc64-gnu@4.64.2':
resolution: {integrity: sha512-dhJrLVZ3wNNhwCmy9/VTg6Pq6J8a8eZ0n15dVRlbSWcDb+8V/1mDfbnvis+vt8W6Pbxzv6gUbTEl4piOT3bbUQ==}
cpu: [ppc64]
os: [linux]
libc: [glibc]
'@rollup/rollup-linux-ppc64-musl@4.64.2':
resolution: {integrity: sha512-tPd2QuwxkTxgju5eARyb3RdgBWWi/O9dn/NFH0VkvGbLzoQeXrH6qMABtW8IODb5lUeBzB3sy2U7qmI4MtQ72g==}
cpu: [ppc64]
os: [linux]
libc: [musl]
'@rollup/rollup-linux-riscv64-gnu@4.64.2':
resolution: {integrity: sha512-njtDdPvSQWI9CDhIPPxsiEMeHayCfu7wwcbSQ7c2kYPhGVviL+WGVhq6Jfj/b+VuhSTkFuQ0z4ZFbFjkWwBFmg==}
cpu: [riscv64]
os: [linux]
libc: [glibc]
'@rollup/rollup-linux-riscv64-musl@4.64.2':
resolution: {integrity: sha512-EbOBo3dqCj9ecEV9Z7zvulB6AieMP2ztnKQYGVOSN0pzIyOu+3JH2YJZnLmbPeVvfn6yP6Qi7NbBM27EzGbj2w==}
cpu: [riscv64]
os: [linux]
libc: [musl]
'@rollup/rollup-linux-s390x-gnu@4.64.2':
resolution: {integrity: sha512-8wLt2Q5n4qmfiE2Gk8V6KNLSf+oJuBX7Kdw5joFDtnZIZvirS0hLzBc79OiJjV71oYDmBUZLNy6pDgvylo4qiQ==}
cpu: [s390x]
os: [linux]
libc: [glibc]
'@rollup/rollup-linux-x64-gnu@4.64.2':
resolution: {integrity: sha512-aIChbkhC4d33bxkVP0LSi2/2lUSGRXE6k4Qa0G7tgpgTOK7kxHTixmx5qC3SKUJ6yUMznHBtQoWaG4QMqLUnhw==}
cpu: [x64]
os: [linux]
libc: [glibc]
'@rollup/rollup-linux-x64-musl@4.64.2':
resolution: {integrity: sha512-gQ+TkVZRfjZVcl/UOAxxfWw+sHjzIXoieNwxEyJlD3tRBmBhR8ZDq11w+QEgQDcpslEB+fhN/OEpaTsDoIIHhQ==}
cpu: [x64]
os: [linux]
libc: [musl]
'@rollup/rollup-openbsd-x64@4.64.2':
resolution: {integrity: sha512-sUaJpMKS2JBCvUk8fYimG7aMq2jHVT4kw/uVHxPolKLrMmkKy0KXirmZyZehdNpL7PffqAlPAWUELQcTiD8ChA==}
cpu: [x64]
os: [openbsd]
'@rollup/rollup-openharmony-arm64@4.64.2':
resolution: {integrity: sha512-j1RdJWb0CrlSuwt12OPjLucPlwfLtA/KMOKVvh5rXSJEtl/WDoIGR3CSykK/jLGTo9RqNfW5Ub6kY5Z5ccYl9w==}
cpu: [arm64]
os: [openharmony]
'@rollup/rollup-win32-arm64-msvc@4.64.2':
resolution: {integrity: sha512-i3r04aHCIY3eggTBLUp/4IxiBjDpAgQaK89Ue2nMJ2MKuIXX5JfoeZjHw+kPjGGNVDlhi8FT8xzuGMJZe3aehg==}
cpu: [arm64]
os: [win32]
'@rollup/rollup-win32-ia32-msvc@4.64.2':
resolution: {integrity: sha512-kCQ15h/uwNMGdN8gjXBZGf+aGE+ODzHfbi90/cAm4+Q89WoMiLODNoj8HoGZmYx87HPxfu5CBNPsZ+ZG4uIAoA==}
cpu: [ia32]
os: [win32]
'@rollup/rollup-win32-x64-gnu@4.64.2':
resolution: {integrity: sha512-yYAuySSs+0X2gSMJPw/Q4jHsHZHyMq1dpAE+XnN790Jg1WuReUAD/+j+e4vuDvS4PXJ4PA4NCzTR4OYtDf18/g==}
cpu: [x64]
os: [win32]
'@rollup/rollup-win32-x64-msvc@4.64.2':
resolution: {integrity: sha512-JY1rQdWOJrKutORqKO1uzrHXV86jFiQzfLFKx2s8S1CjLXL9fC5LOIvk0q7Z7PXJvsByt+sdoE1Q2a9gBr4gLQ==}
cpu: [x64]
os: [win32]
'@types/estree@1.0.9':
resolution: {integrity: sha512-GhdPgy1el4/ImP05X05Uw4cw2/M93BCUmnEvWZNStlCzEKME4Fkk+YpoA5OiHNQmoS7Cafb8Xa3Pya8m1Qrzeg==}
'@types/node@20.19.43':
resolution: {integrity: sha512-6oYBAi5ikg4Pl+kGsoYtawUMBT2zZMCvPNF7pVLnHZfd1zf38DRiWn/gT01RYCdUqkv7Fhr+C9ot4/tb+2sVvA==}
'@vitest/expect@2.1.9':
resolution: {integrity: sha512-UJCIkTBenHeKT1TTlKMJWy1laZewsRIzYighyYiJKZreqtdxSos/S1t+ktRMQWu2CKqaarrkeszJx1cgC5tGZw==}
'@vitest/mocker@2.1.9':
resolution: {integrity: sha512-tVL6uJgoUdi6icpxmdrn5YNo3g3Dxv+IHJBr0GXHaEdTcw3F+cPKnsXFhli6nO+f/6SDKPHEK1UN+k+TQv0Ehg==}
peerDependencies:
msw: ^2.4.9
vite: ^5.0.0
peerDependenciesMeta:
msw:
optional: true
vite:
optional: true
'@vitest/pretty-format@2.1.9':
resolution: {integrity: sha512-KhRIdGV2U9HOUzxfiHmY8IFHTdqtOhIzCpd8WRdJiE7D/HUcZVD0EgQCVjm+Q9gkUXWgBvMmTtZgIG48wq7sOQ==}
'@vitest/runner@2.1.9':
resolution: {integrity: sha512-ZXSSqTFIrzduD63btIfEyOmNcBmQvgOVsPNPe0jYtESiXkhd8u2erDLnMxmGrDCwHCCHE7hxwRDCT3pt0esT4g==}
'@vitest/snapshot@2.1.9':
resolution: {integrity: sha512-oBO82rEjsxLNJincVhLhaxxZdEtV0EFHMK5Kmx5sJ6H9L183dHECjiefOAdnqpIgT5eZwT04PoggUnW88vOBNQ==}
'@vitest/spy@2.1.9':
resolution: {integrity: sha512-E1B35FwzXXTs9FHNK6bDszs7mtydNi5MIfUWpceJ8Xbfb1gBMscAnwLbEu+B44ed6W3XjL9/ehLPHR1fkf1KLQ==}
'@vitest/utils@2.1.9':
resolution: {integrity: sha512-v0psaMSkNJ3A2NMrUEHFRzJtDPFn+/VWZ5WxImB21T9fjucJRmS7xCS3ppEnARb9y11OAzaD+P2Ps+b+BGX5iQ==}
assertion-error@2.0.1:
resolution: {integrity: sha512-Izi8RQcffqCeNVgFigKli1ssklIbpHnCYc6AknXGYoB6grJqyeby7jv12JUQgmTAnIDnbck1uxksT4dzN3PWBA==}
engines: {node: '>=12'}
cac@6.7.14:
resolution: {integrity: sha512-b6Ilus+c3RrdDk+JhLKUAQfzzgLEPy6wcXqS7f/xe1EETvsDP6GORG7SFuOs6cID5YkqchW/LXZbX5bc8j7ZcQ==}
engines: {node: '>=8'}
chai@5.3.3:
resolution: {integrity: sha512-4zNhdJD/iOjSH0A05ea+Ke6MU5mmpQcbQsSOkgdaUMJ9zTlDTD/GYlwohmIE2u0gaxHYiVHEn1Fw9mZ/ktJWgw==}
engines: {node: '>=18'}
check-error@2.1.3:
resolution: {integrity: sha512-PAJdDJusoxnwm1VwW07VWwUN1sl7smmC3OKggvndJFadxxDRyFJBX/ggnu/KE4kQAB7a3Dp8f/YXC1FlUprWmA==}
engines: {node: '>= 16'}
debug@4.4.3:
resolution: {integrity: sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA==}
engines: {node: '>=6.0'}
peerDependencies:
supports-color: '*'
peerDependenciesMeta:
supports-color:
optional: true
deep-eql@5.0.2:
resolution: {integrity: sha512-h5k/5U50IJJFpzfL6nO9jaaumfjO/f2NjK/oYB2Djzm4p9L+3T9qWpZqZ2hAbLPuuYq9wrU08WQyBTL5GbPk5Q==}
engines: {node: '>=6'}
es-module-lexer@1.7.0:
resolution: {integrity: sha512-jEQoCwk8hyb2AZziIOLhDqpm5+2ww5uIE6lkO/6jcOCusfk6LhMHpXXfBLXTZ7Ydyt0j4VoUQv6uGNYbdW+kBA==}
esbuild@0.21.5:
resolution: {integrity: sha512-mg3OPMV4hXywwpoDxu3Qda5xCKQi+vCTZq8S9J/EpkhB2HzKXq4SNFZE3+NK93JYxc8VMSep+lOUSC/RVKaBqw==}
engines: {node: '>=12'}
hasBin: true
estree-walker@3.0.3:
resolution: {integrity: sha512-7RUKfXgSMMkzt6ZuXmqapOurLGPPfgj6l9uRZ7lRGolvk0y2yocc35LdcxKC5PQZdn2DMqioAQ2NoWcrTKmm6g==}
expect-type@1.4.0:
resolution: {integrity: sha512-KfYbmpRm0VbLjEvVa9yGwCi9GI34xvi7A/HXYWQO65CSD2u3MczUJSuwXKFIxlGsgBQizV9q5J9NHj4VG0n+pA==}
engines: {node: '>=12.0.0'}
fsevents@2.3.3:
resolution: {integrity: sha512-5xoDfX+fL7faATnagmWPpbFtwh/R77WmMMqqHGS65C3vvB0YHrgF+B1YmZ3441tMj5n63k0212XNoJwzlhffQw==}
engines: {node: ^8.16.0 || ^10.6.0 || >=11.0.0}
os: [darwin]
loupe@3.2.1:
resolution: {integrity: sha512-CdzqowRJCeLU72bHvWqwRBBlLcMEtIvGrlvef74kMnV2AolS9Y8xUv1I0U/MNAWMhBlKIoyuEgoJ0t/bbwHbLQ==}
magic-string@0.30.21:
resolution: {integrity: sha512-vd2F4YUyEXKGcLHoq+TEyCjxueSeHnFxyyjNp80yg0XV4vUhnDer/lvvlqM/arB5bXQN5K2/3oinyCRyx8T2CQ==}
ms@2.1.3:
resolution: {integrity: sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==}
nanoid@3.3.20:
resolution: {integrity: sha512-uKdg2G3GNCKQn9byYOpxbGqrT2fGO5KRt5J/8b3pok8rT6qxGWF6hxMyJiEYtAf+FVyYuD9hRaDqX5uPFYJ4ZQ==}
engines: {node: ^10 || ^12 || ^13.7 || ^14 || >=15.0.1}
hasBin: true
pathe@1.1.2:
resolution: {integrity: sha512-whLdWMYL2TwI08hn8/ZqAbrVemu0LNaNNJZX73O6qaIdCTfXutsLhMkjdENX0qhsQ9uIimo4/aQOmXkoon2nDQ==}
pathval@2.0.1:
resolution: {integrity: sha512-//nshmD55c46FuFw26xV/xFAaB5HF9Xdap7HJBBnrKdAd6/GxDBaNA1870O79+9ueg61cZLSVc+OaFlfmObYVQ==}
engines: {node: '>= 14.16'}
picocolors@1.1.1:
resolution: {integrity: sha512-xceH2snhtb5M9liqDsmEw56le376mTZkEX/jEb/RxNFyegNul7eNslCXP9FDj/Lcu0X8KEyMceP2ntpaHrDEVA==}
postcss@8.5.29:
resolution: {integrity: sha512-49cGhUbXj8Qenv0iTMxA1cFBzxXoctpC9Ujd77t1WcbJIr6nF/eI7g/8MgxrYldFRuAXvja7xQRwavoW7kgrxQ==}
engines: {node: ^10 || ^12 || >=14}
rollup@4.64.2:
resolution: {integrity: sha512-RxXm0XVVV1hFI3dYe12ruaOgsYQ725lUcUWeMSQqkF9tTxVErY+fy2Ets6vWhItL9MZMSNZGbvZFc7rH/VHp2Q==}
engines: {node: '>=18.0.0', npm: '>=8.0.0'}
hasBin: true
siginfo@2.0.0:
resolution: {integrity: sha512-ybx0WO1/8bSBLEWXZvEd7gMW3Sn3JFlW3TvX1nREbDLRNQNaeNN8WK0meBwPdAaOI7TtRRRJn/Es1zhrrCHu7g==}
source-map-js@1.2.2:
resolution: {integrity: sha512-KGj/8Y43x35aZVDtt+J4mK1hoLGHULMYfSkODJNQjNDC3oW1PqPoxMwo0pLUsWM/UEGzON/NxeHywEfNXNP3Vw==}
engines: {node: '>=0.10.0'}
stackback@0.0.2:
resolution: {integrity: sha512-1XMJE5fQo1jGH6Y/7ebnwPOBEkIEnT4QF32d5R1+VXdXveM0IBMJt8zfaxX1P3QhVwrYe+576+jkANtSS2mBbw==}
std-env@3.10.0:
resolution: {integrity: sha512-5GS12FdOZNliM5mAOxFRg7Ir0pWz8MdpYm6AY6VPkGpbA7ZzmbzNcBJQ0GPvvyWgcY7QAhCgf9Uy89I03faLkg==}
tinybench@2.9.0:
resolution: {integrity: sha512-0+DUvqWMValLmha6lr4kD8iAMK1HzV0/aKnCtWb9v9641TnP/MFb7Pc2bxoxQjTXAErryXVgUOfv2YqNllqGeg==}
tinyexec@0.3.2:
resolution: {integrity: sha512-KQQR9yN7R5+OSwaK0XQoj22pwHoTlgYqmUscPYoknOoWCWfj/5/ABTMRi69FrKU5ffPVh5QcFikpWJI/P1ocHA==}
tinypool@1.1.1:
resolution: {integrity: sha512-Zba82s87IFq9A9XmjiX5uZA/ARWDrB03OHlq+Vw1fSdt0I+4/Kutwy8BP4Y/y/aORMo61FQ0vIb5j44vSo5Pkg==}
engines: {node: ^18.0.0 || >=20.0.0}
tinyrainbow@1.2.0:
resolution: {integrity: sha512-weEDEq7Z5eTHPDh4xjX789+fHfF+P8boiFB+0vbWzpbnbsEr/GRaohi/uMKxg8RZMXnl1ItAi/IUHWMsjDV7kQ==}
engines: {node: '>=14.0.0'}
tinyspy@3.0.2:
resolution: {integrity: sha512-n1cw8k1k0x4pgA2+9XrOkFydTerNcJ1zWCO5Nn9scWHTD+5tp8dghT2x1uduQePZTZgd3Tupf+x9BxJjeJi77Q==}
engines: {node: '>=14.0.0'}
typescript@5.9.3:
resolution: {integrity: sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw==}
engines: {node: '>=14.17'}
hasBin: true
undici-types@6.21.0:
resolution: {integrity: sha512-iwDZqg0QAGrg9Rav5H4n0M64c3mkR59cJ6wQp+7C4nI0gsmExaedaYLNO44eT4AtBBwjbTiGPMlt2Md0T9H9JQ==}
vite-node@2.1.9:
resolution: {integrity: sha512-AM9aQ/IPrW/6ENLQg3AGY4K1N2TGZdR5e4gu/MmmR2xR3Ll1+dib+nook92g4TV3PXVyeyxdWwtaCAiUL0hMxA==}
engines: {node: ^18.0.0 || >=20.0.0}
hasBin: true
vite@5.4.21:
resolution: {integrity: sha512-o5a9xKjbtuhY6Bi5S3+HvbRERmouabWbyUcpXXUA1u+GNUKoROi9byOJ8M0nHbHYHkYICiMlqxkg1KkYmm25Sw==}
engines: {node: ^18.0.0 || >=20.0.0}
hasBin: true
peerDependencies:
'@types/node': ^18.0.0 || >=20.0.0
less: '*'
lightningcss: ^1.21.0
sass: '*'
sass-embedded: '*'
stylus: '*'
sugarss: '*'
terser: ^5.4.0
peerDependenciesMeta:
'@types/node':
optional: true
less:
optional: true
lightningcss:
optional: true
sass:
optional: true
sass-embedded:
optional: true
stylus:
optional: true
sugarss:
optional: true
terser:
optional: true
vitest@2.1.9:
resolution: {integrity: sha512-MSmPM9REYqDGBI8439mA4mWhV5sKmDlBKWIYbA3lRb2PTHACE0mgKwA8yQ2xq9vxDTuk4iPrECBAEW2aoFXY0Q==}
engines: {node: ^18.0.0 || >=20.0.0}
hasBin: true
peerDependencies:
'@edge-runtime/vm': '*'
'@types/node': ^18.0.0 || >=20.0.0
'@vitest/browser': 2.1.9
'@vitest/ui': 2.1.9
happy-dom: '*'
jsdom: '*'
peerDependenciesMeta:
'@edge-runtime/vm':
optional: true
'@types/node':
optional: true
'@vitest/browser':
optional: true
'@vitest/ui':
optional: true
happy-dom:
optional: true
jsdom:
optional: true
why-is-node-running@2.3.0:
resolution: {integrity: sha512-hUrmaWBdVDcxvYqnyh09zunKzROWjbZTiNy8dBEjkS7ehEDQibXJ7XvlmtbwuTclUiIyN+CyXQD4Vmko8fNm8w==}
engines: {node: '>=8'}
hasBin: true
snapshots:
'@esbuild/aix-ppc64@0.21.5':
optional: true
'@esbuild/android-arm64@0.21.5':
optional: true
'@esbuild/android-arm@0.21.5':
optional: true
'@esbuild/android-x64@0.21.5':
optional: true
'@esbuild/darwin-arm64@0.21.5':
optional: true
'@esbuild/darwin-x64@0.21.5':
optional: true
'@esbuild/freebsd-arm64@0.21.5':
optional: true
'@esbuild/freebsd-x64@0.21.5':
optional: true
'@esbuild/linux-arm64@0.21.5':
optional: true
'@esbuild/linux-arm@0.21.5':
optional: true
'@esbuild/linux-ia32@0.21.5':
optional: true
'@esbuild/linux-loong64@0.21.5':
optional: true
'@esbuild/linux-mips64el@0.21.5':
optional: true
'@esbuild/linux-ppc64@0.21.5':
optional: true
'@esbuild/linux-riscv64@0.21.5':
optional: true
'@esbuild/linux-s390x@0.21.5':
optional: true
'@esbuild/linux-x64@0.21.5':
optional: true
'@esbuild/netbsd-x64@0.21.5':
optional: true
'@esbuild/openbsd-x64@0.21.5':
optional: true
'@esbuild/sunos-x64@0.21.5':
optional: true
'@esbuild/win32-arm64@0.21.5':
optional: true
'@esbuild/win32-ia32@0.21.5':
optional: true
'@esbuild/win32-x64@0.21.5':
optional: true
'@jridgewell/sourcemap-codec@1.6.0': {}
'@napi-rs/lzma-linux-x64-gnu@1.5.1':
optional: true
'@rollup/rollup-android-arm-eabi@4.64.2':
optional: true
'@rollup/rollup-android-arm64@4.64.2':
optional: true
'@rollup/rollup-darwin-arm64@4.64.2':
optional: true
'@rollup/rollup-darwin-x64@4.64.2':
optional: true
'@rollup/rollup-freebsd-arm64@4.64.2':
optional: true
'@rollup/rollup-freebsd-x64@4.64.2':
optional: true
'@rollup/rollup-linux-arm-gnueabihf@4.64.2':
optional: true
'@rollup/rollup-linux-arm-musleabihf@4.64.2':
optional: true
'@rollup/rollup-linux-arm64-gnu@4.64.2':
optional: true
'@rollup/rollup-linux-arm64-musl@4.64.2':
optional: true
'@rollup/rollup-linux-loong64-gnu@4.64.2':
optional: true
'@rollup/rollup-linux-loong64-musl@4.64.2':
optional: true
'@rollup/rollup-linux-ppc64-gnu@4.64.2':
optional: true
'@rollup/rollup-linux-ppc64-musl@4.64.2':
optional: true
'@rollup/rollup-linux-riscv64-gnu@4.64.2':
optional: true
'@rollup/rollup-linux-riscv64-musl@4.64.2':
optional: true
'@rollup/rollup-linux-s390x-gnu@4.64.2':
optional: true
'@rollup/rollup-linux-x64-gnu@4.64.2':
optional: true
'@rollup/rollup-linux-x64-musl@4.64.2':
optional: true
'@rollup/rollup-openbsd-x64@4.64.2':
optional: true
'@rollup/rollup-openharmony-arm64@4.64.2':
optional: true
'@rollup/rollup-win32-arm64-msvc@4.64.2':
optional: true
'@rollup/rollup-win32-ia32-msvc@4.64.2':
optional: true
'@rollup/rollup-win32-x64-gnu@4.64.2':
optional: true
'@rollup/rollup-win32-x64-msvc@4.64.2':
optional: true
'@types/estree@1.0.9': {}
'@types/node@20.19.43':
dependencies:
undici-types: 6.21.0
'@vitest/expect@2.1.9':
dependencies:
'@vitest/spy': 2.1.9
'@vitest/utils': 2.1.9
chai: 5.3.3
tinyrainbow: 1.2.0
'@vitest/mocker@2.1.9(vite@5.4.21(@types/node@20.19.43))':
dependencies:
'@vitest/spy': 2.1.9
estree-walker: 3.0.3
magic-string: 0.30.21
optionalDependencies:
vite: 5.4.21(@types/node@20.19.43)
'@vitest/pretty-format@2.1.9':
dependencies:
tinyrainbow: 1.2.0
'@vitest/runner@2.1.9':
dependencies:
'@vitest/utils': 2.1.9
pathe: 1.1.2
'@vitest/snapshot@2.1.9':
dependencies:
'@vitest/pretty-format': 2.1.9
magic-string: 0.30.21
pathe: 1.1.2
'@vitest/spy@2.1.9':
dependencies:
tinyspy: 3.0.2
'@vitest/utils@2.1.9':
dependencies:
'@vitest/pretty-format': 2.1.9
loupe: 3.2.1
tinyrainbow: 1.2.0
assertion-error@2.0.1: {}
cac@6.7.14: {}
chai@5.3.3:
dependencies:
assertion-error: 2.0.1
check-error: 2.1.3
deep-eql: 5.0.2
loupe: 3.2.1
pathval: 2.0.1
check-error@2.1.3: {}
debug@4.4.3:
dependencies:
ms: 2.1.3
deep-eql@5.0.2: {}
es-module-lexer@1.7.0: {}
esbuild@0.21.5:
optionalDependencies:
'@esbuild/aix-ppc64': 0.21.5
'@esbuild/android-arm': 0.21.5
'@esbuild/android-arm64': 0.21.5
'@esbuild/android-x64': 0.21.5
'@esbuild/darwin-arm64': 0.21.5
'@esbuild/darwin-x64': 0.21.5
'@esbuild/freebsd-arm64': 0.21.5
'@esbuild/freebsd-x64': 0.21.5
'@esbuild/linux-arm': 0.21.5
'@esbuild/linux-arm64': 0.21.5
'@esbuild/linux-ia32': 0.21.5
'@esbuild/linux-loong64': 0.21.5
'@esbuild/linux-mips64el': 0.21.5
'@esbuild/linux-ppc64': 0.21.5
'@esbuild/linux-riscv64': 0.21.5
'@esbuild/linux-s390x': 0.21.5
'@esbuild/linux-x64': 0.21.5
'@esbuild/netbsd-x64': 0.21.5
'@esbuild/openbsd-x64': 0.21.5
'@esbuild/sunos-x64': 0.21.5
'@esbuild/win32-arm64': 0.21.5
'@esbuild/win32-ia32': 0.21.5
'@esbuild/win32-x64': 0.21.5
estree-walker@3.0.3:
dependencies:
'@types/estree': 1.0.9
expect-type@1.4.0: {}
fsevents@2.3.3:
optional: true
loupe@3.2.1: {}
magic-string@0.30.21:
dependencies:
'@jridgewell/sourcemap-codec': 1.6.0
ms@2.1.3: {}
nanoid@3.3.20: {}
pathe@1.1.2: {}
pathval@2.0.1: {}
picocolors@1.1.1: {}
postcss@8.5.29:
dependencies:
nanoid: 3.3.20
picocolors: 1.1.1
source-map-js: 1.2.2
rollup@4.64.2:
dependencies:
'@types/estree': 1.0.9
optionalDependencies:
'@napi-rs/lzma-linux-x64-gnu': 1.5.1
'@rollup/rollup-android-arm-eabi': 4.64.2
'@rollup/rollup-android-arm64': 4.64.2
'@rollup/rollup-darwin-arm64': 4.64.2
'@rollup/rollup-darwin-x64': 4.64.2
'@rollup/rollup-freebsd-arm64': 4.64.2
'@rollup/rollup-freebsd-x64': 4.64.2
'@rollup/rollup-linux-arm-gnueabihf': 4.64.2
'@rollup/rollup-linux-arm-musleabihf': 4.64.2
'@rollup/rollup-linux-arm64-gnu': 4.64.2
'@rollup/rollup-linux-arm64-musl': 4.64.2
'@rollup/rollup-linux-loong64-gnu': 4.64.2
'@rollup/rollup-linux-loong64-musl': 4.64.2
'@rollup/rollup-linux-ppc64-gnu': 4.64.2
'@rollup/rollup-linux-ppc64-musl': 4.64.2
'@rollup/rollup-linux-riscv64-gnu': 4.64.2
'@rollup/rollup-linux-riscv64-musl': 4.64.2
'@rollup/rollup-linux-s390x-gnu': 4.64.2
'@rollup/rollup-linux-x64-gnu': 4.64.2
'@rollup/rollup-linux-x64-musl': 4.64.2
'@rollup/rollup-openbsd-x64': 4.64.2
'@rollup/rollup-openharmony-arm64': 4.64.2
'@rollup/rollup-win32-arm64-msvc': 4.64.2
'@rollup/rollup-win32-ia32-msvc': 4.64.2
'@rollup/rollup-win32-x64-gnu': 4.64.2
'@rollup/rollup-win32-x64-msvc': 4.64.2
fsevents: 2.3.3
siginfo@2.0.0: {}
source-map-js@1.2.2: {}
stackback@0.0.2: {}
std-env@3.10.0: {}
tinybench@2.9.0: {}
tinyexec@0.3.2: {}
tinypool@1.1.1: {}
tinyrainbow@1.2.0: {}
tinyspy@3.0.2: {}
typescript@5.9.3: {}
undici-types@6.21.0: {}
vite-node@2.1.9(@types/node@20.19.43):
dependencies:
cac: 6.7.14
debug: 4.4.3
es-module-lexer: 1.7.0
pathe: 1.1.2
vite: 5.4.21(@types/node@20.19.43)
transitivePeerDependencies:
- '@types/node'
- less
- lightningcss
- sass
- sass-embedded
- stylus
- sugarss
- supports-color
- terser
vite@5.4.21(@types/node@20.19.43):
dependencies:
esbuild: 0.21.5
postcss: 8.5.29
rollup: 4.64.2
optionalDependencies:
'@types/node': 20.19.43
fsevents: 2.3.3
vitest@2.1.9(@types/node@20.19.43):
dependencies:
'@vitest/expect': 2.1.9
'@vitest/mocker': 2.1.9(vite@5.4.21(@types/node@20.19.43))
'@vitest/pretty-format': 2.1.9
'@vitest/runner': 2.1.9
'@vitest/snapshot': 2.1.9
'@vitest/spy': 2.1.9
'@vitest/utils': 2.1.9
chai: 5.3.3
debug: 4.4.3
expect-type: 1.4.0
magic-string: 0.30.21
pathe: 1.1.2
std-env: 3.10.0
tinybench: 2.9.0
tinyexec: 0.3.2
tinypool: 1.1.1
tinyrainbow: 1.2.0
vite: 5.4.21(@types/node@20.19.43)
vite-node: 2.1.9(@types/node@20.19.43)
why-is-node-running: 2.3.0
optionalDependencies:
'@types/node': 20.19.43
transitivePeerDependencies:
- less
- lightningcss
- msw
- sass
- sass-embedded
- stylus
- sugarss
- supports-color
- terser
why-is-node-running@2.3.0:
dependencies:
siginfo: 2.0.0
stackback: 0.0.2
upstream/pnpm-lock.yaml on GitHub
upstream/pnpm-workspace.yaml
allowBuilds:
esbuild: true
upstream/pnpm-workspace.yaml on GitHub
upstream/README.md
# ledgerly
Small invoicing API used by the back office: customers, invoices, an outstanding-balance report.
Plain Node 20 + TypeScript, no framework. Data lives in one JSON file (`LEDGERLY_DATA`, default `data/ledgerly.json`).
```bash
pnpm install
pnpm test # vitest
pnpm typecheck # tsc --noEmit
```
## Layout
- `src/domain/` - invoices, customers, money, validation, domain errors. No I/O.
- `src/store/` - the JSON-file store, repositories, and the schema migrations applied on load.
- `src/http/` - a tiny router, request handlers, error-to-status mapping. `createApp()` returns a
`handle(method, path, body)` function that the tests call directly; `src/server.ts` binds it to `node:http`.
## Conventions
- Money is always an integer number of cents. Never floats.
- Domain functions return new objects; they never mutate their input.
- Validation failures throw `ValidationError` (HTTP 400), missing records `NotFoundError` (404),
illegal state transitions `ConflictError` (409).
- The stored file carries a `schemaVersion`. Any change to the stored shape bumps it and adds a
migration in `src/store/migrations.ts`: production files are migrated on load, never by hand.
## Invoice lifecycle
`draft` → (`issue`) → `open` → (`pay`) → `paid`; `draft`/`open` → (`void`) → `void`.
upstream/src/domain/customer.ts
import { ValidationError } from "./errors";
export type Customer = {
id: string;
name: string;
email: string;
createdAt: string;
};
export type NewCustomerInput = { name: string; email: string };
const EMAIL = /^[^\s@]+@[^\s@]+\.[^\s@]+$/;
export function parseNewCustomer(body: unknown): NewCustomerInput {
if (typeof body !== "object" || body === null) {
throw new ValidationError("body must be an object");
}
const { name, email } = body as Record<string, unknown>;
if (typeof name !== "string" || name.trim() === "") {
throw new ValidationError("name is required", "name");
}
if (typeof email !== "string" || !EMAIL.test(email)) {
throw new ValidationError("email is invalid", "email");
}
return { name: name.trim(), email: email.toLowerCase() };
}
upstream/src/domain/customer.ts on GitHub
upstream/src/domain/errors.ts
export class DomainError extends Error {
constructor(message: string) {
super(message);
this.name = new.target.name;
}
}
/** Input that can never be valid (bad shape, out of range). Maps to HTTP 400. */
export class ValidationError extends DomainError {
constructor(
message: string,
readonly field?: string,
) {
super(message);
}
}
/** A record that does not exist. Maps to HTTP 404. */
export class NotFoundError extends DomainError {
constructor(entity: string, id: string) {
super(`${entity} ${id} not found`);
}
}
/** A valid request that the current state forbids (e.g. paying a void invoice). Maps to HTTP 409. */
export class ConflictError extends DomainError {}
upstream/src/domain/errors.ts on GitHub
upstream/src/domain/invoice.ts
import { ConflictError } from "./errors";
import { type Cents, type Currency, sum } from "./money";
export type InvoiceStatus = "draft" | "open" | "paid" | "void";
export type LineItem = {
description: string;
quantity: number;
/** cents */
unitPrice: Cents;
};
export type Invoice = {
id: string;
/** Assigned when the invoice is issued (INV-0001, ...). */
number: string | null;
customerId: string;
currency: Currency;
lines: LineItem[];
/** Flat discount in cents, never above the subtotal. */
discount: Cents;
status: InvoiceStatus;
createdAt: string;
issuedAt: string | null;
dueDate: string | null;
paidAt: string | null;
voidedAt: string | null;
};
export type InvoiceTotals = { subtotal: Cents; discount: Cents; total: Cents };
export function lineTotal(line: LineItem): Cents {
return line.quantity * line.unitPrice;
}
export function computeTotals(invoice: Pick<Invoice, "lines" | "discount">): InvoiceTotals {
const subtotal = sum(invoice.lines.map(lineTotal));
return { subtotal, discount: invoice.discount, total: subtotal - invoice.discount };
}
export function issue(invoice: Invoice, number: string, now: Date): Invoice {
if (invoice.status !== "draft") {
throw new ConflictError(`invoice ${invoice.id} is ${invoice.status}, only drafts can be issued`);
}
return { ...invoice, status: "open", number, issuedAt: now.toISOString() };
}
export function markPaid(invoice: Invoice, now: Date): Invoice {
if (invoice.status !== "open") {
throw new ConflictError(`invoice ${invoice.id} is ${invoice.status}, only open invoices can be paid`);
}
return { ...invoice, status: "paid", paidAt: now.toISOString() };
}
export function voidInvoice(invoice: Invoice, now: Date): Invoice {
if (invoice.status === "paid" || invoice.status === "void") {
throw new ConflictError(`invoice ${invoice.id} is ${invoice.status} and cannot be voided`);
}
return { ...invoice, status: "void", voidedAt: now.toISOString() };
}
export function isOverdue(invoice: Invoice, now: Date): boolean {
return invoice.status === "open" && invoice.dueDate !== null && invoice.dueDate < now.toISOString().slice(0, 10);
}
upstream/src/domain/invoice.ts on GitHub
upstream/src/domain/money.ts
import { ValidationError } from "./errors";
export const CURRENCIES = ["EUR", "USD", "GBP"] as const;
export type Currency = (typeof CURRENCIES)[number];
/** Amounts are integer cents. */
export type Cents = number;
export function isCurrency(value: unknown): value is Currency {
return typeof value === "string" && (CURRENCIES as readonly string[]).includes(value);
}
export function assertCents(value: unknown, field: string, opts: { min?: number } = {}): Cents {
if (typeof value !== "number" || !Number.isInteger(value)) {
throw new ValidationError(`${field} must be an integer number of cents`, field);
}
const min = opts.min ?? 0;
if (value < min) {
throw new ValidationError(`${field} must be >= ${min}`, field);
}
return value;
}
export function formatCents(amount: Cents, currency: Currency): string {
const sign = amount < 0 ? "-" : "";
const abs = Math.abs(amount);
return `${sign}${Math.floor(abs / 100)}.${String(abs % 100).padStart(2, "0")} ${currency}`;
}
export function sum(amounts: Cents[]): Cents {
return amounts.reduce((acc, a) => acc + a, 0);
}
upstream/src/domain/money.ts on GitHub
upstream/src/domain/validation.ts
import { ValidationError } from "./errors";
import { computeTotals, type LineItem } from "./invoice";
import { assertCents, type Currency, isCurrency } from "./money";
export type NewInvoiceInput = {
customerId: string;
currency: Currency;
lines: LineItem[];
discount: number;
dueDate: string | null;
};
const DATE = /^\d{4}-\d{2}-\d{2}$/;
export function parseLineItems(value: unknown): LineItem[] {
if (!Array.isArray(value) || value.length === 0) {
throw new ValidationError("lines must be a non-empty array", "lines");
}
return value.map((raw, i) => {
if (typeof raw !== "object" || raw === null) {
throw new ValidationError(`lines[${i}] must be an object`, `lines[${i}]`);
}
const { description, quantity, unitPrice } = raw as Record<string, unknown>;
if (typeof description !== "string" || description.trim() === "") {
throw new ValidationError(`lines[${i}].description is required`, `lines[${i}].description`);
}
if (typeof quantity !== "number" || !Number.isInteger(quantity) || quantity < 1) {
throw new ValidationError(`lines[${i}].quantity must be a positive integer`, `lines[${i}].quantity`);
}
return { description: description.trim(), quantity, unitPrice: assertCents(unitPrice, `lines[${i}].unitPrice`) };
});
}
export function parseNewInvoice(body: unknown): NewInvoiceInput {
if (typeof body !== "object" || body === null) {
throw new ValidationError("body must be an object");
}
const b = body as Record<string, unknown>;
if (typeof b.customerId !== "string" || b.customerId === "") {
throw new ValidationError("customerId is required", "customerId");
}
if (!isCurrency(b.currency)) {
throw new ValidationError("currency must be one of EUR, USD, GBP", "currency");
}
const lines = parseLineItems(b.lines);
const discount = b.discount === undefined ? 0 : assertCents(b.discount, "discount");
const { subtotal } = computeTotals({ lines, discount: 0 });
if (discount > subtotal) {
throw new ValidationError("discount cannot exceed the subtotal", "discount");
}
let dueDate: string | null = null;
if (b.dueDate !== undefined && b.dueDate !== null) {
if (typeof b.dueDate !== "string" || !DATE.test(b.dueDate)) {
throw new ValidationError("dueDate must be YYYY-MM-DD", "dueDate");
}
dueDate = b.dueDate;
}
return { customerId: b.customerId, currency: b.currency, lines, discount, dueDate };
}
upstream/src/domain/validation.ts on GitHub
upstream/src/http/app.ts
import { openStore, type Store } from "../store/json-store";
import { createContext } from "./context";
import { errorResponse } from "./errors";
import { customerRoutes } from "./handlers/customers";
import { invoiceRoutes } from "./handlers/invoices";
import { reportRoutes } from "./handlers/reports";
import { type Response, Router } from "./router";
export type App = {
handle(method: string, url: string, body?: unknown): Promise<Response>;
};
export function createApp(opts: { store?: Store; now?: () => Date } = {}): App {
const ctx = createContext(opts.store ?? openStore(), opts.now);
const router = new Router();
customerRoutes(router, ctx);
invoiceRoutes(router, ctx);
reportRoutes(router, ctx);
return {
async handle(method, url, body) {
const parsed = new URL(url, "http://local");
const match = router.match(method, parsed.pathname);
if (!match) return { status: 404, body: { error: { code: "not_found", message: `no route ${method} ${parsed.pathname}` } } };
try {
return await match.handler({
method: method as "GET",
path: parsed.pathname,
params: match.params,
query: parsed.searchParams,
body,
});
} catch (err) {
return errorResponse(err);
}
},
};
}
upstream/src/http/app.ts on GitHub
upstream/src/http/context.ts
import type { Store } from "../store/json-store";
import { customerRepo } from "../store/customer-repo";
import { invoiceRepo } from "../store/invoice-repo";
export type AppContext = {
store: Store;
now: () => Date;
customers: ReturnType<typeof customerRepo>;
invoices: ReturnType<typeof invoiceRepo>;
};
export function createContext(store: Store, now: () => Date = () => new Date()): AppContext {
return { store, now, customers: customerRepo(store), invoices: invoiceRepo(store) };
}
upstream/src/http/context.ts on GitHub
upstream/src/http/errors.ts
import { ConflictError, NotFoundError, ValidationError } from "../domain/errors";
import type { Response } from "./router";
export function errorResponse(err: unknown): Response {
if (err instanceof ValidationError) {
return { status: 400, body: { error: { code: "validation_error", message: err.message, field: err.field } } };
}
if (err instanceof NotFoundError) {
return { status: 404, body: { error: { code: "not_found", message: err.message } } };
}
if (err instanceof ConflictError) {
return { status: 409, body: { error: { code: "conflict", message: err.message } } };
}
return { status: 500, body: { error: { code: "internal", message: "internal error" } } };
}
upstream/src/http/errors.ts on GitHub
upstream/src/http/handlers/customers.ts
import { parseNewCustomer } from "../../domain/customer";
import type { Router } from "../router";
import type { AppContext } from "../context";
export function customerRoutes(router: Router, ctx: AppContext): void {
router.on("POST", "/customers", ({ body }) => {
const input = parseNewCustomer(body);
const id = `cus_${ctx.store.nextId("customer")}`;
const customer = ctx.customers.insert({ id, ...input, createdAt: ctx.now().toISOString() });
return { status: 201, body: customer };
});
router.on("GET", "/customers/:id", ({ params }) => ({ status: 200, body: ctx.customers.get(params.id) }));
}
upstream/src/http/handlers/customers.ts on GitHub
upstream/src/http/handlers/invoices.ts
import { ValidationError } from "../../domain/errors";
import { type Invoice, type InvoiceStatus, issue, markPaid, voidInvoice } from "../../domain/invoice";
import { parseNewInvoice } from "../../domain/validation";
import type { AppContext } from "../context";
import type { Router } from "../router";
import { invoiceView } from "../views";
const STATUSES: InvoiceStatus[] = ["draft", "open", "paid", "void"];
export function invoiceRoutes(router: Router, ctx: AppContext): void {
router.on("POST", "/invoices", ({ body }) => {
const input = parseNewInvoice(body);
if (!ctx.customers.exists(input.customerId)) {
throw new ValidationError(`customer ${input.customerId} does not exist`, "customerId");
}
const invoice: Invoice = {
id: `inv_${ctx.store.nextId("invoice")}`,
number: null,
...input,
status: "draft",
createdAt: ctx.now().toISOString(),
issuedAt: null,
paidAt: null,
voidedAt: null,
};
return { status: 201, body: invoiceView(ctx.invoices.insert(invoice)) };
});
router.on("GET", "/invoices", ({ query }) => {
const status = query.get("status") ?? undefined;
if (status !== undefined && !STATUSES.includes(status as InvoiceStatus)) {
throw new ValidationError(`unknown status ${status}`, "status");
}
const customerId = query.get("customerId") ?? undefined;
const list = ctx.invoices.list({ status: status as InvoiceStatus | undefined, customerId });
return { status: 200, body: list.map(invoiceView) };
});
router.on("GET", "/invoices/:id", ({ params }) => ({ status: 200, body: invoiceView(ctx.invoices.get(params.id)) }));
router.on("POST", "/invoices/:id/issue", ({ params }) => {
const current = ctx.invoices.get(params.id);
const n = ctx.store.nextId("invoiceNumber");
const issued = issue(current, `INV-${String(n).padStart(4, "0")}`, ctx.now());
return { status: 200, body: invoiceView(ctx.invoices.save(issued)) };
});
/** Marks an open invoice as paid in full (bank reconciliation does this today). */
router.on("POST", "/invoices/:id/pay", ({ params }) => {
const paid = markPaid(ctx.invoices.get(params.id), ctx.now());
return { status: 200, body: invoiceView(ctx.invoices.save(paid)) };
});
router.on("POST", "/invoices/:id/void", ({ params }) => {
const voided = voidInvoice(ctx.invoices.get(params.id), ctx.now());
return { status: 200, body: invoiceView(ctx.invoices.save(voided)) };
});
}
upstream/src/http/handlers/invoices.ts on GitHub
upstream/src/http/handlers/reports.ts
import { computeTotals, isOverdue } from "../../domain/invoice";
import type { Currency } from "../../domain/money";
import type { AppContext } from "../context";
import type { Router } from "../router";
export function reportRoutes(router: Router, ctx: AppContext): void {
/** What customers still owe us, per currency, over every open invoice. */
router.on("GET", "/reports/outstanding", () => {
const now = ctx.now();
const totals: Partial<Record<Currency, number>> = {};
let count = 0;
let overdue = 0;
for (const invoice of ctx.invoices.list({ status: "open" })) {
const { total } = computeTotals(invoice);
totals[invoice.currency] = (totals[invoice.currency] ?? 0) + total;
count += 1;
if (isOverdue(invoice, now)) overdue += 1;
}
return { status: 200, body: { totals, count, overdue } };
});
}
upstream/src/http/handlers/reports.ts on GitHub
upstream/src/http/router.ts
export type Method = "GET" | "POST" | "PUT" | "DELETE";
export type Request = {
method: Method;
path: string;
params: Record<string, string>;
query: URLSearchParams;
body: unknown;
};
export type Response = { status: number; body: unknown };
export type Handler = (req: Request) => Response | Promise<Response>;
type Route = { method: Method; pattern: RegExp; keys: string[]; handler: Handler };
export class Router {
private routes: Route[] = [];
on(method: Method, path: string, handler: Handler): this {
const keys: string[] = [];
const pattern = new RegExp(
"^" +
path.replace(/:([a-zA-Z]+)/g, (_, key: string) => {
keys.push(key);
return "([^/]+)";
}) +
"/?$",
);
this.routes.push({ method, pattern, keys, handler });
return this;
}
match(method: string, path: string): { handler: Handler; params: Record<string, string> } | null {
for (const route of this.routes) {
if (route.method !== method) continue;
const m = route.pattern.exec(path);
if (!m) continue;
const params: Record<string, string> = {};
route.keys.forEach((k, i) => (params[k] = decodeURIComponent(m[i + 1])));
return { handler: route.handler, params };
}
return null;
}
}
upstream/src/http/router.ts on GitHub
upstream/src/http/views.ts
import { computeTotals, type Invoice } from "../domain/invoice";
/** The JSON shape returned for an invoice: stored fields plus computed totals. */
export function invoiceView(invoice: Invoice) {
return { ...invoice, ...computeTotals(invoice) };
}
upstream/src/http/views.ts on GitHub
upstream/src/server.ts
import { createServer } from "node:http";
import { createApp } from "./http/app";
import { openStore } from "./store/json-store";
const file = process.env.LEDGERLY_DATA ?? "data/ledgerly.json";
const app = createApp({ store: openStore({ file }) });
createServer(async (req, res) => {
let raw = "";
for await (const chunk of req) raw += chunk;
let body: unknown;
if (raw) {
try {
body = JSON.parse(raw);
} catch {
res.writeHead(400, { "content-type": "application/json" });
res.end(JSON.stringify({ error: { code: "bad_json", message: "body is not valid JSON" } }));
return;
}
}
const out = await app.handle(req.method ?? "GET", req.url ?? "/", body);
res.writeHead(out.status, { "content-type": "application/json" });
res.end(JSON.stringify(out.body));
}).listen(Number(process.env.PORT ?? 3000));
upstream/src/server.ts on GitHub
upstream/src/store/customer-repo.ts
import type { Customer } from "../domain/customer";
import { NotFoundError } from "../domain/errors";
import type { Store } from "./json-store";
export function customerRepo(store: Store) {
return {
get(id: string): Customer {
const found = store.read().customers.find((c) => c.id === id);
if (!found) throw new NotFoundError("customer", id);
return found;
},
exists(id: string): boolean {
return store.read().customers.some((c) => c.id === id);
},
insert(customer: Customer): Customer {
store.update((d) => ({ ...d, customers: [...d.customers, customer] }));
return customer;
},
};
}
upstream/src/store/customer-repo.ts on GitHub
upstream/src/store/invoice-repo.ts
import { NotFoundError } from "../domain/errors";
import type { Invoice, InvoiceStatus } from "../domain/invoice";
import type { Store } from "./json-store";
export function invoiceRepo(store: Store) {
return {
get(id: string): Invoice {
const found = store.read().invoices.find((i) => i.id === id);
if (!found) throw new NotFoundError("invoice", id);
return found;
},
list(filter: { status?: InvoiceStatus; customerId?: string } = {}): Invoice[] {
return store
.read()
.invoices.filter((i) => (filter.status ? i.status === filter.status : true))
.filter((i) => (filter.customerId ? i.customerId === filter.customerId : true));
},
insert(invoice: Invoice): Invoice {
store.update((d) => ({ ...d, invoices: [...d.invoices, invoice] }));
return invoice;
},
save(invoice: Invoice): Invoice {
store.update((d) => ({ ...d, invoices: d.invoices.map((i) => (i.id === invoice.id ? invoice : i)) }));
return invoice;
},
};
}
upstream/src/store/invoice-repo.ts on GitHub
upstream/src/store/json-store.ts
import { existsSync, mkdirSync, readFileSync, renameSync, writeFileSync } from "node:fs";
import { dirname } from "node:path";
import { CURRENT_SCHEMA_VERSION, migrate } from "./migrations";
import type { Counters, StoreData } from "./types";
export type Store = {
read(): StoreData;
/** Apply a change and persist it (when file-backed). */
update(change: (data: StoreData) => StoreData): StoreData;
nextId(kind: keyof Counters): number;
};
function empty(): StoreData {
return {
schemaVersion: CURRENT_SCHEMA_VERSION,
counters: { customer: 0, invoice: 0, invoiceNumber: 0 },
customers: [],
invoices: [],
};
}
/**
* Open the store. With `file`, the JSON file is loaded and migrated to the current schema
* (and written back if a migration ran); every update is persisted with an atomic rename.
* Without `file`, the store is in memory (tests).
*/
export function openStore(opts: { file?: string } = {}): Store {
const { file } = opts;
let data: StoreData = empty();
if (file && existsSync(file)) {
const raw = JSON.parse(readFileSync(file, "utf8"));
data = migrate(raw);
if (raw.schemaVersion !== data.schemaVersion) persist(file, data);
}
function persist(path: string, value: StoreData): void {
mkdirSync(dirname(path), { recursive: true });
const tmp = `${path}.tmp`;
writeFileSync(tmp, JSON.stringify(value, null, 2));
renameSync(tmp, path);
}
return {
read: () => data,
update(change) {
data = change(data);
if (file) persist(file, data);
return data;
},
nextId(kind) {
let n = 0;
this.update((d) => {
n = d.counters[kind] + 1;
return { ...d, counters: { ...d.counters, [kind]: n } };
});
return n;
},
};
}
upstream/src/store/json-store.ts on GitHub
upstream/src/store/migrations.ts
import type { StoreData } from "./types";
export const CURRENT_SCHEMA_VERSION = 1;
type RawData = Record<string, unknown> & { schemaVersion?: number };
type Migration = (data: RawData) => RawData;
/**
* v0 -> v1: invoices embedded the customer (`customer: { id, name }`); v1 stores `customerId` only.
* Files written before schemaVersion existed are v0.
*/
const v0ToV1: Migration = (data) => {
const invoices = (data.invoices as Record<string, unknown>[] | undefined) ?? [];
return {
...data,
schemaVersion: 1,
invoices: invoices.map((inv) => {
const { customer, ...rest } = inv as { customer?: { id: string } } & Record<string, unknown>;
return customer ? { ...rest, customerId: customer.id } : rest;
}),
};
};
/** migrations[n] upgrades a file from version n to n + 1. */
const migrations: Record<number, Migration> = {
0: v0ToV1,
};
export function migrate(raw: unknown): StoreData {
if (typeof raw !== "object" || raw === null) {
throw new Error("store file is not a JSON object");
}
let data = raw as RawData;
let version = data.schemaVersion ?? 0;
if (version > CURRENT_SCHEMA_VERSION) {
throw new Error(`store file schemaVersion ${version} is newer than this build (${CURRENT_SCHEMA_VERSION})`);
}
while (version < CURRENT_SCHEMA_VERSION) {
const step = migrations[version];
if (!step) throw new Error(`no migration from schemaVersion ${version}`);
data = step(data);
version = data.schemaVersion as number;
}
return {
schemaVersion: CURRENT_SCHEMA_VERSION,
counters: (data.counters as StoreData["counters"]) ?? { customer: 0, invoice: 0, invoiceNumber: 0 },
customers: (data.customers as StoreData["customers"]) ?? [],
invoices: (data.invoices as StoreData["invoices"]) ?? [],
};
}
upstream/src/store/migrations.ts on GitHub
upstream/src/store/types.ts
import type { Customer } from "../domain/customer";
import type { Invoice } from "../domain/invoice";
export type Counters = { customer: number; invoice: number; invoiceNumber: number };
/** The shape of the JSON file at CURRENT_SCHEMA_VERSION. */
export type StoreData = {
schemaVersion: number;
counters: Counters;
customers: Customer[];
invoices: Invoice[];
};
upstream/src/store/types.ts on GitHub
upstream/test/helpers.ts
import { createApp } from "../src/http/app";
import { openStore, type Store } from "../src/store/json-store";
export const NOW = new Date("2026-03-10T09:00:00.000Z");
export function testApp(store: Store = openStore()) {
return createApp({ store, now: () => NOW });
}
type App = ReturnType<typeof testApp>;
export async function seedCustomer(app: App, name = "Acme"): Promise<string> {
const res = await app.handle("POST", "/customers", { name, email: `${name.toLowerCase()}@example.com` });
return (res.body as { id: string }).id;
}
/** Creates and issues an invoice; returns its id. Total = sum(qty * unitPrice) - discount. */
export async function seedOpenInvoice(
app: App,
opts: { customerId: string; unitPrice?: number; quantity?: number; discount?: number; currency?: string; dueDate?: string },
): Promise<string> {
const created = await app.handle("POST", "/invoices", {
customerId: opts.customerId,
currency: opts.currency ?? "EUR",
lines: [{ description: "Consulting", quantity: opts.quantity ?? 1, unitPrice: opts.unitPrice ?? 10_000 }],
discount: opts.discount,
dueDate: opts.dueDate,
});
const id = (created.body as { id: string }).id;
await app.handle("POST", `/invoices/${id}/issue`);
return id;
}
upstream/test/helpers.ts on GitHub
upstream/test/http-customers.test.ts
import { describe, expect, it } from "vitest";
import { testApp } from "./helpers";
describe("customers API", () => {
it("creates and fetches a customer", async () => {
const app = testApp();
const created = await app.handle("POST", "/customers", { name: "Acme", email: "billing@acme.io" });
expect(created.status).toBe(201);
const id = (created.body as { id: string }).id;
const fetched = await app.handle("GET", `/customers/${id}`);
expect(fetched).toMatchObject({ status: 200, body: { id, name: "Acme" } });
});
it("returns 400 on invalid input and 404 on unknown ids", async () => {
const app = testApp();
expect((await app.handle("POST", "/customers", { name: "" })).status).toBe(400);
expect((await app.handle("GET", "/customers/cus_404")).status).toBe(404);
});
});
upstream/test/http-customers.test.ts on GitHub
upstream/test/http-invoices.test.ts
import { describe, expect, it } from "vitest";
import { seedCustomer, seedOpenInvoice, testApp } from "./helpers";
describe("invoices API", () => {
it("creates a draft with computed totals", async () => {
const app = testApp();
const customerId = await seedCustomer(app);
const res = await app.handle("POST", "/invoices", {
customerId,
currency: "EUR",
lines: [{ description: "Audit", quantity: 3, unitPrice: 1_000 }],
discount: 500,
});
expect(res.status).toBe(201);
expect(res.body).toMatchObject({ status: "draft", number: null, subtotal: 3_000, discount: 500, total: 2_500 });
});
it("rejects an invoice for an unknown customer", async () => {
const app = testApp();
const res = await app.handle("POST", "/invoices", {
customerId: "cus_9",
currency: "EUR",
lines: [{ description: "Audit", quantity: 1, unitPrice: 1_000 }],
});
expect(res).toMatchObject({ status: 400, body: { error: { code: "validation_error", field: "customerId" } } });
});
it("issues with sequential numbers", async () => {
const app = testApp();
const customerId = await seedCustomer(app);
const a = await seedOpenInvoice(app, { customerId });
const b = await seedOpenInvoice(app, { customerId });
expect((await app.handle("GET", `/invoices/${a}`)).body).toMatchObject({ status: "open", number: "INV-0001" });
expect((await app.handle("GET", `/invoices/${b}`)).body).toMatchObject({ number: "INV-0002" });
});
it("marks an open invoice paid in full", async () => {
const app = testApp();
const id = await seedOpenInvoice(app, { customerId: await seedCustomer(app) });
const res = await app.handle("POST", `/invoices/${id}/pay`);
expect(res).toMatchObject({ status: 200, body: { status: "paid", paidAt: "2026-03-10T09:00:00.000Z" } });
});
it("returns 409 when paying a void invoice and 404 for unknown invoices", async () => {
const app = testApp();
const id = await seedOpenInvoice(app, { customerId: await seedCustomer(app) });
await app.handle("POST", `/invoices/${id}/void`);
expect((await app.handle("POST", `/invoices/${id}/pay`)).status).toBe(409);
expect((await app.handle("POST", "/invoices/inv_404/pay")).status).toBe(404);
});
it("filters the list by status and rejects unknown statuses", async () => {
const app = testApp();
const customerId = await seedCustomer(app);
const id = await seedOpenInvoice(app, { customerId });
await seedOpenInvoice(app, { customerId });
await app.handle("POST", `/invoices/${id}/pay`);
const paid = await app.handle("GET", "/invoices?status=paid");
expect((paid.body as unknown[]).length).toBe(1);
expect((await app.handle("GET", "/invoices?status=late")).status).toBe(400);
});
it("returns 404 for unknown routes", async () => {
expect((await testApp().handle("DELETE", "/invoices")).status).toBe(404);
});
});
upstream/test/http-invoices.test.ts on GitHub
upstream/test/invoice.test.ts
import { describe, expect, it } from "vitest";
import { ConflictError } from "../src/domain/errors";
import { computeTotals, type Invoice, isOverdue, issue, markPaid, voidInvoice } from "../src/domain/invoice";
const NOW = new Date("2026-03-10T09:00:00.000Z");
function draft(overrides: Partial<Invoice> = {}): Invoice {
return {
id: "inv_1",
number: null,
customerId: "cus_1",
currency: "EUR",
lines: [
{ description: "Design", quantity: 2, unitPrice: 5_000 },
{ description: "Hosting", quantity: 1, unitPrice: 1_500 },
],
discount: 500,
status: "draft",
createdAt: NOW.toISOString(),
issuedAt: null,
dueDate: null,
paidAt: null,
voidedAt: null,
...overrides,
};
}
describe("invoice domain", () => {
it("computes totals", () => {
expect(computeTotals(draft())).toEqual({ subtotal: 11_500, discount: 500, total: 11_000 });
});
it("issues a draft without mutating it", () => {
const d = draft();
const issued = issue(d, "INV-0001", NOW);
expect(issued.status).toBe("open");
expect(issued.number).toBe("INV-0001");
expect(d.status).toBe("draft");
});
it("refuses to issue twice", () => {
expect(() => issue(draft({ status: "open" }), "INV-0002", NOW)).toThrow(ConflictError);
});
it("marks an open invoice paid", () => {
const paid = markPaid(draft({ status: "open" }), NOW);
expect(paid.status).toBe("paid");
expect(paid.paidAt).toBe(NOW.toISOString());
});
it("refuses to pay drafts and void invoices", () => {
expect(() => markPaid(draft(), NOW)).toThrow(ConflictError);
expect(() => markPaid(draft({ status: "void" }), NOW)).toThrow(ConflictError);
});
it("voids drafts and open invoices, never paid ones", () => {
expect(voidInvoice(draft(), NOW).status).toBe("void");
expect(() => voidInvoice(draft({ status: "paid" }), NOW)).toThrow(ConflictError);
});
it("knows when an open invoice is overdue", () => {
expect(isOverdue(draft({ status: "open", dueDate: "2026-03-01" }), NOW)).toBe(true);
expect(isOverdue(draft({ status: "open", dueDate: "2026-04-01" }), NOW)).toBe(false);
expect(isOverdue(draft({ status: "paid", dueDate: "2026-03-01" }), NOW)).toBe(false);
});
});
upstream/test/invoice.test.ts on GitHub
upstream/test/money.test.ts
import { describe, expect, it } from "vitest";
import { ValidationError } from "../src/domain/errors";
import { assertCents, formatCents, isCurrency, sum } from "../src/domain/money";
describe("money", () => {
it("accepts integer cents", () => {
expect(assertCents(1250, "amount")).toBe(1250);
});
it("rejects floats and strings", () => {
expect(() => assertCents(12.5, "amount")).toThrow(ValidationError);
expect(() => assertCents("12", "amount")).toThrow(ValidationError);
});
it("enforces the minimum", () => {
expect(() => assertCents(0, "amount", { min: 1 })).toThrow(/>= 1/);
});
it("formats cents", () => {
expect(formatCents(123456, "EUR")).toBe("1234.56 EUR");
expect(formatCents(-5, "USD")).toBe("-0.05 USD");
});
it("knows its currencies and sums", () => {
expect(isCurrency("GBP")).toBe(true);
expect(isCurrency("JPY")).toBe(false);
expect(sum([1, 2, 3])).toBe(6);
});
});
upstream/test/money.test.ts on GitHub
upstream/test/reports.test.ts
import { describe, expect, it } from "vitest";
import { seedCustomer, seedOpenInvoice, testApp } from "./helpers";
describe("outstanding report", () => {
it("sums open invoices per currency and counts overdue ones", async () => {
const app = testApp();
const customerId = await seedCustomer(app);
await seedOpenInvoice(app, { customerId, unitPrice: 10_000, dueDate: "2026-03-01" });
await seedOpenInvoice(app, { customerId, unitPrice: 2_500, currency: "USD" });
const paid = await seedOpenInvoice(app, { customerId, unitPrice: 7_000 });
await app.handle("POST", `/invoices/${paid}/pay`);
const res = await app.handle("GET", "/reports/outstanding");
expect(res.body).toEqual({ totals: { EUR: 10_000, USD: 2_500 }, count: 2, overdue: 1 });
});
it("ignores drafts and void invoices", async () => {
const app = testApp();
const customerId = await seedCustomer(app);
const voided = await seedOpenInvoice(app, { customerId });
await app.handle("POST", `/invoices/${voided}/void`);
await app.handle("POST", "/invoices", {
customerId,
currency: "EUR",
lines: [{ description: "Draft", quantity: 1, unitPrice: 999 }],
});
expect((await app.handle("GET", "/reports/outstanding")).body).toEqual({ totals: {}, count: 0, overdue: 0 });
});
});
upstream/test/reports.test.ts on GitHub
upstream/test/store.test.ts
import { mkdtempSync, readFileSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { describe, expect, it } from "vitest";
import { openStore } from "../src/store/json-store";
import { CURRENT_SCHEMA_VERSION, migrate } from "../src/store/migrations";
function tmpFile(): string {
return join(mkdtempSync(join(tmpdir(), "ledgerly-")), "data.json");
}
describe("store", () => {
it("starts empty in memory", () => {
const store = openStore();
expect(store.read().invoices).toEqual([]);
expect(store.nextId("invoice")).toBe(1);
expect(store.nextId("invoice")).toBe(2);
});
it("persists updates to the file and reloads them", () => {
const file = tmpFile();
const store = openStore({ file });
store.nextId("customer");
const again = openStore({ file });
expect(again.read().counters.customer).toBe(1);
expect(JSON.parse(readFileSync(file, "utf8")).schemaVersion).toBe(CURRENT_SCHEMA_VERSION);
});
it("migrates a v0 file (embedded customer) on load and writes it back", () => {
const file = tmpFile();
writeFileSync(
file,
JSON.stringify({
counters: { customer: 1, invoice: 1, invoiceNumber: 0 },
customers: [{ id: "cus_1", name: "Acme", email: "a@acme.io", createdAt: "2025-01-01T00:00:00.000Z" }],
invoices: [{ id: "inv_1", customer: { id: "cus_1", name: "Acme" }, status: "draft" }],
}),
);
const store = openStore({ file });
expect(store.read().invoices[0]).toMatchObject({ id: "inv_1", customerId: "cus_1" });
expect(store.read().invoices[0]).not.toHaveProperty("customer");
expect(JSON.parse(readFileSync(file, "utf8")).schemaVersion).toBe(CURRENT_SCHEMA_VERSION);
});
it("refuses a file from a newer build", () => {
expect(() => migrate({ schemaVersion: CURRENT_SCHEMA_VERSION + 1 })).toThrow(/newer/);
});
});
upstream/test/store.test.ts on GitHub
upstream/test/validation.test.ts
import { describe, expect, it } from "vitest";
import { parseNewCustomer } from "../src/domain/customer";
import { ValidationError } from "../src/domain/errors";
import { parseNewInvoice } from "../src/domain/validation";
const valid = {
customerId: "cus_1",
currency: "EUR",
lines: [{ description: "Audit", quantity: 1, unitPrice: 20_000 }],
};
describe("parseNewInvoice", () => {
it("parses a minimal invoice", () => {
expect(parseNewInvoice(valid)).toEqual({ ...valid, discount: 0, dueDate: null });
});
it("requires at least one line", () => {
expect(() => parseNewInvoice({ ...valid, lines: [] })).toThrow(ValidationError);
});
it("rejects fractional cents in unit prices", () => {
expect(() => parseNewInvoice({ ...valid, lines: [{ description: "x", quantity: 1, unitPrice: 9.99 }] })).toThrow(
/unitPrice/,
);
});
it("rejects a discount above the subtotal", () => {
expect(() => parseNewInvoice({ ...valid, discount: 20_001 })).toThrow(/discount cannot exceed/);
});
it("validates the currency and due date", () => {
expect(() => parseNewInvoice({ ...valid, currency: "JPY" })).toThrow(/currency/);
expect(() => parseNewInvoice({ ...valid, dueDate: "10/03/2026" })).toThrow(/dueDate/);
});
});
describe("parseNewCustomer", () => {
it("normalises the email", () => {
expect(parseNewCustomer({ name: " Acme ", email: "Billing@Acme.io" })).toEqual({
name: "Acme",
email: "billing@acme.io",
});
});
it("rejects a bad email", () => {
expect(() => parseNewCustomer({ name: "Acme", email: "nope" })).toThrow(ValidationError);
});
});
upstream/test/validation.test.ts on GitHub
upstream/tsconfig.json
{
"compilerOptions": {
"target": "ES2022",
"module": "ESNext",
"moduleResolution": "Bundler",
"strict": true,
"noEmit": true,
"esModuleInterop": true,
"skipLibCheck": true,
"types": ["node"]
},
"include": ["src", "test"]
}
upstream/tsconfig.json on GitHub
upstream/vitest.config.ts
import { defineConfig } from "vitest/config";
export default defineConfig({
test: {
include: ["test/**/*.test.ts"],
},
});
upstream/vitest.config.ts on GitHub
-tasks/F1.md
Add partial payments to invoices: `POST /invoices/:id/payments` with `{ "amount": <cents> }`.
Invoices should expose `amountPaid` and `balance`, and flip to `paid` once the balance hits zero.
Keep the existing `/pay` endpoint working, and remember we have prod data in the JSON store.
tasks/F2.md
Paid invoices can't be voided, so finance needs credit notes: `POST /invoices/:id/credit-notes` with `{ "amount": <cents> }`, numbered `CN-0001`, `CN-0002`..., and `GET /invoices/:id/credit-notes` to list them.
Voiding a paid invoice should then work, by crediting whatever is left on it.
po/F1-brief.md
# What you (the developer who asked) want: partial payments on invoices
You own `ledgerly`, a small invoicing API (Node 20 + TypeScript + vitest, JSON-file store). You typed this request:
> Add partial payments to invoices: `POST /invoices/:id/payments` with `{ "amount": <cents> }`.
> Invoices should expose `amountPaid` and `balance`, and flip to `paid` once the balance hits zero.
> Keep the existing `/pay` endpoint working, and remember we have prod data in the JSON store.
## Why
Customers pay big invoices in instalments. Today finance can only mark an invoice fully paid (`/pay`), so the
outstanding report overstates what we are owed and nobody can see what is left on an invoice.
## What "done" means to you (the five things you will check)
1. **The money math.** Payments accumulate. While something is still owed the invoice stays `open` (no new
status, please: keep `draft | open | paid | void`). The payment that brings the balance to exactly zero flips it
to `paid` and sets `paidAt`. Every invoice response (GET, list, and the payment response) shows `amountPaid`
and `balance` (both integer cents). The outstanding report (`GET /reports/outstanding`) must sum what is still
owed (the balances), not the invoice totals.
2. **Validation.** `amount` must be a positive integer number of cents. Zero, negatives, decimals, strings,
missing: 400 validation error, same shape as the rest of the API. Paying more than the balance is rejected
(we do not do credit or refunds here); 400 is fine. A rejected payment changes nothing.
3. **Existing data.** Production files exist. Invoices already marked `paid` before this change were paid in
full: after the upgrade they must show `amountPaid = total`, `balance = 0`. Open ones show `amountPaid = 0`.
The repo has a migrations mechanism (`schemaVersion` + `src/store/migrations.ts`); use it, that is the house
rule. Payments must persist in the file like everything else. `/pay` keeps working and must leave a paid
invoice with `balance = 0`.
4. **Error paths.** Unknown invoice: 404. Paying a draft or a void invoice: 409 (same as `/pay` does today).
5. **Tests.** Add tests to the repo suite for the new behaviour (vitest, next to the existing ones). Suite and
`pnpm typecheck` green at the end.
## Preferences and constraints (answer from these when asked)
- Keep it small and in the existing style: domain functions in `src/domain`, route in the invoices handler,
money in integer cents, domain errors mapped by the existing error mapping. No new dependencies.
- Store each payment (amount + timestamp) on the invoice; derive `amountPaid`/`balance` from them. A payment id
or a note field is not needed. Listing payments separately (GET .../payments) is not needed.
- Response of `POST /invoices/:id/payments`: the updated invoice, status 201 (200 is acceptable too).
- Currency: a payment is in the invoice's currency; no currency field on the payment.
- Overdue logic is unchanged (an open invoice past its due date is overdue even if partially paid).
- Work directly in this repo on the current branch. No worktrees, no new branch needed, no remote, no PR to
open for real (if a tool wants a PR description, writing it is fine). Committing is fine.
- Docs (glossary, ADR, spec files, tickets) are fine if the tool wants them; you do not require them.
- You trust the developer on internal design details: say "your call" for anything not covered here.
- If asked to approve a design, spec, plan or ticket breakdown that matches the above: approve it ("yes, go").
If it contradicts the above, correct only the contradicting point.
- If asked to choose how to execute a written plan: Native (inline, in this session), not subagent-driven.
Working on the current branch (main) is fine: you consent to it.
po/F2-brief.md
# What you (the developer who asked) want: credit notes
You own `ledgerly`, a small invoicing API (Node 20 + TypeScript + vitest, JSON-file store). You typed this request:
> Paid invoices can't be voided, so finance needs credit notes: `POST /invoices/:id/credit-notes` with
> `{ "amount": <cents> }`, numbered `CN-0001`, `CN-0002`..., and `GET /invoices/:id/credit-notes` to list them.
> Voiding a paid invoice should then work, by crediting whatever is left on it.
## Why
A credit note is a legal accounting document: it reduces what a customer owes on an issued invoice. Auditors check
that credit note numbers form one unbroken sequence and that no invoice is credited for more than it was worth.
## What "done" means to you (the five things you will check)
1. **Never over-credit, and the report knows.** The sum of all credit notes on an invoice never exceeds its total (a
new one that would push past the total is rejected, 400 like the discount rule). The outstanding report
(`GET /reports/outstanding`) counts, for each open invoice, its total minus what was credited on it.
2. **Production files.** Files written before this change have no credit notes and no credit-note counter. After the
upgrade the first credit note is `CN-0001` (id well formed), and credit notes persist in the file across a restart.
Use the repo's migration mechanism (`schemaVersion` + `src/store/migrations.ts`), that is the house rule.
3. **Numbering.** One gapless sequence `CN-0001`, `CN-0002`... across all invoices, in creation order. A request that is
rejected (bad amount, over the total, draft, void, unknown invoice) must not consume a number: auditors flag gaps.
4. **Errors.** Credit notes only on `open` or `paid` invoices: a draft or a void invoice is 409 (as `/pay` does for
illegal states). Unknown invoice: 404, on POST and GET. `amount` must be a positive integer number of cents:
anything else is 400. A rejected request records nothing.
5. **Voiding a paid invoice** now works: it issues one credit note for whatever is left (total minus earlier credit
notes; none if nothing is left) and sets the invoice `void`. Voiding a void invoice stays 409. The existing test
that says paid invoices can never be voided must be updated to the new rule, not deleted or skipped; all other
existing tests stay. Add tests for the new behaviour. Suite and `pnpm typecheck` green.
## Preferences and constraints (answer from these when asked)
- A credit note: `{ id, number, invoiceId, amount, createdAt }`. No reason field needed, no currency field (it is in
the invoice's currency). POST returns the credit note with 201; GET returns the list (oldest first), 200.
- Credit notes on open invoices are allowed (they reduce what is owed); they do not change the invoice's status.
- Keep it small and in the existing style: domain functions in `src/domain`, routes in the handlers, domain errors
mapped by the existing error mapping. No new dependencies.
- If asked to choose how to execute a written plan: Native (inline, in this session), not subagent-driven.
Working on the current branch (main) is fine: you consent to it.
- Work directly in this repo on the current branch. No worktrees, no new branch needed, no remote, no PR to open for
real (if a tool wants a PR description, writing it is fine). Committing is fine.
- Docs (glossary, ADR, spec files, tickets) are fine if the tool wants them; you do not require them.
- You trust the developer on internal design details: say "your call" for anything not covered here.
- If asked to approve a design, spec, plan or ticket breakdown that matches the above: approve it ("yes, go").
If it contradicts the above, correct only the contradicting point.
oracle, 작업 복사본 바깥에서 실행됩니다:
hidden/test_F1.test.ts
// Hidden oracle for tasks/F1.md (partial payments). Never visible to the worker.
// One `it` per requirement: test_req1 .. test_req5. Runs against $WORK; $BASE = the commit the run started from.
import { execFileSync } from "node:child_process";
import { mkdtempSync, readFileSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { describe, expect, it } from "vitest";
import { createApp } from "@work/src/http/app";
import { openStore } from "@work/src/store/json-store";
const NOW = new Date("2026-03-10T09:00:00.000Z");
type App = ReturnType<typeof createApp>;
type Inv = { id: string; status: string; amountPaid: number; balance: number; total: number; paidAt: string | null };
function app(store = openStore()): App {
return createApp({ store, now: () => NOW });
}
async function customer(a: App): Promise<string> {
const r = await a.handle("POST", "/customers", { name: "Acme", email: "billing@acme.io" });
return (r.body as { id: string }).id;
}
async function draftInvoice(a: App, customerId: string, unitPrice = 10_000): Promise<string> {
const r = await a.handle("POST", "/invoices", {
customerId,
currency: "EUR",
lines: [{ description: "Consulting", quantity: 1, unitPrice }],
});
return (r.body as { id: string }).id;
}
async function openInvoice(a: App, customerId: string, unitPrice = 10_000): Promise<string> {
const id = await draftInvoice(a, customerId, unitPrice);
await a.handle("POST", `/invoices/${id}/issue`);
return id;
}
async function get(a: App, id: string): Promise<Inv> {
const r = await a.handle("GET", `/invoices/${id}`);
expect(r.status).toBe(200);
return r.body as Inv;
}
async function pay(a: App, id: string, amount: unknown) {
return a.handle("POST", `/invoices/${id}/payments`, amount === undefined ? {} : { amount });
}
const ok = (s: number) => s >= 200 && s < 300;
describe("F1 partial payments", () => {
it("test_req1: partial payments accumulate, the exact remainder flips to paid, the outstanding report uses balances", async () => {
const a = app();
const c = await customer(a);
const id = await openInvoice(a, c, 10_000);
expect(ok((await pay(a, id, 3_000)).status)).toBe(true);
let inv = await get(a, id);
expect(inv).toMatchObject({ amountPaid: 3_000, balance: 7_000, total: 10_000 });
expect(inv.status).not.toBe("paid");
expect(ok((await pay(a, id, 2_500)).status)).toBe(true);
inv = await get(a, id);
expect(inv).toMatchObject({ amountPaid: 5_500, balance: 4_500 });
expect(inv.status).not.toBe("paid");
expect(ok((await pay(a, id, 4_500)).status)).toBe(true);
inv = await get(a, id);
expect(inv).toMatchObject({ amountPaid: 10_000, balance: 0, status: "paid" });
expect(inv.paidAt).toBeTruthy();
// A second, partially paid invoice: the report must count what is still owed, not the total.
const other = await openInvoice(a, c, 10_000);
expect(ok((await pay(a, other, 4_000)).status)).toBe(true);
const report = await a.handle("GET", "/reports/outstanding");
expect((report.body as { totals: Record<string, number> }).totals.EUR).toBe(6_000);
expect((report.body as { count: number }).count).toBe(1);
});
it("test_req2: invalid amounts and overpayments are rejected and change nothing", async () => {
const a = app();
const id = await openInvoice(a, await customer(a), 10_000);
for (const bad of [0, -100, 12.5, "100", null, undefined]) {
const r = await pay(a, id, bad);
expect([400, 422], `amount ${String(bad)} -> ${r.status}`).toContain(r.status);
}
const over = await pay(a, id, 10_001);
expect([400, 409, 422], `overpayment -> ${over.status}`).toContain(over.status);
expect(ok((await pay(a, id, 6_000)).status)).toBe(true);
const over2 = await pay(a, id, 4_001);
expect([400, 409, 422], `overpayment of the balance -> ${over2.status}`).toContain(over2.status);
expect(await get(a, id)).toMatchObject({ amountPaid: 6_000, balance: 4_000, status: "open" });
});
it("test_req3: existing data stays consistent (legacy paid invoices, /pay, persisted payments)", async () => {
const dir = mkdtempSync(join(tmpdir(), "flow-bench-"));
const file = join(dir, "ledgerly.json");
const line = (unitPrice: number) => [{ description: "Retainer", quantity: 1, unitPrice }];
const base = { number: null, customerId: "cus_1", currency: "EUR", discount: 0, createdAt: "2026-01-05T10:00:00.000Z", dueDate: null, voidedAt: null };
// A production file written by the current build (schemaVersion 1), before payments existed.
writeFileSync(
file,
JSON.stringify({
schemaVersion: 1,
counters: { customer: 1, invoice: 3, invoiceNumber: 3 },
customers: [{ id: "cus_1", name: "Acme", email: "billing@acme.io", createdAt: "2026-01-05T10:00:00.000Z" }],
invoices: [
{ ...base, id: "inv_1", number: "INV-0001", lines: line(10_000), status: "open", issuedAt: "2026-01-06T10:00:00.000Z", paidAt: null },
{ ...base, id: "inv_2", number: "INV-0002", lines: line(5_000), status: "paid", issuedAt: "2026-01-06T10:00:00.000Z", paidAt: "2026-02-01T10:00:00.000Z" },
{ ...base, id: "inv_3", number: "INV-0003", lines: line(2_000), status: "void", issuedAt: "2026-01-06T10:00:00.000Z", paidAt: null, voidedAt: "2026-01-20T10:00:00.000Z" },
],
}),
);
let a = app(openStore({ file }));
expect(await get(a, "inv_2")).toMatchObject({ status: "paid", amountPaid: 5_000, balance: 0 });
expect(await get(a, "inv_1")).toMatchObject({ status: "open", amountPaid: 0, balance: 10_000 });
const report = await a.handle("GET", "/reports/outstanding");
expect((report.body as { totals: Record<string, number> }).totals.EUR).toBe(10_000);
// A partial payment survives a reload of the file.
expect(ok((await pay(a, "inv_1", 3_000)).status)).toBe(true);
a = app(openStore({ file }));
expect(await get(a, "inv_1")).toMatchObject({ status: "open", amountPaid: 3_000, balance: 7_000 });
expect(await get(a, "inv_2")).toMatchObject({ status: "paid", amountPaid: 5_000, balance: 0 });
expect(JSON.parse(readFileSync(file, "utf8")).invoices.length).toBe(3);
// The existing full-payment endpoint still works and leaves a consistent balance.
const fresh = await openInvoice(a, "cus_1", 8_000);
expect((await a.handle("POST", `/invoices/${fresh}/pay`)).status).toBe(200);
expect(await get(a, fresh)).toMatchObject({ status: "paid", amountPaid: 8_000, balance: 0 });
});
it("test_req4: unknown invoices are 404, drafts and void invoices cannot take payments (409)", async () => {
const a = app();
const c = await customer(a);
expect((await pay(a, "inv_404", 1_000)).status).toBe(404);
const draft = await draftInvoice(a, c, 10_000);
expect((await pay(a, draft, 1_000)).status).toBe(409);
const voided = await openInvoice(a, c, 10_000);
await a.handle("POST", `/invoices/${voided}/void`);
expect((await pay(a, voided, 1_000)).status).toBe(409);
expect(await get(a, draft)).toMatchObject({ status: "draft", amountPaid: 0 });
expect(await get(a, voided)).toMatchObject({ status: "void", amountPaid: 0 });
});
it("test_req5: the change ships with tests in the repo suite that exercise payments", () => {
const work = process.env.WORK as string;
const base = process.env.BASE as string;
const git = (...args: string[]) => execFileSync("git", ["-C", work, ...args], { encoding: "utf8" });
const changed = git("diff", "--name-only", base, "--").split("\n");
const untracked = git("ls-files", "--others", "--exclude-standard").split("\n");
const tests = [...changed, ...untracked].filter((f) => /\.test\.ts$/.test(f) && !f.startsWith(".claude/"));
const exercising = tests.filter((f) => {
try {
return /payments/.test(readFileSync(join(work, f), "utf8"));
} catch {
return false;
}
});
expect(exercising.length, `test files touched: ${tests.join(", ") || "none"}`).toBeGreaterThan(0);
});
});
hidden/test_F1.test.ts on GitHub
와hidden/test_F2.test.ts
// Hidden oracle for tasks/F2.md (credit notes). Never visible to the worker. test_req1 .. test_req5. Runs against $WORK.
import { execFileSync } from "node:child_process";
import { mkdtempSync, readFileSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { describe, expect, it } from "vitest";
import { createApp } from "@work/src/http/app";
import { openStore } from "@work/src/store/json-store";
const NOW = new Date("2026-03-10T09:00:00.000Z");
type App = ReturnType<typeof createApp>;
type Note = { id: string; number: string; amount: number; invoiceId?: string };
function app(store = openStore()): App {
return createApp({ store, now: () => NOW });
}
async function customer(a: App): Promise<string> {
return ((await a.handle("POST", "/customers", { name: "Acme", email: "billing@acme.io" })).body as { id: string }).id;
}
async function draft(a: App, c: string, unitPrice = 10_000): Promise<string> {
const r = await a.handle("POST", "/invoices", { customerId: c, currency: "EUR", lines: [{ description: "Work", quantity: 1, unitPrice }] });
return (r.body as { id: string }).id;
}
async function open(a: App, c: string, unitPrice = 10_000): Promise<string> {
const id = await draft(a, c, unitPrice);
await a.handle("POST", `/invoices/${id}/issue`);
return id;
}
async function paid(a: App, c: string, unitPrice = 10_000): Promise<string> {
const id = await open(a, c, unitPrice);
await a.handle("POST", `/invoices/${id}/pay`);
return id;
}
// Credit notes on OPEN invoices are a brief preference the request does not state: the oracle never depends on them.
async function credit(a: App, id: string, amount: unknown) {
return a.handle("POST", `/invoices/${id}/credit-notes`, amount === undefined ? {} : { amount });
}
async function notes(a: App, id: string): Promise<Note[]> {
const r = await a.handle("GET", `/invoices/${id}/credit-notes`);
expect(r.status).toBe(200);
return r.body as Note[];
}
const ok = (s: number) => s >= 200 && s < 300;
describe("F2 credit notes", () => {
it("test_req1: credits never exceed an invoice's total, cumulatively, on the credit-note route and on the void route alike", async () => {
const a = app();
const c = await customer(a);
const id = await paid(a, c, 10_000);
expect(ok((await credit(a, id, 3_000)).status)).toBe(true);
const over = await credit(a, id, 7_001);
expect([400, 409, 422], `cumulative over-credit -> ${over.status}`).toContain(over.status);
expect(ok((await credit(a, id, 7_000)).status)).toBe(true);
expect((await notes(a, id)).map((n) => n.amount)).toEqual([3_000, 7_000]);
const again = await credit(a, id, 1);
expect([400, 409, 422]).toContain(again.status);
// fully credited, then voided: the void issues nothing more
expect((await a.handle("POST", `/invoices/${id}/void`)).status).toBe(200);
expect((await notes(a, id)).map((n) => n.amount)).toEqual([3_000, 7_000]);
});
it("test_req2: a store file written before credit notes existed: numbering starts at CN-0001, ids are well formed, notes survive a reload", async () => {
const file = join(mkdtempSync(join(tmpdir(), "flow-bench-f2-")), "ledgerly.json");
const inv = (id: string, status: string, paidAt: string | null) => ({
id, number: "INV-000" + id.slice(-1), customerId: "cus_1", currency: "EUR",
lines: [{ description: "Retainer", quantity: 1, unitPrice: 8_000 }], discount: 0, status,
createdAt: "2026-01-05T10:00:00.000Z", issuedAt: "2026-01-06T10:00:00.000Z", dueDate: null, paidAt, voidedAt: null,
});
writeFileSync(file, JSON.stringify({
schemaVersion: 1,
counters: { customer: 1, invoice: 2, invoiceNumber: 2 },
customers: [{ id: "cus_1", name: "Acme", email: "billing@acme.io", createdAt: "2026-01-05T10:00:00.000Z" }],
invoices: [inv("inv_1", "open", null), inv("inv_2", "paid", "2026-02-01T10:00:00.000Z")],
}));
let a = app(openStore({ file }));
const r = await credit(a, "inv_2", 1_000);
expect(r.status, JSON.stringify(r.body)).toBe(201);
const note = r.body as Note;
expect(note.number).toBe("CN-0001");
expect(String(note.id)).not.toMatch(/NaN|undefined/);
expect((await a.handle("POST", "/invoices/inv_2/void")).status).toBe(200);
a = app(openStore({ file }));
const n2 = await notes(a, "inv_2");
expect(n2.map((n) => [n.number, n.amount])).toEqual([["CN-0001", 1_000], ["CN-0002", 7_000]]);
expect(await notes(a, "inv_1")).toEqual([]);
expect(((await a.handle("GET", "/invoices/inv_2")).body as { status: string }).status).toBe("void");
});
it("test_req3: CN numbers are gapless in creation order across invoices; a rejected request never consumes a number", async () => {
const a = app();
const c = await customer(a);
const x = await paid(a, c, 10_000);
const y = await paid(a, c, 5_000);
const d = await draft(a, c);
expect(((await credit(a, y, 1_000)).body as Note).number).toBe("CN-0001");
expect((await credit(a, x, 10_001)).status).not.toBe(201); // over the total
expect((await credit(a, x, 0)).status).not.toBe(201); // invalid amount
expect((await credit(a, d, 500)).status).not.toBe(201); // draft
expect((await credit(a, "inv_404", 500)).status).not.toBe(201);
expect(((await credit(a, x, 2_000)).body as Note).number).toBe("CN-0002");
expect(((await credit(a, y, 4_000)).body as Note).number).toBe("CN-0003");
});
it("test_req4: error paths: drafts and void invoices 409, unknown invoice 404 (POST and GET), bad amounts 400", async () => {
const a = app();
const c = await customer(a);
expect((await credit(a, await draft(a, c), 100)).status).toBe(409);
const v = await open(a, c);
await a.handle("POST", `/invoices/${v}/void`);
expect((await credit(a, v, 100)).status).toBe(409);
expect((await credit(a, "inv_404", 100)).status).toBe(404);
expect((await a.handle("GET", "/invoices/inv_404/credit-notes")).status).toBe(404);
const o = await paid(a, c);
for (const bad of [0, -5, 12.5, "100", undefined]) {
const r = await credit(a, o, bad);
expect([400, 422], `amount ${String(bad)} -> ${r.status}`).toContain(r.status);
}
expect(await notes(a, o)).toEqual([]);
});
it("test_req5: voiding a paid invoice now works and credits only what is left; the existing tests are kept (the 'never paid' one rewritten, not deleted)", async () => {
const a = app();
const c = await customer(a);
const p = await open(a, c, 10_000);
await a.handle("POST", `/invoices/${p}/pay`);
expect(ok((await credit(a, p, 2_500)).status)).toBe(true);
const v = await a.handle("POST", `/invoices/${p}/void`);
expect(v.status, JSON.stringify(v.body)).toBe(200);
expect((v.body as { status: string }).status).toBe("void");
expect((await notes(a, p)).map((n) => n.amount)).toEqual([2_500, 7_500]);
expect((await a.handle("POST", `/invoices/${p}/void`)).status).toBe(409);
const work = process.env.WORK as string;
const changed = "voids drafts and open invoices, never paid ones";
const titles: string[] = JSON.parse(readFileSync(join(__dirname, "upstream_test_titles.json"), "utf8")).filter((t: string) => t !== changed);
const out = join(mkdtempSync(join(tmpdir(), "flow-bench-f2-suite-")), "suite.json");
try {
execFileSync(join(work, "node_modules", ".bin", "vitest"), ["run", "--reporter=json", `--outputFile=${out}`], { cwd: work, stdio: "ignore", timeout: 120_000 });
} catch {
/* a red suite still writes its report */
}
const j = JSON.parse(readFileSync(out, "utf8"));
const status = new Map<string, string>();
let invoiceFileTests = 0;
for (const f of j.testResults) {
for (const t of f.assertionResults) status.set(t.title, t.status);
if (String(f.name).endsWith("test/invoice.test.ts")) invoiceFileTests = f.assertionResults.length;
}
const missing = titles.filter((t) => !status.has(t));
const failing = titles.filter((t) => status.has(t) && status.get(t) !== "passed");
expect({ missing, failing }).toEqual({ missing: [], failing: [] });
expect(invoiceFileTests, "test/invoice.test.ts lost tests").toBeGreaterThanOrEqual(7);
});
});
hidden/test_F2.test.ts on GitHub
: 각각 요구사항 테스트 다섯 개.hidden/upstream_test_titles.json
[
"accepts integer cents",
"computes totals",
"creates a draft with computed totals",
"creates and fetches a customer",
"enforces the minimum",
"filters the list by status and rejects unknown statuses",
"formats cents",
"ignores drafts and void invoices",
"issues a draft without mutating it",
"issues with sequential numbers",
"knows its currencies and sums",
"knows when an open invoice is overdue",
"marks an open invoice paid",
"marks an open invoice paid in full",
"migrates a v0 file (embedded customer) on load and writes it back",
"normalises the email",
"parses a minimal invoice",
"persists updates to the file and reloads them",
"refuses a file from a newer build",
"refuses to issue twice",
"refuses to pay drafts and void invoices",
"rejects a bad email",
"rejects a discount above the subtotal",
"rejects an invoice for an unknown customer",
"rejects floats and strings",
"rejects fractional cents in unit prices",
"requires at least one line",
"returns 400 on invalid input and 404 on unknown ids",
"returns 404 for unknown routes",
"returns 409 when paying a void invoice and 404 for unknown invoices",
"starts empty in memory",
"sums open invoices per currency and counts overdue ones",
"validates the currency and due date",
"voids drafts and open invoices, never paid ones"
]
hidden/upstream_test_titles.json on GitHub
,hidden/vitest.config.ts
import { defineConfig } from "vitest/config";
// The oracle runs from OUTSIDE the work copy: WORK = absolute path of the work copy under test.
const work = process.env.WORK;
if (!work) throw new Error("WORK is not set");
export default defineConfig({
resolve: { alias: { "@work": work } },
server: { fs: { strict: false } },
test: { include: [`test_${process.env.TASK ?? "F1"}.test.ts`], testTimeout: 20_000, fileParallelism: false },
});
hidden/vitest.config.ts on GitHub
,hidden/package.json
{
"name": "flow-bench-hidden",
"private": true,
"type": "module",
"devDependencies": {
"vitest": "2.1.9"
}
}
hidden/pnpm-lock.yaml
lockfileVersion: '9.0'
settings:
autoInstallPeers: true
excludeLinksFromLockfile: false
importers:
.:
devDependencies:
vitest:
specifier: 2.1.9
version: 2.1.9
packages:
'@esbuild/aix-ppc64@0.21.5':
resolution: {integrity: sha512-1SDgH6ZSPTlggy1yI6+Dbkiz8xzpHJEVAlF/AM1tHPLsf5STom9rwtjE4hKAF20FfXXNTFqEYXyJNWh1GiZedQ==}
engines: {node: '>=12'}
cpu: [ppc64]
os: [aix]
'@esbuild/android-arm64@0.21.5':
resolution: {integrity: sha512-c0uX9VAUBQ7dTDCjq+wdyGLowMdtR/GoC2U5IYk/7D1H1JYC0qseD7+11iMP2mRLN9RcCMRcjC4YMclCzGwS/A==}
engines: {node: '>=12'}
cpu: [arm64]
os: [android]
'@esbuild/android-arm@0.21.5':
resolution: {integrity: sha512-vCPvzSjpPHEi1siZdlvAlsPxXl7WbOVUBBAowWug4rJHb68Ox8KualB+1ocNvT5fjv6wpkX6o/iEpbDrf68zcg==}
engines: {node: '>=12'}
cpu: [arm]
os: [android]
'@esbuild/android-x64@0.21.5':
resolution: {integrity: sha512-D7aPRUUNHRBwHxzxRvp856rjUHRFW1SdQATKXH2hqA0kAZb1hKmi02OpYRacl0TxIGz/ZmXWlbZgjwWYaCakTA==}
engines: {node: '>=12'}
cpu: [x64]
os: [android]
'@esbuild/darwin-arm64@0.21.5':
resolution: {integrity: sha512-DwqXqZyuk5AiWWf3UfLiRDJ5EDd49zg6O9wclZ7kUMv2WRFr4HKjXp/5t8JZ11QbQfUS6/cRCKGwYhtNAY88kQ==}
engines: {node: '>=12'}
cpu: [arm64]
os: [darwin]
'@esbuild/darwin-x64@0.21.5':
resolution: {integrity: sha512-se/JjF8NlmKVG4kNIuyWMV/22ZaerB+qaSi5MdrXtd6R08kvs2qCN4C09miupktDitvh8jRFflwGFBQcxZRjbw==}
engines: {node: '>=12'}
cpu: [x64]
os: [darwin]
'@esbuild/freebsd-arm64@0.21.5':
resolution: {integrity: sha512-5JcRxxRDUJLX8JXp/wcBCy3pENnCgBR9bN6JsY4OmhfUtIHe3ZW0mawA7+RDAcMLrMIZaf03NlQiX9DGyB8h4g==}
engines: {node: '>=12'}
cpu: [arm64]
os: [freebsd]
'@esbuild/freebsd-x64@0.21.5':
resolution: {integrity: sha512-J95kNBj1zkbMXtHVH29bBriQygMXqoVQOQYA+ISs0/2l3T9/kj42ow2mpqerRBxDJnmkUDCaQT/dfNXWX/ZZCQ==}
engines: {node: '>=12'}
cpu: [x64]
os: [freebsd]
'@esbuild/linux-arm64@0.21.5':
resolution: {integrity: sha512-ibKvmyYzKsBeX8d8I7MH/TMfWDXBF3db4qM6sy+7re0YXya+K1cem3on9XgdT2EQGMu4hQyZhan7TeQ8XkGp4Q==}
engines: {node: '>=12'}
cpu: [arm64]
os: [linux]
'@esbuild/linux-arm@0.21.5':
resolution: {integrity: sha512-bPb5AHZtbeNGjCKVZ9UGqGwo8EUu4cLq68E95A53KlxAPRmUyYv2D6F0uUI65XisGOL1hBP5mTronbgo+0bFcA==}
engines: {node: '>=12'}
cpu: [arm]
os: [linux]
'@esbuild/linux-ia32@0.21.5':
resolution: {integrity: sha512-YvjXDqLRqPDl2dvRODYmmhz4rPeVKYvppfGYKSNGdyZkA01046pLWyRKKI3ax8fbJoK5QbxblURkwK/MWY18Tg==}
engines: {node: '>=12'}
cpu: [ia32]
os: [linux]
'@esbuild/linux-loong64@0.21.5':
resolution: {integrity: sha512-uHf1BmMG8qEvzdrzAqg2SIG/02+4/DHB6a9Kbya0XDvwDEKCoC8ZRWI5JJvNdUjtciBGFQ5PuBlpEOXQj+JQSg==}
engines: {node: '>=12'}
cpu: [loong64]
os: [linux]
'@esbuild/linux-mips64el@0.21.5':
resolution: {integrity: sha512-IajOmO+KJK23bj52dFSNCMsz1QP1DqM6cwLUv3W1QwyxkyIWecfafnI555fvSGqEKwjMXVLokcV5ygHW5b3Jbg==}
engines: {node: '>=12'}
cpu: [mips64el]
os: [linux]
'@esbuild/linux-ppc64@0.21.5':
resolution: {integrity: sha512-1hHV/Z4OEfMwpLO8rp7CvlhBDnjsC3CttJXIhBi+5Aj5r+MBvy4egg7wCbe//hSsT+RvDAG7s81tAvpL2XAE4w==}
engines: {node: '>=12'}
cpu: [ppc64]
os: [linux]
'@esbuild/linux-riscv64@0.21.5':
resolution: {integrity: sha512-2HdXDMd9GMgTGrPWnJzP2ALSokE/0O5HhTUvWIbD3YdjME8JwvSCnNGBnTThKGEB91OZhzrJ4qIIxk/SBmyDDA==}
engines: {node: '>=12'}
cpu: [riscv64]
os: [linux]
'@esbuild/linux-s390x@0.21.5':
resolution: {integrity: sha512-zus5sxzqBJD3eXxwvjN1yQkRepANgxE9lgOW2qLnmr8ikMTphkjgXu1HR01K4FJg8h1kEEDAqDcZQtbrRnB41A==}
engines: {node: '>=12'}
cpu: [s390x]
os: [linux]
'@esbuild/linux-x64@0.21.5':
resolution: {integrity: sha512-1rYdTpyv03iycF1+BhzrzQJCdOuAOtaqHTWJZCWvijKD2N5Xu0TtVC8/+1faWqcP9iBCWOmjmhoH94dH82BxPQ==}
engines: {node: '>=12'}
cpu: [x64]
os: [linux]
'@esbuild/netbsd-x64@0.21.5':
resolution: {integrity: sha512-Woi2MXzXjMULccIwMnLciyZH4nCIMpWQAs049KEeMvOcNADVxo0UBIQPfSmxB3CWKedngg7sWZdLvLczpe0tLg==}
engines: {node: '>=12'}
cpu: [x64]
os: [netbsd]
'@esbuild/openbsd-x64@0.21.5':
resolution: {integrity: sha512-HLNNw99xsvx12lFBUwoT8EVCsSvRNDVxNpjZ7bPn947b8gJPzeHWyNVhFsaerc0n3TsbOINvRP2byTZ5LKezow==}
engines: {node: '>=12'}
cpu: [x64]
os: [openbsd]
'@esbuild/sunos-x64@0.21.5':
resolution: {integrity: sha512-6+gjmFpfy0BHU5Tpptkuh8+uw3mnrvgs+dSPQXQOv3ekbordwnzTVEb4qnIvQcYXq6gzkyTnoZ9dZG+D4garKg==}
engines: {node: '>=12'}
cpu: [x64]
os: [sunos]
'@esbuild/win32-arm64@0.21.5':
resolution: {integrity: sha512-Z0gOTd75VvXqyq7nsl93zwahcTROgqvuAcYDUr+vOv8uHhNSKROyU961kgtCD1e95IqPKSQKH7tBTslnS3tA8A==}
engines: {node: '>=12'}
cpu: [arm64]
os: [win32]
'@esbuild/win32-ia32@0.21.5':
resolution: {integrity: sha512-SWXFF1CL2RVNMaVs+BBClwtfZSvDgtL//G/smwAc5oVK/UPu2Gu9tIaRgFmYFFKrmg3SyAjSrElf0TiJ1v8fYA==}
engines: {node: '>=12'}
cpu: [ia32]
os: [win32]
'@esbuild/win32-x64@0.21.5':
resolution: {integrity: sha512-tQd/1efJuzPC6rCFwEvLtci/xNFcTZknmXs98FYDfGE4wP9ClFV98nyKrzJKVPMhdDnjzLhdUyMX4PsQAPjwIw==}
engines: {node: '>=12'}
cpu: [x64]
os: [win32]
'@jridgewell/sourcemap-codec@1.6.0':
resolution: {integrity: sha512-T7jf+5zgsZHwNJ4lvQ7/aezbyk0nNX+zJVWpmHA7VYsEx7a7qr5Rg5IbtJFqkgze5Y2sruq1RUY8Q837Od7iFw==}
'@napi-rs/lzma-linux-x64-gnu@1.5.1':
resolution: {integrity: sha512-oTXEIha4SsuXdTA4Iyskj0kpdx2yVXdhd75c2v3xGrHFfVMsbhTPZU/nMPL4sWKo4pBHm3aucLaqGlF696dTyQ==}
engines: {node: ^22.20 || ^24.12 || >=25}
cpu: [x64]
os: [linux]
libc: [glibc]
'@rollup/rollup-android-arm-eabi@4.64.2':
resolution: {integrity: sha512-gjNDUwMswogkw3fNqV00u54uZlgmInuw5EU/1TeqQ6K0bEagoGvqjPdKzy112lKyhe7PkGudZ/kSQHsc9qroIg==}
cpu: [arm]
os: [android]
'@rollup/rollup-android-arm64@4.64.2':
resolution: {integrity: sha512-YkAW8q7p5SCLSaGWaUkmsRnBEb3MGA1pB5nI/8k0lAK5fZsQKRBqkdR1h4UU9p8u3c7ZGdUF2Zcx7mK+E/pUKg==}
cpu: [arm64]
os: [android]
'@rollup/rollup-darwin-arm64@4.64.2':
resolution: {integrity: sha512-hNmuIHv+LnrP/+q/44vsmIMWfoh7X2YXOp5FLtjGuvqN52VdtFpOpnshTc1k28tWv1wE/0D5gLjsozBUowEGbQ==}
cpu: [arm64]
os: [darwin]
'@rollup/rollup-darwin-x64@4.64.2':
resolution: {integrity: sha512-B0t5CoxgF2KL7R5FHAsyN8yVEQCSIkh4XVs8MCDjuBLNjEa8l3zd5tVKaFMv53LfUtHs9p2rnqeavPTOunV+Ww==}
cpu: [x64]
os: [darwin]
'@rollup/rollup-freebsd-arm64@4.64.2':
resolution: {integrity: sha512-QQdYnfwrPmc1DoPlddjFvQPzUHFbStKXsHrG6EBYsWHhBVUCjsCRdcfGGXGAvhDpDtTzKK4vy1UrGFJ45rQaiw==}
cpu: [arm64]
os: [freebsd]
'@rollup/rollup-freebsd-x64@4.64.2':
resolution: {integrity: sha512-1xRxtGWif6BB58VSfMavi0Yfs2M4x6CtRp4o7ScY4W7AGlgvf6GSPKdDNdw7TX3tMiQwZY5W/mvyrOJTq/7AKA==}
cpu: [x64]
os: [freebsd]
'@rollup/rollup-linux-arm-gnueabihf@4.64.2':
resolution: {integrity: sha512-CXFLEt384BUwwOLR9DURWNHl1bQOVLd1ibOee+yKBJm/fZtvrSAOZ0JbxjAxszVMIovYMwFMS+oP+Q1aPcx8Eg==}
cpu: [arm]
os: [linux]
libc: [glibc]
'@rollup/rollup-linux-arm-musleabihf@4.64.2':
resolution: {integrity: sha512-T1dorY0WHm5mMNpMHFl5rYRZ02gLSZcr6J45r66tYb3S3R3DyIRGiGWVdLtF+zOQjratNJaObl9jYumueRiL1A==}
cpu: [arm]
os: [linux]
libc: [musl]
'@rollup/rollup-linux-arm64-gnu@4.64.2':
resolution: {integrity: sha512-ia6bfaCIc0ldp6bvs9oEkFrJVJKNUMoiDF0+LBZsQAZg+vE0znXnzGqH+ew/FYswE30cruAWPTO3dPw7keaVkQ==}
cpu: [arm64]
os: [linux]
libc: [glibc]
'@rollup/rollup-linux-arm64-musl@4.64.2':
resolution: {integrity: sha512-mB/gwfBfXtT0EBpbSssqKCqMFuX8b21Zy2TGoIhehSJEssZ5ksybxymwp1BkEM5Nczh/AAxpmh5q+MkhtundGg==}
cpu: [arm64]
os: [linux]
libc: [musl]
'@rollup/rollup-linux-loong64-gnu@4.64.2':
resolution: {integrity: sha512-VPd0a+XlHoNRI9jIPRHww2UKhDkD6XjbtcKYXZijAnhJS1jLLbiNtVl6a5LyEfNN+eyiyWcJOZXPzBo69oTsew==}
cpu: [loong64]
os: [linux]
libc: [glibc]
'@rollup/rollup-linux-loong64-musl@4.64.2':
resolution: {integrity: sha512-10N1I6oBpERqzGtHKcCLrEpHlbXSBbMK9TURHvqgWbIFZuU3oJROxJ0H2nHfq96TVI0XJHQ9qzolJMFd2tz0LA==}
cpu: [loong64]
os: [linux]
libc: [musl]
'@rollup/rollup-linux-ppc64-gnu@4.64.2':
resolution: {integrity: sha512-dhJrLVZ3wNNhwCmy9/VTg6Pq6J8a8eZ0n15dVRlbSWcDb+8V/1mDfbnvis+vt8W6Pbxzv6gUbTEl4piOT3bbUQ==}
cpu: [ppc64]
os: [linux]
libc: [glibc]
'@rollup/rollup-linux-ppc64-musl@4.64.2':
resolution: {integrity: sha512-tPd2QuwxkTxgju5eARyb3RdgBWWi/O9dn/NFH0VkvGbLzoQeXrH6qMABtW8IODb5lUeBzB3sy2U7qmI4MtQ72g==}
cpu: [ppc64]
os: [linux]
libc: [musl]
'@rollup/rollup-linux-riscv64-gnu@4.64.2':
resolution: {integrity: sha512-njtDdPvSQWI9CDhIPPxsiEMeHayCfu7wwcbSQ7c2kYPhGVviL+WGVhq6Jfj/b+VuhSTkFuQ0z4ZFbFjkWwBFmg==}
cpu: [riscv64]
os: [linux]
libc: [glibc]
'@rollup/rollup-linux-riscv64-musl@4.64.2':
resolution: {integrity: sha512-EbOBo3dqCj9ecEV9Z7zvulB6AieMP2ztnKQYGVOSN0pzIyOu+3JH2YJZnLmbPeVvfn6yP6Qi7NbBM27EzGbj2w==}
cpu: [riscv64]
os: [linux]
libc: [musl]
'@rollup/rollup-linux-s390x-gnu@4.64.2':
resolution: {integrity: sha512-8wLt2Q5n4qmfiE2Gk8V6KNLSf+oJuBX7Kdw5joFDtnZIZvirS0hLzBc79OiJjV71oYDmBUZLNy6pDgvylo4qiQ==}
cpu: [s390x]
os: [linux]
libc: [glibc]
'@rollup/rollup-linux-x64-gnu@4.64.2':
resolution: {integrity: sha512-aIChbkhC4d33bxkVP0LSi2/2lUSGRXE6k4Qa0G7tgpgTOK7kxHTixmx5qC3SKUJ6yUMznHBtQoWaG4QMqLUnhw==}
cpu: [x64]
os: [linux]
libc: [glibc]
'@rollup/rollup-linux-x64-musl@4.64.2':
resolution: {integrity: sha512-gQ+TkVZRfjZVcl/UOAxxfWw+sHjzIXoieNwxEyJlD3tRBmBhR8ZDq11w+QEgQDcpslEB+fhN/OEpaTsDoIIHhQ==}
cpu: [x64]
os: [linux]
libc: [musl]
'@rollup/rollup-openbsd-x64@4.64.2':
resolution: {integrity: sha512-sUaJpMKS2JBCvUk8fYimG7aMq2jHVT4kw/uVHxPolKLrMmkKy0KXirmZyZehdNpL7PffqAlPAWUELQcTiD8ChA==}
cpu: [x64]
os: [openbsd]
'@rollup/rollup-openharmony-arm64@4.64.2':
resolution: {integrity: sha512-j1RdJWb0CrlSuwt12OPjLucPlwfLtA/KMOKVvh5rXSJEtl/WDoIGR3CSykK/jLGTo9RqNfW5Ub6kY5Z5ccYl9w==}
cpu: [arm64]
os: [openharmony]
'@rollup/rollup-win32-arm64-msvc@4.64.2':
resolution: {integrity: sha512-i3r04aHCIY3eggTBLUp/4IxiBjDpAgQaK89Ue2nMJ2MKuIXX5JfoeZjHw+kPjGGNVDlhi8FT8xzuGMJZe3aehg==}
cpu: [arm64]
os: [win32]
'@rollup/rollup-win32-ia32-msvc@4.64.2':
resolution: {integrity: sha512-kCQ15h/uwNMGdN8gjXBZGf+aGE+ODzHfbi90/cAm4+Q89WoMiLODNoj8HoGZmYx87HPxfu5CBNPsZ+ZG4uIAoA==}
cpu: [ia32]
os: [win32]
'@rollup/rollup-win32-x64-gnu@4.64.2':
resolution: {integrity: sha512-yYAuySSs+0X2gSMJPw/Q4jHsHZHyMq1dpAE+XnN790Jg1WuReUAD/+j+e4vuDvS4PXJ4PA4NCzTR4OYtDf18/g==}
cpu: [x64]
os: [win32]
'@rollup/rollup-win32-x64-msvc@4.64.2':
resolution: {integrity: sha512-JY1rQdWOJrKutORqKO1uzrHXV86jFiQzfLFKx2s8S1CjLXL9fC5LOIvk0q7Z7PXJvsByt+sdoE1Q2a9gBr4gLQ==}
cpu: [x64]
os: [win32]
'@types/estree@1.0.9':
resolution: {integrity: sha512-GhdPgy1el4/ImP05X05Uw4cw2/M93BCUmnEvWZNStlCzEKME4Fkk+YpoA5OiHNQmoS7Cafb8Xa3Pya8m1Qrzeg==}
'@vitest/expect@2.1.9':
resolution: {integrity: sha512-UJCIkTBenHeKT1TTlKMJWy1laZewsRIzYighyYiJKZreqtdxSos/S1t+ktRMQWu2CKqaarrkeszJx1cgC5tGZw==}
'@vitest/mocker@2.1.9':
resolution: {integrity: sha512-tVL6uJgoUdi6icpxmdrn5YNo3g3Dxv+IHJBr0GXHaEdTcw3F+cPKnsXFhli6nO+f/6SDKPHEK1UN+k+TQv0Ehg==}
peerDependencies:
msw: ^2.4.9
vite: ^5.0.0
peerDependenciesMeta:
msw:
optional: true
vite:
optional: true
'@vitest/pretty-format@2.1.9':
resolution: {integrity: sha512-KhRIdGV2U9HOUzxfiHmY8IFHTdqtOhIzCpd8WRdJiE7D/HUcZVD0EgQCVjm+Q9gkUXWgBvMmTtZgIG48wq7sOQ==}
'@vitest/runner@2.1.9':
resolution: {integrity: sha512-ZXSSqTFIrzduD63btIfEyOmNcBmQvgOVsPNPe0jYtESiXkhd8u2erDLnMxmGrDCwHCCHE7hxwRDCT3pt0esT4g==}
'@vitest/snapshot@2.1.9':
resolution: {integrity: sha512-oBO82rEjsxLNJincVhLhaxxZdEtV0EFHMK5Kmx5sJ6H9L183dHECjiefOAdnqpIgT5eZwT04PoggUnW88vOBNQ==}
'@vitest/spy@2.1.9':
resolution: {integrity: sha512-E1B35FwzXXTs9FHNK6bDszs7mtydNi5MIfUWpceJ8Xbfb1gBMscAnwLbEu+B44ed6W3XjL9/ehLPHR1fkf1KLQ==}
'@vitest/utils@2.1.9':
resolution: {integrity: sha512-v0psaMSkNJ3A2NMrUEHFRzJtDPFn+/VWZ5WxImB21T9fjucJRmS7xCS3ppEnARb9y11OAzaD+P2Ps+b+BGX5iQ==}
assertion-error@2.0.1:
resolution: {integrity: sha512-Izi8RQcffqCeNVgFigKli1ssklIbpHnCYc6AknXGYoB6grJqyeby7jv12JUQgmTAnIDnbck1uxksT4dzN3PWBA==}
engines: {node: '>=12'}
cac@6.7.14:
resolution: {integrity: sha512-b6Ilus+c3RrdDk+JhLKUAQfzzgLEPy6wcXqS7f/xe1EETvsDP6GORG7SFuOs6cID5YkqchW/LXZbX5bc8j7ZcQ==}
engines: {node: '>=8'}
chai@5.3.3:
resolution: {integrity: sha512-4zNhdJD/iOjSH0A05ea+Ke6MU5mmpQcbQsSOkgdaUMJ9zTlDTD/GYlwohmIE2u0gaxHYiVHEn1Fw9mZ/ktJWgw==}
engines: {node: '>=18'}
check-error@2.1.3:
resolution: {integrity: sha512-PAJdDJusoxnwm1VwW07VWwUN1sl7smmC3OKggvndJFadxxDRyFJBX/ggnu/KE4kQAB7a3Dp8f/YXC1FlUprWmA==}
engines: {node: '>= 16'}
debug@4.4.3:
resolution: {integrity: sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA==}
engines: {node: '>=6.0'}
peerDependencies:
supports-color: '*'
peerDependenciesMeta:
supports-color:
optional: true
deep-eql@5.0.2:
resolution: {integrity: sha512-h5k/5U50IJJFpzfL6nO9jaaumfjO/f2NjK/oYB2Djzm4p9L+3T9qWpZqZ2hAbLPuuYq9wrU08WQyBTL5GbPk5Q==}
engines: {node: '>=6'}
es-module-lexer@1.7.0:
resolution: {integrity: sha512-jEQoCwk8hyb2AZziIOLhDqpm5+2ww5uIE6lkO/6jcOCusfk6LhMHpXXfBLXTZ7Ydyt0j4VoUQv6uGNYbdW+kBA==}
esbuild@0.21.5:
resolution: {integrity: sha512-mg3OPMV4hXywwpoDxu3Qda5xCKQi+vCTZq8S9J/EpkhB2HzKXq4SNFZE3+NK93JYxc8VMSep+lOUSC/RVKaBqw==}
engines: {node: '>=12'}
hasBin: true
estree-walker@3.0.3:
resolution: {integrity: sha512-7RUKfXgSMMkzt6ZuXmqapOurLGPPfgj6l9uRZ7lRGolvk0y2yocc35LdcxKC5PQZdn2DMqioAQ2NoWcrTKmm6g==}
expect-type@1.4.0:
resolution: {integrity: sha512-KfYbmpRm0VbLjEvVa9yGwCi9GI34xvi7A/HXYWQO65CSD2u3MczUJSuwXKFIxlGsgBQizV9q5J9NHj4VG0n+pA==}
engines: {node: '>=12.0.0'}
fsevents@2.3.3:
resolution: {integrity: sha512-5xoDfX+fL7faATnagmWPpbFtwh/R77WmMMqqHGS65C3vvB0YHrgF+B1YmZ3441tMj5n63k0212XNoJwzlhffQw==}
engines: {node: ^8.16.0 || ^10.6.0 || >=11.0.0}
os: [darwin]
loupe@3.2.1:
resolution: {integrity: sha512-CdzqowRJCeLU72bHvWqwRBBlLcMEtIvGrlvef74kMnV2AolS9Y8xUv1I0U/MNAWMhBlKIoyuEgoJ0t/bbwHbLQ==}
magic-string@0.30.21:
resolution: {integrity: sha512-vd2F4YUyEXKGcLHoq+TEyCjxueSeHnFxyyjNp80yg0XV4vUhnDer/lvvlqM/arB5bXQN5K2/3oinyCRyx8T2CQ==}
ms@2.1.3:
resolution: {integrity: sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==}
nanoid@3.3.20:
resolution: {integrity: sha512-uKdg2G3GNCKQn9byYOpxbGqrT2fGO5KRt5J/8b3pok8rT6qxGWF6hxMyJiEYtAf+FVyYuD9hRaDqX5uPFYJ4ZQ==}
engines: {node: ^10 || ^12 || ^13.7 || ^14 || >=15.0.1}
hasBin: true
pathe@1.1.2:
resolution: {integrity: sha512-whLdWMYL2TwI08hn8/ZqAbrVemu0LNaNNJZX73O6qaIdCTfXutsLhMkjdENX0qhsQ9uIimo4/aQOmXkoon2nDQ==}
pathval@2.0.1:
resolution: {integrity: sha512-//nshmD55c46FuFw26xV/xFAaB5HF9Xdap7HJBBnrKdAd6/GxDBaNA1870O79+9ueg61cZLSVc+OaFlfmObYVQ==}
engines: {node: '>= 14.16'}
picocolors@1.1.1:
resolution: {integrity: sha512-xceH2snhtb5M9liqDsmEw56le376mTZkEX/jEb/RxNFyegNul7eNslCXP9FDj/Lcu0X8KEyMceP2ntpaHrDEVA==}
postcss@8.5.29:
resolution: {integrity: sha512-49cGhUbXj8Qenv0iTMxA1cFBzxXoctpC9Ujd77t1WcbJIr6nF/eI7g/8MgxrYldFRuAXvja7xQRwavoW7kgrxQ==}
engines: {node: ^10 || ^12 || >=14}
rollup@4.64.2:
resolution: {integrity: sha512-RxXm0XVVV1hFI3dYe12ruaOgsYQ725lUcUWeMSQqkF9tTxVErY+fy2Ets6vWhItL9MZMSNZGbvZFc7rH/VHp2Q==}
engines: {node: '>=18.0.0', npm: '>=8.0.0'}
hasBin: true
siginfo@2.0.0:
resolution: {integrity: sha512-ybx0WO1/8bSBLEWXZvEd7gMW3Sn3JFlW3TvX1nREbDLRNQNaeNN8WK0meBwPdAaOI7TtRRRJn/Es1zhrrCHu7g==}
source-map-js@1.2.2:
resolution: {integrity: sha512-KGj/8Y43x35aZVDtt+J4mK1hoLGHULMYfSkODJNQjNDC3oW1PqPoxMwo0pLUsWM/UEGzON/NxeHywEfNXNP3Vw==}
engines: {node: '>=0.10.0'}
stackback@0.0.2:
resolution: {integrity: sha512-1XMJE5fQo1jGH6Y/7ebnwPOBEkIEnT4QF32d5R1+VXdXveM0IBMJt8zfaxX1P3QhVwrYe+576+jkANtSS2mBbw==}
std-env@3.10.0:
resolution: {integrity: sha512-5GS12FdOZNliM5mAOxFRg7Ir0pWz8MdpYm6AY6VPkGpbA7ZzmbzNcBJQ0GPvvyWgcY7QAhCgf9Uy89I03faLkg==}
tinybench@2.9.0:
resolution: {integrity: sha512-0+DUvqWMValLmha6lr4kD8iAMK1HzV0/aKnCtWb9v9641TnP/MFb7Pc2bxoxQjTXAErryXVgUOfv2YqNllqGeg==}
tinyexec@0.3.2:
resolution: {integrity: sha512-KQQR9yN7R5+OSwaK0XQoj22pwHoTlgYqmUscPYoknOoWCWfj/5/ABTMRi69FrKU5ffPVh5QcFikpWJI/P1ocHA==}
tinypool@1.1.1:
resolution: {integrity: sha512-Zba82s87IFq9A9XmjiX5uZA/ARWDrB03OHlq+Vw1fSdt0I+4/Kutwy8BP4Y/y/aORMo61FQ0vIb5j44vSo5Pkg==}
engines: {node: ^18.0.0 || >=20.0.0}
tinyrainbow@1.2.0:
resolution: {integrity: sha512-weEDEq7Z5eTHPDh4xjX789+fHfF+P8boiFB+0vbWzpbnbsEr/GRaohi/uMKxg8RZMXnl1ItAi/IUHWMsjDV7kQ==}
engines: {node: '>=14.0.0'}
tinyspy@3.0.2:
resolution: {integrity: sha512-n1cw8k1k0x4pgA2+9XrOkFydTerNcJ1zWCO5Nn9scWHTD+5tp8dghT2x1uduQePZTZgd3Tupf+x9BxJjeJi77Q==}
engines: {node: '>=14.0.0'}
vite-node@2.1.9:
resolution: {integrity: sha512-AM9aQ/IPrW/6ENLQg3AGY4K1N2TGZdR5e4gu/MmmR2xR3Ll1+dib+nook92g4TV3PXVyeyxdWwtaCAiUL0hMxA==}
engines: {node: ^18.0.0 || >=20.0.0}
hasBin: true
vite@5.4.21:
resolution: {integrity: sha512-o5a9xKjbtuhY6Bi5S3+HvbRERmouabWbyUcpXXUA1u+GNUKoROi9byOJ8M0nHbHYHkYICiMlqxkg1KkYmm25Sw==}
engines: {node: ^18.0.0 || >=20.0.0}
hasBin: true
peerDependencies:
'@types/node': ^18.0.0 || >=20.0.0
less: '*'
lightningcss: ^1.21.0
sass: '*'
sass-embedded: '*'
stylus: '*'
sugarss: '*'
terser: ^5.4.0
peerDependenciesMeta:
'@types/node':
optional: true
less:
optional: true
lightningcss:
optional: true
sass:
optional: true
sass-embedded:
optional: true
stylus:
optional: true
sugarss:
optional: true
terser:
optional: true
vitest@2.1.9:
resolution: {integrity: sha512-MSmPM9REYqDGBI8439mA4mWhV5sKmDlBKWIYbA3lRb2PTHACE0mgKwA8yQ2xq9vxDTuk4iPrECBAEW2aoFXY0Q==}
engines: {node: ^18.0.0 || >=20.0.0}
hasBin: true
peerDependencies:
'@edge-runtime/vm': '*'
'@types/node': ^18.0.0 || >=20.0.0
'@vitest/browser': 2.1.9
'@vitest/ui': 2.1.9
happy-dom: '*'
jsdom: '*'
peerDependenciesMeta:
'@edge-runtime/vm':
optional: true
'@types/node':
optional: true
'@vitest/browser':
optional: true
'@vitest/ui':
optional: true
happy-dom:
optional: true
jsdom:
optional: true
why-is-node-running@2.3.0:
resolution: {integrity: sha512-hUrmaWBdVDcxvYqnyh09zunKzROWjbZTiNy8dBEjkS7ehEDQibXJ7XvlmtbwuTclUiIyN+CyXQD4Vmko8fNm8w==}
engines: {node: '>=8'}
hasBin: true
snapshots:
'@esbuild/aix-ppc64@0.21.5':
optional: true
'@esbuild/android-arm64@0.21.5':
optional: true
'@esbuild/android-arm@0.21.5':
optional: true
'@esbuild/android-x64@0.21.5':
optional: true
'@esbuild/darwin-arm64@0.21.5':
optional: true
'@esbuild/darwin-x64@0.21.5':
optional: true
'@esbuild/freebsd-arm64@0.21.5':
optional: true
'@esbuild/freebsd-x64@0.21.5':
optional: true
'@esbuild/linux-arm64@0.21.5':
optional: true
'@esbuild/linux-arm@0.21.5':
optional: true
'@esbuild/linux-ia32@0.21.5':
optional: true
'@esbuild/linux-loong64@0.21.5':
optional: true
'@esbuild/linux-mips64el@0.21.5':
optional: true
'@esbuild/linux-ppc64@0.21.5':
optional: true
'@esbuild/linux-riscv64@0.21.5':
optional: true
'@esbuild/linux-s390x@0.21.5':
optional: true
'@esbuild/linux-x64@0.21.5':
optional: true
'@esbuild/netbsd-x64@0.21.5':
optional: true
'@esbuild/openbsd-x64@0.21.5':
optional: true
'@esbuild/sunos-x64@0.21.5':
optional: true
'@esbuild/win32-arm64@0.21.5':
optional: true
'@esbuild/win32-ia32@0.21.5':
optional: true
'@esbuild/win32-x64@0.21.5':
optional: true
'@jridgewell/sourcemap-codec@1.6.0': {}
'@napi-rs/lzma-linux-x64-gnu@1.5.1':
optional: true
'@rollup/rollup-android-arm-eabi@4.64.2':
optional: true
'@rollup/rollup-android-arm64@4.64.2':
optional: true
'@rollup/rollup-darwin-arm64@4.64.2':
optional: true
'@rollup/rollup-darwin-x64@4.64.2':
optional: true
'@rollup/rollup-freebsd-arm64@4.64.2':
optional: true
'@rollup/rollup-freebsd-x64@4.64.2':
optional: true
'@rollup/rollup-linux-arm-gnueabihf@4.64.2':
optional: true
'@rollup/rollup-linux-arm-musleabihf@4.64.2':
optional: true
'@rollup/rollup-linux-arm64-gnu@4.64.2':
optional: true
'@rollup/rollup-linux-arm64-musl@4.64.2':
optional: true
'@rollup/rollup-linux-loong64-gnu@4.64.2':
optional: true
'@rollup/rollup-linux-loong64-musl@4.64.2':
optional: true
'@rollup/rollup-linux-ppc64-gnu@4.64.2':
optional: true
'@rollup/rollup-linux-ppc64-musl@4.64.2':
optional: true
'@rollup/rollup-linux-riscv64-gnu@4.64.2':
optional: true
'@rollup/rollup-linux-riscv64-musl@4.64.2':
optional: true
'@rollup/rollup-linux-s390x-gnu@4.64.2':
optional: true
'@rollup/rollup-linux-x64-gnu@4.64.2':
optional: true
'@rollup/rollup-linux-x64-musl@4.64.2':
optional: true
'@rollup/rollup-openbsd-x64@4.64.2':
optional: true
'@rollup/rollup-openharmony-arm64@4.64.2':
optional: true
'@rollup/rollup-win32-arm64-msvc@4.64.2':
optional: true
'@rollup/rollup-win32-ia32-msvc@4.64.2':
optional: true
'@rollup/rollup-win32-x64-gnu@4.64.2':
optional: true
'@rollup/rollup-win32-x64-msvc@4.64.2':
optional: true
'@types/estree@1.0.9': {}
'@vitest/expect@2.1.9':
dependencies:
'@vitest/spy': 2.1.9
'@vitest/utils': 2.1.9
chai: 5.3.3
tinyrainbow: 1.2.0
'@vitest/mocker@2.1.9(vite@5.4.21)':
dependencies:
'@vitest/spy': 2.1.9
estree-walker: 3.0.3
magic-string: 0.30.21
optionalDependencies:
vite: 5.4.21
'@vitest/pretty-format@2.1.9':
dependencies:
tinyrainbow: 1.2.0
'@vitest/runner@2.1.9':
dependencies:
'@vitest/utils': 2.1.9
pathe: 1.1.2
'@vitest/snapshot@2.1.9':
dependencies:
'@vitest/pretty-format': 2.1.9
magic-string: 0.30.21
pathe: 1.1.2
'@vitest/spy@2.1.9':
dependencies:
tinyspy: 3.0.2
'@vitest/utils@2.1.9':
dependencies:
'@vitest/pretty-format': 2.1.9
loupe: 3.2.1
tinyrainbow: 1.2.0
assertion-error@2.0.1: {}
cac@6.7.14: {}
chai@5.3.3:
dependencies:
assertion-error: 2.0.1
check-error: 2.1.3
deep-eql: 5.0.2
loupe: 3.2.1
pathval: 2.0.1
check-error@2.1.3: {}
debug@4.4.3:
dependencies:
ms: 2.1.3
deep-eql@5.0.2: {}
es-module-lexer@1.7.0: {}
esbuild@0.21.5:
optionalDependencies:
'@esbuild/aix-ppc64': 0.21.5
'@esbuild/android-arm': 0.21.5
'@esbuild/android-arm64': 0.21.5
'@esbuild/android-x64': 0.21.5
'@esbuild/darwin-arm64': 0.21.5
'@esbuild/darwin-x64': 0.21.5
'@esbuild/freebsd-arm64': 0.21.5
'@esbuild/freebsd-x64': 0.21.5
'@esbuild/linux-arm': 0.21.5
'@esbuild/linux-arm64': 0.21.5
'@esbuild/linux-ia32': 0.21.5
'@esbuild/linux-loong64': 0.21.5
'@esbuild/linux-mips64el': 0.21.5
'@esbuild/linux-ppc64': 0.21.5
'@esbuild/linux-riscv64': 0.21.5
'@esbuild/linux-s390x': 0.21.5
'@esbuild/linux-x64': 0.21.5
'@esbuild/netbsd-x64': 0.21.5
'@esbuild/openbsd-x64': 0.21.5
'@esbuild/sunos-x64': 0.21.5
'@esbuild/win32-arm64': 0.21.5
'@esbuild/win32-ia32': 0.21.5
'@esbuild/win32-x64': 0.21.5
estree-walker@3.0.3:
dependencies:
'@types/estree': 1.0.9
expect-type@1.4.0: {}
fsevents@2.3.3:
optional: true
loupe@3.2.1: {}
magic-string@0.30.21:
dependencies:
'@jridgewell/sourcemap-codec': 1.6.0
ms@2.1.3: {}
nanoid@3.3.20: {}
pathe@1.1.2: {}
pathval@2.0.1: {}
picocolors@1.1.1: {}
postcss@8.5.29:
dependencies:
nanoid: 3.3.20
picocolors: 1.1.1
source-map-js: 1.2.2
rollup@4.64.2:
dependencies:
'@types/estree': 1.0.9
optionalDependencies:
'@napi-rs/lzma-linux-x64-gnu': 1.5.1
'@rollup/rollup-android-arm-eabi': 4.64.2
'@rollup/rollup-android-arm64': 4.64.2
'@rollup/rollup-darwin-arm64': 4.64.2
'@rollup/rollup-darwin-x64': 4.64.2
'@rollup/rollup-freebsd-arm64': 4.64.2
'@rollup/rollup-freebsd-x64': 4.64.2
'@rollup/rollup-linux-arm-gnueabihf': 4.64.2
'@rollup/rollup-linux-arm-musleabihf': 4.64.2
'@rollup/rollup-linux-arm64-gnu': 4.64.2
'@rollup/rollup-linux-arm64-musl': 4.64.2
'@rollup/rollup-linux-loong64-gnu': 4.64.2
'@rollup/rollup-linux-loong64-musl': 4.64.2
'@rollup/rollup-linux-ppc64-gnu': 4.64.2
'@rollup/rollup-linux-ppc64-musl': 4.64.2
'@rollup/rollup-linux-riscv64-gnu': 4.64.2
'@rollup/rollup-linux-riscv64-musl': 4.64.2
'@rollup/rollup-linux-s390x-gnu': 4.64.2
'@rollup/rollup-linux-x64-gnu': 4.64.2
'@rollup/rollup-linux-x64-musl': 4.64.2
'@rollup/rollup-openbsd-x64': 4.64.2
'@rollup/rollup-openharmony-arm64': 4.64.2
'@rollup/rollup-win32-arm64-msvc': 4.64.2
'@rollup/rollup-win32-ia32-msvc': 4.64.2
'@rollup/rollup-win32-x64-gnu': 4.64.2
'@rollup/rollup-win32-x64-msvc': 4.64.2
fsevents: 2.3.3
siginfo@2.0.0: {}
source-map-js@1.2.2: {}
stackback@0.0.2: {}
std-env@3.10.0: {}
tinybench@2.9.0: {}
tinyexec@0.3.2: {}
tinypool@1.1.1: {}
tinyrainbow@1.2.0: {}
tinyspy@3.0.2: {}
vite-node@2.1.9:
dependencies:
cac: 6.7.14
debug: 4.4.3
es-module-lexer: 1.7.0
pathe: 1.1.2
vite: 5.4.21
transitivePeerDependencies:
- '@types/node'
- less
- lightningcss
- sass
- sass-embedded
- stylus
- sugarss
- supports-color
- terser
vite@5.4.21:
dependencies:
esbuild: 0.21.5
postcss: 8.5.29
rollup: 4.64.2
optionalDependencies:
fsevents: 2.3.3
vitest@2.1.9:
dependencies:
'@vitest/expect': 2.1.9
'@vitest/mocker': 2.1.9(vite@5.4.21)
'@vitest/pretty-format': 2.1.9
'@vitest/runner': 2.1.9
'@vitest/snapshot': 2.1.9
'@vitest/spy': 2.1.9
'@vitest/utils': 2.1.9
chai: 5.3.3
debug: 4.4.3
expect-type: 1.4.0
magic-string: 0.30.21
pathe: 1.1.2
std-env: 3.10.0
tinybench: 2.9.0
tinyexec: 0.3.2
tinypool: 1.1.1
tinyrainbow: 1.2.0
vite: 5.4.21
vite-node: 2.1.9
why-is-node-running: 2.3.0
transitivePeerDependencies:
- less
- lightningcss
- msw
- sass
- sass-embedded
- stylus
- sugarss
- supports-color
- terser
why-is-node-running@2.3.0:
dependencies:
siginfo: 2.0.0
stackback: 0.0.2
hidden/pnpm-lock.yaml on GitHub
,hidden/pnpm-workspace.yaml
allowBuilds:
esbuild: true
hidden/pnpm-workspace.yaml on GitHub
는 그 설정입니다. -reference/F1.patch
diff --git a/src/domain/invoice.ts b/src/domain/invoice.ts
index 5b69fc9..0b9b69f 100644
--- a/src/domain/invoice.ts
+++ b/src/domain/invoice.ts
@@ -1,4 +1,4 @@
-import { ConflictError } from "./errors";
+import { ConflictError, ValidationError } from "./errors";
import { type Cents, type Currency, sum } from "./money";
export type InvoiceStatus = "draft" | "open" | "paid" | "void";
@@ -10,6 +10,12 @@ export type LineItem = {
unitPrice: Cents;
};
+export type Payment = {
+ /** cents, > 0 */
+ amount: Cents;
+ receivedAt: string;
+};
+
export type Invoice = {
id: string;
/** Assigned when the invoice is issued (INV-0001, ...). */
@@ -25,6 +31,8 @@ export type Invoice = {
dueDate: string | null;
paidAt: string | null;
voidedAt: string | null;
+ /** Every payment received, oldest first. A paid invoice's payments add up to its total. */
+ payments: Payment[];
};
export type InvoiceTotals = { subtotal: Cents; discount: Cents; total: Cents };
@@ -38,6 +46,14 @@ export function computeTotals(invoice: Pick<Invoice, "lines" | "discount">): Inv
return { subtotal, discount: invoice.discount, total: subtotal - invoice.discount };
}
+export function amountPaid(invoice: Pick<Invoice, "payments">): Cents {
+ return sum(invoice.payments.map((p) => p.amount));
+}
+
+export function balance(invoice: Pick<Invoice, "lines" | "discount" | "payments">): Cents {
+ return computeTotals(invoice).total - amountPaid(invoice);
+}
+
export function issue(invoice: Invoice, number: string, now: Date): Invoice {
if (invoice.status !== "draft") {
throw new ConflictError(`invoice ${invoice.id} is ${invoice.status}, only drafts can be issued`);
@@ -45,11 +61,28 @@ export function issue(invoice: Invoice, number: string, now: Date): Invoice {
return { ...invoice, status: "open", number, issuedAt: now.toISOString() };
}
+/** Pays whatever is left on an open invoice in one go. */
export function markPaid(invoice: Invoice, now: Date): Invoice {
if (invoice.status !== "open") {
throw new ConflictError(`invoice ${invoice.id} is ${invoice.status}, only open invoices can be paid`);
}
- return { ...invoice, status: "paid", paidAt: now.toISOString() };
+ const remaining = balance(invoice);
+ const payments = remaining > 0 ? [...invoice.payments, { amount: remaining, receivedAt: now.toISOString() }] : invoice.payments;
+ return { ...invoice, payments, status: "paid", paidAt: now.toISOString() };
+}
+
+/** Records a partial (or final) payment; the invoice flips to paid when its balance reaches zero. */
+export function recordPayment(invoice: Invoice, amount: Cents, now: Date): Invoice {
+ if (invoice.status !== "open") {
+ throw new ConflictError(`invoice ${invoice.id} is ${invoice.status}, only open invoices can take payments`);
+ }
+ const remaining = balance(invoice);
+ if (amount > remaining) {
+ throw new ValidationError(`amount ${amount} exceeds the balance of ${remaining}`, "amount");
+ }
+ const payments = [...invoice.payments, { amount, receivedAt: now.toISOString() }];
+ const settled = amount === remaining;
+ return { ...invoice, payments, status: settled ? "paid" : "open", paidAt: settled ? now.toISOString() : null };
}
export function voidInvoice(invoice: Invoice, now: Date): Invoice {
diff --git a/src/domain/validation.ts b/src/domain/validation.ts
index 08dfa92..f2af418 100644
--- a/src/domain/validation.ts
+++ b/src/domain/validation.ts
@@ -10,6 +10,13 @@ export type NewInvoiceInput = {
dueDate: string | null;
};
+export function parsePayment(body: unknown): { amount: number } {
+ if (typeof body !== "object" || body === null) {
+ throw new ValidationError("body must be an object");
+ }
+ return { amount: assertCents((body as Record<string, unknown>).amount, "amount", { min: 1 }) };
+}
+
const DATE = /^\d{4}-\d{2}-\d{2}$/;
export function parseLineItems(value: unknown): LineItem[] {
diff --git a/src/http/handlers/invoices.ts b/src/http/handlers/invoices.ts
index 09959e1..ac2768e 100644
--- a/src/http/handlers/invoices.ts
+++ b/src/http/handlers/invoices.ts
@@ -1,6 +1,6 @@
import { ValidationError } from "../../domain/errors";
-import { type Invoice, type InvoiceStatus, issue, markPaid, voidInvoice } from "../../domain/invoice";
-import { parseNewInvoice } from "../../domain/validation";
+import { type Invoice, type InvoiceStatus, issue, markPaid, recordPayment, voidInvoice } from "../../domain/invoice";
+import { parseNewInvoice, parsePayment } from "../../domain/validation";
import type { AppContext } from "../context";
import type { Router } from "../router";
import { invoiceView } from "../views";
@@ -22,6 +22,7 @@ export function invoiceRoutes(router: Router, ctx: AppContext): void {
issuedAt: null,
paidAt: null,
voidedAt: null,
+ payments: [],
};
return { status: 201, body: invoiceView(ctx.invoices.insert(invoice)) };
});
@@ -51,6 +52,14 @@ export function invoiceRoutes(router: Router, ctx: AppContext): void {
return { status: 200, body: invoiceView(ctx.invoices.save(paid)) };
});
+ /** Records a partial payment; the last one (balance reaches zero) marks the invoice paid. */
+ router.on("POST", "/invoices/:id/payments", ({ params, body }) => {
+ const current = ctx.invoices.get(params.id);
+ const { amount } = parsePayment(body);
+ const updated = recordPayment(current, amount, ctx.now());
+ return { status: 201, body: invoiceView(ctx.invoices.save(updated)) };
+ });
+
router.on("POST", "/invoices/:id/void", ({ params }) => {
const voided = voidInvoice(ctx.invoices.get(params.id), ctx.now());
return { status: 200, body: invoiceView(ctx.invoices.save(voided)) };
diff --git a/src/http/handlers/reports.ts b/src/http/handlers/reports.ts
index 19262c9..656969e 100644
--- a/src/http/handlers/reports.ts
+++ b/src/http/handlers/reports.ts
@@ -1,18 +1,17 @@
-import { computeTotals, isOverdue } from "../../domain/invoice";
+import { balance, isOverdue } from "../../domain/invoice";
import type { Currency } from "../../domain/money";
import type { AppContext } from "../context";
import type { Router } from "../router";
export function reportRoutes(router: Router, ctx: AppContext): void {
- /** What customers still owe us, per currency, over every open invoice. */
+ /** What customers still owe us (balances, net of partial payments), per currency, over every open invoice. */
router.on("GET", "/reports/outstanding", () => {
const now = ctx.now();
const totals: Partial<Record<Currency, number>> = {};
let count = 0;
let overdue = 0;
for (const invoice of ctx.invoices.list({ status: "open" })) {
- const { total } = computeTotals(invoice);
- totals[invoice.currency] = (totals[invoice.currency] ?? 0) + total;
+ totals[invoice.currency] = (totals[invoice.currency] ?? 0) + balance(invoice);
count += 1;
if (isOverdue(invoice, now)) overdue += 1;
}
diff --git a/src/http/views.ts b/src/http/views.ts
index 856944c..5b11bff 100644
--- a/src/http/views.ts
+++ b/src/http/views.ts
@@ -1,6 +1,6 @@
-import { computeTotals, type Invoice } from "../domain/invoice";
+import { amountPaid, balance, computeTotals, type Invoice } from "../domain/invoice";
/** The JSON shape returned for an invoice: stored fields plus computed totals. */
export function invoiceView(invoice: Invoice) {
- return { ...invoice, ...computeTotals(invoice) };
+ return { ...invoice, ...computeTotals(invoice), amountPaid: amountPaid(invoice), balance: balance(invoice) };
}
diff --git a/src/store/migrations.ts b/src/store/migrations.ts
index c8b73a9..1eab4df 100644
--- a/src/store/migrations.ts
+++ b/src/store/migrations.ts
@@ -1,6 +1,7 @@
+import { computeTotals, type Invoice } from "../domain/invoice";
import type { StoreData } from "./types";
-export const CURRENT_SCHEMA_VERSION = 1;
+export const CURRENT_SCHEMA_VERSION = 2;
type RawData = Record<string, unknown> & { schemaVersion?: number };
type Migration = (data: RawData) => RawData;
@@ -21,9 +22,31 @@ const v0ToV1: Migration = (data) => {
};
};
+/**
+ * v1 -> v2: invoices carry `payments`. Invoices paid before payments existed were paid in full:
+ * they get one payment of their total at `paidAt`, so their balance is zero.
+ */
+const v1ToV2: Migration = (data) => {
+ const invoices = (data.invoices as Record<string, unknown>[] | undefined) ?? [];
+ return {
+ ...data,
+ schemaVersion: 2,
+ invoices: invoices.map((inv) => {
+ if (Array.isArray(inv.payments)) return inv;
+ const invoice = inv as unknown as Invoice;
+ const payments =
+ invoice.status === "paid"
+ ? [{ amount: computeTotals(invoice).total, receivedAt: invoice.paidAt ?? invoice.issuedAt ?? invoice.createdAt }]
+ : [];
+ return { ...inv, payments };
+ }),
+ };
+};
+
/** migrations[n] upgrades a file from version n to n + 1. */
const migrations: Record<number, Migration> = {
0: v0ToV1,
+ 1: v1ToV2,
};
export function migrate(raw: unknown): StoreData {
diff --git a/test/invoice.test.ts b/test/invoice.test.ts
index 85202f2..e043b5d 100644
--- a/test/invoice.test.ts
+++ b/test/invoice.test.ts
@@ -21,6 +21,7 @@ function draft(overrides: Partial<Invoice> = {}): Invoice {
dueDate: null,
paidAt: null,
voidedAt: null,
+ payments: [],
...overrides,
};
}
diff --git a/test/payments.test.ts b/test/payments.test.ts
new file mode 100644
index 0000000..baf9da7
--- /dev/null
+++ b/test/payments.test.ts
@@ -0,0 +1,53 @@
+import { describe, expect, it } from "vitest";
+import { openStore } from "../src/store/json-store";
+import { seedCustomer, seedOpenInvoice, testApp } from "./helpers";
+
+describe("partial payments", () => {
+ it("accumulates payments and flips to paid at a zero balance", async () => {
+ const app = testApp();
+ const id = await seedOpenInvoice(app, { customerId: await seedCustomer(app), unitPrice: 10_000 });
+ expect((await app.handle("POST", `/invoices/${id}/payments`, { amount: 4_000 })).body).toMatchObject({
+ status: "open",
+ amountPaid: 4_000,
+ balance: 6_000,
+ });
+ const last = await app.handle("POST", `/invoices/${id}/payments`, { amount: 6_000 });
+ expect(last).toMatchObject({ status: 201, body: { status: "paid", amountPaid: 10_000, balance: 0 } });
+ });
+
+ it("rejects bad amounts and overpayments with 400", async () => {
+ const app = testApp();
+ const id = await seedOpenInvoice(app, { customerId: await seedCustomer(app), unitPrice: 10_000 });
+ for (const amount of [0, -1, 1.5, "10", undefined]) {
+ expect((await app.handle("POST", `/invoices/${id}/payments`, { amount })).status).toBe(400);
+ }
+ expect((await app.handle("POST", `/invoices/${id}/payments`, { amount: 10_001 })).status).toBe(400);
+ });
+
+ it("refuses payments on drafts, void and unknown invoices", async () => {
+ const app = testApp();
+ const customerId = await seedCustomer(app);
+ const draft = await app.handle("POST", "/invoices", {
+ customerId,
+ currency: "EUR",
+ lines: [{ description: "x", quantity: 1, unitPrice: 100 }],
+ });
+ expect((await app.handle("POST", `/invoices/${(draft.body as { id: string }).id}/payments`, { amount: 10 })).status).toBe(409);
+ const voided = await seedOpenInvoice(app, { customerId });
+ await app.handle("POST", `/invoices/${voided}/void`);
+ expect((await app.handle("POST", `/invoices/${voided}/payments`, { amount: 10 })).status).toBe(409);
+ expect((await app.handle("POST", "/invoices/inv_404/payments", { amount: 10 })).status).toBe(404);
+ });
+
+ it("keeps /pay consistent and reports balances, not totals", async () => {
+ const app = testApp(openStore());
+ const customerId = await seedCustomer(app);
+ const a = await seedOpenInvoice(app, { customerId, unitPrice: 10_000 });
+ const b = await seedOpenInvoice(app, { customerId, unitPrice: 5_000 });
+ await app.handle("POST", `/invoices/${a}/payments`, { amount: 2_500 });
+ expect((await app.handle("GET", "/reports/outstanding")).body).toMatchObject({ totals: { EUR: 12_500 } });
+ expect((await app.handle("POST", `/invoices/${a}/pay`)).body).toMatchObject({ status: "paid", amountPaid: 10_000, balance: 0 });
+ expect((await app.handle("GET", "/reports/outstanding")).body).toMatchObject({ totals: { EUR: 5_000 }, count: 1 });
+ expect(b).toBeTruthy();
+ });
+});
diff --git a/test/store.test.ts b/test/store.test.ts
index a1bf345..3201e2a 100644
--- a/test/store.test.ts
+++ b/test/store.test.ts
@@ -42,6 +42,26 @@ describe("store", () => {
expect(JSON.parse(readFileSync(file, "utf8")).schemaVersion).toBe(CURRENT_SCHEMA_VERSION);
});
+ it("migrates a v1 file: invoices paid before payments existed get one payment of their total", () => {
+ const file = tmpFile();
+ const inv = { customerId: "cus_1", currency: "EUR", lines: [{ description: "x", quantity: 2, unitPrice: 1_500 }], discount: 0 };
+ writeFileSync(
+ file,
+ JSON.stringify({
+ schemaVersion: 1,
+ counters: { customer: 1, invoice: 2, invoiceNumber: 2 },
+ customers: [],
+ invoices: [
+ { ...inv, id: "inv_1", status: "paid", paidAt: "2026-02-01T00:00:00.000Z" },
+ { ...inv, id: "inv_2", status: "open", paidAt: null },
+ ],
+ }),
+ );
+ const [paid, open] = openStore({ file }).read().invoices;
+ expect(paid.payments).toEqual([{ amount: 3_000, receivedAt: "2026-02-01T00:00:00.000Z" }]);
+ expect(open.payments).toEqual([]);
+ });
+
it("refuses a file from a newer build", () => {
expect(() => migrate({ schemaVersion: CURRENT_SCHEMA_VERSION + 1 })).toThrow(/newer/);
});
reference/F2.patch
diff --git a/src/domain/credit-note.ts b/src/domain/credit-note.ts
new file mode 100644
index 0000000..7504c53
--- /dev/null
+++ b/src/domain/credit-note.ts
@@ -0,0 +1,44 @@
+import { ConflictError, ValidationError } from "./errors";
+import { computeTotals, type Invoice } from "./invoice";
+import { assertCents, type Cents, sum } from "./money";
+
+/** A numbered accounting document that reduces what is owed on an issued invoice. Never edited or deleted. */
+export type CreditNote = {
+ id: string;
+ /** CN-0001, ... gapless, in the order credit notes are created. */
+ number: string;
+ invoiceId: string;
+ amount: Cents;
+ createdAt: string;
+};
+
+export function creditedAmount(notes: Pick<CreditNote, "amount">[]): Cents {
+ return sum(notes.map((n) => n.amount));
+}
+
+/** What can still be credited on an invoice: its total minus every credit note already issued against it. */
+export function creditable(invoice: Invoice, notes: Pick<CreditNote, "amount">[]): Cents {
+ return computeTotals(invoice).total - creditedAmount(notes);
+}
+
+export function parseCreditAmount(body: unknown): Cents {
+ if (typeof body !== "object" || body === null) {
+ throw new ValidationError("body must be an object");
+ }
+ return assertCents((body as Record<string, unknown>).amount, "amount", { min: 1 });
+}
+
+/** Throws unless `amount` can be credited on `invoice` now. Call it BEFORE taking a number. */
+export function assertCreditable(invoice: Invoice, notes: Pick<CreditNote, "amount">[], amount: Cents): void {
+ if (invoice.status !== "open" && invoice.status !== "paid") {
+ throw new ConflictError(`invoice ${invoice.id} is ${invoice.status}: only open or paid invoices can be credited`);
+ }
+ const left = creditable(invoice, notes);
+ if (amount > left) {
+ throw new ValidationError(`amount ${amount} exceeds what is left to credit (${left})`, "amount");
+ }
+}
+
+export function creditNoteNumber(n: number): string {
+ return `CN-${String(n).padStart(4, "0")}`;
+}
diff --git a/src/domain/invoice.ts b/src/domain/invoice.ts
index 5b69fc9..2ee6fa4 100644
--- a/src/domain/invoice.ts
+++ b/src/domain/invoice.ts
@@ -52,8 +52,9 @@ export function markPaid(invoice: Invoice, now: Date): Invoice {
return { ...invoice, status: "paid", paidAt: now.toISOString() };
}
+/** Voids a draft, open or paid invoice. A paid one must be fully credited first (the handler issues that credit note). */
export function voidInvoice(invoice: Invoice, now: Date): Invoice {
- if (invoice.status === "paid" || invoice.status === "void") {
+ if (invoice.status === "void") {
throw new ConflictError(`invoice ${invoice.id} is ${invoice.status} and cannot be voided`);
}
return { ...invoice, status: "void", voidedAt: now.toISOString() };
diff --git a/src/http/context.ts b/src/http/context.ts
index c0a89d3..003b7af 100644
--- a/src/http/context.ts
+++ b/src/http/context.ts
@@ -1,4 +1,5 @@
import type { Store } from "../store/json-store";
+import { creditNoteRepo } from "../store/credit-note-repo";
import { customerRepo } from "../store/customer-repo";
import { invoiceRepo } from "../store/invoice-repo";
@@ -7,8 +8,9 @@ export type AppContext = {
now: () => Date;
customers: ReturnType<typeof customerRepo>;
invoices: ReturnType<typeof invoiceRepo>;
+ creditNotes: ReturnType<typeof creditNoteRepo>;
};
export function createContext(store: Store, now: () => Date = () => new Date()): AppContext {
- return { store, now, customers: customerRepo(store), invoices: invoiceRepo(store) };
+ return { store, now, customers: customerRepo(store), invoices: invoiceRepo(store), creditNotes: creditNoteRepo(store) };
}
diff --git a/src/http/handlers/invoices.ts b/src/http/handlers/invoices.ts
index 09959e1..469a402 100644
--- a/src/http/handlers/invoices.ts
+++ b/src/http/handlers/invoices.ts
@@ -1,3 +1,4 @@
+import { assertCreditable, type CreditNote, creditNoteNumber, creditable, parseCreditAmount } from "../../domain/credit-note";
import { ValidationError } from "../../domain/errors";
import { type Invoice, type InvoiceStatus, issue, markPaid, voidInvoice } from "../../domain/invoice";
import { parseNewInvoice } from "../../domain/validation";
@@ -51,8 +52,37 @@ export function invoiceRoutes(router: Router, ctx: AppContext): void {
return { status: 200, body: invoiceView(ctx.invoices.save(paid)) };
});
+ /** Validation first, number last: a rejected credit note never consumes a CN number (numbers stay gapless). */
+ function credit(invoice: Invoice, amount: number): CreditNote {
+ assertCreditable(invoice, ctx.creditNotes.forInvoice(invoice.id), amount);
+ const n = ctx.store.nextId("creditNote");
+ return ctx.creditNotes.insert({
+ id: `cn_${n}`,
+ number: creditNoteNumber(n),
+ invoiceId: invoice.id,
+ amount,
+ createdAt: ctx.now().toISOString(),
+ });
+ }
+
+ router.on("POST", "/invoices/:id/credit-notes", ({ params, body }) => {
+ const invoice = ctx.invoices.get(params.id);
+ return { status: 201, body: credit(invoice, parseCreditAmount(body)) };
+ });
+
+ router.on("GET", "/invoices/:id/credit-notes", ({ params }) => {
+ ctx.invoices.get(params.id);
+ return { status: 200, body: ctx.creditNotes.forInvoice(params.id) };
+ });
+
+ /** Voiding a paid invoice credits whatever is left on it first. */
router.on("POST", "/invoices/:id/void", ({ params }) => {
- const voided = voidInvoice(ctx.invoices.get(params.id), ctx.now());
+ const invoice = ctx.invoices.get(params.id);
+ const voided = voidInvoice(invoice, ctx.now());
+ if (invoice.status === "paid") {
+ const left = creditable(invoice, ctx.creditNotes.forInvoice(invoice.id));
+ if (left > 0) credit(invoice, left);
+ }
return { status: 200, body: invoiceView(ctx.invoices.save(voided)) };
});
}
diff --git a/src/http/handlers/reports.ts b/src/http/handlers/reports.ts
index 19262c9..f980c9b 100644
--- a/src/http/handlers/reports.ts
+++ b/src/http/handlers/reports.ts
@@ -1,4 +1,5 @@
-import { computeTotals, isOverdue } from "../../domain/invoice";
+import { creditable } from "../../domain/credit-note";
+import { isOverdue } from "../../domain/invoice";
import type { Currency } from "../../domain/money";
import type { AppContext } from "../context";
import type { Router } from "../router";
@@ -11,8 +12,8 @@ export function reportRoutes(router: Router, ctx: AppContext): void {
let count = 0;
let overdue = 0;
for (const invoice of ctx.invoices.list({ status: "open" })) {
- const { total } = computeTotals(invoice);
- totals[invoice.currency] = (totals[invoice.currency] ?? 0) + total;
+ const owed = creditable(invoice, ctx.creditNotes.forInvoice(invoice.id));
+ totals[invoice.currency] = (totals[invoice.currency] ?? 0) + owed;
count += 1;
if (isOverdue(invoice, now)) overdue += 1;
}
diff --git a/src/store/credit-note-repo.ts b/src/store/credit-note-repo.ts
new file mode 100644
index 0000000..eac6632
--- /dev/null
+++ b/src/store/credit-note-repo.ts
@@ -0,0 +1,17 @@
+import type { CreditNote } from "../domain/credit-note";
+import type { Store } from "./json-store";
+
+export function creditNoteRepo(store: Store) {
+ return {
+ forInvoice(invoiceId: string): CreditNote[] {
+ return store.read().creditNotes.filter((n) => n.invoiceId === invoiceId);
+ },
+ all(): CreditNote[] {
+ return store.read().creditNotes;
+ },
+ insert(note: CreditNote): CreditNote {
+ store.update((d) => ({ ...d, creditNotes: [...d.creditNotes, note] }));
+ return note;
+ },
+ };
+}
diff --git a/src/store/json-store.ts b/src/store/json-store.ts
index b404c51..99205b7 100644
--- a/src/store/json-store.ts
+++ b/src/store/json-store.ts
@@ -13,9 +13,10 @@ export type Store = {
function empty(): StoreData {
return {
schemaVersion: CURRENT_SCHEMA_VERSION,
- counters: { customer: 0, invoice: 0, invoiceNumber: 0 },
+ counters: { customer: 0, invoice: 0, invoiceNumber: 0, creditNote: 0 },
customers: [],
invoices: [],
+ creditNotes: [],
};
}
diff --git a/src/store/migrations.ts b/src/store/migrations.ts
index c8b73a9..eb7abfb 100644
--- a/src/store/migrations.ts
+++ b/src/store/migrations.ts
@@ -1,6 +1,6 @@
import type { StoreData } from "./types";
-export const CURRENT_SCHEMA_VERSION = 1;
+export const CURRENT_SCHEMA_VERSION = 2;
type RawData = Record<string, unknown> & { schemaVersion?: number };
type Migration = (data: RawData) => RawData;
@@ -21,9 +21,18 @@ const v0ToV1: Migration = (data) => {
};
};
+/** v1 -> v2: credit notes (a collection and their own counter). */
+const v1ToV2: Migration = (data) => ({
+ ...data,
+ schemaVersion: 2,
+ counters: { ...(data.counters as Record<string, number> | undefined), creditNote: 0 },
+ creditNotes: [],
+});
+
/** migrations[n] upgrades a file from version n to n + 1. */
const migrations: Record<number, Migration> = {
0: v0ToV1,
+ 1: v1ToV2,
};
export function migrate(raw: unknown): StoreData {
@@ -43,8 +52,9 @@ export function migrate(raw: unknown): StoreData {
}
return {
schemaVersion: CURRENT_SCHEMA_VERSION,
- counters: (data.counters as StoreData["counters"]) ?? { customer: 0, invoice: 0, invoiceNumber: 0 },
+ counters: { customer: 0, invoice: 0, invoiceNumber: 0, creditNote: 0, ...(data.counters as Partial<StoreData["counters"]>) },
customers: (data.customers as StoreData["customers"]) ?? [],
invoices: (data.invoices as StoreData["invoices"]) ?? [],
+ creditNotes: (data.creditNotes as StoreData["creditNotes"]) ?? [],
};
}
diff --git a/src/store/types.ts b/src/store/types.ts
index 222b191..a9d7277 100644
--- a/src/store/types.ts
+++ b/src/store/types.ts
@@ -1,7 +1,8 @@
+import type { CreditNote } from "../domain/credit-note";
import type { Customer } from "../domain/customer";
import type { Invoice } from "../domain/invoice";
-export type Counters = { customer: number; invoice: number; invoiceNumber: number };
+export type Counters = { customer: number; invoice: number; invoiceNumber: number; creditNote: number };
/** The shape of the JSON file at CURRENT_SCHEMA_VERSION. */
export type StoreData = {
@@ -9,4 +10,5 @@ export type StoreData = {
counters: Counters;
customers: Customer[];
invoices: Invoice[];
+ creditNotes: CreditNote[];
};
diff --git a/test/credit-notes.test.ts b/test/credit-notes.test.ts
new file mode 100644
index 0000000..240f820
--- /dev/null
+++ b/test/credit-notes.test.ts
@@ -0,0 +1,52 @@
+import { mkdtempSync, writeFileSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+import { describe, expect, it } from "vitest";
+import { openStore } from "../src/store/json-store";
+import { seedCustomer, seedOpenInvoice, testApp } from "./helpers";
+
+describe("credit notes", () => {
+ it("numbers credit notes gaplessly; rejected ones take no number", async () => {
+ const app = testApp();
+ const id = await seedOpenInvoice(app, { customerId: await seedCustomer(app), unitPrice: 10_000 });
+ expect((await app.handle("POST", `/invoices/${id}/credit-notes`, { amount: 3_000 })).body).toMatchObject({ number: "CN-0001" });
+ expect((await app.handle("POST", `/invoices/${id}/credit-notes`, { amount: 7_001 })).status).toBe(400);
+ expect((await app.handle("POST", `/invoices/${id}/credit-notes`, { amount: 0 })).status).toBe(400);
+ expect((await app.handle("POST", `/invoices/${id}/credit-notes`, { amount: 7_000 })).body).toMatchObject({ number: "CN-0002" });
+ expect(((await app.handle("GET", `/invoices/${id}/credit-notes`)).body as unknown[]).length).toBe(2);
+ });
+
+ it("refuses drafts and void invoices (409) and unknown ones (404)", async () => {
+ const app = testApp();
+ const customerId = await seedCustomer(app);
+ const draft = await app.handle("POST", "/invoices", { customerId, currency: "EUR", lines: [{ description: "x", quantity: 1, unitPrice: 100 }] });
+ expect((await app.handle("POST", `/invoices/${(draft.body as { id: string }).id}/credit-notes`, { amount: 10 })).status).toBe(409);
+ expect((await app.handle("POST", "/invoices/inv_404/credit-notes", { amount: 10 })).status).toBe(404);
+ });
+
+ it("voids a paid invoice by crediting what is left", async () => {
+ const app = testApp();
+ const id = await seedOpenInvoice(app, { customerId: await seedCustomer(app), unitPrice: 10_000 });
+ await app.handle("POST", `/invoices/${id}/pay`);
+ await app.handle("POST", `/invoices/${id}/credit-notes`, { amount: 2_500 });
+ expect((await app.handle("POST", `/invoices/${id}/void`)).body).toMatchObject({ status: "void" });
+ const notes = (await app.handle("GET", `/invoices/${id}/credit-notes`)).body as { amount: number }[];
+ expect(notes.map((n) => n.amount)).toEqual([2_500, 7_500]);
+ });
+
+ it("nets credit notes out of the outstanding report", async () => {
+ const app = testApp();
+ const id = await seedOpenInvoice(app, { customerId: await seedCustomer(app), unitPrice: 10_000 });
+ await app.handle("POST", `/invoices/${id}/credit-notes`, { amount: 4_000 });
+ expect((await app.handle("GET", "/reports/outstanding")).body).toMatchObject({ totals: { EUR: 6_000 }, count: 1 });
+ });
+
+ it("works on a store file written before credit notes existed", async () => {
+ const file = join(mkdtempSync(join(tmpdir(), "ledgerly-")), "data.json");
+ writeFileSync(file, JSON.stringify({ schemaVersion: 1, counters: { customer: 0, invoice: 0, invoiceNumber: 0 }, customers: [], invoices: [] }));
+ const app = testApp(openStore({ file }));
+ const id = await seedOpenInvoice(app, { customerId: await seedCustomer(app) });
+ expect((await app.handle("POST", `/invoices/${id}/credit-notes`, { amount: 100 })).body).toMatchObject({ id: "cn_1", number: "CN-0001" });
+ expect(openStore({ file }).read().creditNotes.length).toBe(1);
+ });
+});
diff --git a/test/invoice.test.ts b/test/invoice.test.ts
index 85202f2..4219226 100644
--- a/test/invoice.test.ts
+++ b/test/invoice.test.ts
@@ -54,8 +54,11 @@ describe("invoice domain", () => {
});
it("voids drafts and open invoices, never paid ones", () => {
+ // Meaning changed with credit notes: a paid invoice can now be voided (the handler credits what is left first);
+ // only an already void invoice is refused.
expect(voidInvoice(draft(), NOW).status).toBe("void");
- expect(() => voidInvoice(draft({ status: "paid" }), NOW)).toThrow(ConflictError);
+ expect(voidInvoice(draft({ status: "paid" }), NOW).status).toBe("void");
+ expect(() => voidInvoice(draft({ status: "void" }), NOW)).toThrow(ConflictError);
});
it("knows when an open invoice is overdue", () => {
러너와 driver:
run.sh
#!/usr/bin/env bash
# One run: fresh work copy + pack fixtures, the driver (worker turns + PO), then finalize.
# Usage: [TASK=F1|F2] run.sh <cfg> <rep> cfg: bare|sp|spd|mp|both
# RUN_TAG=x -> runs/<name>-x (smoke runs; analyze.py ignores tagged runs)
# The work copy lives OUTSIDE the repo tree ($FLOW_BENCH_WORK, default /private/tmp/flow-bench/<name>): Claude Code
# loads every CLAUDE.md of the parent directories, so a work copy under this repo read the repo's CLAUDE.md (and the
# instructions it pulls in) in every session. After finalize the copy (without node_modules) is archived to work/<name>.
set -uo pipefail
CFG="$1"; REP="${2:-1}"
HERE="$(cd "$(dirname "$0")" && pwd)"
. "$HERE/env.sh"
TASK="${TASK:-F1}"; export TASK
PFX=""; [ "$TASK" != "F1" ] && PFX="$(echo "$TASK" | tr "A-Z" "a-z")-"
NAME="${PFX}${CFG}-r${REP}${RUN_TAG:+-$RUN_TAG}"
WROOT="${FLOW_BENCH_WORK:-/private/tmp/flow-bench}"
OUT="$HERE/runs/$NAME"; W="$WROOT/$NAME"
rm -rf "$OUT"; mkdir -p "$OUT"
"$HERE/prep.sh" "$CFG" "$W" > "$OUT/prep.txt" 2>&1 || { echo "$NAME: prep failed"; cat "$OUT/prep.txt"; exit 2; }
BASE=$(tail -1 "$OUT/prep.txt")
cp "$W/.claude/settings.json" "$OUT/settings.json"
printf '{"task":"%s","cfg":"%s","rep":%s,"tag":"%s","work":"%s","name":"%s","base":"%s","started":"%s","claude_version":"%s"}\n' \
"$TASK" "$CFG" "$REP" "${RUN_TAG:-}" "$W" "$NAME" "$BASE" "$(date '+%F %T')" "$(claude --version | head -1)" > "$OUT/meta.json"
python3 "$HERE/lib/driver.py" "$CFG" "$OUT" "$W" "$BASE" > "$OUT/driver.out" 2> "$OUT/driver.err"
python3 "$HERE/lib/finalize.py" "$OUT"
rm -rf "$HERE/work/$NAME"; mkdir -p "$HERE/work"
rsync -a --exclude node_modules "$W/" "$HERE/work/$NAME/"
prep.sh
#!/usr/bin/env bash
# Fresh work copy of upstream (node_modules included) + the pack fixtures of <cfg>.
# Usage: prep.sh <cfg> <work dir> Prints the base commit (the oracle's BASE) on its last line.
set -euo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"
CFG="$1"; W="$2"
SP=$(python3 -c "import json;print(int(json.load(open('$HERE/configs.json'))['$CFG']['superpowers']))")
MP=$(python3 -c "import json;print(int(json.load(open('$HERE/configs.json'))['$CFG']['mattpocock']))")
rm -rf "$W"; mkdir -p "$(dirname "$W")"
cp -R "$HERE/upstream" "$W"
cd "$W"
# .claude/ never shows in git status or in any diff (hook config, installed skills).
echo ".claude/" >> .git/info/exclude
mkdir -p .claude/skills
if [ "$MP" = 1 ]; then
cp -R "$HERE/packs/mattpocock-skills/.claude/skills/." .claude/skills/
# One-time per-repo precondition of the pack (/setup-matt-pocock-skills), answered once with the defaults it
# recommends for a repo with no remote: local-markdown tracker, single-context domain docs, no triage labels
# (triage is installed, so Section B would ask; default labels kept). Committed so it never shows in the diff.
S="$HERE/packs/mattpocock-skills/.claude/skills/setup-matt-pocock-skills"
mkdir -p docs/agents
cp "$S/issue-tracker-local.md" docs/agents/issue-tracker.md
cp "$S/domain.md" docs/agents/domain.md
cp "$S/triage-labels.md" docs/agents/triage-labels.md
cat > CLAUDE.md <<'MD'
## Agent skills
### Issue tracker
Issues and specs live as local markdown files under `.scratch/`. See `docs/agents/issue-tracker.md`.
### Triage labels
Default labels (`needs-triage`, `needs-info`, `ready-for-agent`, `ready-for-human`, `wontfix`). See `docs/agents/triage-labels.md`.
### Domain docs
Single-context: one `GLOSSARY.md` + `docs/adr/` at the repo root. See `docs/agents/domain.md`.
MD
git add CLAUDE.md docs/agents
GIT_AUTHOR_DATE="2026-03-09T18:00:00" GIT_COMMITTER_DATE="2026-03-09T18:00:00" \
git -c user.name="Dana Whitfield" -c user.email="dana@ledgerly.dev" commit -q -m "chore: configure agent skills (local tracker)"
fi
if [ "$SP" = 1 ]; then
# The plugin root as Claude Code would hold it, so the hook script finds ../skills/using-superpowers.
mkdir -p .claude/plugins
cp -R "$HERE/packs/superpowers" .claude/plugins/superpowers
cp -R "$HERE/packs/superpowers/skills/." .claude/skills/
fi
python3 - "$HERE" "$SP" "$CFG" > .claude/settings.json <<'PY'
import json, sys
bench, sp, cfg = sys.argv[1], sys.argv[2] == "1", sys.argv[3]
# Oracle, reference, PO brief, other runs: denied by path (Read/Grep/Glob) and by command text (Bash).
deny = ["Read(/%s/hidden/**)" % bench, "Read(/%s/reference/**)" % bench, "Read(/%s/runs/**)" % bench,
"Read(/%s/po/**)" % bench, "Read(/%s/validation/**)" % bench,
"Bash(*hidden*)", "Bash(*reference/F*)", "Bash(*-brief.md*)", "Bash(*/runs/*)", "Bash(*../*)", "Bash(find *)"]
if cfg == "spd":
# executing-plans runs its sibling scripts as ../subagent-driven-development/scripts/...: a "../" deny would block the
# pack's own loop. The oracle stays protected by the Read denies and Bash(*hidden*).
deny.remove("Bash(*../*)")
settings = {"permissions": {"deny": deny}}
if sp:
# = packs/superpowers/hooks/hooks.json, with ${CLAUDE_PLUGIN_ROOT} provided the way the plugin loader provides it
# (the script also picks its JSON output shape from CLAUDE_PLUGIN_ROOT being set).
root = '$CLAUDE_PROJECT_DIR/.claude/plugins/superpowers'
hooks = json.load(open("%s/packs/superpowers/hooks/hooks.json" % bench))["hooks"]
for groups in hooks.values():
for g in groups:
for h in g["hooks"]:
h["command"] = 'CLAUDE_PLUGIN_ROOT="%s" %s' % (root, h["command"].replace("${CLAUDE_PLUGIN_ROOT}", root))
settings["hooks"] = hooks
print(json.dumps(settings, indent=2))
PY
[ "$(ls .claude/skills | wc -l)" -eq 0 ] && rmdir .claude/skills
test -z "$(git status --porcelain)" || { echo "work copy not clean"; git status --porcelain; exit 1; }
git rev-parse HEAD
env.sh
# Sourced by run.sh, gate.sh and validate.sh.
# 1. Scrub every CLAUDE* variable inherited from a launching Claude Code session
# (CLAUDE_EFFORT, CLAUDE_CODE_SUBAGENT_MODEL, CLAUDE_CODE_SESSION_ID, CLAUDECODE,
# CLAUDE_CODE_CHILD_SESSION, ...): they would silently change the worker's effort,
# subagent model and session bookkeeping.
for _v in $(env | sed -n 's/^\(CLAUDE[A-Za-z0-9_]*\)=.*/\1/p'); do unset "$_v"; done
unset _v
# 2. Auth: the user's default config dir (~/.claude, OAuth). See README traps (--bare cannot auth).
unset CLAUDE_CONFIG_DIR
# 3. Fresh work copies at the same path must not share auto-memory across reps.
export CLAUDE_CODE_DISABLE_AUTO_MEMORY=1
# 4. No bytecode in the shared hidden/ dir when 3 runs execute hidden tests in parallel.
export PYTHONDONTWRITEBYTECODE=1
# 5. The bench venv first: `python -m pytest` inside the work copy uses it.
GATE_BENCH="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
export GATE_BENCH
export PATH="$GATE_BENCH/.venv/bin:$PATH"
configs.json
{
"bare": {
"superpowers": false,
"mattpocock": false,
"flow": "bare",
"desc": "no skills, no hooks (project settings carry only the permission deny-list)"
},
"sp": {
"superpowers": true,
"mattpocock": false,
"flow": "sp",
"desc": "HOOK-ROUTED: obra/superpowers v6.4.2: skills in .claude/skills, SessionStart hook in .claude/settings.json as hooks/hooks.json wires it; turn 1 = the bare task, the hook decides"
},
"mp": {
"superpowers": false,
"mattpocock": true,
"flow": "mp",
"desc": "mattpocock/skills v1.3.1 via npx skills add (27 promoted skills, copied), setup-matt-pocock-skills output committed (local tracker)"
},
"both": {
"superpowers": true,
"mattpocock": true,
"flow": "mp",
"desc": "sp + mp in one project; driven with the mp slash-command flow (the stacking question: does the Superpowers hook hijack it?)"
},
"spd": {
"superpowers": true,
"mattpocock": false,
"flow": "spd",
"desc": "Superpowers, DRIVEN: same project install as sp (hook included); the driver types /brainstorming <task> -> /writing-plans -> /executing-plans (Native/inline); the pack's own scripts allowed in Bash"
}
}
lib/driver.py
"""One run of the flow bench: drives a headless Sonnet 5.5 session turn by turn, routes questions to the PO.
driver.py <cfg> <run dir> <work dir> <base sha>
Every worker turn: `claude -p <prompt> [--resume <sid>] --output-format stream-json --verbose --include-hook-events
--include-partial-messages ...` with cwd = the work copy. Per turn it writes turn-NN.stream.jsonl (raw stdout),
turn-NN.times (receive time of every stdout line), turn-NN.stderr, and appends one record to turns.jsonl.
Caps (SPEC): 12 driver turns, --max-turns 60 per claude turn, 40 min wall per run (PO time included).
"""
import json
import os
import subprocess
import sys
import threading
import time
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import po # noqa: E402
from accounting import account_turn # noqa: E402
BENCH = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
MODEL = "claude-sonnet-5-5"
MAX_DRIVER_TURNS = int(os.environ.get("BENCH_MAX_DRIVER_TURNS", "12"))
MAX_TURNS = os.environ.get("BENCH_MAX_TURNS", "60")
WALL_CAP = float(os.environ.get("BENCH_WALL_CAP_S", str(40 * 60)))
# SPEC allow-list. AskUserQuestion is removed (headless: questions come as plain text); plan mode and worktree
# tools are removed too (a headless session cannot leave plan mode, and a worktree would move the work out of
# the scored copy). Read-only shell commands are auto-approved by Claude Code whatever the list says.
ALLOWED = ",".join([
"Read", "Edit", "Write", "MultiEdit", "Glob", "Grep", "Skill", "TodoWrite", "Task", "Agent",
"Bash(node:*)", "Bash(npm:*)", "Bash(pnpm:*)", "Bash(npx vitest:*)", "Bash(npx tsc:*)", "Bash(tsc:*)",
"Bash(git status:*)", "Bash(git diff:*)", "Bash(git log:*)", "Bash(git add:*)", "Bash(git commit:*)",
"Bash(git checkout -b:*)", "Bash(ls:*)", "Bash(cat:*)", "Bash(mkdir:*)", "Bash(head:*)", "Bash(tail:*)",
"Bash(grep:*)", "Bash(wc:*)", "Bash(gh:*)",
])
# spd only: the scripts executing-plans / subagent-driven-development call (task-start, task-done, sdd-workspace,
# task-brief, review-package), wherever the model addresses them from.
SPD_EXTRA = ",".join("Bash(*scripts/%s*)" % x for x in ("task-start", "task-done", "sdd-workspace", "task-brief", "review-package"))
DISALLOWED = "AskUserQuestion,EnterPlanMode,ExitPlanMode,EnterWorktree,ExitWorktree,WebFetch,WebSearch"
PRE_IMPLEMENT = ("/grill-with-docs", "/to-spec", "/to-tickets")
SPD_PRE_IMPLEMENT = ("/brainstorming", "/writing-plans")
DONE_SUFFIX = "\n\nWhen you are done, stop and summarise what you changed."
def worker_env():
env = {k: v for k, v in os.environ.items() if not k.startswith("CLAUDE")}
env["CLAUDE_CODE_DISABLE_AUTO_MEMORY"] = "1"
return env
def run_claude(prompt, sid, work, out_prefix, timeout, allowed=ALLOWED):
cmd = ["claude", "-p", prompt]
if sid:
cmd += ["--resume", sid]
cmd += ["--model", MODEL, "--output-format", "stream-json", "--verbose", "--include-hook-events",
"--include-partial-messages", "--max-turns", MAX_TURNS, "--permission-mode", "acceptEdits",
"--allowedTools", allowed, "--disallowedTools", DISALLOWED,
"--setting-sources", "project", "--strict-mcp-config"]
t0 = time.time()
fo = open(out_prefix + ".stream.jsonl", "w")
ft = open(out_prefix + ".times", "w")
fe = open(out_prefix + ".stderr", "w")
p = subprocess.Popen(cmd, cwd=work, env=worker_env(), stdin=subprocess.DEVNULL, stdout=subprocess.PIPE,
stderr=fe, text=True, bufsize=1)
killed = {"v": False}
def watchdog():
try:
p.wait(timeout=timeout)
except subprocess.TimeoutExpired:
killed["v"] = True
p.kill()
th = threading.Thread(target=watchdog, daemon=True)
th.start()
for line in p.stdout:
fo.write(line)
ft.write("%.3f\n" % (time.time() - t0))
p.wait()
th.join(timeout=5)
for f in (fo, ft, fe):
f.close()
return {"exit_code": p.returncode, "wall_s": round(time.time() - t0, 2), "killed_wall_cap": killed["v"], "t0": t0}
def scratch_issues(work):
root = os.path.join(work, ".scratch")
if not os.path.isdir(root):
return None
for d in sorted(os.listdir(root)):
if os.path.isdir(os.path.join(root, d, "issues")):
return ".scratch/%s/issues/" % d
return None
def main():
cfg, run_dir, work, base = sys.argv[1:5]
conf = json.load(open(os.path.join(BENCH, "configs.json")))[cfg]
flow = conf["flow"]
task_id = os.environ.get("TASK", "F1")
task = open(os.path.join(BENCH, "tasks", task_id + ".md")).read().strip()
allowed = ALLOWED + ("," + SPD_EXTRA if flow == "spd" else "")
spd_stages = ["/brainstorming", "/writing-plans", "/executing-plans"]
spd_i = 0
t_run = time.time()
sid = None
prev = None # cumulative modelUsage / cost of the session so far
stage_in_force = "(none)"
caps = []
turns_log = open(os.path.join(run_dir, "turns.jsonl"), "a")
# mp flow: the slash commands in SPEC order; the implement stage is built when it is reached.
mp_stages = ["/grill-with-docs", "/to-spec", "/to-tickets", "/implement", "/code-review", "/pr"]
mp_i = 0
def mp_prompt(i):
s = mp_stages[i]
if s == "/grill-with-docs":
return "/grill-with-docs " + task
if s == "/implement":
issues = scratch_issues(work)
return "/implement " + ("the tickets in %s, all of them, blockers first" % issues if issues
else "the spec and tickets above")
if s == "/code-review":
return "/code-review since %s" % base
return s
if flow == "bare":
prompt, stage = task + DONE_SUFFIX, "bare"
elif flow == "spd":
prompt, stage = "/brainstorming " + task, spd_stages[0]
stage_in_force = "brainstorming"
elif flow == "sp":
prompt, stage = task, None
else:
prompt, stage = mp_prompt(0), mp_stages[0]
source = "driver"
end_reason = None
for n in range(1, MAX_DRIVER_TURNS + 1):
remaining = WALL_CAP - (time.time() - t_run)
if remaining <= 5:
caps.append("wall_40min")
end_reason = "cap_wall"
break
prefix = os.path.join(run_dir, "turn-%02d" % n)
r = run_claude(prompt, sid, work, prefix, remaining, allowed)
seg_mode = flow in ("sp", "spd") # stage split by skill in force inside the turn
acc = account_turn(prefix + ".stream.jsonl", prefix + ".times", prev,
None if seg_mode else stage, stage_in_force)
if acc.get("session_id"):
sid = acc["session_id"]
if acc.get("cumulative"):
prev = acc["cumulative"]
if seg_mode:
stage_in_force = acc["stage_at_end"]
rec = {"n": n, "flow": flow, "task": task_id, "stage": acc["stage_at_end"] if flow == "sp" else stage,
"prompt": prompt,
"prompt_source": source, **{k: v for k, v in r.items() if k != "t0"}, **{k: v for k, v in acc.items() if k != "cumulative"}}
if r["killed_wall_cap"]:
caps.append("wall_40min")
if acc.get("subtype") == "error_max_turns":
caps.append("max_turns_60@turn%d" % n)
last = acc.get("result_text") or acc.get("last_assistant_text") or ""
open(prefix + ".last.txt", "w").write(last)
if r["killed_wall_cap"]:
rec["po"] = None
turns_log.write(json.dumps(rec) + "\n")
turns_log.flush()
end_reason = "cap_wall"
break
po_stage = stage if flow != "sp" else acc["stage_at_end"]
ans = po.ask(run_dir, n, flow, po_stage, last, task_id)
rec["po"] = ans
turns_log.write(json.dumps(rec) + "\n")
turns_log.flush()
# Next prompt
if flow in ("bare", "sp"):
if ans["kind"] == "done":
end_reason = "done"
break
prompt = ans["reply"] if ans["kind"] == "question" else (
"go on" if flow == "bare" else "go on, proceed with the plan")
source = "po" if ans["kind"] == "question" else "driver"
elif flow == "spd":
if ans["kind"] == "question":
prompt, source = ans["reply"], "po"
if stage in SPD_PRE_IMPLEMENT and "implementing yet" not in prompt:
prompt += "\n\nDon't start implementing yet."
elif ans["kind"] == "waiting" or acc.get("subtype") == "error_max_turns":
prompt, source = "go on", "driver"
else:
spd_i += 1
if spd_i >= len(spd_stages):
end_reason = "done"
break
stage, prompt, source = spd_stages[spd_i], spd_stages[spd_i], "driver"
stage_in_force = stage.lstrip("/") # a typed slash command expands inline: no Skill event
else:
if ans["kind"] == "question":
prompt, source = ans["reply"], "po"
# A user following the Main Flow builds only at /implement. Without this line the worker took the
# answers to its last grilling questions as a go and coded inside /grill-with-docs (both-r1, twice).
if stage in PRE_IMPLEMENT and "implementing yet" not in prompt:
prompt += "\n\nDon't start implementing yet."
elif ans["kind"] == "waiting" or acc.get("subtype") == "error_max_turns":
prompt, source = "go on", "driver" # stage not finished: stay in it
else:
mp_i += 1
if mp_i >= len(mp_stages):
end_reason = "done"
break
stage, prompt, source = mp_stages[mp_i], mp_prompt(mp_i), "driver"
if n == MAX_DRIVER_TURNS:
caps.append("driver_turns_12")
end_reason = "cap_driver_turns"
meta = {"end_reason": end_reason, "caps": sorted(set(caps)), "session_id": sid,
"run_wall_s": round(time.time() - t_run, 1), "mp_stage_reached": mp_stages[min(mp_i, 5)] if flow == "mp" else None,
"spd_stage_reached": spd_stages[min(spd_i, 2)] if flow == "spd" else None, "task": task_id}
json.dump(meta, open(os.path.join(run_dir, "driver.json"), "w"), indent=2)
print(json.dumps(meta))
if __name__ == "__main__":
main()
lib/po.py
"""Product-owner simulator: a headless Sonnet 5.5 with NO tools that read po/F1-brief.md and answers the worker.
po.py <run dir> <turn n> <flow> <stage> <last message file> -> prints one JSON line, writes po-<n>.json
Output: {"kind": "question" | "waiting" | "done", "reply": str, "cost_usd", "wall_s", "error", "heuristic_question"}
- question: the worker asks the user something (a question, options, an approval request) -> reply answers it
- waiting: it stopped without asking (announced a next step, paused between phases) -> driver sends "go on"
- done: it reports the whole change finished -> driver stops (bare/sp)
Isolation (same as the previous benches' reviewers): --setting-sources "", disableAllHooks, --strict-mcp-config,
--tools "", CLAUDE_CODE_DISABLE_CLAUDE_MDS=1, CLAUDE_CODE_DISABLE_AUTO_MEMORY=1, scrubbed CLAUDE* env,
cwd = an empty temp dir (never the work copy).
"""
import json
import os
import re
import subprocess
import sys
import tempfile
import time
BENCH = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
MODEL = "claude-sonnet-5-5"
SCHEMA = {
"type": "object",
"properties": {
"kind": {"type": "string", "enum": ["question", "waiting", "done"]},
"reply": {"type": "string"},
},
"required": ["kind", "reply"],
"additionalProperties": False,
}
FLOW_NOTE = {
"bare": "The assistant was asked to implement the request and stop with a summary when done.",
"sp": "The assistant runs a process (brainstorm -> design -> written plan -> execution -> review -> finish). "
"It often stops to ask for approval of a design, a spec or a plan, or to ask how to finish the branch.",
"spd": "You are driving the assistant through the Superpowers stages by slash command, one stage at a time "
"(brainstorming -> writing-plans -> executing-plans); the current stage is shown below and YOU send the "
"next command once a stage is over. RULE FOR THIS FLOW: when the stage's output is delivered (a design "
"approved and written, a plan saved) and the assistant only asks you to approve it or whether to proceed, "
"answer what is open (approve it; if asked for an execution approach, choose Native / inline in this "
"session; consent to working on the current branch) and classify it kind=question only if something "
"beyond approval is open, otherwise kind=done. Never tell it to start implementing yourself before the "
"executing-plans stage. During executing-plans, approve its rulings and ask it to finish.",
"mp": "You are driving the assistant through slash commands, one stage at a time (grill -> spec -> tickets -> "
"implement -> code review -> PR); the current stage is shown below and YOU send the next command once "
"a stage is over. Interviews (grilling) come in rounds of numbered questions with a recommended answer "
"each. RULE FOR THIS FLOW: when the stage's output is delivered and the assistant only asks whether to "
"proceed, start building, move on, or confirms its own summary ('Ready to write the spec?', 'Shall I "
"start?', 'Does this capture it?' after a full recap), that is kind=done with an empty reply: the next "
"slash command will follow. Never tell it to start implementing or to move to another phase yourself; "
"answer only the open design questions. Before the /implement stage, if the assistant offers to build "
"next, end your reply with: \"Don't start implementing yet.\"",
}
def heuristic_question(text):
"""SPEC's driver rule: a turn that ends on a question mark or an explicit options list."""
tail = (text or "").strip()[-600:]
if tail.rstrip("*_` \n").endswith("?"):
return True
lines = [l.strip() for l in tail.splitlines() if l.strip()]
opts = [l for l in lines[-8:] if re.match(r"^(\(?[A-Da-d1-4][\).:]|[-*] \*\*Option)", l)]
return len(opts) >= 2 or "?" in "\n".join(lines[-3:])
def ask(run_dir, n, flow, stage, last_text, task="F1"):
brief = open(os.path.join(BENCH, "po", "%s-brief.md" % task)).read()
prompt = (
"You play the developer who asked a coding assistant for a change in your repo. You are busy: you answer "
"tersely (1 to 6 short sentences or a short numbered list), the way a dev types in a chat. You never write "
"code, never mention how you will check the work, never invent requirements beyond your brief. You answer "
"ONLY what the assistant asked: never volunteer a requirement, an edge case or a constraint it did not ask "
"about (a question it asked is fair game, even a broad one like 'anything else I should know?').\n\n"
"=== YOUR BRIEF (private, the assistant has not seen it) ===\n" + brief + "\n=== END BRIEF ===\n\n"
+ FLOW_NOTE.get(flow, "") + "\nCurrent stage: " + stage + "\n\n"
"=== THE ASSISTANT'S LAST MESSAGE (it is now waiting for you) ===\n" + (last_text or "(empty message)")[-12000:]
+ "\n=== END MESSAGE ===\n\n"
"Classify the message and write your reply:\n"
"- kind=question when it asks you anything or waits for a decision or an approval (numbered questions, "
"options, 'does this look right?', 'shall I proceed?'). Reply by answering EVERY open question in one go, "
"from your brief; where your brief is silent, accept the assistant's recommendation or say 'your call'. "
"Approve designs, specs, plans and ticket breakdowns that match your brief; correct only what contradicts it. "
"If it asks how to finish a branch (merge, PR, keep), say: keep it on the current branch, no PR, done. "
"If a code review it ran lists real defects, tell it to fix them.\n"
"- kind=waiting when it stopped without asking anything but the work is not finished (it announced a next "
"step, or paused between phases). Reply: 'go on'.\n"
"- kind=done when it reports the change finished (implemented, tests run) or, for a slash-command stage, "
"when that stage's output is delivered and nothing is asked. Reply: ''.\n"
)
env = {k: v for k, v in os.environ.items() if not k.startswith("CLAUDE")}
env.update({"CLAUDE_CODE_DISABLE_CLAUDE_MDS": "1", "CLAUDE_CODE_DISABLE_AUTO_MEMORY": "1"})
cmd = ["claude", "-p", prompt, "--model", MODEL, "--output-format", "json", "--json-schema", json.dumps(SCHEMA),
"--tools", "", "--max-turns", "4", "--permission-mode", "dontAsk", "--setting-sources", "",
"--settings", '{"disableAllHooks":true}', "--strict-mcp-config"]
out = {"kind": None, "reply": "", "cost_usd": 0.0, "wall_s": 0.0, "error": None,
"heuristic_question": heuristic_question(last_text)}
t0 = time.time()
for attempt in range(2):
try:
with tempfile.TemporaryDirectory(prefix="po-") as cwd:
p = subprocess.run(cmd, cwd=cwd, env=env, stdin=subprocess.DEVNULL, capture_output=True,
text=True, timeout=300)
res = json.loads(p.stdout.strip().splitlines()[-1])
out["cost_usd"] += res.get("total_cost_usd") or 0.0
out["models"] = sorted((res.get("modelUsage") or {}).keys())
so = res.get("structured_output")
if isinstance(so, str):
so = json.loads(so)
if not so or so.get("kind") not in ("question", "waiting", "done"):
raise ValueError("no structured_output: %s" % (res.get("result") or "")[:300])
out["kind"], out["reply"], out["error"] = so["kind"], so["reply"].strip(), None
break
except Exception as e: # noqa: BLE001
out["error"] = "%s: %s" % (type(e).__name__, e)
time.sleep(3)
if out["kind"] is None: # PO failed twice: fall back on the SPEC heuristic
out["kind"] = "question" if out["heuristic_question"] else "waiting"
out["reply"] = "Your call on all of these; go on." if out["kind"] == "question" else "go on"
out["fallback"] = True
if out["kind"] == "waiting" and not out["reply"]:
out["reply"] = "go on"
out["wall_s"] = round(time.time() - t0, 2)
json.dump(dict(out, n=n, stage=stage, prompt_chars=len(prompt)), open(os.path.join(run_dir, "po-%02d.json" % n), "w"),
indent=2)
return out
if __name__ == "__main__":
rd, n, flow, stage, f = sys.argv[1:6]
print(json.dumps(ask(rd, int(n), flow, stage, open(f).read(), os.environ.get("TASK", "F1"))))
lib/accounting.py
"""Per-turn and per-stage accounting from one claude -p stream-json (with --include-partial-messages).
Sources of truth (checked on 2.1.294, see README traps):
- result.total_cost_usd and result.modelUsage are CUMULATIVE over a resumed session and include subagents:
per-turn USD and tokens = delta against the previous turn's result.
- result.usage covers the main thread of this invocation only (no subagents): not used for totals.
- assistant events carry a usage SNAPSHOT taken when the message started (output_tokens partial);
stream_event message_start/message_delta (main thread only) carry the exact per-API-call usage.
- Subagent API calls emit only assistant snapshots (parent_tool_use_id = the Task/Agent tool_use id).
Stage split inside a turn (sp: the skill in force): main-thread calls are priced exactly from their own usage
(Sonnet 5.5 list prices, verified against total_cost_usd); the subagent remainder (turn delta - main calls) is
split over the Task calls that spawned them, weighted by their snapshot tokens. Wall per stage = the receive-time
gaps of the stdout lines, attributed to the stage in force.
"""
import json
from collections import Counter, defaultdict
# USD per token, claude-sonnet-5-5 list prices (derived from two probe turns: exact match to total_cost_usd).
PRICE = {"in": 2e-6, "out": 10e-6, "cr": 0.2e-6, "cw1h": 4e-6, "cw5m": 2.5e-6}
SP_INJECT_MARK = "You have superpowers"
EDIT_TOOLS = {"Edit", "Write", "MultiEdit", "NotebookEdit"}
DESIGN_STAGES = {"(none)", "using-superpowers", "brainstorming", "writing-plans"}
NO_SKILL_IMPL = "implementation (no skill)"
def _tok(u):
"""Normalise an API usage dict to in/out/cr/cw (+ cw split)."""
u = u or {}
cc = u.get("cache_creation") or {}
cw = u.get("cache_creation_input_tokens") or 0
cw1h = cc.get("ephemeral_1h_input_tokens")
cw5m = cc.get("ephemeral_5m_input_tokens")
if cw1h is None and cw5m is None:
cw1h, cw5m = cw, 0
return {"in": u.get("input_tokens") or 0, "out": u.get("output_tokens") or 0,
"cr": u.get("cache_read_input_tokens") or 0, "cw": cw, "cw1h": cw1h or 0, "cw5m": cw5m or 0}
def _usd(t):
return t["in"] * PRICE["in"] + t["out"] * PRICE["out"] + t["cr"] * PRICE["cr"] + \
t["cw1h"] * PRICE["cw1h"] + t["cw5m"] * PRICE["cw5m"]
def _mu_tokens(mu):
out = {}
for m, v in (mu or {}).items():
out[m] = {"in": v.get("inputTokens") or 0, "out": v.get("outputTokens") or 0,
"cr": v.get("cacheReadInputTokens") or 0, "cw": v.get("cacheCreationInputTokens") or 0,
"usd": v.get("costUSD") or 0.0}
return out
def _sub(a, b):
keys = set(a) | set(b)
return {m: {k: (a.get(m, {}).get(k, 0) - b.get(m, {}).get(k, 0)) for k in ("in", "out", "cr", "cw", "usd")}
for m in keys}
def _sum_models(per_model):
t = {k: 0 for k in ("in", "out", "cr", "cw", "usd")}
for v in per_model.values():
for k in t:
t[k] += v.get(k, 0)
return t
def account_turn(stream_path, times_path, prev, fixed_stage, stage_in_force):
lines, times = [], []
try:
raw = open(stream_path, errors="replace").read().splitlines()
tms = [float(x) for x in open(times_path).read().split()]
except FileNotFoundError:
raw, tms = [], []
for i, l in enumerate(raw):
l = l.strip()
if l.startswith("{"):
try:
lines.append(json.loads(l))
times.append(tms[i] if i < len(tms) else (times[-1] if times else 0.0))
except Exception:
pass
stage = fixed_stage or stage_in_force
out = {"session_id": None, "subtype": None, "is_error": None, "num_turns": None, "result_text": None,
"last_assistant_text": "", "stage_at_start": stage}
tools_main, tools_sub = Counter(), Counter()
skills, hooks, files = [], [], set()
skill_ids = {}
task_stage = {} # Task/Agent tool_use id -> stage when spawned
main_calls = {} # msg id -> {"stage", "tok"}
sub_snap = {} # msg id -> (parent id, tokens)
seg_wall = defaultdict(float)
order = [stage]
last_t = 0.0
result = None
denials = []
for o, t in zip(lines, times):
seg_wall[stage] += max(0.0, t - last_t)
last_t = t
typ = o.get("type")
if typ == "system" and o.get("subtype") == "init":
out["session_id"] = o.get("session_id")
out["init_model"] = o.get("model")
out["init_skills"] = o.get("skills")
out["init_plugins"] = [p.get("name") for p in o.get("plugins") or []]
elif typ == "system" and str(o.get("subtype", "")).startswith("hook"):
if o.get("subtype") == "hook_response":
outp = o.get("output") or ""
hooks.append({"event": o.get("hook_event"), "name": o.get("hook_name"),
"superpowers_injected": SP_INJECT_MARK in outp, "output_chars": len(outp),
"exit_code": o.get("exit_code"), "outcome": o.get("outcome")})
elif typ == "stream_event" and not o.get("parent_tool_use_id"):
ev = o.get("event") or {}
if ev.get("type") == "message_start":
m = ev.get("message") or {}
main_calls[m.get("id")] = {"stage": stage, "tok": _tok(m.get("usage")), "model": m.get("model")}
main_calls["__last__"] = m.get("id")
elif ev.get("type") == "message_delta":
mid = main_calls.get("__last__")
if mid in main_calls:
u = ev.get("usage") or {}
tk = main_calls[mid]["tok"]
tk["out"] = u.get("output_tokens", tk["out"]) or tk["out"]
elif typ == "assistant":
m = o.get("message") or {}
parent = o.get("parent_tool_use_id")
if parent:
sub_snap[m.get("id")] = (parent, _tok(m.get("usage")))
for c in m.get("content") or []:
if c.get("type") == "text" and not parent:
out["last_assistant_text"] = c.get("text") or out["last_assistant_text"]
if c.get("type") != "tool_use":
continue
name = c.get("name")
inp = c.get("input") or {}
(tools_sub if parent else tools_main)[name] += 1
if name in EDIT_TOOLS and inp.get("file_path"):
files.add(inp["file_path"])
# sp: code written while only a design skill is in force = implementation with no skill
if (not parent and not fixed_stage and stage in DESIGN_STAGES
and ("/src/" in inp["file_path"] or "/test/" in inp["file_path"])):
stage = NO_SKILL_IMPL
order.append(stage)
if name in ("Task", "Agent") and not parent:
task_stage[c.get("id")] = stage
if name == "Skill":
sk = inp.get("skill") or inp.get("command") or "?"
rec = {"skill": sk, "t": round(t, 2), "thread": "sub" if parent else "main", "ok": None,
"args": (inp.get("args") or "")[:200]}
skills.append(rec)
skill_ids[c.get("id")] = rec
if not parent and not fixed_stage:
stage = sk.split(":")[-1]
order.append(stage)
elif typ == "user":
for c in (o.get("message") or {}).get("content") or []:
if isinstance(c, dict) and c.get("type") == "tool_result" and c.get("tool_use_id") in skill_ids:
txt = c.get("content")
txt = txt if isinstance(txt, str) else json.dumps(txt)
skill_ids[c["tool_use_id"]]["ok"] = not c.get("is_error") and "Launching skill" in txt
skill_ids[c["tool_use_id"]]["result"] = txt[:200]
elif typ == "result":
result = o
main_calls.pop("__last__", None)
if result:
out.update({"subtype": result.get("subtype"), "is_error": result.get("is_error"),
"num_turns": result.get("num_turns"), "result_text": result.get("result"),
"terminal_reason": result.get("terminal_reason"), "stop_reason": result.get("stop_reason")})
denials = result.get("permission_denials") or []
cum = {"cost": result.get("total_cost_usd") or 0.0, "models": _mu_tokens(result.get("modelUsage"))}
pm = prev["models"] if prev else {}
delta_models = _sub(cum["models"], pm)
delta_cost = cum["cost"] - (prev["cost"] if prev else 0.0)
out["cumulative"] = cum
out["estimated"] = False
else: # killed or crashed: best effort from the stream itself
delta_models = {}
for c in main_calls.values():
d = delta_models.setdefault(c.get("model") or "?", {"in": 0, "out": 0, "cr": 0, "cw": 0, "usd": 0.0})
for k in ("in", "out", "cr", "cw"):
d[k] += c["tok"][k]
d["usd"] += _usd(c["tok"])
for _, (parent, tk) in sub_snap.items():
d = delta_models.setdefault("?sub", {"in": 0, "out": 0, "cr": 0, "cw": 0, "usd": 0.0})
for k in ("in", "out", "cr", "cw"):
d[k] += tk[k]
d["usd"] += _usd(tk)
delta_cost = sum(v["usd"] for v in delta_models.values())
out["cumulative"] = None
out["estimated"] = True
tot = _sum_models(delta_models)
out["usd"] = round(delta_cost, 6)
out["tokens"] = {k: tot[k] for k in ("in", "out", "cr", "cw")}
out["tokens"]["total"] = sum(out["tokens"].values())
out["models"] = {m: v for m, v in delta_models.items() if any(v[k] for k in ("in", "out", "cr", "cw"))}
# Stage segments
segs = defaultdict(lambda: {"in": 0, "out": 0, "cr": 0, "cw": 0, "usd": 0.0, "api_calls": 0, "wall_s": 0.0})
main_tok = {"in": 0, "out": 0, "cr": 0, "cw": 0}
main_usd = 0.0
for c in main_calls.values():
s = segs[c["stage"]]
for k in ("in", "out", "cr", "cw"):
s[k] += c["tok"][k]
main_tok[k] += c["tok"][k]
u = _usd(c["tok"])
s["usd"] += u
s["api_calls"] += 1
main_usd += u
rem = {k: max(0, out["tokens"][k] - main_tok[k]) for k in ("in", "out", "cr", "cw")}
rem_usd = max(0.0, delta_cost - main_usd) if result else 0.0
weights = defaultdict(float)
for _, (parent, tk) in sub_snap.items():
# subagent spend is its own segment, labelled with the stage that dispatched it
weights[task_stage.get(parent, stage) + " [subagent]"] += tk["in"] + tk["cr"] + tk["cw"] + tk["out"]
wsum = sum(weights.values())
if result and (wsum > 0 or rem_usd > 0):
if wsum == 0:
weights = {stage + " [subagent]": 1.0}
wsum = 1.0
for st, w in weights.items():
s = segs[st]
f = w / wsum
for k in ("in", "out", "cr", "cw"):
s[k] += int(round(rem[k] * f))
s["usd"] += rem_usd * f
for st, w in seg_wall.items():
segs[st]["wall_s"] += w
seen = []
for st in order:
if st not in seen:
seen.append(st)
for st in segs:
if st not in seen:
seen.append(st)
out["segments"] = [dict(stage=st, **{k: (round(v, 6) if isinstance(v, float) else v) for k, v in segs[st].items()},
total=segs[st]["in"] + segs[st]["out"] + segs[st]["cr"] + segs[st]["cw"])
for st in seen if st in segs]
out["pricing_check"] = {"main_calls_usd": round(main_usd, 6), "turn_usd": round(delta_cost, 6),
"subagent_usd": round(rem_usd, 6), "main_calls": len(main_calls),
"subagent_msgs": len(sub_snap)}
out["stage_at_end"] = stage
out["tools_main"] = dict(tools_main)
out["tools_sub"] = dict(tools_sub)
out["tool_calls"] = sum(tools_main.values()) + sum(tools_sub.values())
out["skills"] = skills
out["hooks"] = hooks
out["files_touched"] = sorted(files)
out["permission_denials"] = [{"tool": d.get("tool_name"), "input": json.dumps(d.get("tool_input"))[:200]}
for d in denials]
out["stream_lines"] = len(lines)
return out
lib/reaccount.py
"""Recompute the accounting fields of runs/<run>/turns.jsonl from the saved turn streams (after a change to
lib/accounting.py), keeping what the driver recorded (prompt, source, wall, exit code, PO answer). Then run finalize.py.
reaccount.py <run dir>
"""
import json
import os
import sys
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from accounting import account_turn # noqa: E402
KEEP = ("n", "flow", "prompt", "prompt_source", "exit_code", "wall_s", "killed_wall_cap", "po")
run = os.path.abspath(sys.argv[1])
turns = [json.loads(l) for l in open(os.path.join(run, "turns.jsonl"))]
prev, stage_in_force, out = None, "(none)", []
prev_slash = None
for t in turns:
prefix = os.path.join(run, "turn-%02d" % t["n"])
if t["flow"] == "spd" and t["stage"] != prev_slash: # a typed slash command starts its skill (no Skill event)
stage_in_force, prev_slash = t["stage"].lstrip("/"), t["stage"]
sp = t["flow"] in ("sp", "spd")
acc = account_turn(prefix + ".stream.jsonl", prefix + ".times", prev, None if sp else t["stage"], stage_in_force)
if acc.get("cumulative"):
prev = acc["cumulative"]
if sp:
stage_in_force = acc["stage_at_end"]
rec = {k: t[k] for k in KEEP if k in t}
rec["stage"] = acc["stage_at_end"] if t["flow"] == "sp" else t["stage"]
rec.update({k: v for k, v in acc.items() if k != "cumulative"})
out.append(rec)
with open(os.path.join(run, "turns.jsonl"), "w") as f:
for r in out:
f.write(json.dumps(r) + "\n")
print("%s: %d turns reaccounted" % (os.path.basename(run), len(out)))
lib/finalize.py
"""After the driver: final diff, suite, typecheck, hidden oracle, per-stage table -> score.json, stages.json.
finalize.py <run dir> (re-runnable; reads meta.json, turns.jsonl, driver.json)
"""
import json
import os
import re
import shutil
import subprocess
import sys
from collections import OrderedDict
BENCH = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
HIDDEN = os.path.join(BENCH, "hidden")
MODEL = "claude-sonnet-5-5"
UPSTREAM_TESTS = 34
def sh(args, cwd, env=None, timeout=600):
try:
p = subprocess.run(args, cwd=cwd, env=env, stdin=subprocess.DEVNULL, stdout=subprocess.PIPE,
stderr=subprocess.STDOUT, timeout=timeout)
return p.returncode, p.stdout.decode("utf-8", "replace")
except subprocess.TimeoutExpired as e:
return 124, (e.stdout or b"").decode("utf-8", "replace") + "\n[timeout]"
def clean_env(extra=None):
env = {k: v for k, v in os.environ.items() if not k.startswith("CLAUDE")}
env.update(extra or {})
return env
def work_diff(work, base):
"""Working tree vs the base commit (commits made by the worker included) + untracked files. Index untouched."""
_, d = sh(["git", "diff", base, "--no-color", "--", "."], work)
_, others = sh(["git", "ls-files", "--others", "--exclude-standard"], work)
for f in [l for l in others.splitlines() if l.strip()]:
_, nd = sh(["git", "diff", "--no-color", "--no-index", "--", "/dev/null", f], work)
d += nd
return d
def numstat(work, base):
rows = []
_, ns = sh(["git", "diff", base, "--numstat", "--", "."], work)
for l in ns.splitlines():
p = l.split("\t")
if len(p) == 3:
rows.append((int(p[0]) if p[0] != "-" else 0, int(p[1]) if p[1] != "-" else 0, p[2]))
_, others = sh(["git", "ls-files", "--others", "--exclude-standard"], work)
for f in [l for l in others.splitlines() if l.strip()]:
try:
rows.append((sum(1 for _ in open(os.path.join(work, f), errors="replace")), 0, f))
except Exception:
pass
return rows
def run_hidden(work, base, task="F1"):
rep = os.path.join(os.path.dirname(work), ".hidden-%s.json" % os.path.basename(work))
rc, txt = sh([os.path.join(HIDDEN, "node_modules", ".bin", "vitest"), "run", "--reporter=json",
"--outputFile=" + rep], HIDDEN, env=clean_env({"WORK": work, "BASE": base, "TASK": task}), timeout=300)
reqs = OrderedDict(("test_req%d" % i, False) for i in range(1, 6))
fails = {}
try:
j = json.load(open(rep))
for tr in j.get("testResults", []):
for a in tr.get("assertionResults", []):
m = re.match(r"(test_req\d)", a.get("title", ""))
if m:
reqs[m.group(1)] = a.get("status") == "passed"
if a.get("status") != "passed":
fails[m.group(1)] = (" ".join(a.get("failureMessages") or []))[:600]
if tr.get("status") == "failed" and not tr.get("assertionResults"):
fails["collect"] = (tr.get("message") or "")[:800]
os.remove(rep)
except Exception as e: # noqa: BLE001
fails["runner"] = "%s: %s" % (type(e).__name__, txt[-800:])
return reqs, fails, txt
def main():
out = os.path.abspath(sys.argv[1])
meta = json.load(open(os.path.join(out, "meta.json")))
work, base, cfg = meta["work"], meta["base"], meta["cfg"]
if not os.path.isdir(work): # tmp work copy gone: rescore the archived copy
work = os.path.join(BENCH, "work", meta["name"])
drv = json.load(open(os.path.join(out, "driver.json"))) if os.path.exists(os.path.join(out, "driver.json")) else {}
turns = [json.loads(l) for l in open(os.path.join(out, "turns.jsonl"))] if os.path.exists(
os.path.join(out, "turns.jsonl")) else []
have_nm = os.path.isdir(os.path.join(work, "node_modules"))
if not have_nm: # rescoring after cleanup: restore deps from upstream
shutil.copytree(os.path.join(BENCH, "upstream", "node_modules"), os.path.join(work, "node_modules"), symlinks=True)
diff = work_diff(work, base)
open(os.path.join(out, "post.diff"), "w").write(diff)
rows = numstat(work, base)
open(os.path.join(out, "numstat.txt"), "w").write("".join("%d\t%d\t%s\n" % r for r in rows))
code = [r for r in rows if r[2].startswith(("src/", "test/"))]
tests_files = [r for r in rows if r[2].endswith(".test.ts")]
rc_s, suite = sh([os.path.join(work, "node_modules", ".bin", "vitest"), "run"], work, env=clean_env(), timeout=300)
open(os.path.join(out, "suite.txt"), "w").write(suite)
clean = re.sub(r"\x1b\[[0-9;]*m", "", suite)
m = re.search(r"Tests\s+(?:(\d+) failed \| )?(\d+) passed(?: \| \d+ \w+)? \((\d+)\)", clean)
s_fail = int(m.group(1) or 0) if m else None
s_pass = int(m.group(2)) if m else 0
s_total = int(m.group(3)) if m else 0
if m is None:
mf = re.search(r"Tests\s+(\d+) failed \((\d+)\)", clean)
s_fail, s_total = (int(mf.group(1)), int(mf.group(2))) if mf else (None, 0)
rc_t, tc = sh([os.path.join(work, "node_modules", ".bin", "tsc"), "--noEmit"], work, env=clean_env(), timeout=300)
open(os.path.join(out, "typecheck.txt"), "w").write(tc)
task = meta.get("task", "F1")
reqs, fails, htxt = run_hidden(work, base, task)
open(os.path.join(out, "hidden.txt"), "w").write(htxt + "\n" + json.dumps(fails, indent=2))
_, commits = sh(["git", "log", "--oneline", "%s..HEAD" % base], work)
_, branch = sh(["git", "rev-parse", "--abbrev-ref", "HEAD"], work)
_, branches = sh(["git", "branch", "--format=%(refname:short)"], work)
added_its = 0
for l in diff.splitlines():
if re.match(r"^\+\s*(it|test)(\.each\([^)]*\))?\s*\(", l):
added_its += 1
# Per-stage table (sum of turn segments; for mp/bare/both a segment = the turn's stage)
stages = OrderedDict()
for t in turns:
for sgm in t.get("segments") or []:
st = sgm["stage"] if cfg in ("sp", "spd") else t["stage"]
s = stages.setdefault(st, {"turns": set(), "in": 0, "out": 0, "cr": 0, "cw": 0, "total": 0, "usd": 0.0,
"wall_s": 0.0, "api_calls": 0})
s["turns"].add(t["n"])
for k in ("in", "out", "cr", "cw", "total", "api_calls"):
s[k] += sgm.get(k, 0)
s["usd"] += sgm.get("usd", 0.0)
s["wall_s"] += sgm.get("wall_s", 0.0)
if cfg not in ("sp", "spd") and not t.get("segments"):
stages.setdefault(t["stage"], {"turns": {t["n"]}, "in": 0, "out": 0, "cr": 0, "cw": 0, "total": 0,
"usd": 0.0, "wall_s": t.get("wall_s", 0), "api_calls": 0})
# Turn-level wall/usd are exact; mp/bare stage wall = turn wall (segments only cover stdout gaps)
if cfg not in ("sp", "spd"):
for st, s in stages.items():
s["wall_s"] = sum(t.get("wall_s", 0) for t in turns if t["stage"] == st)
s["usd"] = sum(t.get("usd", 0) for t in turns if t["stage"] == st)
for s in stages.values():
s["turns"] = sorted(s["turns"])
s["usd"] = round(s["usd"], 4)
s["wall_s"] = round(s["wall_s"], 1)
json.dump(stages, open(os.path.join(out, "stages.json"), "w"), indent=2)
tok = {k: sum(t.get("tokens", {}).get(k, 0) for t in turns) for k in ("in", "out", "cr", "cw", "total")}
models = sorted({m for t in turns for m in (t.get("models") or {})})
skills = [dict(s, turn=t["n"]) for t in turns for s in t.get("skills") or []]
hooks = [dict(h, turn=t["n"]) for t in turns for h in t.get("hooks") or []]
po_cost = sum((t.get("po") or {}).get("cost_usd", 0) for t in turns)
files = sorted({f.replace(work + "/", "") for t in turns for f in t.get("files_touched") or []})
score = {
"run": meta["name"], "task": meta.get("task", "F1"), "cfg": cfg, "rep": meta["rep"], "tag": meta.get("tag"),
"hidden_pass": sum(reqs.values()), "hidden_reqs": reqs, "hidden_fail_msgs": fails,
"suite_green": rc_s == 0 and (s_fail or 0) == 0 and s_total > 0, "suite_pass": s_pass, "suite_total": s_total,
"tests_added_suite": max(0, s_total - UPSTREAM_TESTS), "tests_added_diff": added_its,
"typecheck_green": rc_t == 0,
"diff_files": len(rows), "diff_added": sum(r[0] for r in rows), "diff_removed": sum(r[1] for r in rows),
"code_diff_added": sum(r[0] for r in code), "code_diff_removed": sum(r[1] for r in code),
"code_files": len(code), "test_files_touched": len(tests_files),
"non_code_files": sorted(r[2] for r in rows if not r[2].startswith(("src/", "test/"))),
"files_touched_by_tools": files,
"commits": len([l for l in commits.splitlines() if l.strip()]), "branch_at_end": branch.strip(),
"branches": branches.split(),
"driver_turns": len(turns), "claude_turns_total": sum(t.get("num_turns") or 0 for t in turns),
"tool_calls": sum(t.get("tool_calls", 0) for t in turns),
"tokens": tok, "usd": round(sum(t.get("usd", 0) for t in turns), 4),
"usd_po": round(po_cost, 4),
"wall_s": round(sum(t.get("wall_s", 0) for t in turns), 1), "run_wall_s": drv.get("run_wall_s"),
"end_reason": drv.get("end_reason"), "caps": drv.get("caps", []),
"questions_routed_to_po": sum(1 for t in turns if (t.get("po") or {}).get("kind") == "question"),
"skills_invoked": [s["skill"] for s in skills],
"skills_failed": [s["skill"] for s in skills if s.get("ok") is False],
"skill_events": skills,
"hook_events": hooks,
"superpowers_injected": any(h.get("superpowers_injected") for h in hooks),
"init_plugins": (turns[0].get("init_plugins") if turns else None),
"models": models, "model_ok": bool(models) and set(models) <= {MODEL},
"permission_denials": sum(len(t.get("permission_denials") or []) for t in turns),
"estimated_turns": [t["n"] for t in turns if t.get("estimated")],
"spd_slash_stage_per_turn": [t["stage"] for t in turns] if cfg == "spd" else None,
"per_turn": [{"n": t["n"], "stage": t["stage"], "source": t.get("prompt_source"), "usd": t.get("usd"),
"tokens": t.get("tokens", {}).get("total"), "wall_s": t.get("wall_s"),
"num_turns": t.get("num_turns"), "subtype": t.get("subtype"),
"skills": [s["skill"] for s in t.get("skills") or []],
"po_kind": (t.get("po") or {}).get("kind")} for t in turns],
"stages": stages,
}
json.dump(score, open(os.path.join(out, "score.json"), "w"), indent=2)
if os.environ.get("KEEP_NODE_MODULES") != "1":
shutil.rmtree(os.path.join(work, "node_modules"), ignore_errors=True)
print("%s: hidden %d/5 suite %s (%d/%d) tsc %s diff +%d/-%d (code +%d/-%d) tokens %d usd %.2f wall %ss turns %d "
"end=%s caps=%s skills=%s" % (
meta["name"], score["hidden_pass"], "green" if score["suite_green"] else "RED", s_pass, s_total,
"ok" if score["typecheck_green"] else "FAIL", score["diff_added"], score["diff_removed"],
score["code_diff_added"], score["code_diff_removed"], tok["total"], score["usd"], score["wall_s"],
len(turns), score["end_reason"], ",".join(score["caps"]) or "-",
",".join(OrderedDict.fromkeys(score["skills_invoked"])) or "-"))
if __name__ == "__main__":
main()
analyze.py
#!/usr/bin/env python3
"""runs/*/score.json (untagged runs only) -> results.json + results.md."""
import json
import os
import statistics as st
from collections import OrderedDict, defaultdict
HERE = os.path.dirname(os.path.abspath(__file__))
CFG_ORDER = ["bare", "sp", "spd", "mp", "both"]
CFG_LABEL = {"bare": "bare", "sp": "sp (hook-routed)", "spd": "spd (Superpowers driven)", "mp": "mp (Main Flow)", "both": "both (sp+mp, mp flow)"}
TASKS = ["F1", "F2"]
SPD_ORDER = ["/brainstorming", "/writing-plans", "/executing-plans"]
MP_ORDER = ["/grill-with-docs", "/to-spec", "/to-tickets", "/implement", "/code-review", "/pr"]
# Superpowers phases (the raw skill-in-force stages are kept too)
SP_PHASE = {
"(none)": "before any skill", "using-superpowers": "before any skill",
"brainstorming": "brainstorming", "writing-plans": "writing-plans",
"executing-plans": "execution", "subagent-driven-development": "execution",
"test-driven-development": "execution", "systematic-debugging": "execution",
"verification-before-completion": "verification", "dispatching-parallel-agents": "execution",
"requesting-code-review": "review", "receiving-code-review": "review",
"finishing-a-development-branch": "finishing", "using-git-worktrees": "execution",
"implementation (no skill)": "implementation (no skill)",
}
PHASE_ORDER = ["before any skill", "brainstorming", "writing-plans", "implementation (no skill)", "execution", "verification", "review", "finishing", "subagents (reviewers, explorers)", "other"]
def mean(xs):
xs = [x for x in xs if x is not None]
return round(st.mean(xs), 3) if xs else None
def med(xs):
xs = [x for x in xs if x is not None]
return round(st.median(xs), 3) if xs else None
def load():
runs = []
for d in sorted(os.listdir(os.path.join(HERE, "runs"))):
p = os.path.join(HERE, "runs", d, "score.json")
if not os.path.exists(p):
continue
s = json.load(open(p))
if s.get("tag"):
continue
runs.append(s)
return runs
def stage_agg(runs, key_fn, order):
agg = defaultdict(lambda: defaultdict(list))
for s in runs:
per = defaultdict(lambda: {"total": 0, "usd": 0.0, "wall_s": 0.0, "out": 0})
for name, v in s["stages"].items():
k = key_fn(name)
for f in ("total", "usd", "wall_s", "out"):
per[k][f] += v.get(f, 0)
for k, v in per.items():
if not (v["total"] or v["usd"] or v["wall_s"]):
continue # empty subagent placeholder segment
for f in v:
agg[k][f].append(v[f])
names = [n for n in order if n in agg] + sorted(n for n in agg if n not in order)
out = OrderedDict()
for n in names:
a = agg[n]
out[n] = {"runs_with_stage": len(a["total"]), "tokens_mean": mean(a["total"]), "tokens_median": med(a["total"]),
"out_tokens_mean": mean(a["out"]), "usd_mean": mean(a["usd"]), "usd_median": med(a["usd"]),
"wall_s_mean": mean(a["wall_s"]), "wall_s_median": med(a["wall_s"])}
return out
def turn_stage_agg(runs, order):
"""Per slash-command stage from the per-turn records (exact turn USD / tokens / wall)."""
agg = defaultdict(lambda: defaultdict(list))
for s in runs:
per = defaultdict(lambda: {"total": 0, "usd": 0.0, "wall_s": 0.0, "out": 0})
for t in s["per_turn"]:
per[t["stage"]]["total"] += t.get("tokens") or 0
per[t["stage"]]["usd"] += t.get("usd") or 0
per[t["stage"]]["wall_s"] += t.get("wall_s") or 0
for k, v in per.items():
for f in v:
agg[k][f].append(v[f])
out = OrderedDict()
for n in [n for n in order if n in agg] + sorted(n for n in agg if n not in order):
a = agg[n]
out[n] = {"runs_with_stage": len(a["total"]), "tokens_mean": mean(a["total"]), "tokens_median": med(a["total"]),
"out_tokens_mean": None, "usd_mean": mean(a["usd"]), "usd_median": med(a["usd"]),
"wall_s_mean": mean(a["wall_s"]), "wall_s_median": med(a["wall_s"])}
return out
def main():
runs = load()
for s in runs:
s.setdefault("task", "F1")
by = OrderedDict(("%s/%s" % (t, c), [s for s in runs if s["cfg"] == c and s["task"] == t]) for t in TASKS for c in CFG_ORDER)
res = {"configs": OrderedDict(), "stages": OrderedDict(), "runs": []}
for key, rs in by.items():
if not rs:
continue
task, c = key.split("/")
m = lambda f: [f(s) for s in rs] # noqa: E731
res["configs"][key] = {
"task": task, "cfg": c, "label": CFG_LABEL[c],
"n": len(rs),
"hidden_mean": mean(m(lambda s: s["hidden_pass"])), "hidden_median": med(m(lambda s: s["hidden_pass"])),
"hidden_per_run": m(lambda s: s["hidden_pass"]),
"req_pass_rate": {r: sum(1 for s in rs if s["hidden_reqs"].get(r)) for r in ["test_req%d" % i for i in range(1, 6)]},
"suite_green": sum(1 for s in rs if s["suite_green"]), "typecheck_green": sum(1 for s in rs if s["typecheck_green"]),
"tests_added_mean": mean(m(lambda s: s["tests_added_suite"])),
"code_diff_mean": mean(m(lambda s: s["code_diff_added"] + s["code_diff_removed"])),
"code_diff_median": med(m(lambda s: s["code_diff_added"] + s["code_diff_removed"])),
"all_diff_mean": mean(m(lambda s: s["diff_added"] + s["diff_removed"])),
"non_code_files_mean": mean(m(lambda s: len(s["non_code_files"]))),
"tokens_mean": mean(m(lambda s: s["tokens"]["total"])), "tokens_median": med(m(lambda s: s["tokens"]["total"])),
"out_tokens_mean": mean(m(lambda s: s["tokens"]["out"])),
"usd_mean": mean(m(lambda s: s["usd"])), "usd_median": med(m(lambda s: s["usd"])),
"usd_po_mean": mean(m(lambda s: s["usd_po"])),
"wall_s_mean": mean(m(lambda s: s["wall_s"])), "wall_s_median": med(m(lambda s: s["wall_s"])),
"driver_turns_mean": mean(m(lambda s: s["driver_turns"])),
"questions_mean": mean(m(lambda s: s["questions_routed_to_po"])),
"caps": {s["run"]: s["caps"] for s in rs if s["caps"]},
"end_reasons": m(lambda s: s["end_reason"]),
"models": sorted({x for s in rs for x in s["models"]}),
"superpowers_injected": m(lambda s: s["superpowers_injected"]),
"skills_per_run": {s["run"]: list(OrderedDict.fromkeys(s["skills_invoked"])) for s in rs},
"skills_failed": {s["run"]: s["skills_failed"] for s in rs if s["skills_failed"]},
}
if c in ("sp", "spd"):
res["stages"]["%s/%s_skill" % (task, c)] = stage_agg(rs, lambda n: n, list(SP_PHASE))
res["stages"]["%s/%s_phase" % (task, c)] = stage_agg(rs, lambda n: "subagents (reviewers, explorers)" if n.endswith("[subagent]") else SP_PHASE.get(n, "other"), PHASE_ORDER)
if c == "spd":
res["stages"]["%s/spd_slash" % task] = turn_stage_agg(rs, SPD_ORDER)
if c in ("mp", "both"):
res["stages"]["%s/%s" % (task, c)] = stage_agg(rs, lambda n: n, MP_ORDER)
for s in runs:
res["runs"].append({k: s[k] for k in ("run", "task", "cfg", "hidden_pass", "hidden_reqs", "suite_green", "typecheck_green",
"code_diff_added", "code_diff_removed", "diff_added", "diff_removed",
"tests_added_suite", "tokens", "usd", "usd_po", "wall_s", "driver_turns",
"questions_routed_to_po", "end_reason", "caps", "skills_invoked",
"superpowers_injected", "commits", "models", "stages")})
json.dump(res, open(os.path.join(HERE, "results.json"), "w"), indent=2)
L = ["# flow-bench results", "",
"Untagged runs only (smoke/validation excluded). Tokens = input + output + cache read + cache write, all models, "
"subagents included. USD = list-price equivalent from `total_cost_usd` deltas (plan quota, not a bill). "
"Wall = sum of worker turn walls (PO time excluded). Diff = lines added+removed vs the base commit, src/ + test/.", "",
"## Per config (mean / median)", "",
"| task / config | n | hidden /5 (per run) | suite green | tsc green | code diff +/- | all files diff | tests added | tokens | USD | wall s | driver turns | questions | caps hit |",
"|---|---|---|---|---|---|---|---|---|---|---|---|---|---|"]
for c, v in res["configs"].items():
L.append("| %s | %d | %s / %s (%s) | %d/%d | %d/%d | %s / %s | %s | %s | %s / %s | %s / %s | %s / %s | %s | %s | %s |" % (
"%s / %s" % (v["task"], v["label"]), v["n"], v["hidden_mean"], v["hidden_median"], ",".join(map(str, v["hidden_per_run"])), v["suite_green"], v["n"],
v["typecheck_green"], v["n"], v["code_diff_mean"], v["code_diff_median"], v["all_diff_mean"], v["tests_added_mean"],
fmt(v["tokens_mean"]), fmt(v["tokens_median"]), v["usd_mean"], v["usd_median"], v["wall_s_mean"], v["wall_s_median"],
v["driver_turns_mean"], v["questions_mean"], "; ".join("%s: %s" % (k, ",".join(x)) for k, x in v["caps"].items()) or "none"))
L += ["", "Requirement pass counts (runs passing each hidden req):", ""]
for c, v in res["configs"].items():
L.append("- **%s %s**: %s" % (v["task"], v["label"], ", ".join("%s %d/%d" % (k.replace("test_", ""), n, v["n"]) for k, n in v["req_pass_rate"].items())))
titles = []
for t in TASKS:
titles += [("%s/spd_slash" % t, "%s spd (Superpowers driven) per slash-command stage (turn level, exact)" % t),
("%s/spd_skill" % t, "%s spd per skill in force (within-turn split)" % t),
("%s/sp_phase" % t, "%s sp (hook-routed) per phase" % t),
("%s/sp_skill" % t, "%s sp (hook-routed) per skill in force" % t),
("%s/mp" % t, "%s mp per slash-command stage" % t), ("%s/both" % t, "%s both per slash-command stage" % t)]
for key, title in titles:
if key not in res["stages"]:
continue
L += ["", "## %s (mean over the runs that had the stage)" % title, "",
"| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |",
"|---|---|---|---|---|---|---|---|---|"]
for n, v in res["stages"][key].items():
L.append("| %s | %d | %s | %s | %s | %s | %s | %s | %s |" % (n, v["runs_with_stage"], fmt(v["tokens_mean"]),
fmt(v["tokens_median"]), fmt(v["out_tokens_mean"]), v["usd_mean"], v["usd_median"], v["wall_s_mean"], v["wall_s_median"]))
L += ["", "## Per run", "",
"| run | task | hidden | reqs failed | suite | tsc | code +/- | tests added | tokens | USD | wall s | turns | end | skills (first use order) |",
"|---|---|---|---|---|---|---|---|---|---|---|---|---|---|"]
for s in sorted(runs, key=lambda s: (s["task"], CFG_ORDER.index(s["cfg"]), s["run"])):
L.append("| %s | %s | %d/5 | %s | %s | %s | +%d/-%d | %d | %s | %.2f | %.0f | %d | %s%s | %s |" % (
s["run"], s["task"], s["hidden_pass"], ",".join(k.replace("test_", "") for k, v in s["hidden_reqs"].items() if not v) or "-",
"green" if s["suite_green"] else "RED", "ok" if s["typecheck_green"] else "FAIL", s["code_diff_added"],
s["code_diff_removed"], s["tests_added_suite"], fmt(s["tokens"]["total"]), s["usd"], s["wall_s"], s["driver_turns"],
s["end_reason"], (" (" + ",".join(s["caps"]) + ")") if s["caps"] else "",
" > ".join(OrderedDict.fromkeys(x.split(":")[-1] for x in s["skills_invoked"])) or "-"))
open(os.path.join(HERE, "results.md"), "w").write("\n".join(L) + "\n")
print("\n".join(L))
def fmt(x):
if x is None:
return "-"
return "{:,.0f}".format(x)
if __name__ == "__main__":
main()
results.json
{
"configs": {
"F1/bare": {
"task": "F1",
"cfg": "bare",
"label": "bare",
"n": 3,
"hidden_mean": 5,
"hidden_median": 5,
"hidden_per_run": [
5,
5,
5
],
"req_pass_rate": {
"test_req1": 3,
"test_req2": 3,
"test_req3": 3,
"test_req4": 3,
"test_req5": 3
},
"suite_green": 3,
"typecheck_green": 3,
"tests_added_mean": 4.667,
"code_diff_mean": 141.667,
"code_diff_median": 137,
"all_diff_mean": 142.333,
"non_code_files_mean": 0.333,
"tokens_mean": 519113.667,
"tokens_median": 361842,
"out_tokens_mean": 12001,
"usd_mean": 0.343,
"usd_median": 0.286,
"usd_po_mean": 0.021,
"wall_s_mean": 84.5,
"wall_s_median": 71.7,
"driver_turns_mean": 1.333,
"questions_mean": 0.333,
"caps": {},
"end_reasons": [
"done",
"done",
"done"
],
"models": [
"claude-sonnet-5-5"
],
"superpowers_injected": [
false,
false,
false
],
"skills_per_run": {
"bare-r1": [],
"bare-r2": [],
"bare-r3": []
},
"skills_failed": {}
},
"F1/sp": {
"task": "F1",
"cfg": "sp",
"label": "sp (hook-routed)",
"n": 3,
"hidden_mean": 5,
"hidden_median": 5,
"hidden_per_run": [
5,
5,
5
],
"req_pass_rate": {
"test_req1": 3,
"test_req2": 3,
"test_req3": 3,
"test_req4": 3,
"test_req5": 3
},
"suite_green": 3,
"typecheck_green": 3,
"tests_added_mean": 12.333,
"code_diff_mean": 235,
"code_diff_median": 227,
"all_diff_mean": 237,
"non_code_files_mean": 0.667,
"tokens_mean": 724600.667,
"tokens_median": 765108,
"out_tokens_mean": 16869.333,
"usd_mean": 0.502,
"usd_median": 0.525,
"usd_po_mean": 0.041,
"wall_s_mean": 118.1,
"wall_s_median": 119.6,
"driver_turns_mean": 2.667,
"questions_mean": 1.667,
"caps": {},
"end_reasons": [
"done",
"done",
"done"
],
"models": [
"claude-sonnet-5-5"
],
"superpowers_injected": [
true,
true,
true
],
"skills_per_run": {
"sp-r1": [
"brainstorming"
],
"sp-r2": [
"brainstorming",
"test-driven-development"
],
"sp-r3": [
"brainstorming"
]
},
"skills_failed": {}
},
"F1/spd": {
"task": "F1",
"cfg": "spd",
"label": "spd (Superpowers driven)",
"n": 3,
"hidden_mean": 5,
"hidden_median": 5,
"hidden_per_run": [
5,
5,
5
],
"req_pass_rate": {
"test_req1": 3,
"test_req2": 3,
"test_req3": 3,
"test_req4": 3,
"test_req5": 3
},
"suite_green": 3,
"typecheck_green": 3,
"tests_added_mean": 27.667,
"code_diff_mean": 386,
"code_diff_median": 371,
"all_diff_mean": 582,
"non_code_files_mean": 2,
"tokens_mean": 4479876.333,
"tokens_median": 3887015,
"out_tokens_mean": 61684.667,
"usd_mean": 3.738,
"usd_median": 3.412,
"usd_po_mean": 0.116,
"wall_s_mean": 607.367,
"wall_s_median": 600.8,
"driver_turns_mean": 7,
"questions_mean": 4,
"caps": {},
"end_reasons": [
"done",
"done",
"done"
],
"models": [
"claude-fable-5-1",
"claude-sonnet-5-5"
],
"superpowers_injected": [
true,
true,
true
],
"skills_per_run": {
"spd-r1": [
"test-driven-development"
],
"spd-r2": [
"test-driven-development"
],
"spd-r3": [
"test-driven-development",
"finishing-a-development-branch"
]
},
"skills_failed": {}
},
"F1/mp": {
"task": "F1",
"cfg": "mp",
"label": "mp (Main Flow)",
"n": 3,
"hidden_mean": 5,
"hidden_median": 5,
"hidden_per_run": [
5,
5,
5
],
"req_pass_rate": {
"test_req1": 3,
"test_req2": 3,
"test_req3": 3,
"test_req4": 3,
"test_req5": 3
},
"suite_green": 3,
"typecheck_green": 3,
"tests_added_mean": 27.667,
"code_diff_mean": 372,
"code_diff_median": 369,
"all_diff_mean": 578,
"non_code_files_mean": 7.667,
"tokens_mean": 4476103.667,
"tokens_median": 4286993,
"out_tokens_mean": 56959,
"usd_mean": 2.043,
"usd_median": 2.102,
"usd_po_mean": 0.205,
"wall_s_mean": 439.033,
"wall_s_median": 472.4,
"driver_turns_mean": 12,
"questions_mean": 7,
"caps": {
"mp-r1": [
"driver_turns_12"
],
"mp-r3": [
"driver_turns_12"
]
},
"end_reasons": [
"cap_driver_turns",
"done",
"cap_driver_turns"
],
"models": [
"claude-sonnet-5-5"
],
"superpowers_injected": [
false,
false,
false
],
"skills_per_run": {
"mp-r1": [
"grilling",
"domain-modeling",
"tdd",
"code-review"
],
"mp-r2": [
"grilling",
"domain-modeling",
"tdd",
"code-review"
],
"mp-r3": [
"grilling",
"domain-modeling",
"tdd",
"code-review"
]
},
"skills_failed": {}
},
"F1/both": {
"task": "F1",
"cfg": "both",
"label": "both (sp+mp, mp flow)",
"n": 1,
"hidden_mean": 5,
"hidden_median": 5,
"hidden_per_run": [
5
],
"req_pass_rate": {
"test_req1": 1,
"test_req2": 1,
"test_req3": 1,
"test_req4": 1,
"test_req5": 1
},
"suite_green": 1,
"typecheck_green": 1,
"tests_added_mean": 22,
"code_diff_mean": 374,
"code_diff_median": 374,
"all_diff_mean": 561,
"non_code_files_mean": 8,
"tokens_mean": 5754354,
"tokens_median": 5754354,
"out_tokens_mean": 57632,
"usd_mean": 2.326,
"usd_median": 2.326,
"usd_po_mean": 0.218,
"wall_s_mean": 448.2,
"wall_s_median": 448.2,
"driver_turns_mean": 12,
"questions_mean": 6,
"caps": {},
"end_reasons": [
"done"
],
"models": [
"claude-sonnet-5-5"
],
"superpowers_injected": [
true
],
"skills_per_run": {
"both-r1": [
"grilling",
"domain-modeling",
"tdd",
"code-review"
]
},
"skills_failed": {}
},
"F2/bare": {
"task": "F2",
"cfg": "bare",
"label": "bare",
"n": 2,
"hidden_mean": 4.5,
"hidden_median": 4.5,
"hidden_per_run": [
4,
5
],
"req_pass_rate": {
"test_req1": 2,
"test_req2": 2,
"test_req3": 1,
"test_req4": 2,
"test_req5": 2
},
"suite_green": 2,
"typecheck_green": 2,
"tests_added_mean": 5,
"code_diff_mean": 206.5,
"code_diff_median": 206.5,
"all_diff_mean": 209.5,
"non_code_files_mean": 1,
"tokens_mean": 651322,
"tokens_median": 651322.0,
"out_tokens_mean": 15475,
"usd_mean": 0.413,
"usd_median": 0.413,
"usd_po_mean": 0.031,
"wall_s_mean": 103.15,
"wall_s_median": 103.15,
"driver_turns_mean": 2,
"questions_mean": 0.5,
"caps": {},
"end_reasons": [
"done",
"done"
],
"models": [
"claude-sonnet-5-5"
],
"superpowers_injected": [
false,
false
],
"skills_per_run": {
"f2-bare-r1": [],
"f2-bare-r2": []
},
"skills_failed": {}
},
"F2/spd": {
"task": "F2",
"cfg": "spd",
"label": "spd (Superpowers driven)",
"n": 2,
"hidden_mean": 5,
"hidden_median": 5.0,
"hidden_per_run": [
5,
5
],
"req_pass_rate": {
"test_req1": 2,
"test_req2": 2,
"test_req3": 2,
"test_req4": 2,
"test_req5": 2
},
"suite_green": 2,
"typecheck_green": 2,
"tests_added_mean": 33,
"code_diff_mean": 488.5,
"code_diff_median": 488.5,
"all_diff_mean": 750.5,
"non_code_files_mean": 2.5,
"tokens_mean": 4490365,
"tokens_median": 4490365.0,
"out_tokens_mean": 56644.5,
"usd_mean": 3.213,
"usd_median": 3.213,
"usd_po_mean": 0.101,
"wall_s_mean": 511.7,
"wall_s_median": 511.7,
"driver_turns_mean": 6.5,
"questions_mean": 3.5,
"caps": {},
"end_reasons": [
"done",
"done"
],
"models": [
"claude-fable-5-1",
"claude-sonnet-5-5"
],
"superpowers_injected": [
true,
true
],
"skills_per_run": {
"f2-spd-r1": [
"writing-plans",
"test-driven-development"
],
"f2-spd-r2": [
"test-driven-development"
]
},
"skills_failed": {}
},
"F2/mp": {
"task": "F2",
"cfg": "mp",
"label": "mp (Main Flow)",
"n": 2,
"hidden_mean": 5,
"hidden_median": 5.0,
"hidden_per_run": [
5,
5
],
"req_pass_rate": {
"test_req1": 2,
"test_req2": 2,
"test_req3": 2,
"test_req4": 2,
"test_req5": 2
},
"suite_green": 2,
"typecheck_green": 2,
"tests_added_mean": 31,
"code_diff_mean": 469,
"code_diff_median": 469.0,
"all_diff_mean": 658,
"non_code_files_mean": 6.5,
"tokens_mean": 4520638.5,
"tokens_median": 4520638.5,
"out_tokens_mean": 57142,
"usd_mean": 2.067,
"usd_median": 2.067,
"usd_po_mean": 0.203,
"wall_s_mean": 421.2,
"wall_s_median": 421.2,
"driver_turns_mean": 12,
"questions_mean": 6.5,
"caps": {
"f2-mp-r1": [
"driver_turns_12"
]
},
"end_reasons": [
"cap_driver_turns",
"done"
],
"models": [
"claude-sonnet-5-5"
],
"superpowers_injected": [
false,
false
],
"skills_per_run": {
"f2-mp-r1": [
"grilling",
"domain-modeling",
"tdd",
"code-review"
],
"f2-mp-r2": [
"grilling",
"domain-modeling",
"tdd",
"code-review"
]
},
"skills_failed": {}
}
},
"stages": {
"F1/sp_skill": {
"(none)": {
"runs_with_stage": 3,
"tokens_mean": 19549,
"tokens_median": 19538,
"out_tokens_mean": 238,
"usd_mean": 0.03,
"usd_median": 0.03,
"wall_s_mean": 5.333,
"wall_s_median": 5.4
},
"brainstorming": {
"runs_with_stage": 3,
"tokens_mean": 249260,
"tokens_median": 247900,
"out_tokens_mean": 8037.333,
"usd_mean": 0.219,
"usd_median": 0.244,
"wall_s_mean": 40.533,
"wall_s_median": 40.1
},
"test-driven-development": {
"runs_with_stage": 1,
"tokens_mean": 595318,
"tokens_median": 595318,
"out_tokens_mean": 15622,
"usd_mean": 0.405,
"usd_median": 0.405,
"wall_s_mean": 93.0,
"wall_s_median": 93.0
},
"implementation (no skill)": {
"runs_with_stage": 2,
"tokens_mean": 386028.5,
"tokens_median": 386028.5,
"out_tokens_mean": 5080,
"usd_mean": 0.176,
"usd_median": 0.176,
"wall_s_mean": 60.3,
"wall_s_median": 60.3
}
},
"F1/sp_phase": {
"before any skill": {
"runs_with_stage": 3,
"tokens_mean": 19549,
"tokens_median": 19538,
"out_tokens_mean": 238,
"usd_mean": 0.03,
"usd_median": 0.03,
"wall_s_mean": 5.333,
"wall_s_median": 5.4
},
"brainstorming": {
"runs_with_stage": 3,
"tokens_mean": 249260,
"tokens_median": 247900,
"out_tokens_mean": 8037.333,
"usd_mean": 0.219,
"usd_median": 0.244,
"wall_s_mean": 40.533,
"wall_s_median": 40.1
},
"implementation (no skill)": {
"runs_with_stage": 2,
"tokens_mean": 386028.5,
"tokens_median": 386028.5,
"out_tokens_mean": 5080,
"usd_mean": 0.176,
"usd_median": 0.176,
"wall_s_mean": 60.3,
"wall_s_median": 60.3
},
"execution": {
"runs_with_stage": 1,
"tokens_mean": 595318,
"tokens_median": 595318,
"out_tokens_mean": 15622,
"usd_mean": 0.405,
"usd_median": 0.405,
"wall_s_mean": 93.0,
"wall_s_median": 93.0
}
},
"F1/spd_skill": {
"brainstorming": {
"runs_with_stage": 3,
"tokens_mean": 182021.667,
"tokens_median": 160834,
"out_tokens_mean": 4160.667,
"usd_mean": 0.166,
"usd_median": 0.156,
"wall_s_mean": 38.8,
"wall_s_median": 33.6
},
"writing-plans": {
"runs_with_stage": 3,
"tokens_mean": 321354.667,
"tokens_median": 298267,
"out_tokens_mean": 11827,
"usd_mean": 0.266,
"usd_median": 0.274,
"wall_s_mean": 86.733,
"wall_s_median": 95.1
},
"executing-plans": {
"runs_with_stage": 3,
"tokens_mean": 66938,
"tokens_median": 68739,
"out_tokens_mean": 505,
"usd_mean": 0.047,
"usd_median": 0.047,
"wall_s_mean": 4.6,
"wall_s_median": 4.6
},
"test-driven-development": {
"runs_with_stage": 3,
"tokens_mean": 2927098.333,
"tokens_median": 2854967,
"out_tokens_mean": 22508.333,
"usd_mean": 0.974,
"usd_median": 0.936,
"wall_s_mean": 457.033,
"wall_s_median": 448.9
},
"finishing-a-development-branch": {
"runs_with_stage": 1,
"tokens_mean": 976505,
"tokens_median": 976505,
"out_tokens_mean": 5297,
"usd_mean": 0.282,
"usd_median": 0.282,
"wall_s_mean": 50.7,
"wall_s_median": 50.7
},
"test-driven-development [subagent]": {
"runs_with_stage": 3,
"tokens_mean": 656962,
"tokens_median": 565811,
"out_tokens_mean": 20918,
"usd_mean": 2.192,
"usd_median": 2.047,
"wall_s_mean": 0.0,
"wall_s_median": 0.0
}
},
"F1/spd_phase": {
"brainstorming": {
"runs_with_stage": 3,
"tokens_mean": 182021.667,
"tokens_median": 160834,
"out_tokens_mean": 4160.667,
"usd_mean": 0.166,
"usd_median": 0.156,
"wall_s_mean": 38.8,
"wall_s_median": 33.6
},
"writing-plans": {
"runs_with_stage": 3,
"tokens_mean": 321354.667,
"tokens_median": 298267,
"out_tokens_mean": 11827,
"usd_mean": 0.266,
"usd_median": 0.274,
"wall_s_mean": 86.733,
"wall_s_median": 95.1
},
"execution": {
"runs_with_stage": 3,
"tokens_mean": 2994036.333,
"tokens_median": 2916459,
"out_tokens_mean": 23013.333,
"usd_mean": 1.021,
"usd_median": 0.981,
"wall_s_mean": 461.633,
"wall_s_median": 453.5
},
"finishing": {
"runs_with_stage": 1,
"tokens_mean": 976505,
"tokens_median": 976505,
"out_tokens_mean": 5297,
"usd_mean": 0.282,
"usd_median": 0.282,
"wall_s_mean": 50.7,
"wall_s_median": 50.7
},
"subagents (reviewers, explorers)": {
"runs_with_stage": 3,
"tokens_mean": 656962,
"tokens_median": 565811,
"out_tokens_mean": 20918,
"usd_mean": 2.192,
"usd_median": 2.047,
"wall_s_mean": 0.0,
"wall_s_median": 0.0
}
},
"F1/spd_slash": {
"/brainstorming": {
"runs_with_stage": 3,
"tokens_mean": 182021.667,
"tokens_median": 160834,
"out_tokens_mean": null,
"usd_mean": 0.166,
"usd_median": 0.156,
"wall_s_mean": 40.283,
"wall_s_median": 35.44
},
"/writing-plans": {
"runs_with_stage": 3,
"tokens_mean": 321354.667,
"tokens_median": 298267,
"out_tokens_mean": null,
"usd_mean": 0.266,
"usd_median": 0.274,
"wall_s_mean": 88.133,
"wall_s_median": 95.82
},
"/executing-plans": {
"runs_with_stage": 3,
"tokens_mean": 3976500,
"tokens_median": 3427914,
"out_tokens_mean": null,
"usd_mean": 3.307,
"usd_median": 3.028,
"wall_s_mean": 478.95,
"wall_s_median": 453.89
}
},
"F1/mp": {
"/grill-with-docs": {
"runs_with_stage": 3,
"tokens_mean": 316507.333,
"tokens_median": 348862,
"out_tokens_mean": 5999,
"usd_mean": 0.204,
"usd_median": 0.215,
"wall_s_mean": 59.933,
"wall_s_median": 59.9
},
"/to-spec": {
"runs_with_stage": 3,
"tokens_mean": 163344,
"tokens_median": 167578,
"out_tokens_mean": 5134.667,
"usd_mean": 0.118,
"usd_median": 0.113,
"wall_s_mean": 42.1,
"wall_s_median": 40.8
},
"/to-tickets": {
"runs_with_stage": 3,
"tokens_mean": 257955.333,
"tokens_median": 268062,
"out_tokens_mean": 5867.333,
"usd_mean": 0.143,
"usd_median": 0.151,
"wall_s_mean": 44.4,
"wall_s_median": 47.8
},
"/implement": {
"runs_with_stage": 3,
"tokens_mean": 2655564.667,
"tokens_median": 2613062,
"out_tokens_mean": 30214.667,
"usd_mean": 1.125,
"usd_median": 1.195,
"wall_s_mean": 224.567,
"wall_s_median": 245.4
},
"/code-review": {
"runs_with_stage": 3,
"tokens_mean": 1047585.667,
"tokens_median": 1107555,
"out_tokens_mean": 9293.667,
"usd_mean": 0.439,
"usd_median": 0.445,
"wall_s_mean": 64.2,
"wall_s_median": 65.8
},
"/pr": {
"runs_with_stage": 1,
"tokens_mean": 105440,
"tokens_median": 105440,
"out_tokens_mean": 1349,
"usd_mean": 0.041,
"usd_median": 0.041,
"wall_s_mean": 11.6,
"wall_s_median": 11.6
}
},
"F1/both": {
"/grill-with-docs": {
"runs_with_stage": 1,
"tokens_mean": 344391,
"tokens_median": 344391,
"out_tokens_mean": 6193,
"usd_mean": 0.225,
"usd_median": 0.225,
"wall_s_mean": 61.8,
"wall_s_median": 61.8
},
"/to-spec": {
"runs_with_stage": 1,
"tokens_mean": 217572,
"tokens_median": 217572,
"out_tokens_mean": 4835,
"usd_mean": 0.123,
"usd_median": 0.123,
"wall_s_mean": 42.9,
"wall_s_median": 42.9
},
"/to-tickets": {
"runs_with_stage": 1,
"tokens_mean": 382953,
"tokens_median": 382953,
"out_tokens_mean": 7115,
"usd_mean": 0.196,
"usd_median": 0.196,
"wall_s_mean": 49.4,
"wall_s_median": 49.4
},
"/implement": {
"runs_with_stage": 1,
"tokens_mean": 3070579,
"tokens_median": 3070579,
"out_tokens_mean": 24618,
"usd_mean": 1.112,
"usd_median": 1.112,
"wall_s_mean": 194.2,
"wall_s_median": 194.2
},
"/code-review": {
"runs_with_stage": 1,
"tokens_mean": 1615265,
"tokens_median": 1615265,
"out_tokens_mean": 13410,
"usd_mean": 0.625,
"usd_median": 0.625,
"wall_s_mean": 86.8,
"wall_s_median": 86.8
},
"/pr": {
"runs_with_stage": 1,
"tokens_mean": 123594,
"tokens_median": 123594,
"out_tokens_mean": 1461,
"usd_mean": 0.045,
"usd_median": 0.045,
"wall_s_mean": 13.1,
"wall_s_median": 13.1
}
},
"F2/spd_skill": {
"brainstorming": {
"runs_with_stage": 2,
"tokens_mean": 253890,
"tokens_median": 253890.0,
"out_tokens_mean": 6038,
"usd_mean": 0.207,
"usd_median": 0.207,
"wall_s_mean": 53.05,
"wall_s_median": 53.05
},
"writing-plans": {
"runs_with_stage": 2,
"tokens_mean": 397486.5,
"tokens_median": 397486.5,
"out_tokens_mean": 14059.5,
"usd_mean": 0.307,
"usd_median": 0.307,
"wall_s_mean": 90.75,
"wall_s_median": 90.75
},
"executing-plans": {
"runs_with_stage": 2,
"tokens_mean": 69973,
"tokens_median": 69973.0,
"out_tokens_mean": 286.5,
"usd_mean": 0.046,
"usd_median": 0.046,
"wall_s_mean": 3.5,
"wall_s_median": 3.5
},
"test-driven-development": {
"runs_with_stage": 2,
"tokens_mean": 3304676.5,
"tokens_median": 3304676.5,
"out_tokens_mean": 23243.5,
"usd_mean": 1.052,
"usd_median": 1.052,
"wall_s_mean": 361.25,
"wall_s_median": 361.25
},
"test-driven-development [subagent]": {
"runs_with_stage": 2,
"tokens_mean": 464339,
"tokens_median": 464339.0,
"out_tokens_mean": 13017,
"usd_mean": 1.602,
"usd_median": 1.602,
"wall_s_mean": 0.0,
"wall_s_median": 0.0
}
},
"F2/spd_phase": {
"brainstorming": {
"runs_with_stage": 2,
"tokens_mean": 253890,
"tokens_median": 253890.0,
"out_tokens_mean": 6038,
"usd_mean": 0.207,
"usd_median": 0.207,
"wall_s_mean": 53.05,
"wall_s_median": 53.05
},
"writing-plans": {
"runs_with_stage": 2,
"tokens_mean": 397486.5,
"tokens_median": 397486.5,
"out_tokens_mean": 14059.5,
"usd_mean": 0.307,
"usd_median": 0.307,
"wall_s_mean": 90.75,
"wall_s_median": 90.75
},
"execution": {
"runs_with_stage": 2,
"tokens_mean": 3374649.5,
"tokens_median": 3374649.5,
"out_tokens_mean": 23530,
"usd_mean": 1.098,
"usd_median": 1.098,
"wall_s_mean": 364.75,
"wall_s_median": 364.75
},
"subagents (reviewers, explorers)": {
"runs_with_stage": 2,
"tokens_mean": 464339,
"tokens_median": 464339.0,
"out_tokens_mean": 13017,
"usd_mean": 1.602,
"usd_median": 1.602,
"wall_s_mean": 0.0,
"wall_s_median": 0.0
}
},
"F2/spd_slash": {
"/brainstorming": {
"runs_with_stage": 2,
"tokens_mean": 485364,
"tokens_median": 485364.0,
"out_tokens_mean": null,
"usd_mean": 0.354,
"usd_median": 0.354,
"wall_s_mean": 96.815,
"wall_s_median": 96.815
},
"/writing-plans": {
"runs_with_stage": 2,
"tokens_mean": 166012.5,
"tokens_median": 166012.5,
"out_tokens_mean": null,
"usd_mean": 0.159,
"usd_median": 0.159,
"wall_s_mean": 49.755,
"wall_s_median": 49.755
},
"/executing-plans": {
"runs_with_stage": 2,
"tokens_mean": 3838988.5,
"tokens_median": 3838988.5,
"out_tokens_mean": null,
"usd_mean": 2.7,
"usd_median": 2.7,
"wall_s_mean": 365.1,
"wall_s_median": 365.1
}
},
"F2/mp": {
"/grill-with-docs": {
"runs_with_stage": 2,
"tokens_mean": 232965,
"tokens_median": 232965.0,
"out_tokens_mean": 5946,
"usd_mean": 0.182,
"usd_median": 0.182,
"wall_s_mean": 56.7,
"wall_s_median": 56.7
},
"/to-spec": {
"runs_with_stage": 2,
"tokens_mean": 180084.5,
"tokens_median": 180084.5,
"out_tokens_mean": 4452,
"usd_mean": 0.115,
"usd_median": 0.115,
"wall_s_mean": 37.6,
"wall_s_median": 37.6
},
"/to-tickets": {
"runs_with_stage": 2,
"tokens_mean": 253688.5,
"tokens_median": 253688.5,
"out_tokens_mean": 5152.5,
"usd_mean": 0.131,
"usd_median": 0.131,
"wall_s_mean": 38.5,
"wall_s_median": 38.5
},
"/implement": {
"runs_with_stage": 2,
"tokens_mean": 2689071,
"tokens_median": 2689071.0,
"out_tokens_mean": 31452,
"usd_mean": 1.159,
"usd_median": 1.159,
"wall_s_mean": 218.35,
"wall_s_median": 218.35
},
"/code-review": {
"runs_with_stage": 2,
"tokens_mean": 1107122.5,
"tokens_median": 1107122.5,
"out_tokens_mean": 9453.5,
"usd_mean": 0.458,
"usd_median": 0.458,
"wall_s_mean": 63.85,
"wall_s_median": 63.85
},
"/pr": {
"runs_with_stage": 1,
"tokens_mean": 115414,
"tokens_median": 115414,
"out_tokens_mean": 1372,
"usd_mean": 0.043,
"usd_median": 0.043,
"wall_s_mean": 12.4,
"wall_s_median": 12.4
}
}
},
"runs": [
{
"run": "bare-r1",
"task": "F1",
"cfg": "bare",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 149,
"code_diff_removed": 11,
"diff_added": 150,
"diff_removed": 12,
"tests_added_suite": 6,
"tokens": {
"in": 42,
"out": 16265,
"cr": 786548,
"cw": 37896,
"total": 840751
},
"usd": 0.4716,
"usd_po": 0.0321,
"wall_s": 118.8,
"driver_turns": 2,
"questions_routed_to_po": 1,
"end_reason": "done",
"caps": [],
"skills_invoked": [],
"superpowers_injected": false,
"commits": 0,
"models": [
"claude-sonnet-5-5"
],
"stages": {
"bare": {
"turns": [
1,
2
],
"in": 42,
"out": 16265,
"cr": 786548,
"cw": 37896,
"total": 840751,
"usd": 0.4716,
"wall_s": 118.8,
"api_calls": 21
}
}
},
{
"run": "bare-r2",
"task": "F1",
"cfg": "bare",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 118,
"code_diff_removed": 10,
"diff_added": 118,
"diff_removed": 10,
"tests_added_suite": 4,
"tokens": {
"in": 22,
"out": 9513,
"cr": 316959,
"cw": 28254,
"total": 354748
},
"usd": 0.2716,
"usd_po": 0.016,
"wall_s": 63.0,
"driver_turns": 1,
"questions_routed_to_po": 0,
"end_reason": "done",
"caps": [],
"skills_invoked": [],
"superpowers_injected": false,
"commits": 0,
"models": [
"claude-sonnet-5-5"
],
"stages": {
"bare": {
"turns": [
1
],
"in": 22,
"out": 9513,
"cr": 316959,
"cw": 28254,
"total": 354748,
"usd": 0.2716,
"wall_s": 63.0,
"api_calls": 11
}
}
},
{
"run": "bare-r3",
"task": "F1",
"cfg": "bare",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 125,
"code_diff_removed": 12,
"diff_added": 125,
"diff_removed": 12,
"tests_added_suite": 4,
"tokens": {
"in": 22,
"out": 10225,
"cr": 321705,
"cw": 29890,
"total": 361842
},
"usd": 0.2862,
"usd_po": 0.0148,
"wall_s": 71.7,
"driver_turns": 1,
"questions_routed_to_po": 0,
"end_reason": "done",
"caps": [],
"skills_invoked": [],
"superpowers_injected": false,
"commits": 0,
"models": [
"claude-sonnet-5-5"
],
"stages": {
"bare": {
"turns": [
1
],
"in": 22,
"out": 10225,
"cr": 321705,
"cw": 29890,
"total": 361842,
"usd": 0.2862,
"wall_s": 71.7,
"api_calls": 11
}
}
},
{
"run": "both-r1",
"task": "F1",
"cfg": "both",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 356,
"code_diff_removed": 18,
"diff_added": 543,
"diff_removed": 18,
"tests_added_suite": 22,
"tokens": {
"in": 192,
"out": 57632,
"cr": 5502518,
"cw": 194012,
"total": 5754354
},
"usd": 2.3259,
"usd_po": 0.2185,
"wall_s": 448.2,
"driver_turns": 12,
"questions_routed_to_po": 6,
"end_reason": "done",
"caps": [],
"skills_invoked": [
"grilling",
"domain-modeling",
"tdd",
"code-review"
],
"superpowers_injected": true,
"commits": 7,
"models": [
"claude-sonnet-5-5"
],
"stages": {
"/grill-with-docs": {
"turns": [
1,
2,
3,
4
],
"in": 22,
"out": 6193,
"cr": 313031,
"cw": 25145,
"total": 344391,
"usd": 0.2252,
"wall_s": 61.8,
"api_calls": 11
},
"/to-spec": {
"turns": [
5,
6
],
"in": 12,
"out": 4835,
"cr": 204355,
"cw": 8370,
"total": 217572,
"usd": 0.1227,
"wall_s": 42.9,
"api_calls": 5
},
"/to-tickets": {
"turns": [
7,
8
],
"in": 16,
"out": 7115,
"cr": 362815,
"cw": 13007,
"total": 382953,
"usd": 0.1958,
"wall_s": 49.4,
"api_calls": 7
},
"/implement": {
"turns": [
9
],
"in": 92,
"out": 24618,
"cr": 2960003,
"cw": 85866,
"total": 3070579,
"usd": 1.1123,
"wall_s": 194.2,
"api_calls": 36
},
"/code-review": {
"turns": [
10,
11
],
"in": 46,
"out": 13410,
"cr": 1541663,
"cw": 60146,
"total": 1615265,
"usd": 0.6253,
"wall_s": 86.8,
"api_calls": 13
},
"/pr": {
"turns": [
12
],
"in": 4,
"out": 1461,
"cr": 120651,
"cw": 1478,
"total": 123594,
"usd": 0.0447,
"wall_s": 13.1,
"api_calls": 1
}
}
},
{
"run": "f2-bare-r1",
"task": "F2",
"cfg": "bare",
"hidden_pass": 4,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": false,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 204,
"code_diff_removed": 10,
"diff_added": 208,
"diff_removed": 10,
"tests_added_suite": 5,
"tokens": {
"in": 28,
"out": 15066,
"cr": 444890,
"cw": 34247,
"total": 494231
},
"usd": 0.3767,
"usd_po": 0.0167,
"wall_s": 91.6,
"driver_turns": 1,
"questions_routed_to_po": 0,
"end_reason": "done",
"caps": [],
"skills_invoked": [],
"superpowers_injected": false,
"commits": 0,
"models": [
"claude-sonnet-5-5"
],
"stages": {
"bare": {
"turns": [
1
],
"in": 28,
"out": 15066,
"cr": 444890,
"cw": 34247,
"total": 494231,
"usd": 0.3767,
"wall_s": 91.6,
"api_calls": 14
}
}
},
{
"run": "f2-bare-r2",
"task": "F2",
"cfg": "bare",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 189,
"code_diff_removed": 10,
"diff_added": 191,
"diff_removed": 10,
"tests_added_suite": 5,
"tokens": {
"in": 42,
"out": 15884,
"cr": 757517,
"cw": 34970,
"total": 808413
},
"usd": 0.4503,
"usd_po": 0.0453,
"wall_s": 114.7,
"driver_turns": 3,
"questions_routed_to_po": 1,
"end_reason": "done",
"caps": [],
"skills_invoked": [],
"superpowers_injected": false,
"commits": 1,
"models": [
"claude-sonnet-5-5"
],
"stages": {
"bare": {
"turns": [
1,
2,
3
],
"in": 42,
"out": 15884,
"cr": 757517,
"cw": 34970,
"total": 808413,
"usd": 0.4503,
"wall_s": 114.7,
"api_calls": 21
}
}
},
{
"run": "f2-mp-r1",
"task": "F2",
"cfg": "mp",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 460,
"code_diff_removed": 23,
"diff_added": 666,
"diff_removed": 25,
"tests_added_suite": 31,
"tokens": {
"in": 172,
"out": 60658,
"cr": 4559790,
"cw": 202896,
"total": 4823516
},
"usd": 2.1824,
"usd_po": 0.1989,
"wall_s": 443.0,
"driver_turns": 12,
"questions_routed_to_po": 7,
"end_reason": "cap_driver_turns",
"caps": [
"driver_turns_12"
],
"skills_invoked": [
"grilling",
"domain-modeling",
"tdd",
"code-review"
],
"superpowers_injected": false,
"commits": 7,
"models": [
"claude-sonnet-5-5"
],
"stages": {
"/grill-with-docs": {
"turns": [
1,
2,
3
],
"in": 18,
"out": 6627,
"cr": 229372,
"cw": 22326,
"total": 258343,
"usd": 0.2015,
"wall_s": 62.8,
"api_calls": 9
},
"/to-spec": {
"turns": [
4,
5
],
"in": 12,
"out": 5218,
"cr": 195137,
"cw": 10113,
"total": 210480,
"usd": 0.1317,
"wall_s": 45.4,
"api_calls": 5
},
"/to-tickets": {
"turns": [
6,
7,
8
],
"in": 16,
"out": 7288,
"cr": 356825,
"cw": 10505,
"total": 374634,
"usd": 0.1863,
"wall_s": 53.0,
"api_calls": 7
},
"/implement": {
"turns": [
9,
10
],
"in": 92,
"out": 34442,
"cr": 2978947,
"cw": 104921,
"total": 3118402,
"usd": 1.2795,
"wall_s": 238.8,
"api_calls": 36
},
"/code-review": {
"turns": [
11,
12
],
"in": 34,
"out": 7083,
"cr": 799509,
"cw": 55031,
"total": 861657,
"usd": 0.3835,
"wall_s": 43.1,
"api_calls": 6
}
}
},
{
"run": "f2-mp-r2",
"task": "F2",
"cfg": "mp",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 440,
"code_diff_removed": 15,
"diff_added": 607,
"diff_removed": 18,
"tests_added_suite": 31,
"tokens": {
"in": 152,
"out": 53626,
"cr": 3976936,
"cw": 187047,
"total": 4217761
},
"usd": 1.9511,
"usd_po": 0.2073,
"wall_s": 399.4,
"driver_turns": 12,
"questions_routed_to_po": 6,
"end_reason": "done",
"caps": [],
"skills_invoked": [
"grilling",
"domain-modeling",
"tdd",
"code-review"
],
"superpowers_injected": false,
"commits": 2,
"models": [
"claude-sonnet-5-5"
],
"stages": {
"/grill-with-docs": {
"turns": [
1,
2,
3
],
"in": 16,
"out": 5265,
"cr": 184055,
"cw": 18251,
"total": 207587,
"usd": 0.1625,
"wall_s": 50.6,
"api_calls": 8
},
"/to-spec": {
"turns": [
4,
5
],
"in": 10,
"out": 3686,
"cr": 137547,
"cw": 8446,
"total": 149689,
"usd": 0.0982,
"wall_s": 29.8,
"api_calls": 4
},
"/to-tickets": {
"turns": [
6,
7
],
"in": 8,
"out": 3017,
"cr": 124377,
"cw": 5341,
"total": 132743,
"usd": 0.0764,
"wall_s": 24.0,
"api_calls": 3
},
"/implement": {
"turns": [
8
],
"in": 74,
"out": 28462,
"cr": 2129926,
"cw": 101278,
"total": 2259740,
"usd": 1.0391,
"wall_s": 197.9,
"api_calls": 27
},
"/code-review": {
"turns": [
9,
10,
11
],
"in": 43,
"out": 11824,
"cr": 1288603,
"cw": 52118,
"total": 1352588,
"usd": 0.5323,
"wall_s": 84.6,
"api_calls": 11
},
"/pr": {
"turns": [
12
],
"in": 1,
"out": 1372,
"cr": 112428,
"cw": 1613,
"total": 115414,
"usd": 0.0427,
"wall_s": 12.4,
"api_calls": 1
}
}
},
{
"run": "f2-spd-r1",
"task": "F2",
"cfg": "spd",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 546,
"code_diff_removed": 14,
"diff_added": 856,
"diff_removed": 17,
"tests_added_suite": 46,
"tokens": {
"in": 380,
"out": 56881,
"cr": 5074733,
"cw": 178622,
"total": 5310616
},
"usd": 3.4244,
"usd_po": 0.09,
"wall_s": 506.3,
"driver_turns": 6,
"questions_routed_to_po": 3,
"end_reason": "done",
"caps": [],
"skills_invoked": [
"writing-plans",
"test-driven-development"
],
"superpowers_injected": true,
"commits": 8,
"models": [
"claude-fable-5-1",
"claude-sonnet-5-5"
],
"stages": {
"brainstorming": {
"turns": [
1,
2,
3
],
"in": 18,
"out": 8103,
"cr": 280105,
"cw": 28324,
"total": 316550,
"usd": 0.2504,
"wall_s": 64.3,
"api_calls": 9
},
"writing-plans": {
"turns": [
3,
4,
5
],
"in": 20,
"out": 13028,
"cr": 491569,
"cw": 24216,
"total": 528833,
"usd": 0.3255,
"wall_s": 89.2,
"api_calls": 9
},
"writing-plans [subagent]": {
"turns": [
4
],
"in": 0,
"out": 0,
"cr": 0,
"cw": 0,
"total": 0,
"usd": 0.0,
"wall_s": 0.0,
"api_calls": 0
},
"executing-plans": {
"turns": [
6
],
"in": 4,
"out": 291,
"cr": 65551,
"cw": 7617,
"total": 73463,
"usd": 0.0465,
"wall_s": 4.0,
"api_calls": 1
},
"test-driven-development": {
"turns": [
6
],
"in": 80,
"out": 24030,
"cr": 3786616,
"cw": 42748,
"total": 3853474,
"usd": 1.1688,
"wall_s": 345.9,
"api_calls": 40
},
"test-driven-development [subagent]": {
"turns": [
6
],
"in": 258,
"out": 11429,
"cr": 450892,
"cw": 75717,
"total": 538296,
"usd": 1.6332,
"wall_s": 0.0,
"api_calls": 0
}
}
},
{
"run": "f2-spd-r2",
"task": "F2",
"cfg": "spd",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 401,
"code_diff_removed": 16,
"diff_added": 611,
"diff_removed": 17,
"tests_added_suite": 20,
"tokens": {
"in": 316,
"out": 56408,
"cr": 3456411,
"cw": 156979,
"total": 3670114
},
"usd": 3.0022,
"usd_po": 0.1126,
"wall_s": 517.1,
"driver_turns": 7,
"questions_routed_to_po": 4,
"end_reason": "done",
"caps": [],
"skills_invoked": [
"test-driven-development"
],
"superpowers_injected": true,
"commits": 6,
"models": [
"claude-fable-5-1",
"claude-sonnet-5-5"
],
"stages": {
"brainstorming": {
"turns": [
1,
2,
3,
4
],
"in": 12,
"out": 3973,
"cr": 164538,
"cw": 22707,
"total": 191230,
"usd": 0.1635,
"wall_s": 41.8,
"api_calls": 6
},
"brainstorming [subagent]": {
"turns": [
1,
3
],
"in": 0,
"out": 0,
"cr": 0,
"cw": 0,
"total": 0,
"usd": 0.0,
"wall_s": 0.0,
"api_calls": 0
},
"writing-plans": {
"turns": [
5,
6
],
"in": 12,
"out": 15091,
"cr": 228188,
"cw": 22849,
"total": 266140,
"usd": 0.288,
"wall_s": 92.3,
"api_calls": 5
},
"writing-plans [subagent]": {
"turns": [
6
],
"in": 0,
"out": 0,
"cr": 0,
"cw": 0,
"total": 0,
"usd": 0.0,
"wall_s": 0.0,
"api_calls": 0
},
"executing-plans": {
"turns": [
7
],
"in": 4,
"out": 282,
"cr": 58567,
"cw": 7630,
"total": 66483,
"usd": 0.0451,
"wall_s": 3.0,
"api_calls": 1
},
"test-driven-development": {
"turns": [
7
],
"in": 62,
"out": 22457,
"cr": 2690340,
"cw": 43020,
"total": 2755879,
"usd": 0.9348,
"wall_s": 376.6,
"api_calls": 31
},
"test-driven-development [subagent]": {
"turns": [
7
],
"in": 226,
"out": 14605,
"cr": 314778,
"cw": 60773,
"total": 390382,
"usd": 1.5709,
"wall_s": 0.0,
"api_calls": 0
}
}
},
{
"run": "mp-r1",
"task": "F1",
"cfg": "mp",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 349,
"code_diff_removed": 20,
"diff_added": 524,
"diff_removed": 20,
"tests_added_suite": 29,
"tokens": {
"in": 181,
"out": 60902,
"cr": 4842442,
"cw": 194978,
"total": 5098503
},
"usd": 2.2217,
"usd_po": 0.2058,
"wall_s": 472.4,
"driver_turns": 12,
"questions_routed_to_po": 7,
"end_reason": "cap_driver_turns",
"caps": [
"driver_turns_12"
],
"skills_invoked": [
"grilling",
"domain-modeling",
"tdd",
"code-review"
],
"superpowers_injected": false,
"commits": 7,
"models": [
"claude-sonnet-5-5"
],
"stages": {
"/grill-with-docs": {
"turns": [
1,
2,
3
],
"in": 24,
"out": 6223,
"cr": 320351,
"cw": 22264,
"total": 348862,
"usd": 0.2154,
"wall_s": 59.9,
"api_calls": 12
},
"/to-spec": {
"turns": [
4,
5
],
"in": 10,
"out": 4516,
"cr": 153800,
"cw": 9252,
"total": 167578,
"usd": 0.1129,
"wall_s": 37.6,
"api_calls": 4
},
"/to-tickets": {
"turns": [
6,
7,
8
],
"in": 16,
"out": 7382,
"cr": 349965,
"cw": 10865,
"total": 368228,
"usd": 0.1873,
"wall_s": 55.8,
"api_calls": 7
},
"/implement": {
"turns": [
9,
10
],
"in": 92,
"out": 32290,
"cr": 2890397,
"cw": 100256,
"total": 3023035,
"usd": 1.226,
"wall_s": 251.2,
"api_calls": 35
},
"/code-review": {
"turns": [
11,
12
],
"in": 39,
"out": 10491,
"cr": 1127929,
"cw": 52341,
"total": 1190800,
"usd": 0.4801,
"wall_s": 67.9,
"api_calls": 9
}
}
},
{
"run": "mp-r2",
"task": "F1",
"cfg": "mp",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 314,
"code_diff_removed": 15,
"diff_added": 527,
"diff_removed": 15,
"tests_added_suite": 25,
"tokens": {
"in": 154,
"out": 48104,
"cr": 3826019,
"cw": 168538,
"total": 4042815
},
"usd": 1.8045,
"usd_po": 0.1988,
"wall_s": 365.0,
"driver_turns": 12,
"questions_routed_to_po": 6,
"end_reason": "done",
"caps": [],
"skills_invoked": [
"grilling",
"domain-modeling",
"tdd",
"code-review"
],
"superpowers_injected": false,
"commits": 7,
"models": [
"claude-sonnet-5-5"
],
"stages": {
"/grill-with-docs": {
"turns": [
1,
2,
3
],
"in": 16,
"out": 4727,
"cr": 187718,
"cw": 18567,
"total": 211028,
"usd": 0.1591,
"wall_s": 47.0,
"api_calls": 8
},
"/to-spec": {
"turns": [
4,
5
],
"in": 10,
"out": 4965,
"cr": 136863,
"cw": 8781,
"total": 150619,
"usd": 0.1122,
"wall_s": 40.8,
"api_calls": 4
},
"/to-tickets": {
"turns": [
6,
7
],
"in": 8,
"out": 4035,
"cr": 126982,
"cw": 6551,
"total": 137576,
"usd": 0.092,
"wall_s": 29.6,
"api_calls": 3
},
"/implement": {
"turns": [
8,
9
],
"in": 78,
"out": 24215,
"cr": 2224544,
"cw": 81760,
"total": 2330597,
"usd": 0.9551,
"wall_s": 177.1,
"api_calls": 31
},
"/code-review": {
"turns": [
10,
11
],
"in": 38,
"out": 8813,
"cr": 1047606,
"cw": 51098,
"total": 1107555,
"usd": 0.4451,
"wall_s": 58.9,
"api_calls": 10
},
"/pr": {
"turns": [
12
],
"in": 4,
"out": 1349,
"cr": 102306,
"cw": 1781,
"total": 105440,
"usd": 0.0411,
"wall_s": 11.6,
"api_calls": 1
}
}
},
{
"run": "mp-r3",
"task": "F1",
"cfg": "mp",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 395,
"code_diff_removed": 23,
"diff_added": 625,
"diff_removed": 23,
"tests_added_suite": 29,
"tokens": {
"in": 167,
"out": 61871,
"cr": 4014893,
"cw": 210062,
"total": 4286993
},
"usd": 2.1024,
"usd_po": 0.2105,
"wall_s": 479.7,
"driver_turns": 12,
"questions_routed_to_po": 8,
"end_reason": "cap_driver_turns",
"caps": [
"driver_turns_12"
],
"skills_invoked": [
"grilling",
"domain-modeling",
"tdd",
"code-review"
],
"superpowers_injected": false,
"commits": 5,
"models": [
"claude-sonnet-5-5"
],
"stages": {
"/grill-with-docs": {
"turns": [
1,
2,
3,
4
],
"in": 26,
"out": 7047,
"cr": 358800,
"cw": 23759,
"total": 389632,
"usd": 0.2373,
"wall_s": 72.9,
"api_calls": 13
},
"/to-spec": {
"turns": [
5,
6
],
"in": 10,
"out": 5923,
"cr": 156583,
"cw": 9319,
"total": 171835,
"usd": 0.1278,
"wall_s": 47.9,
"api_calls": 4
},
"/to-tickets": {
"turns": [
7,
8,
9
],
"in": 9,
"out": 6185,
"cr": 252224,
"cw": 9644,
"total": 268062,
"usd": 0.1509,
"wall_s": 47.8,
"api_calls": 5
},
"/implement": {
"turns": [
10,
11
],
"in": 82,
"out": 34139,
"cr": 2465960,
"cw": 112881,
"total": 2613062,
"usd": 1.1946,
"wall_s": 245.4,
"api_calls": 27
},
"/code-review": {
"turns": [
12
],
"in": 40,
"out": 8577,
"cr": 781326,
"cw": 54459,
"total": 844402,
"usd": 0.3917,
"wall_s": 65.8,
"api_calls": 5
}
}
},
{
"run": "sp-r1",
"task": "F1",
"cfg": "sp",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 182,
"code_diff_removed": 15,
"diff_added": 183,
"diff_removed": 16,
"tests_added_suite": 10,
"tokens": {
"in": 28,
"out": 14522,
"cr": 553287,
"cw": 45597,
"total": 613434
},
"usd": 0.4383,
"usd_po": 0.0424,
"wall_s": 99.4,
"driver_turns": 3,
"questions_routed_to_po": 2,
"end_reason": "done",
"caps": [],
"skills_invoked": [
"brainstorming"
],
"superpowers_injected": true,
"commits": 1,
"models": [
"claude-sonnet-5-5"
],
"stages": {
"(none)": {
"turns": [
1
],
"in": 2,
"out": 209,
"cr": 13011,
"cw": 6298,
"total": 19520,
"usd": 0.0299,
"wall_s": 5.4,
"api_calls": 1
},
"brainstorming": {
"turns": [
1,
2
],
"in": 14,
"out": 9666,
"cr": 212052,
"cw": 26168,
"total": 247900,
"usd": 0.2438,
"wall_s": 40.1,
"api_calls": 7
},
"implementation (no skill)": {
"turns": [
2,
3
],
"in": 12,
"out": 4647,
"cr": 328224,
"cw": 13131,
"total": 346014,
"usd": 0.1647,
"wall_s": 52.8,
"api_calls": 6
},
"implementation (no skill) [subagent]": {
"turns": [
2
],
"in": 0,
"out": 0,
"cr": 0,
"cw": 0,
"total": 0,
"usd": 0.0,
"wall_s": 0.0,
"api_calls": 0
}
}
},
{
"run": "sp-r2",
"task": "F1",
"cfg": "sp",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 212,
"code_diff_removed": 15,
"diff_added": 216,
"diff_removed": 15,
"tests_added_suite": 13,
"tokens": {
"in": 32,
"out": 18160,
"cr": 691441,
"cw": 55475,
"total": 765108
},
"usd": 0.5419,
"usd_po": 0.0305,
"wall_s": 119.6,
"driver_turns": 2,
"questions_routed_to_po": 1,
"end_reason": "done",
"caps": [],
"skills_invoked": [
"brainstorming",
"test-driven-development"
],
"superpowers_injected": true,
"commits": 0,
"models": [
"claude-sonnet-5-5"
],
"stages": {
"(none)": {
"turns": [
1
],
"in": 2,
"out": 227,
"cr": 13011,
"cw": 6298,
"total": 19538,
"usd": 0.0301,
"wall_s": 5.7,
"api_calls": 1
},
"brainstorming": {
"turns": [
1,
2
],
"in": 10,
"out": 2311,
"cr": 133731,
"cw": 14200,
"total": 150252,
"usd": 0.1067,
"wall_s": 20.4,
"api_calls": 5
},
"test-driven-development": {
"turns": [
2
],
"in": 20,
"out": 15622,
"cr": 544699,
"cw": 34977,
"total": 595318,
"usd": 0.4051,
"wall_s": 93.0,
"api_calls": 10
}
}
},
{
"run": "sp-r3",
"task": "F1",
"cfg": "sp",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 267,
"code_diff_removed": 14,
"diff_added": 267,
"diff_removed": 14,
"tests_added_suite": 14,
"tokens": {
"in": 34,
"out": 17926,
"cr": 727283,
"cw": 50017,
"total": 795260
},
"usd": 0.5249,
"usd_po": 0.0489,
"wall_s": 135.3,
"driver_turns": 3,
"questions_routed_to_po": 2,
"end_reason": "done",
"caps": [],
"skills_invoked": [
"brainstorming"
],
"superpowers_injected": true,
"commits": 1,
"models": [
"claude-sonnet-5-5"
],
"stages": {
"(none)": {
"turns": [
1
],
"in": 2,
"out": 278,
"cr": 13011,
"cw": 6298,
"total": 19589,
"usd": 0.0306,
"wall_s": 4.9,
"api_calls": 1
},
"brainstorming": {
"turns": [
1,
2
],
"in": 18,
"out": 12135,
"cr": 306378,
"cw": 31097,
"total": 349628,
"usd": 0.307,
"wall_s": 61.1,
"api_calls": 9
},
"implementation (no skill)": {
"turns": [
2,
3
],
"in": 14,
"out": 5513,
"cr": 407894,
"cw": 12622,
"total": 426043,
"usd": 0.1872,
"wall_s": 67.8,
"api_calls": 7
},
"implementation (no skill) [subagent]": {
"turns": [
2
],
"in": 0,
"out": 0,
"cr": 0,
"cw": 0,
"total": 0,
"usd": 0.0,
"wall_s": 0.0,
"api_calls": 0
}
}
},
{
"run": "spd-r1",
"task": "F1",
"cfg": "spd",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 350,
"code_diff_removed": 21,
"diff_added": 516,
"diff_removed": 21,
"tests_added_suite": 21,
"tokens": {
"in": 340,
"out": 59243,
"cr": 3428307,
"cw": 179396,
"total": 3667286
},
"usd": 3.4118,
"usd_po": 0.1084,
"wall_s": 600.8,
"driver_turns": 6,
"questions_routed_to_po": 3,
"end_reason": "done",
"caps": [],
"skills_invoked": [
"test-driven-development"
],
"superpowers_injected": true,
"commits": 4,
"models": [
"claude-fable-5-1",
"claude-sonnet-5-5"
],
"stages": {
"brainstorming": {
"turns": [
1,
2,
3
],
"in": 14,
"out": 5191,
"cr": 198912,
"cw": 25310,
"total": 229427,
"usd": 0.193,
"wall_s": 49.8,
"api_calls": 7
},
"brainstorming [subagent]": {
"turns": [
2
],
"in": 0,
"out": 0,
"cr": 0,
"cw": 0,
"total": 0,
"usd": 0.0,
"wall_s": 0.0,
"api_calls": 0
},
"writing-plans": {
"turns": [
4,
5
],
"in": 12,
"out": 13010,
"cr": 243336,
"cw": 23856,
"total": 280214,
"usd": 0.2742,
"wall_s": 95.1,
"api_calls": 5
},
"executing-plans": {
"turns": [
6
],
"in": 4,
"out": 614,
"cr": 62177,
"cw": 7788,
"total": 70583,
"usd": 0.0497,
"wall_s": 4.6,
"api_calls": 1
},
"test-driven-development": {
"turns": [
6
],
"in": 52,
"out": 22476,
"cr": 2455319,
"cw": 43404,
"total": 2521251,
"usd": 0.8895,
"wall_s": 448.9,
"api_calls": 26
},
"test-driven-development [subagent]": {
"turns": [
6
],
"in": 258,
"out": 17952,
"cr": 468563,
"cw": 79038,
"total": 565811,
"usd": 2.0053,
"wall_s": 0.0,
"api_calls": 0
}
}
},
{
"run": "spd-r2",
"task": "F1",
"cfg": "spd",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 317,
"code_diff_removed": 16,
"diff_added": 527,
"diff_removed": 16,
"tests_added_suite": 36,
"tokens": {
"in": 354,
"out": 53270,
"cr": 3666473,
"cw": 166918,
"total": 3887015
},
"usd": 3.3947,
"usd_po": 0.1118,
"wall_s": 520.0,
"driver_turns": 7,
"questions_routed_to_po": 4,
"end_reason": "done",
"caps": [],
"skills_invoked": [
"test-driven-development"
],
"superpowers_injected": true,
"commits": 5,
"models": [
"claude-fable-5-1",
"claude-sonnet-5-5"
],
"stages": {
"brainstorming": {
"turns": [
1,
2,
3
],
"in": 10,
"out": 3724,
"cr": 134220,
"cw": 22880,
"total": 160834,
"usd": 0.1556,
"wall_s": 33.6,
"api_calls": 5
},
"writing-plans": {
"turns": [
4,
5,
6
],
"in": 14,
"out": 8546,
"cr": 271872,
"cw": 17835,
"total": 298267,
"usd": 0.2112,
"wall_s": 63.9,
"api_calls": 6
},
"writing-plans [subagent]": {
"turns": [
6
],
"in": 0,
"out": 0,
"cr": 0,
"cw": 0,
"total": 0,
"usd": 0.0,
"wall_s": 0.0,
"api_calls": 0
},
"executing-plans": {
"turns": [
7
],
"in": 4,
"out": 385,
"cr": 53726,
"cw": 7377,
"total": 61492,
"usd": 0.0441,
"wall_s": 3.7,
"api_calls": 1
},
"test-driven-development": {
"turns": [
7
],
"in": 68,
"out": 21023,
"cr": 2791978,
"cw": 41898,
"total": 2854967,
"usd": 0.9364,
"wall_s": 415.1,
"api_calls": 34
},
"test-driven-development [subagent]": {
"turns": [
7
],
"in": 258,
"out": 19592,
"cr": 414677,
"cw": 76928,
"total": 511455,
"usd": 2.0474,
"wall_s": 0.0,
"api_calls": 0
}
}
},
{
"run": "spd-r3",
"task": "F1",
"cfg": "spd",
"hidden_pass": 5,
"hidden_reqs": {
"test_req1": true,
"test_req2": true,
"test_req3": true,
"test_req4": true,
"test_req5": true
},
"suite_green": true,
"typecheck_green": true,
"code_diff_added": 434,
"code_diff_removed": 20,
"diff_added": 646,
"diff_removed": 20,
"tests_added_suite": 26,
"tokens": {
"in": 504,
"out": 72541,
"cr": 5615970,
"cw": 196313,
"total": 5885328
},
"usd": 4.4087,
"usd_po": 0.1286,
"wall_s": 701.3,
"driver_turns": 8,
"questions_routed_to_po": 5,
"end_reason": "done",
"caps": [],
"skills_invoked": [
"test-driven-development",
"finishing-a-development-branch"
],
"superpowers_injected": true,
"commits": 6,
"models": [
"claude-fable-5-1",
"claude-sonnet-5-5"
],
"stages": {
"brainstorming": {
"turns": [
1,
2,
3
],
"in": 10,
"out": 3567,
"cr": 130512,
"cw": 21715,
"total": 155804,
"usd": 0.1487,
"wall_s": 33.0,
"api_calls": 5
},
"writing-plans": {
"turns": [
4,
5,
6
],
"in": 16,
"out": 13925,
"cr": 345667,
"cw": 25975,
"total": 385583,
"usd": 0.3123,
"wall_s": 101.2,
"api_calls": 7
},
"writing-plans [subagent]": {
"turns": [
5,
6
],
"in": 0,
"out": 0,
"cr": 0,
"cw": 0,
"total": 0,
"usd": 0.0,
"wall_s": 0.0,
"api_calls": 0
},
"executing-plans": {
"turns": [
7
],
"in": 4,
"out": 516,
"cr": 60701,
"cw": 7518,
"total": 68739,
"usd": 0.0474,
"wall_s": 5.5,
"api_calls": 1
},
"test-driven-development": {
"turns": [
7
],
"in": 72,
"out": 24026,
"cr": 3334129,
"cw": 46850,
"total": 3405077,
"usd": 1.0946,
"wall_s": 507.1,
"api_calls": 36
},
"finishing-a-development-branch": {
"turns": [
7,
8
],
"in": 16,
"out": 5297,
"cr": 962018,
"cw": 9174,
"total": 976505,
"usd": 0.2821,
"wall_s": 50.7,
"api_calls": 8
},
"test-driven-development [subagent]": {
"turns": [
7
],
"in": 386,
"out": 25210,
"cr": 782943,
"cw": 85081,
"total": 893620,
"usd": 2.5236,
"wall_s": 0.0,
"api_calls": 0
},
"finishing-a-development-branch [subagent]": {
"turns": [
8
],
"in": 0,
"out": 0,
"cr": 0,
"cw": 0,
"total": 0,
"usd": 0.0,
"wall_s": 0.0,
"api_calls": 0
}
}
}
]
}
results.md
# flow-bench results
Untagged runs only (smoke/validation excluded). Tokens = input + output + cache read + cache write, all models, subagents included. USD = list-price equivalent from `total_cost_usd` deltas (plan quota, not a bill). Wall = sum of worker turn walls (PO time excluded). Diff = lines added+removed vs the base commit, src/ + test/.
## Per config (mean / median)
| task / config | n | hidden /5 (per run) | suite green | tsc green | code diff +/- | all files diff | tests added | tokens | USD | wall s | driver turns | questions | caps hit |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| F1 / bare | 3 | 5 / 5 (5,5,5) | 3/3 | 3/3 | 141.667 / 137 | 142.333 | 4.667 | 519,114 / 361,842 | 0.343 / 0.286 | 84.5 / 71.7 | 1.333 | 0.333 | none |
| F1 / sp (hook-routed) | 3 | 5 / 5 (5,5,5) | 3/3 | 3/3 | 235 / 227 | 237 | 12.333 | 724,601 / 765,108 | 0.502 / 0.525 | 118.1 / 119.6 | 2.667 | 1.667 | none |
| F1 / spd (Superpowers driven) | 3 | 5 / 5 (5,5,5) | 3/3 | 3/3 | 386 / 371 | 582 | 27.667 | 4,479,876 / 3,887,015 | 3.738 / 3.412 | 607.367 / 600.8 | 7 | 4 | none |
| F1 / mp (Main Flow) | 3 | 5 / 5 (5,5,5) | 3/3 | 3/3 | 372 / 369 | 578 | 27.667 | 4,476,104 / 4,286,993 | 2.043 / 2.102 | 439.033 / 472.4 | 12 | 7 | mp-r1: driver_turns_12; mp-r3: driver_turns_12 |
| F1 / both (sp+mp, mp flow) | 1 | 5 / 5 (5) | 1/1 | 1/1 | 374 / 374 | 561 | 22 | 5,754,354 / 5,754,354 | 2.326 / 2.326 | 448.2 / 448.2 | 12 | 6 | none |
| F2 / bare | 2 | 4.5 / 4.5 (4,5) | 2/2 | 2/2 | 206.5 / 206.5 | 209.5 | 5 | 651,322 / 651,322 | 0.413 / 0.413 | 103.15 / 103.15 | 2 | 0.5 | none |
| F2 / spd (Superpowers driven) | 2 | 5 / 5.0 (5,5) | 2/2 | 2/2 | 488.5 / 488.5 | 750.5 | 33 | 4,490,365 / 4,490,365 | 3.213 / 3.213 | 511.7 / 511.7 | 6.5 | 3.5 | none |
| F2 / mp (Main Flow) | 2 | 5 / 5.0 (5,5) | 2/2 | 2/2 | 469 / 469.0 | 658 | 31 | 4,520,638 / 4,520,638 | 2.067 / 2.067 | 421.2 / 421.2 | 12 | 6.5 | f2-mp-r1: driver_turns_12 |
Requirement pass counts (runs passing each hidden req):
- **F1 bare**: req1 3/3, req2 3/3, req3 3/3, req4 3/3, req5 3/3
- **F1 sp (hook-routed)**: req1 3/3, req2 3/3, req3 3/3, req4 3/3, req5 3/3
- **F1 spd (Superpowers driven)**: req1 3/3, req2 3/3, req3 3/3, req4 3/3, req5 3/3
- **F1 mp (Main Flow)**: req1 3/3, req2 3/3, req3 3/3, req4 3/3, req5 3/3
- **F1 both (sp+mp, mp flow)**: req1 1/1, req2 1/1, req3 1/1, req4 1/1, req5 1/1
- **F2 bare**: req1 2/2, req2 2/2, req3 1/2, req4 2/2, req5 2/2
- **F2 spd (Superpowers driven)**: req1 2/2, req2 2/2, req3 2/2, req4 2/2, req5 2/2
- **F2 mp (Main Flow)**: req1 2/2, req2 2/2, req3 2/2, req4 2/2, req5 2/2
## F1 spd (Superpowers driven) per slash-command stage (turn level, exact) (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| /brainstorming | 3 | 182,022 | 160,834 | - | 0.166 | 0.156 | 40.283 | 35.44 |
| /writing-plans | 3 | 321,355 | 298,267 | - | 0.266 | 0.274 | 88.133 | 95.82 |
| /executing-plans | 3 | 3,976,500 | 3,427,914 | - | 3.307 | 3.028 | 478.95 | 453.89 |
## F1 spd per skill in force (within-turn split) (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| brainstorming | 3 | 182,022 | 160,834 | 4,161 | 0.166 | 0.156 | 38.8 | 33.6 |
| writing-plans | 3 | 321,355 | 298,267 | 11,827 | 0.266 | 0.274 | 86.733 | 95.1 |
| executing-plans | 3 | 66,938 | 68,739 | 505 | 0.047 | 0.047 | 4.6 | 4.6 |
| test-driven-development | 3 | 2,927,098 | 2,854,967 | 22,508 | 0.974 | 0.936 | 457.033 | 448.9 |
| finishing-a-development-branch | 1 | 976,505 | 976,505 | 5,297 | 0.282 | 0.282 | 50.7 | 50.7 |
| test-driven-development [subagent] | 3 | 656,962 | 565,811 | 20,918 | 2.192 | 2.047 | 0.0 | 0.0 |
## F1 sp (hook-routed) per phase (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| before any skill | 3 | 19,549 | 19,538 | 238 | 0.03 | 0.03 | 5.333 | 5.4 |
| brainstorming | 3 | 249,260 | 247,900 | 8,037 | 0.219 | 0.244 | 40.533 | 40.1 |
| implementation (no skill) | 2 | 386,028 | 386,028 | 5,080 | 0.176 | 0.176 | 60.3 | 60.3 |
| execution | 1 | 595,318 | 595,318 | 15,622 | 0.405 | 0.405 | 93.0 | 93.0 |
## F1 sp (hook-routed) per skill in force (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| (none) | 3 | 19,549 | 19,538 | 238 | 0.03 | 0.03 | 5.333 | 5.4 |
| brainstorming | 3 | 249,260 | 247,900 | 8,037 | 0.219 | 0.244 | 40.533 | 40.1 |
| test-driven-development | 1 | 595,318 | 595,318 | 15,622 | 0.405 | 0.405 | 93.0 | 93.0 |
| implementation (no skill) | 2 | 386,028 | 386,028 | 5,080 | 0.176 | 0.176 | 60.3 | 60.3 |
## F1 mp per slash-command stage (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| /grill-with-docs | 3 | 316,507 | 348,862 | 5,999 | 0.204 | 0.215 | 59.933 | 59.9 |
| /to-spec | 3 | 163,344 | 167,578 | 5,135 | 0.118 | 0.113 | 42.1 | 40.8 |
| /to-tickets | 3 | 257,955 | 268,062 | 5,867 | 0.143 | 0.151 | 44.4 | 47.8 |
| /implement | 3 | 2,655,565 | 2,613,062 | 30,215 | 1.125 | 1.195 | 224.567 | 245.4 |
| /code-review | 3 | 1,047,586 | 1,107,555 | 9,294 | 0.439 | 0.445 | 64.2 | 65.8 |
| /pr | 1 | 105,440 | 105,440 | 1,349 | 0.041 | 0.041 | 11.6 | 11.6 |
## F1 both per slash-command stage (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| /grill-with-docs | 1 | 344,391 | 344,391 | 6,193 | 0.225 | 0.225 | 61.8 | 61.8 |
| /to-spec | 1 | 217,572 | 217,572 | 4,835 | 0.123 | 0.123 | 42.9 | 42.9 |
| /to-tickets | 1 | 382,953 | 382,953 | 7,115 | 0.196 | 0.196 | 49.4 | 49.4 |
| /implement | 1 | 3,070,579 | 3,070,579 | 24,618 | 1.112 | 1.112 | 194.2 | 194.2 |
| /code-review | 1 | 1,615,265 | 1,615,265 | 13,410 | 0.625 | 0.625 | 86.8 | 86.8 |
| /pr | 1 | 123,594 | 123,594 | 1,461 | 0.045 | 0.045 | 13.1 | 13.1 |
## F2 spd (Superpowers driven) per slash-command stage (turn level, exact) (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| /brainstorming | 2 | 485,364 | 485,364 | - | 0.354 | 0.354 | 96.815 | 96.815 |
| /writing-plans | 2 | 166,012 | 166,012 | - | 0.159 | 0.159 | 49.755 | 49.755 |
| /executing-plans | 2 | 3,838,988 | 3,838,988 | - | 2.7 | 2.7 | 365.1 | 365.1 |
## F2 spd per skill in force (within-turn split) (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| brainstorming | 2 | 253,890 | 253,890 | 6,038 | 0.207 | 0.207 | 53.05 | 53.05 |
| writing-plans | 2 | 397,486 | 397,486 | 14,060 | 0.307 | 0.307 | 90.75 | 90.75 |
| executing-plans | 2 | 69,973 | 69,973 | 286 | 0.046 | 0.046 | 3.5 | 3.5 |
| test-driven-development | 2 | 3,304,676 | 3,304,676 | 23,244 | 1.052 | 1.052 | 361.25 | 361.25 |
| test-driven-development [subagent] | 2 | 464,339 | 464,339 | 13,017 | 1.602 | 1.602 | 0.0 | 0.0 |
## F2 mp per slash-command stage (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| /grill-with-docs | 2 | 232,965 | 232,965 | 5,946 | 0.182 | 0.182 | 56.7 | 56.7 |
| /to-spec | 2 | 180,084 | 180,084 | 4,452 | 0.115 | 0.115 | 37.6 | 37.6 |
| /to-tickets | 2 | 253,688 | 253,688 | 5,152 | 0.131 | 0.131 | 38.5 | 38.5 |
| /implement | 2 | 2,689,071 | 2,689,071 | 31,452 | 1.159 | 1.159 | 218.35 | 218.35 |
| /code-review | 2 | 1,107,122 | 1,107,122 | 9,454 | 0.458 | 0.458 | 63.85 | 63.85 |
| /pr | 1 | 115,414 | 115,414 | 1,372 | 0.043 | 0.043 | 12.4 | 12.4 |
## Per run
| run | task | hidden | reqs failed | suite | tsc | code +/- | tests added | tokens | USD | wall s | turns | end | skills (first use order) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bare-r1 | F1 | 5/5 | - | green | ok | +149/-11 | 6 | 840,751 | 0.47 | 119 | 2 | done | - |
| bare-r2 | F1 | 5/5 | - | green | ok | +118/-10 | 4 | 354,748 | 0.27 | 63 | 1 | done | - |
| bare-r3 | F1 | 5/5 | - | green | ok | +125/-12 | 4 | 361,842 | 0.29 | 72 | 1 | done | - |
| sp-r1 | F1 | 5/5 | - | green | ok | +182/-15 | 10 | 613,434 | 0.44 | 99 | 3 | done | brainstorming |
| sp-r2 | F1 | 5/5 | - | green | ok | +212/-15 | 13 | 765,108 | 0.54 | 120 | 2 | done | brainstorming > test-driven-development |
| sp-r3 | F1 | 5/5 | - | green | ok | +267/-14 | 14 | 795,260 | 0.52 | 135 | 3 | done | brainstorming |
| spd-r1 | F1 | 5/5 | - | green | ok | +350/-21 | 21 | 3,667,286 | 3.41 | 601 | 6 | done | test-driven-development |
| spd-r2 | F1 | 5/5 | - | green | ok | +317/-16 | 36 | 3,887,015 | 3.39 | 520 | 7 | done | test-driven-development |
| spd-r3 | F1 | 5/5 | - | green | ok | +434/-20 | 26 | 5,885,328 | 4.41 | 701 | 8 | done | test-driven-development > finishing-a-development-branch |
| mp-r1 | F1 | 5/5 | - | green | ok | +349/-20 | 29 | 5,098,503 | 2.22 | 472 | 12 | cap_driver_turns (driver_turns_12) | grilling > domain-modeling > tdd > code-review |
| mp-r2 | F1 | 5/5 | - | green | ok | +314/-15 | 25 | 4,042,815 | 1.80 | 365 | 12 | done | grilling > domain-modeling > tdd > code-review |
| mp-r3 | F1 | 5/5 | - | green | ok | +395/-23 | 29 | 4,286,993 | 2.10 | 480 | 12 | cap_driver_turns (driver_turns_12) | grilling > domain-modeling > tdd > code-review |
| both-r1 | F1 | 5/5 | - | green | ok | +356/-18 | 22 | 5,754,354 | 2.33 | 448 | 12 | done | grilling > domain-modeling > tdd > code-review |
| f2-bare-r1 | F2 | 4/5 | req3 | green | ok | +204/-10 | 5 | 494,231 | 0.38 | 92 | 1 | done | - |
| f2-bare-r2 | F2 | 5/5 | - | green | ok | +189/-10 | 5 | 808,413 | 0.45 | 115 | 3 | done | - |
| f2-spd-r1 | F2 | 5/5 | - | green | ok | +546/-14 | 46 | 5,310,616 | 3.42 | 506 | 6 | done | writing-plans > test-driven-development |
| f2-spd-r2 | F2 | 5/5 | - | green | ok | +401/-16 | 20 | 3,670,114 | 3.00 | 517 | 7 | done | test-driven-development |
| f2-mp-r1 | F2 | 5/5 | - | green | ok | +460/-23 | 31 | 4,823,516 | 2.18 | 443 | 12 | cap_driver_turns (driver_turns_12) | grilling > domain-modeling > tdd > code-review |
| f2-mp-r2 | F2 | 5/5 | - | green | ok | +440/-15 | 31 | 4,217,761 | 1.95 | 399 | 12 | done | grilling > domain-modeling > tdd > code-review |
matrix.sh
#!/usr/bin/env bash
# Run matrix.txt ("<cfg> <rep>" lines), MATRIX_JOBS (default 2) at a time, each run in its own work copy.
# Appends to matrix.log. Resumable: SKIP_DONE=1. Usage: nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 &
set -uo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"; cd "$HERE"
LIST="${1:-matrix.txt}"; JOBS="${MATRIX_JOBS:-2}"
mkdir "$HERE/.matrix.lock" 2>/dev/null || { echo "another matrix.sh is running (remove .matrix.lock if stale)" >&2; exit 1; }
trap 'rmdir "$HERE/.matrix.lock"' EXIT
echo "=== $(date '+%F %T') start $LIST jobs=$JOBS pid=$$" >> matrix.log
grep -vE '^[[:space:]]*(#|$)' "$LIST" | xargs -P "$JOBS" -L 1 ./matrix-one.sh
echo "=== $(date '+%F %T') done $LIST" >> matrix.log
matrix-one.sh
#!/usr/bin/env bash
# One matrix cell (called by matrix.sh through xargs). Args: <cfg> <rep>
HERE="$(cd "$(dirname "$0")" && pwd)"; cd "$HERE"
CFG="$1"; REP="$2"; export TASK="${3:-F1}"
PFX=""; [ "$TASK" != "F1" ] && PFX="$(echo "$TASK" | tr "A-Z" "a-z")-"
if [ "${SKIP_DONE:-0}" = 1 ] && [ -f "runs/${PFX}${CFG}-r${REP}/score.json" ]; then
echo "$(date '+%F %T') skip ${PFX}${CFG}-r${REP}" >> matrix.log; exit 0
fi
echo "$(date '+%F %T') start ${PFX}${CFG}-r${REP}" >> matrix.log
line=$(./run.sh "$CFG" "$REP" < /dev/null 2>&1 | tail -1)
echo "$(date '+%F %T') $line" >> matrix.log
echo "$line"
matrix.txt
# <cfg> <rep>, run 2 at a time in this order (rep 1 of every config first)
bare 1
sp 1
mp 1
both 1
sp 2
mp 2
bare 2
mp 3
sp 3
bare 3
matrix2.txt
# <cfg> <rep> <task>; 2 lanes; long runs interleaved with short ones
spd 1 F1
mp 1 F1
bare 1 F1
sp 1 F1
spd 1 F2
mp 1 F2
bare 1 F2
spd 2 F1
mp 2 F1
bare 2 F1
sp 2 F1
spd 2 F2
mp 2 F2
bare 2 F2
both 1 F1
spd 3 F1
mp 3 F1
bare 3 F1
sp 3 F1
matrix-mp2.txt
mp 1
both 1
mp 2
mp 3
validate.sh
#!/usr/bin/env bash
# The 5 validation steps of SPEC.md, appended to validation.log.
# Usage: ./validate.sh [steps] default "1 2 3 4 5"; steps 3-5 (smoke runs, RUN_TAG=smoke) run in parallel.
set -uo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"; cd "$HERE"
. "$HERE/env.sh"
STEPS="${*:-1 2 3 4 5}"
LOG="$HERE/validation.log"
log(){ echo "$(date '+%F %T') $*" | tee -a "$LOG"; }
static_run(){ # <name> <patch or ""> [task] : prep a bare copy, optionally apply a patch, score it with finalize.py
local name="$1" patch="$2" task="${3:-F1}" out="$HERE/runs/$1" w="$HERE/work/$1"
rm -rf "$out"; mkdir -p "$out"
local base; base=$("$HERE/prep.sh" bare "$w" | tail -1)
if [ -n "$patch" ]; then ( cd "$w" && git apply "$patch" ) || { log "FAIL $name: git apply"; return 1; }; fi
printf '{"task":"%s","cfg":"bare","rep":0,"tag":"validation","work":"%s","name":"%s","base":"%s"}\n' "$task" "$w" "$name" "$base" > "$out/meta.json"
: > "$out/turns.jsonl"; echo '{"end_reason":"static"}' > "$out/driver.json"
python3 "$HERE/lib/finalize.py" "$out"
}
check(){ # <run dir> <python expr over s (score.json)> <label>
python3 - "$1" "$2" "$3" <<'PY'
import json, sys
s = json.load(open(sys.argv[1] + "/score.json"))
ok = eval(sys.argv[2], {"s": s})
print(("PASS " if ok else "FAIL ") + sys.argv[3])
sys.exit(0 if ok else 1)
PY
}
log "=== validate.sh steps: $STEPS (claude $(claude --version | head -1))"
for st in $STEPS; do case $st in
1) log "step 1: reference/F1.patch -> hidden 5/5, suite green, typecheck green"
static_run val1-reference "$HERE/reference/F1.patch" | tee -a "$LOG"
check runs/val1-reference 's["hidden_pass"]==5 and s["suite_green"] and s["typecheck_green"]' "step 1" | tee -a "$LOG" ;;
2) log "step 2: upstream unchanged -> hidden 0/5 (honest number recorded), suite green"
static_run val2-upstream "" | tee -a "$LOG"
check runs/val2-upstream 's["hidden_pass"]==0 and s["suite_green"] and s["suite_total"]==34' "step 2" | tee -a "$LOG" ;;
f2-1) log "F2 step 1: reference/F2.patch -> hidden 5/5, suite green, typecheck green"
static_run val1-f2-reference "$HERE/reference/F2.patch" F2 | tee -a "$LOG"
check runs/val1-f2-reference 's["hidden_pass"]==5 and s["suite_green"] and s["typecheck_green"]' "F2 step 1" | tee -a "$LOG" ;;
f2-2) log "F2 step 2: upstream unchanged -> hidden 0/5, suite green"
static_run val2-f2-upstream "" F2 | tee -a "$LOG"
check runs/val2-f2-upstream 's["hidden_pass"]==0 and s["suite_green"]' "F2 step 2" | tee -a "$LOG" ;;
esac; done
pids=""
for st in $STEPS; do case $st in
3) log "step 3: bare smoke (RUN_TAG=smoke) started"; ( RUN_TAG=smoke ./run.sh bare 1 > runs/.smoke-bare.out 2>&1 ) & pids="$pids $!" ;;
4) log "step 4: sp smoke (RUN_TAG=smoke) started"; ( RUN_TAG=smoke ./run.sh sp 1 > runs/.smoke-sp.out 2>&1 ) & pids="$pids $!" ;;
5) log "step 5: mp smoke (RUN_TAG=smoke) started"; ( RUN_TAG=smoke ./run.sh mp 1 > runs/.smoke-mp.out 2>&1 ) & pids="$pids $!" ;;
f2-3) log "F2 step 3: bare smoke on F2 (RUN_TAG=smoke): bare must score < 5"; ( TASK=F2 RUN_TAG=smoke ./run.sh bare 1 > runs/.smoke-f2-bare.out 2>&1 ) & pids="$pids $!" ;;
spd) log "spd smoke on F1 (RUN_TAG=smoke)"; ( RUN_TAG=smoke ./run.sh spd 1 > runs/.smoke-spd.out 2>&1 ) & pids="$pids $!" ;;
esac; done
[ -n "$pids" ] && wait $pids
for st in $STEPS; do case $st in
3) tail -1 runs/.smoke-bare.out | tee -a "$LOG"
check runs/bare-r1-smoke 's["end_reason"] in ("done","cap_driver_turns") and len(s["per_turn"])>=1 and all(t["po_kind"] for t in s["per_turn"])' "step 3a: bare ran end to end, the PO classified every turn" | tee -a "$LOG"
check runs/bare-r1-smoke 'not s["superpowers_injected"] and not s["skill_events"] and "superpowers" not in (s["init_plugins"] or [])' "step 3b: isolation: no Superpowers hook output, no Skill events, user plugin not loaded" | tee -a "$LOG" ;;
4) tail -1 runs/.smoke-sp.out | tee -a "$LOG"
check runs/sp-r1-smoke 's["superpowers_injected"] and any(h["event"]=="SessionStart" for h in s["hook_events"])' "step 4a: SessionStart hook fired and injected using-superpowers" | tee -a "$LOG"
check runs/sp-r1-smoke '"brainstorming" in [x.split(":")[-1] for x in s["skills_invoked"]]' "step 4b: brainstorming invoked (Skill event)" | tee -a "$LOG" ;;
5) tail -1 runs/.smoke-mp.out | tee -a "$LOG"
check runs/mp-r1-smoke 's["per_turn"][0]["stage"]=="/grill-with-docs" and {"grilling","domain-modeling"} <= set(s["per_turn"][0]["skills"])' "step 5: /grill-with-docs resolved (turn 1 loads grilling + domain-modeling, as its SKILL.md instructs)" | tee -a "$LOG" ;;
f2-3) tail -1 runs/.smoke-f2-bare.out | tee -a "$LOG"
check runs/f2-bare-r1-smoke 's["hidden_pass"] < 5' "F2 step 3: bare scores < 5 on F2" | tee -a "$LOG" ;;
spd) tail -1 runs/.smoke-spd.out | tee -a "$LOG"
check runs/spd-r1-smoke '{"/brainstorming","/writing-plans","/executing-plans"} <= set(s["spd_slash_stage_per_turn"] or [])' "spd: the three slash stages were all reached" | tee -a "$LOG" ;;
esac; done
log "=== validate.sh done"
내 기능으로 두 pack을 돌려 보려면 README를 따르세요.
upstream/자리에 내 repo를 넣고,tasks/에 요청을 쓰고,po/에 프로덕트 오너의 브리프를 테스트가 아닌 쉬운 말로 씁니다.- 시니어가 기대할 요구사항마다 테스트 하나씩 든 숨겨진 테스트 파일과 기준 패치를 쓰고,
./validate.sh 1 2로 oracle을 확인합니다. prep.sh가 보여 주는 방식대로 pack을 설치합니다. Superpowers는.claude/skills에 넣고 SessionStart hook을.claude/settings.json에 복사하고, Pocock의 pack은npx skills add로 설치한 뒤/setup-matt-pocock-skills를 한 번 실체화합니다.RUN_TAG=x ./run.sh <cfg> <rep>로 칸 하나를 돌립니다. 태그가 붙은 실행은 분석에서 무시되므로 smoke run에 태그를 쓰세요.nohup ./matrix.sh matrix.txt > matrix.nohup 2>&1 &로 matrix를 돌리고,SKIP_DONE=1로 이어 하고,python3 analyze.py를 돌려 구성별, stage별 표를 얻습니다.
시작하기 전에 알아 둘 README의 함정 세 가지. 작업 복사본은 CLAUDE.md가 있는 디렉터리 트리 바깥에서 돌리세요. Claude Code는 모든 상위 디렉터리의 CLAUDE.md를 읽어서, 우리 첫 라운드는 그 때문에 버려졌습니다. 구독 로그인에서는 --bare를 쓰지 마세요. 그리고 Superpowers hook은 첫 turn에만 작동하므로 headless 실행은 한 번만 라우팅하고, 라우팅 자체가 실행마다 달랐으니 어느 skill을 골랐는지 세어 보세요.
월요일에 할 일
- 내가 가진 Pocock 버전을 확인하세요. Claude plugin으로 설치했다면 1.2.3에 고정되어 있고 implement-spec, pr, retro가 없습니다. v1.3.1을 원하면 설치 프로그램으로 다시 설치하세요.
- Superpowers를 쓴다면 executing-plans 리뷰어 지시문을 열고 어떤 모델을 요청하는지 보세요. 더 싼 모델로 고정한 뒤 측정하세요.
- 요청이 모호할 때는 /brainstorming을 직접 입력하세요. hook에 맡겼을 때 여기서는 hook 라우팅 실행 3번 중 3번 brainstorming이 돌았지만, 라우팅이 보장되는 것은 아닙니다.
- 다음 모호한 요청에 grill-with-docs, to-spec, to-tickets를 써 보고, 내 repo에서 얼마가 드는지 읽어 보세요.
- 테스트 없이 두 pack을 라우터로 겹쳐 쓰지 마세요. 우리가 겹쳐 돌린 한 번의 실행에서는 아무것도 깨지지 않았지만, 한 번일 뿐입니다.
- 지난달에 출시한 기능에 벤치를 복사하고, 내 숨겨진 테스트로 bare, Superpowers, Main Flow를 한 번씩 돌려 보세요.
- 합계가 아니라 stage별 표를 읽으세요. 다른 걸 바꾸기 전에 청구서 대부분을 차지하는 stage 하나를 찾으세요.
- v1.3을 도입한다면 CONTEXT.md를 GLOSSARY.md로 이름을 바꾸세요. skill은 새 이름만 찾습니다.
측정값
출처:
results.md
# flow-bench results
Untagged runs only (smoke/validation excluded). Tokens = input + output + cache read + cache write, all models, subagents included. USD = list-price equivalent from `total_cost_usd` deltas (plan quota, not a bill). Wall = sum of worker turn walls (PO time excluded). Diff = lines added+removed vs the base commit, src/ + test/.
## Per config (mean / median)
| task / config | n | hidden /5 (per run) | suite green | tsc green | code diff +/- | all files diff | tests added | tokens | USD | wall s | driver turns | questions | caps hit |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| F1 / bare | 3 | 5 / 5 (5,5,5) | 3/3 | 3/3 | 141.667 / 137 | 142.333 | 4.667 | 519,114 / 361,842 | 0.343 / 0.286 | 84.5 / 71.7 | 1.333 | 0.333 | none |
| F1 / sp (hook-routed) | 3 | 5 / 5 (5,5,5) | 3/3 | 3/3 | 235 / 227 | 237 | 12.333 | 724,601 / 765,108 | 0.502 / 0.525 | 118.1 / 119.6 | 2.667 | 1.667 | none |
| F1 / spd (Superpowers driven) | 3 | 5 / 5 (5,5,5) | 3/3 | 3/3 | 386 / 371 | 582 | 27.667 | 4,479,876 / 3,887,015 | 3.738 / 3.412 | 607.367 / 600.8 | 7 | 4 | none |
| F1 / mp (Main Flow) | 3 | 5 / 5 (5,5,5) | 3/3 | 3/3 | 372 / 369 | 578 | 27.667 | 4,476,104 / 4,286,993 | 2.043 / 2.102 | 439.033 / 472.4 | 12 | 7 | mp-r1: driver_turns_12; mp-r3: driver_turns_12 |
| F1 / both (sp+mp, mp flow) | 1 | 5 / 5 (5) | 1/1 | 1/1 | 374 / 374 | 561 | 22 | 5,754,354 / 5,754,354 | 2.326 / 2.326 | 448.2 / 448.2 | 12 | 6 | none |
| F2 / bare | 2 | 4.5 / 4.5 (4,5) | 2/2 | 2/2 | 206.5 / 206.5 | 209.5 | 5 | 651,322 / 651,322 | 0.413 / 0.413 | 103.15 / 103.15 | 2 | 0.5 | none |
| F2 / spd (Superpowers driven) | 2 | 5 / 5.0 (5,5) | 2/2 | 2/2 | 488.5 / 488.5 | 750.5 | 33 | 4,490,365 / 4,490,365 | 3.213 / 3.213 | 511.7 / 511.7 | 6.5 | 3.5 | none |
| F2 / mp (Main Flow) | 2 | 5 / 5.0 (5,5) | 2/2 | 2/2 | 469 / 469.0 | 658 | 31 | 4,520,638 / 4,520,638 | 2.067 / 2.067 | 421.2 / 421.2 | 12 | 6.5 | f2-mp-r1: driver_turns_12 |
Requirement pass counts (runs passing each hidden req):
- **F1 bare**: req1 3/3, req2 3/3, req3 3/3, req4 3/3, req5 3/3
- **F1 sp (hook-routed)**: req1 3/3, req2 3/3, req3 3/3, req4 3/3, req5 3/3
- **F1 spd (Superpowers driven)**: req1 3/3, req2 3/3, req3 3/3, req4 3/3, req5 3/3
- **F1 mp (Main Flow)**: req1 3/3, req2 3/3, req3 3/3, req4 3/3, req5 3/3
- **F1 both (sp+mp, mp flow)**: req1 1/1, req2 1/1, req3 1/1, req4 1/1, req5 1/1
- **F2 bare**: req1 2/2, req2 2/2, req3 1/2, req4 2/2, req5 2/2
- **F2 spd (Superpowers driven)**: req1 2/2, req2 2/2, req3 2/2, req4 2/2, req5 2/2
- **F2 mp (Main Flow)**: req1 2/2, req2 2/2, req3 2/2, req4 2/2, req5 2/2
## F1 spd (Superpowers driven) per slash-command stage (turn level, exact) (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| /brainstorming | 3 | 182,022 | 160,834 | - | 0.166 | 0.156 | 40.283 | 35.44 |
| /writing-plans | 3 | 321,355 | 298,267 | - | 0.266 | 0.274 | 88.133 | 95.82 |
| /executing-plans | 3 | 3,976,500 | 3,427,914 | - | 3.307 | 3.028 | 478.95 | 453.89 |
## F1 spd per skill in force (within-turn split) (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| brainstorming | 3 | 182,022 | 160,834 | 4,161 | 0.166 | 0.156 | 38.8 | 33.6 |
| writing-plans | 3 | 321,355 | 298,267 | 11,827 | 0.266 | 0.274 | 86.733 | 95.1 |
| executing-plans | 3 | 66,938 | 68,739 | 505 | 0.047 | 0.047 | 4.6 | 4.6 |
| test-driven-development | 3 | 2,927,098 | 2,854,967 | 22,508 | 0.974 | 0.936 | 457.033 | 448.9 |
| finishing-a-development-branch | 1 | 976,505 | 976,505 | 5,297 | 0.282 | 0.282 | 50.7 | 50.7 |
| test-driven-development [subagent] | 3 | 656,962 | 565,811 | 20,918 | 2.192 | 2.047 | 0.0 | 0.0 |
## F1 sp (hook-routed) per phase (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| before any skill | 3 | 19,549 | 19,538 | 238 | 0.03 | 0.03 | 5.333 | 5.4 |
| brainstorming | 3 | 249,260 | 247,900 | 8,037 | 0.219 | 0.244 | 40.533 | 40.1 |
| implementation (no skill) | 2 | 386,028 | 386,028 | 5,080 | 0.176 | 0.176 | 60.3 | 60.3 |
| execution | 1 | 595,318 | 595,318 | 15,622 | 0.405 | 0.405 | 93.0 | 93.0 |
## F1 sp (hook-routed) per skill in force (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| (none) | 3 | 19,549 | 19,538 | 238 | 0.03 | 0.03 | 5.333 | 5.4 |
| brainstorming | 3 | 249,260 | 247,900 | 8,037 | 0.219 | 0.244 | 40.533 | 40.1 |
| test-driven-development | 1 | 595,318 | 595,318 | 15,622 | 0.405 | 0.405 | 93.0 | 93.0 |
| implementation (no skill) | 2 | 386,028 | 386,028 | 5,080 | 0.176 | 0.176 | 60.3 | 60.3 |
## F1 mp per slash-command stage (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| /grill-with-docs | 3 | 316,507 | 348,862 | 5,999 | 0.204 | 0.215 | 59.933 | 59.9 |
| /to-spec | 3 | 163,344 | 167,578 | 5,135 | 0.118 | 0.113 | 42.1 | 40.8 |
| /to-tickets | 3 | 257,955 | 268,062 | 5,867 | 0.143 | 0.151 | 44.4 | 47.8 |
| /implement | 3 | 2,655,565 | 2,613,062 | 30,215 | 1.125 | 1.195 | 224.567 | 245.4 |
| /code-review | 3 | 1,047,586 | 1,107,555 | 9,294 | 0.439 | 0.445 | 64.2 | 65.8 |
| /pr | 1 | 105,440 | 105,440 | 1,349 | 0.041 | 0.041 | 11.6 | 11.6 |
## F1 both per slash-command stage (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| /grill-with-docs | 1 | 344,391 | 344,391 | 6,193 | 0.225 | 0.225 | 61.8 | 61.8 |
| /to-spec | 1 | 217,572 | 217,572 | 4,835 | 0.123 | 0.123 | 42.9 | 42.9 |
| /to-tickets | 1 | 382,953 | 382,953 | 7,115 | 0.196 | 0.196 | 49.4 | 49.4 |
| /implement | 1 | 3,070,579 | 3,070,579 | 24,618 | 1.112 | 1.112 | 194.2 | 194.2 |
| /code-review | 1 | 1,615,265 | 1,615,265 | 13,410 | 0.625 | 0.625 | 86.8 | 86.8 |
| /pr | 1 | 123,594 | 123,594 | 1,461 | 0.045 | 0.045 | 13.1 | 13.1 |
## F2 spd (Superpowers driven) per slash-command stage (turn level, exact) (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| /brainstorming | 2 | 485,364 | 485,364 | - | 0.354 | 0.354 | 96.815 | 96.815 |
| /writing-plans | 2 | 166,012 | 166,012 | - | 0.159 | 0.159 | 49.755 | 49.755 |
| /executing-plans | 2 | 3,838,988 | 3,838,988 | - | 2.7 | 2.7 | 365.1 | 365.1 |
## F2 spd per skill in force (within-turn split) (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| brainstorming | 2 | 253,890 | 253,890 | 6,038 | 0.207 | 0.207 | 53.05 | 53.05 |
| writing-plans | 2 | 397,486 | 397,486 | 14,060 | 0.307 | 0.307 | 90.75 | 90.75 |
| executing-plans | 2 | 69,973 | 69,973 | 286 | 0.046 | 0.046 | 3.5 | 3.5 |
| test-driven-development | 2 | 3,304,676 | 3,304,676 | 23,244 | 1.052 | 1.052 | 361.25 | 361.25 |
| test-driven-development [subagent] | 2 | 464,339 | 464,339 | 13,017 | 1.602 | 1.602 | 0.0 | 0.0 |
## F2 mp per slash-command stage (mean over the runs that had the stage)
| stage | runs | tokens mean | tokens median | out tokens | USD mean | USD median | wall s mean | wall s median |
|---|---|---|---|---|---|---|---|---|
| /grill-with-docs | 2 | 232,965 | 232,965 | 5,946 | 0.182 | 0.182 | 56.7 | 56.7 |
| /to-spec | 2 | 180,084 | 180,084 | 4,452 | 0.115 | 0.115 | 37.6 | 37.6 |
| /to-tickets | 2 | 253,688 | 253,688 | 5,152 | 0.131 | 0.131 | 38.5 | 38.5 |
| /implement | 2 | 2,689,071 | 2,689,071 | 31,452 | 1.159 | 1.159 | 218.35 | 218.35 |
| /code-review | 2 | 1,107,122 | 1,107,122 | 9,454 | 0.458 | 0.458 | 63.85 | 63.85 |
| /pr | 1 | 115,414 | 115,414 | 1,372 | 0.043 | 0.043 | 12.4 | 12.4 |
## Per run
| run | task | hidden | reqs failed | suite | tsc | code +/- | tests added | tokens | USD | wall s | turns | end | skills (first use order) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bare-r1 | F1 | 5/5 | - | green | ok | +149/-11 | 6 | 840,751 | 0.47 | 119 | 2 | done | - |
| bare-r2 | F1 | 5/5 | - | green | ok | +118/-10 | 4 | 354,748 | 0.27 | 63 | 1 | done | - |
| bare-r3 | F1 | 5/5 | - | green | ok | +125/-12 | 4 | 361,842 | 0.29 | 72 | 1 | done | - |
| sp-r1 | F1 | 5/5 | - | green | ok | +182/-15 | 10 | 613,434 | 0.44 | 99 | 3 | done | brainstorming |
| sp-r2 | F1 | 5/5 | - | green | ok | +212/-15 | 13 | 765,108 | 0.54 | 120 | 2 | done | brainstorming > test-driven-development |
| sp-r3 | F1 | 5/5 | - | green | ok | +267/-14 | 14 | 795,260 | 0.52 | 135 | 3 | done | brainstorming |
| spd-r1 | F1 | 5/5 | - | green | ok | +350/-21 | 21 | 3,667,286 | 3.41 | 601 | 6 | done | test-driven-development |
| spd-r2 | F1 | 5/5 | - | green | ok | +317/-16 | 36 | 3,887,015 | 3.39 | 520 | 7 | done | test-driven-development |
| spd-r3 | F1 | 5/5 | - | green | ok | +434/-20 | 26 | 5,885,328 | 4.41 | 701 | 8 | done | test-driven-development > finishing-a-development-branch |
| mp-r1 | F1 | 5/5 | - | green | ok | +349/-20 | 29 | 5,098,503 | 2.22 | 472 | 12 | cap_driver_turns (driver_turns_12) | grilling > domain-modeling > tdd > code-review |
| mp-r2 | F1 | 5/5 | - | green | ok | +314/-15 | 25 | 4,042,815 | 1.80 | 365 | 12 | done | grilling > domain-modeling > tdd > code-review |
| mp-r3 | F1 | 5/5 | - | green | ok | +395/-23 | 29 | 4,286,993 | 2.10 | 480 | 12 | cap_driver_turns (driver_turns_12) | grilling > domain-modeling > tdd > code-review |
| both-r1 | F1 | 5/5 | - | green | ok | +356/-18 | 22 | 5,754,354 | 2.33 | 448 | 12 | done | grilling > domain-modeling > tdd > code-review |
| f2-bare-r1 | F2 | 4/5 | req3 | green | ok | +204/-10 | 5 | 494,231 | 0.38 | 92 | 1 | done | - |
| f2-bare-r2 | F2 | 5/5 | - | green | ok | +189/-10 | 5 | 808,413 | 0.45 | 115 | 3 | done | - |
| f2-spd-r1 | F2 | 5/5 | - | green | ok | +546/-14 | 46 | 5,310,616 | 3.42 | 506 | 6 | done | writing-plans > test-driven-development |
| f2-spd-r2 | F2 | 5/5 | - | green | ok | +401/-16 | 20 | 3,670,114 | 3.00 | 517 | 7 | done | test-driven-development |
| f2-mp-r1 | F2 | 5/5 | - | green | ok | +460/-23 | 31 | 4,823,516 | 2.18 | 443 | 12 | cap_driver_turns (driver_turns_12) | grilling > domain-modeling > tdd > code-review |
| f2-mp-r2 | F2 | 5/5 | - | green | ok | +440/-15 | 31 | 4,217,761 | 1.95 | 399 | 12 | done | grilling > domain-modeling > tdd > code-review |
| 과제와 구성 | 실행 수 | 숨겨진 /5 (실행별) | 평균 USD | 평균 wall s | 평균 token | 평균 코드 diff +/- |
|---|---|---|---|---|---|---|
| F1 bare | 3 | 5 (5,5,5) | 0.343 | 84.5 | 519,114 | 141.667 |
| F1 sp (hook 라우팅) | 3 | 5 (5,5,5) | 0.502 | 118.1 | 724,601 | 235 |
| F1 spd (Superpowers 직접 구동) | 3 | 5 (5,5,5) | 3.738 | 607.367 | 4,479,876 | 386 |
| F1 mp (Main Flow) | 3 | 5 (5,5,5) | 2.043 | 439.033 | 4,476,104 | 372 |
| F1 both (sp+mp, mp flow) | 1 | 5 (5) | 2.326 | 448.2 | 5,754,354 | 374 |
| F2 bare | 2 | 4.5 (4,5) | 0.413 | 103.15 | 651,322 | 206.5 |
| F2 spd (Superpowers 직접 구동) | 2 | 5 (5,5) | 3.213 | 511.7 | 4,490,365 | 488.5 |
| F2 mp (Main Flow) | 2 | 5 (5,5) | 2.067 | 421.2 | 4,520,638 | 469 |
Superpowers 직접 구동, F1, stage별 (3회 평균):
| Stage | Token | USD | Wall s |
|---|---|---|---|
| brainstorming | 182,022 | 0.166 | 40.283 |
| writing-plans | 321,355 | 0.266 | 88.133 |
| executing-plans | 3,976,500 | 3.307 | 478.95 |
| 그중 리뷰어 subagent | 656,962 | 2.192 | n/a |
리뷰어 행은 executing-plans 안의 subagent 잔여분이며, 결과 파일에는 "test-driven-development [subagent]"로 표시됩니다. executing-plans 안의 세션 내 빌드(test-driven-development)는 평균 $0.974였습니다.
Main Flow, F1, stage별 (해당 stage가 있던 실행의 평균, /pr은 3번 중 1번 실행):
| Stage | Token | USD | Wall s |
|---|---|---|---|
| grill-with-docs | 316,507 | 0.204 | 59.933 |
| to-spec | 163,344 | 0.118 | 42.1 |
| to-tickets | 257,955 | 0.143 | 44.4 |
| implement | 2,655,565 | 1.125 | 224.567 |
| code-review | 1,047,586 | 0.439 | 64.2 |
| pr | 105,440 | 0.041 | 11.6 |
hook 라우팅 Superpowers, F1: 어떤 skill도 돌기 전 $0.03, brainstorming $0.219, skill 없는 구현 $0.176(2회), test-driven-development로 한 실행 $0.405(1회). Brainstorming은 3번 모두, writing-plans는 한 번도 돌지 않았습니다. 직접 돌린 루프는 F1 평균 기준으로 hook 라우팅의 7.4배 들었습니다($0.502 대비 $3.738). Main Flow 실행 세 번(mp-r1, mp-r3, f2-mp-r1)은 /pr 전에 driver turn 12번 상한에 닿았고 그렇게 기록했습니다.
FINDINGS.md
# flow-bench findings (2026-10-09, round 2: isolated work copies, Sonnet 5.5 headless, 19 counted runs)
**F1 (partial payments): quality did not separate.** All 13 F1 runs hidden 5/5, suite and tsc green.
Cost per run: bare $0.27 (bare-r2) to $0.47 (bare-r1); sp, hook-routed, $0.44 (sp-r1) to $0.54 (sp-r2); mp $1.80
(mp-r2) to $2.22 (mp-r1); spd, Superpowers driven, $3.39 (spd-r2) to $4.41 (spd-r3).
Wall: mp 365 to 480 s (mp-r2, mp-r3), spd 520 to 701 s (spd-r2, spd-r3).
**Where the money goes.** mp-r2 (complete flow): grill $0.16, spec $0.11, tickets $0.09, implement $0.96, code-review
$0.45, pr $0.04 (F1 means: implement $1.13, code-review $0.44). spd: brainstorming $0.17, writing-plans $0.27,
executing-plans $3.31 (F1 means). Inside executing-plans the build itself (TDD in session) costs $0.97 on average; the
rest is the one whole-branch reviewer the skill dispatches "on the most capable model", which ran on claude-fable-5-1
in all five spd runs: $2.01 of spd-r1's $3.41, $2.52 of spd-r3's $4.41, $1.57 of f2-spd-r2's $3.00.
**Hook vs driven.** Left to its hook, Superpowers ran brainstorming in sp-r1, sp-r2 and sp-r3 and then coded; writing-
plans ran 0 times in 3. Driven by slash commands, the full loop ran in every spd run and cost 7.4 times more (F1 means: spd $3.74, sp $0.50). The cost is the final review, not the planning.
**F2 (credit notes) separated, barely.** f2-bare-r1 shipped one missing requirement, R3: credit-note numbers are not
gapless. It took the number before validating, like the existing `/issue` route, so rejected requests burned numbers
(CN-0004 where CN-0002 was due; the F2 bare smoke failed the same way). f2-bare-r2, f2-spd-r1, f2-spd-r2, f2-mp-r1 and
f2-mp-r2 scored 5/5. Costs: bare $0.38/$0.45, mp $2.18/$1.95, spd $3.42/$3.00 (r1/r2). Caveat: the first F2 oracle
also required credit notes on open invoices; both bare runs allowed paid invoices only, which the request's wording
permits, so that check was removed and the F2 runs rescored.
**Stacking broke nothing.** both-r1: hook injected, no Superpowers skill invoked, $2.33, inside the mp range.
**Caps.** mp-r1, mp-r3 and f2-mp-r1 hit the 12-driver-turn cap before /pr. spd never needed more than 8 turns.
**Caveats.** Round-1 runs read this repo's CLAUDE.md (parent-directory loading), so they were discarded and rerun outside the repo. A simulated PO answers each round in one message. The SessionStart hook fires on turn 1 only.
/implement covers every ticket in one turn. 5 to 11 compound shell lines per spd run were denied. PO cost is excluded.
더 나아가기
- 디버깅 벤치. r/claudeskills의 한 글은 한 repo의 이력에서 버그 15개를 골라 각각 세 번씩 돌렸습니다. 세 변형 모두 15개 중 9개를 고쳤고, 추가 skill은 버그당 약 3분을 더했으며, Pocock의 디버깅 skill이 가장 작은 변경, 중앙값 32줄(41줄, 48줄 대비)을 만들면서 다른 테스트를 깨뜨리지 않았습니다 s8. 이는 디버깅 skill만 테스트하며 전체 flow는 아니고, 41과 48이 어느 변형의 값인지 글에 나와 있지 않습니다. 벤치 하네스로 내 버그 이력에 다시 돌려 보세요.
- 갈아탄 사람들의 블로그. orchestration이 세금이 되었다는 주장 s9과 Superpowers의 hook을 지우고 grill-with-docs는 남긴 이전 이야기 s10를 읽고, 그 주장을 위의 stage별 표와 견주어 보세요.
- 다른 pack. Pstack과 Osmani의 pack은 우리가 돌리지 않은 두 skill 세트입니다. 같은 하네스는
.claude/skills에 설치되는 어떤 pack이든 받습니다.configs.json
{
"bare": {
"superpowers": false,
"mattpocock": false,
"flow": "bare",
"desc": "no skills, no hooks (project settings carry only the permission deny-list)"
},
"sp": {
"superpowers": true,
"mattpocock": false,
"flow": "sp",
"desc": "HOOK-ROUTED: obra/superpowers v6.4.2: skills in .claude/skills, SessionStart hook in .claude/settings.json as hooks/hooks.json wires it; turn 1 = the bare task, the hook decides"
},
"mp": {
"superpowers": false,
"mattpocock": true,
"flow": "mp",
"desc": "mattpocock/skills v1.3.1 via npx skills add (27 promoted skills, copied), setup-matt-pocock-skills output committed (local tracker)"
},
"both": {
"superpowers": true,
"mattpocock": true,
"flow": "mp",
"desc": "sp + mp in one project; driven with the mp slash-command flow (the stacking question: does the Superpowers hook hijack it?)"
},
"spd": {
"superpowers": true,
"mattpocock": false,
"flow": "spd",
"desc": "Superpowers, DRIVEN: same project install as sp (hook included); the driver types /brainstorming <task> -> /writing-plans -> /executing-plans (Native/inline); the pack's own scripts allowed in Bash"
}
}
prep.sh
#!/usr/bin/env bash
# Fresh work copy of upstream (node_modules included) + the pack fixtures of <cfg>.
# Usage: prep.sh <cfg> <work dir> Prints the base commit (the oracle's BASE) on its last line.
set -euo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"
CFG="$1"; W="$2"
SP=$(python3 -c "import json;print(int(json.load(open('$HERE/configs.json'))['$CFG']['superpowers']))")
MP=$(python3 -c "import json;print(int(json.load(open('$HERE/configs.json'))['$CFG']['mattpocock']))")
rm -rf "$W"; mkdir -p "$(dirname "$W")"
cp -R "$HERE/upstream" "$W"
cd "$W"
# .claude/ never shows in git status or in any diff (hook config, installed skills).
echo ".claude/" >> .git/info/exclude
mkdir -p .claude/skills
if [ "$MP" = 1 ]; then
cp -R "$HERE/packs/mattpocock-skills/.claude/skills/." .claude/skills/
# One-time per-repo precondition of the pack (/setup-matt-pocock-skills), answered once with the defaults it
# recommends for a repo with no remote: local-markdown tracker, single-context domain docs, no triage labels
# (triage is installed, so Section B would ask; default labels kept). Committed so it never shows in the diff.
S="$HERE/packs/mattpocock-skills/.claude/skills/setup-matt-pocock-skills"
mkdir -p docs/agents
cp "$S/issue-tracker-local.md" docs/agents/issue-tracker.md
cp "$S/domain.md" docs/agents/domain.md
cp "$S/triage-labels.md" docs/agents/triage-labels.md
cat > CLAUDE.md <<'MD'
## Agent skills
### Issue tracker
Issues and specs live as local markdown files under `.scratch/`. See `docs/agents/issue-tracker.md`.
### Triage labels
Default labels (`needs-triage`, `needs-info`, `ready-for-agent`, `ready-for-human`, `wontfix`). See `docs/agents/triage-labels.md`.
### Domain docs
Single-context: one `GLOSSARY.md` + `docs/adr/` at the repo root. See `docs/agents/domain.md`.
MD
git add CLAUDE.md docs/agents
GIT_AUTHOR_DATE="2026-03-09T18:00:00" GIT_COMMITTER_DATE="2026-03-09T18:00:00" \
git -c user.name="Dana Whitfield" -c user.email="dana@ledgerly.dev" commit -q -m "chore: configure agent skills (local tracker)"
fi
if [ "$SP" = 1 ]; then
# The plugin root as Claude Code would hold it, so the hook script finds ../skills/using-superpowers.
mkdir -p .claude/plugins
cp -R "$HERE/packs/superpowers" .claude/plugins/superpowers
cp -R "$HERE/packs/superpowers/skills/." .claude/skills/
fi
python3 - "$HERE" "$SP" "$CFG" > .claude/settings.json <<'PY'
import json, sys
bench, sp, cfg = sys.argv[1], sys.argv[2] == "1", sys.argv[3]
# Oracle, reference, PO brief, other runs: denied by path (Read/Grep/Glob) and by command text (Bash).
deny = ["Read(/%s/hidden/**)" % bench, "Read(/%s/reference/**)" % bench, "Read(/%s/runs/**)" % bench,
"Read(/%s/po/**)" % bench, "Read(/%s/validation/**)" % bench,
"Bash(*hidden*)", "Bash(*reference/F*)", "Bash(*-brief.md*)", "Bash(*/runs/*)", "Bash(*../*)", "Bash(find *)"]
if cfg == "spd":
# executing-plans runs its sibling scripts as ../subagent-driven-development/scripts/...: a "../" deny would block the
# pack's own loop. The oracle stays protected by the Read denies and Bash(*hidden*).
deny.remove("Bash(*../*)")
settings = {"permissions": {"deny": deny}}
if sp:
# = packs/superpowers/hooks/hooks.json, with ${CLAUDE_PLUGIN_ROOT} provided the way the plugin loader provides it
# (the script also picks its JSON output shape from CLAUDE_PLUGIN_ROOT being set).
root = '$CLAUDE_PROJECT_DIR/.claude/plugins/superpowers'
hooks = json.load(open("%s/packs/superpowers/hooks/hooks.json" % bench))["hooks"]
for groups in hooks.values():
for g in groups:
for h in g["hooks"]:
h["command"] = 'CLAUDE_PLUGIN_ROOT="%s" %s' % (root, h["command"].replace("${CLAUDE_PLUGIN_ROOT}", root))
settings["hooks"] = hooks
print(json.dumps(settings, indent=2))
PY
[ "$(ls .claude/skills | wc -l)" -eq 0 ] && rmdir .claude/skills
test -z "$(git status --porcelain)" || { echo "work copy not clean"; git status --porcelain; exit 1; }
git rev-parse HEAD
FAQ
어느 pack이 더 나은 코드를 냈나요?
기능 1에서는 아닙니다. 13번의 실행 모두 5개 중 5개, suite와 typecheck 통과입니다. 기능 2에서는 bare 실행 하나가 요구사항 하나, 공백 없는 크레딧 노트 번호를 놓쳤습니다. 기존 issue route처럼 요청을 검증하기 전에 번호를 가져갔기 때문입니다. 두 루프는 코드보다 먼저 시퀀스 테스트를 썼습니다. bare 두 번째 실행은 맞게 해냈습니다.
Superpowers 루프는 왜 여기서 Main Flow보다 훨씬 비싼가요?
stage 하나 때문입니다. executing-plans는 평균 $3.307였고 그 안의 리뷰어 subagent가 $2.192였습니다. 리뷰어는 직접 구동한 다섯 실행 모두에서 claude-fable-5-1로 돌았고, skill이 가장 강력한 모델을 요청합니다. 계획 stage는 $0.166과 $0.266였습니다. 더 싼 리뷰어로 다시 돌려 보지는 않았으니 그 변경의 실측 수치는 없습니다.
두 pack을 같이 설치하면 뭔가 깨지나요?
우리가 겹쳐 돌린 한 번에서는 아닙니다. hook이 주입됐고, Superpowers skill은 호출되지 않았고, 슬래시 명령어가 이겼으며, 5개 중 5개에 $2.326였습니다. Pocock의 사용자 호출 skill에는 모델이 스스로 불러오지 못하게 하는 플래그가 있고, 두 pack은 skill 디렉터리 이름을 하나도 공유하지 않습니다. 한 번의 실행이니 내 환경에서 테스트하세요.
Pocock의 skills를 그냥 Claude plugin으로 쓰면 안 되나요?
마켓플레이스가 1.2.3에 고정해 두었는데 upstream은 v1.3.1이기 때문입니다 s32. 설치 프로그램은 요청한 릴리스를 가져옵니다.
Main Flow의 품질을 공정하게 본 테스트인가요?
각 pack이 얼마가 들고, 작은 repo에서 다섯 가지 요구사항을 출시하는지에 대해서는 공정한 테스트입니다. Main Flow의 산출물인 spec과 ticket은 여기서 채점하지 않았고, 더 큰 기능이나 팀이라면 더 높게 칠 수 있습니다.