Claude Code subagents को Haiku 5.5 पर route करें: table, दो फ़ाइलें, harness

संक्षेप में

• Claude Code के subagents तब तक Haiku 5.5 पर नहीं चलते जब तक कोई फ़ाइल ऐसा न कहे। built-in Explore agent और हर custom subagent main मॉडल विरासत में लेते हैं, और अकेला subagent model variable Explore को छूता तक नहीं।
• दो फ़ाइलें इसे ठीक करती हैं: सिर्फ़ explorer के लिए project की Explore.md agent फ़ाइल (Haiku 5.5, effort low), और हर subagent को Haiku पर धकेलने के लिए CLAUDE_CODE_SUBAGENT_MODEL_FORCE वाला settings.json env block।
• हमारा matrix, एक असली repo पर 182 runs: explore, tests, lint, commit, refactor, review और summary, सब Haiku 5.5 पर low और medium effort दोनों में किसी छिपे test या तय नियम से स्वीकार हुए (हर एक में 70 में से 70)।
• सातों को effort low पर Haiku 5.5 को सौंपिए। जटिल agentic coding के लिए Sonnet 5.5 या Opus 5.5 रखिए, जो Anthropic की अपनी सलाह है।
• असली plan meter मुश्किल से हिला: 70 Haiku low runs में five-hour window 21% से 21%, सभी 182 runs में साप्ताहिक window 67%। /usage breakdown मॉडलों को local weights से रैंक करता है, billing से नहीं।
• पूरा model id पिन कीजिए। Bedrock, Vertex और Foundry पर haiku alias Haiku 4.5 है।

स्रोत क्या कहते हैं

Haiku 5.5 की कीमत 100k-token तक के prompts पर $0.10 प्रति दस लाख input tokens और उसके ऊपर $0.50 है, output $0.50 और $2.50 पर। Haiku 4.5 की कीमत $1.00 और $5.00 है, Sonnet 5.5 की $2.00 और $10.00 s1। pricing page Opus 5.5 को $4 और $20 प्रति दस लाख tokens पर सूचीबद्ध करता है और बताता है कि बाकी 1M-context मॉडलों के उलट Haiku 5.5 की कीमत prompt की लंबाई से तय होती है s20। सुर्खी वाला 90% पिछले Haiku के उन requests का हिस्सा है जिनके prompts 100k tokens से छोटे थे, कीमत में कटौती नहीं s1।

Anthropic Haiku 5.5 को Opus 5.5 और Sonnet 5.5 के बगल में एक subagent के रूप में रखता है, और कहता है कि जटिल agentic coding के लिए Sonnet 5.5 और Opus 5.5 बेहतर विकल्प बने रहते हैं, जबकि Haiku 5.5 सीमित दायरे के काम के लिए है s1। Terminal-Bench 4.0 पर launch post Haiku 5.5 के लिए 39.2% और Sonnet 5.5 के लिए 70.6% देती है s1।

कुछ भी अपने आप Haiku पर नहीं जाता। v2.1.198 से built-in Explore agent का डिफ़ॉल्ट Haiku नहीं रहा और वह main मॉडल विरासत में लेता है s24। subagent के मॉडल का दस्तावेज़ी क्रम यह है: प्रति-invocation parameter, definition का frontmatter, CLAUDE_CODE_SUBAGENT_MODEL, फिर मुख्य बातचीत का मॉडल s2। अकेला CLAUDE_CODE_SUBAGENT_MODEL built-in Explore और Plan subagents को नहीं बदलता s2। Explore नाम का user या project subagent built-in को override करता है, और docs उसे Haiku मॉडल के साथ परिभाषित करने का सुझाव देते हैं s2। CLAUDE_CODE_SUBAGENT_MODEL_FORCE=1 v2.1.257 से हर subagent पर एक ही मॉडल लागू करता है s2 s24।

costs doc सरल subagent कामों के लिए model: haiku की सलाह देता है, और कहता है कि subagent requests फिर भी उसी usage से कटते हैं s3। साप्ताहिक meters सभी मॉडलों के लिए एक हैं, और जहाँ शामिल हो वहाँ Fable के लिए एक और s22; दूसरे मॉडल उन्हीं सीमाओं से खींचते हैं s23।

Claude Code में Haiku 5.5, Opus 5.5, Sonnet 5.5 या Fable के लिए thinking बंद नहीं की जा सकती: alwaysThinkingEnabled को false करने और MAX_THINKING_TOKENS=0 का कोई असर नहीं होता। Effort low से max तक चलता है और Haiku 5.5 का डिफ़ॉल्ट medium है s21। haiku alias Anthropic API पर Haiku 5.5 और Platform on AWS, Bedrock, Vertex और Foundry पर Haiku 4.5 में बदलता है s21।

एक practitioner का Pro-plan setup (medium पर Opus 5.5 main, एक Sonnet 5.5 implementer, एक Haiku 5.5 read-only scoper, low पर एक Haiku 5.5 shipper) दस sessions में 13 PRs बताता है, हर एक पर five-hour window का लगभग 6%, और एक-PR session के लिए साप्ताहिक counter का लगभग 1% s4। उनकी अपनी चेतावनी: बचत main context के छोटा रहने से आती है, सस्ते मॉडल से नहीं, क्योंकि subagent tokens को Opus की दरों पर आँकने से सिर्फ़ 5% बढ़ता है s4।

दो फ़ाइलें

दो फ़ाइलें, एक लें या दोनों। project की agent फ़ाइल .claude/agents/Explore.md का नाम Explore है, इसलिए वह built-in explorer की जगह लेती है। पूरा model id, effort low, सिर्फ़ पढ़ने और खोजने वाले tools।

Explore.md

---
name: Explore
description: Read-only codebase explorer. Use it to find where something is defined or called and to summarize what it found, with file paths and function names.
model: claude-haiku-5-5
effort: low
tools: Read, Glob, Grep, Bash
---

You explore a codebase and report back. Search with Glob and Grep, read only the files you need, and never edit anything. Answer with file paths and function names, then stop.

Explore.md on GitHub

settings फ़ाइल .claude/settings.json में env block है: subagent मॉडल, और force flag जो उसे हर subagent पर लागू करता है।

settings.json

{
  "env": {
    "CLAUDE_CODE_SUBAGENT_MODEL": "claude-haiku-5-5",
    "CLAUDE_CODE_SUBAGENT_MODEL_FORCE": "1"
  }
}

settings.json on GitHub

किसके लिए कौन-सी:

• जब हर subagent सस्ता होना चाहिए तब env block इस्तेमाल कीजिए। FORCE हर agent को Haiku पर खींच लाता है, उन्हें भी जिन्हें आपने जानबूझकर Opus पर रखा था।
• जब सिर्फ़ explorer को हिलना हो तब Explore फ़ाइल इस्तेमाल कीजिए। बाकी subagents अपने-अपने मॉडल रखते हैं।
• FORCE के बिना CLAUDE_CODE_SUBAGENT_MODEL सेट करने से Explore के लिए कुछ नहीं होता। हमारे दो headless runs, main session Fable पर, "use the Explore agent to find where the query cache is cleared": बिना किसी setting के tool result ने Opus 5.5 बताया, और variable को Haiku 5.5 पर सेट करने और कुछ और न करने पर फिर Opus 5.5। Explore फ़ाइल के साथ, या दोनों env variables के साथ, उसने Haiku 5.5 पढ़ा।

जाँच: /tasks दिखाता है कि हर subagent किस मॉडल और effort पर चला (v2.1.243 और बाद में s24), और headless run का Agent tool result resolvedModel साथ लाता है। उस पंक्ति पर भरोसा कीजिए, फ़ाइल पर कभी नहीं।

alias का जाल: claude-haiku-5-5 लिखिए, haiku नहीं। alias सिर्फ़ Anthropic API पर Haiku 5.5 है; Bedrock, Vertex और Foundry पर इसका मतलब अब भी Haiku 4.5 है s21।

एक पुरानी खुली रिपोर्ट कहती है कि subagent के frontmatter का मॉडल चुपचाप नज़रअंदाज़ हो गया s25, और एक बाद की रिपोर्ट 2.1.277 से 2.1.290 पर 1,371 में से 1,149 subagents को parent के Opus मॉडल पर चलता गिनती है s10। Claude Code 2.1.293 पर हमारे सभी 182 runs ने configured मॉडल ही इस्तेमाल किया (results table में model_ok True)। फिर भी जाँचिए।

routing table

tinydb पर सात रोज़मर्रा के काम, commit 18d73a1, 226 tests, headless claude -p, हर काम के लिए एक निर्धारक accept या reject check। हर cell में स्वीकार हुए runs, प्रति run औसत list price और औसत wall seconds हैं। Haiku cells प्रति काम 10 runs के हैं, Sonnet और Opus cells प्रति काम 3 runs के।

"refactor" काम वही छोटा feature है: tinydb.operations में push और pull update operations जोड़ना, जिसे एक छिपी test फ़ाइल ग्रेड करती है। इसकी लागत Haiku low पर $0.0052 और Opus पर $0.1675 रही, 17.1 s बनाम 30.3 s में।

Sonnet या Opus पर क्या रहता है: जटिल agentic coding, Anthropic के अपने शब्दों में s1। ये सातों सीमित काम हैं, हर एक एक मिनट से कम का, एक छोटी repo पर।

harness

वह benchmark जिसने यह table बनाया। इसे एक सादे terminal से चलाइए और उस समय account पर कुछ और न चलने दीजिए, क्योंकि plan meter साझा है। tinydb checkout, छिपे tests और reference patches शामिल नहीं हैं: इसे अपनी repo की ओर मोड़िए और अपने checks लिखिए।

Runner और meter:

• README.md: harness क्या है और क्या छोड़ा गया है।

  README.md

# routing bench

The benchmark harness behind the routing table: run.sh runs one cell headless and reads the real meters from the rate_limit_event, matrix.sh runs cells sequentially, check.py decides accept or reject per task, analyze.py writes the results table. tasks/ holds the seven prompts, inject/ the planted bugs. The tinydb checkout (commit 18d73a1), the hidden tests and the reference patches are not included: bring your own repo and checks.

README.md on GitHub

run.sh

#!/usr/bin/env bash
# One run: fresh tinydb work copy + task injection, headless claude -p, extract tokens/meter, check.
# Usage: run.sh <config> <task> <rep>
#   config: a key of configs.json (haiku-med | haiku-low | sonnet | opus)
#   task:   a key of tasks.json   (explore | tests | lint | commit | refactor | review | summary)
set -uo pipefail
CFG="$1"; TASK="$2"; REP="${3:-1}"
HERE="$(cd "$(dirname "$0")" && pwd)"
. "$HERE/env.sh"
PY="$HERE/.venv/bin/python"
MAX_TURNS="${BENCH_MAX_TURNS:-40}"
NAME="${TASK}-${CFG}-r${REP}"
OUT="$HERE/runs/$NAME"; W="$HERE/work/$NAME"
rm -rf "$OUT"; mkdir -p "$OUT"
"$HERE/prep.sh" "$TASK" "$W" > "$OUT/prep.txt" 2>&1 || { echo "$NAME: prep failed"; cat "$OUT/prep.txt"; exit 2; }
( cd "$W" && git diff HEAD > "$OUT/pre.diff" )

# Config args (model pinned by full id, optional --effort)
CARGS=()
while IFS= read -r a; do CARGS+=("$a"); done < <("$PY" -c "
import json; [print(a) for a in json.load(open('$HERE/configs.json'))['$CFG']['args']]")
[ ${#CARGS[@]} -gt 0 ] || { echo "bad config $CFG" >&2; exit 2; }

# Fixed allow-list for every cell, including Bash(ruff:*) for the lint task.
TOOLS="Read,Edit,Write,MultiEdit,Glob,Grep,Skill,TodoWrite,Bash(uv:*),Bash(python:*),Bash(python3:*),Bash(pytest:*),Bash(ruff:*),Bash(git status:*),Bash(git diff:*),Bash(git log:*),Bash(ls:*),Bash(cat:*),Bash(mkdir:*),Bash(mktemp:*),Bash(head:*),Bash(tail:*),Bash(grep:*),Bash(wc:*)"

cd "$W"
START=$(date +%s)
claude -p "$(cat "$HERE/tasks/$TASK.md")" "${CARGS[@]}" \
  --output-format stream-json --verbose --max-turns "$MAX_TURNS" \
  --permission-mode acceptEdits --allowedTools "$TOOLS" \
  --setting-sources project --strict-mcp-config \
  > "$OUT/stream.jsonl" 2> "$OUT/stderr.txt"
echo "$?" > "$OUT/exit_code.txt"
END=$(date +%s)
echo "$((END-START))" > "$OUT/wall_s.txt"

# After-meter: one cheap Haiku probe (its own stream is kept for audit).
"$HERE/probe.sh" "$OUT/probe.jsonl" > "$OUT/after.json" 2>/dev/null

# Diff of the agent's work (tracked + untracked) vs the post-injection state
# (index left untouched: the commit task's staged changes stay staged for the check)
git diff HEAD -- . ':(exclude).coverage' > "$OUT/post.diff"
git diff HEAD --numstat -- . ':(exclude).coverage' > "$OUT/numstat.txt"
git ls-files --others --exclude-standard | while IFS= read -r f; do
  git diff --no-index -- /dev/null "$f" >> "$OUT/post.diff"
  echo "$(wc -l < "$f" | tr -d ' ')	0	$f" >> "$OUT/numstat.txt"
done

"$PY" "$HERE/extract.py" "$OUT" "$CFG" "$TASK" "$REP" > "$OUT/result.json"
"$PY" "$HERE/check.py" "$TASK" "$W" "$OUT/result.json" > "$OUT/check.txt"; echo "$?" >> "$OUT/check.txt"
"$PY" "$HERE/extract.py" "$OUT" "$CFG" "$TASK" "$REP" --score

run.sh on GitHub

matrix.sh

#!/usr/bin/env bash
# Run an ordered list of "<config> <task> <rep>" lines SEQUENTIALLY (the plan meter is shared:
# never run two cells at once). Blank lines and #comments are skipped. Appends to matrix.log.
# Usage: matrix.sh [matrix-file]   (default: matrix.txt)
#   SKIP_DONE=1 matrix.sh ...      skip cells whose runs/<task>-<config>-r<rep>/score.json exists (resume)
set -uo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"; cd "$HERE"
LIST="${1:-matrix.txt}"
# one matrix at a time: the meter is shared
mkdir "$HERE/.matrix.lock" 2>/dev/null || { echo "another matrix.sh is running (remove .matrix.lock if stale)" >&2; exit 1; }
trap 'rmdir "$HERE/.matrix.lock"' EXIT
echo "=== $(date '+%F %T') start $LIST" >> matrix.log
grep -vE '^\s*(#|$)' "$LIST" | while read -r CFG TASK REP; do
  if [ "${SKIP_DONE:-0}" = 1 ] && [ -f "runs/${TASK}-${CFG}-r${REP}/score.json" ]; then
    echo "$(date '+%F %T') skip ${TASK}-${CFG}-r${REP}" >> matrix.log; continue
  fi
  line=$(./run.sh "$CFG" "$TASK" "$REP" < /dev/null 2>&1 | tail -1)
  echo "$(date '+%F %T') $line" | tee -a matrix.log
done
echo "=== $(date '+%F %T') done $LIST" >> matrix.log

matrix.sh on GitHub

gen_matrix.py

#!/usr/bin/env python3
"""Write the ordered matrix file.

gen_matrix.py [--reps 3] [--passes 2] [--configs haiku-low,haiku-med,sonnet,opus] [--out matrix.txt]

Order: configs run as contiguous BLOCKS (all tasks of one rep set), cheap configs first; tasks are
rotated inside each block so no task always runs first. With --passes 2 the second pass runs the
configs in reverse order (ABBA), which interleaves configs over time without breaking the
contiguous segments that analyze.py needs to beat the 1% meter resolution.
"""
import argparse, json, os

HERE = os.path.dirname(os.path.abspath(__file__))
ap = argparse.ArgumentParser()
ap.add_argument("--reps", type=int, default=3, help="reps per config x task per pass")
ap.add_argument("--passes", type=int, default=2)
ap.add_argument("--configs", default="haiku-low,haiku-med,sonnet,opus")
ap.add_argument("--start-rep", type=int, default=1, help="first rep number (use 2 to keep the r1 smoke runs)")
ap.add_argument("--out", default=os.path.join(HERE, "matrix.txt"))
a = ap.parse_args()
tasks = list(json.load(open(os.path.join(HERE, "tasks.json"))))
cfgs = a.configs.split(",")
lines, rep = [], a.start_rep
for p in range(a.passes):
    order = cfgs if p % 2 == 0 else list(reversed(cfgs))
    lines.append(f"# pass {p + 1}: {' -> '.join(order)}")
    for c in order:
        lines.append(f"# block {c}")
        for r in range(a.reps):
            k = rep + r
            rot = (k - 1) % len(tasks)
            for t in tasks[rot:] + tasks[:rot]:
                lines.append(f"{c} {t} {k}")
    rep += a.reps
open(a.out, "w").write("\n".join(lines) + "\n")
n = sum(1 for l in lines if not l.startswith("#"))
print(f"{a.out}: {n} runs ({len(cfgs)} configs x {len(tasks)} tasks x {a.reps} reps x {a.passes} passes)")

gen_matrix.py on GitHub

prep.sh

#!/usr/bin/env bash
# Build a fresh work copy for one task: upstream tinydb, baseline commit, task injection.
# Usage: prep.sh <task> <workdir>
set -euo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"
TASK="$1"; W="$2"
read -r INJ MODE < <("$HERE/.venv/bin/python" -c "
import json,sys; t=json.load(open('$HERE/tasks.json'))['$TASK']; print(t['inject'] or '-', t['mode'] or '-')")
rm -rf "$W"; mkdir -p "$(dirname "$W")"
cp -R "$HERE/upstream" "$W"
cd "$W"
/usr/bin/find . -name __pycache__ -type d -prune -exec rm -rf {} +
rm -f .coverage
git update-index -q --refresh || true
G() { git -c user.email=dev@example.com -c user.name=dev "$@"; }
# Injections that belong to the baseline are folded into the top upstream commit
# (amend, same message/author/date), so `git log` shows no extra "bug" commit.
if [ "$MODE" = "commit" ]; then
  git apply "$HERE/$INJ"
  git add -A
  G commit -q --amend --no-edit --no-verify
fi
case "$MODE" in
  stage)    git apply --index "$HERE/$INJ" ;;
  worktree) git apply "$HERE/$INJ" ;;
esac

prep.sh on GitHub

env.sh

# Sourced by run.sh and probe.sh.
# Auth: an isolated CLAUDE_CONFIG_DIR is not logged in
# (`claude auth status` -> loggedIn false), so runs use the user's default config dir:
# CLAUDE_CONFIG_DIR is deliberately left unset.
unset CLAUDE_CONFIG_DIR
# Fresh work copies at the same path must not inherit memory written by an earlier rep.
export CLAUDE_CODE_DISABLE_AUTO_MEMORY=1
# The bench venv (pytest, ruff, mypy) comes first, so `python -m pytest` and `ruff` just work.
export PATH="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/.venv/bin:$PATH"

env.sh on GitHub

probe.sh

#!/usr/bin/env bash
# Cheapest possible call: one Haiku turn answering "ok". Prints the current plan
# meter read from its rate_limit_event as one JSON line. The probe itself costs a
# few hundred tokens of Haiku (well under the 1% meter resolution).
# Usage: probe.sh [raw-stream-out-file]
set -uo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"
[ -f "$HERE/env.sh" ] && . "$HERE/env.sh"
RAW="${1:-$(mktemp -t hrb-probe)}"; case "$RAW" in /*) ;; *) RAW="$PWD/$RAW";; esac
cd "$(mktemp -d -t hrb-probe-cwd)"
claude --model claude-haiku-5-5 -p "ok" --max-turns 1 --output-format stream-json --verbose \
  --setting-sources project --strict-mcp-config > "$RAW" 2>/dev/null
"$HERE/.venv/bin/python" "$HERE/meter.py" "$RAW"

probe.sh on GitHub

meter.py

#!/usr/bin/env python3
"""Read a stream-json file and print the plan meter carried by its rate_limit_event lines.

Output (one JSON line): {"ts", "five_hour", "five_hour_resets_at", "seven_day",
"seven_day_resets_at", "n_events", "raw_first", "raw_last"}: utilizations are the
values of the FIRST event (the meter as the run began). `--last` uses the last event.
"""
import json, sys, time

def events(path):
    out = []
    for line in open(path, errors="replace"):
        line = line.strip()
        if not line.startswith("{"):
            continue
        try:
            o = json.loads(line)
        except Exception:
            continue
        if o.get("type") == "rate_limit_event":
            out.append(o)
    return out

def windows(ev):
    info = ev.get("rate_limit_info") or {}
    uw = info.get("unifiedWindows") or {}
    def w(name):
        x = uw.get(name) or {}
        return x.get("utilization"), x.get("resetsAt")
    fh, fhr = w("five_hour")
    sd, sdr = w("seven_day")
    return {"five_hour": fh, "five_hour_resets_at": fhr, "seven_day": sd, "seven_day_resets_at": sdr}

def meter(path, which="first"):
    evs = events(path)
    if not evs:
        return {"ts": int(time.time()), "n_events": 0, "five_hour": None, "seven_day": None}
    ev = evs[0] if which == "first" else evs[-1]
    d = {"ts": int(time.time()), "n_events": len(evs)}
    d.update(windows(ev))
    d["raw_first"] = evs[0].get("rate_limit_info")
    d["raw_last"] = evs[-1].get("rate_limit_info")
    return d

if __name__ == "__main__":
    which = "last" if "--last" in sys.argv else "first"
    path = [a for a in sys.argv[1:] if not a.startswith("--")][0]
    print(json.dumps(meter(path, which)))

meter.py on GitHub

extract.py

#!/usr/bin/env python3
"""Extract one run's numbers from runs/<name>/stream.jsonl.

extract.py <outdir> <config> <task> <rep>          -> prints the stream's `result` line (result.json)
extract.py <outdir> <config> <task> <rep> --score  -> writes score.json, prints a one-line summary
"""
import json, os, sys

HERE = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, HERE)
from meter import meter  # noqa: E402

out, cfg, task, rep = sys.argv[1:5]
score_mode = "--score" in sys.argv
stream = os.path.join(out, "stream.jsonl")

lines = []
for raw in open(stream, errors="replace") if os.path.exists(stream) else []:
    raw = raw.strip()
    if raw.startswith("{"):
        try:
            lines.append(json.loads(raw))
        except Exception:
            pass
result = next((o for o in reversed(lines) if o.get("type") == "result"), None)
init = next((o for o in lines if o.get("type") == "system" and o.get("subtype") == "init"), {})

if not score_mode:
    print(json.dumps(result or {"type": "result", "result": "", "missing": True}))
    sys.exit(0)

def rd(name, default=None):
    p = os.path.join(out, name)
    try:
        return open(p).read().strip()
    except Exception:
        return default

def jl(name):
    try:
        return json.loads(rd(name) or "null")
    except Exception:
        return None

cfgs = json.load(open(os.path.join(HERE, "configs.json")))
before = meter(stream, "first") if os.path.exists(stream) else None
after = jl("after.json")
r = result or {}
usage = r.get("usage") or {}
tok = {k: usage.get(k) or 0 for k in ("input_tokens", "cache_read_input_tokens", "cache_creation_input_tokens", "output_tokens")}
mu = r.get("modelUsage") or {}
assistant_models = sorted({(o.get("message") or {}).get("model") for o in lines
                           if o.get("type") == "assistant" and (o.get("message") or {}).get("model") not in (None, "<synthetic>")})
check_txt = (rd("check.txt", "") or "").split("\n")
verdict_line = check_txt[0] if check_txt else ""
numstat = [l.split("\t") for l in (rd("numstat.txt", "") or "").split("\n") if l.strip()]
added = sum(int(a) for a, d, f in numstat if a.isdigit())
deleted = sum(int(d) for a, d, f in numstat if d.isdigit())
changed_by_agent = (rd("pre.diff", "") or "") != (rd("post.diff", "") or "")

def strip_raw(m):
    if not m:
        return m
    return {k: v for k, v in m.items() if not k.startswith("raw_")}

s = {
    "run": os.path.basename(out), "config": cfg, "task": task, "rep": int(rep),
    "expected_model": cfgs[cfg]["model"], "config_args": cfgs[cfg]["args"],
    "init_model": init.get("model"), "assistant_models": assistant_models,
    "model_usage_keys": sorted(mu.keys()),
    "model_ok": init.get("model") == cfgs[cfg]["model"] and set(mu.keys()) <= {cfgs[cfg]["model"]} and bool(mu),
    "claude_code_version": init.get("claude_code_version"),
    "meter_before": strip_raw(before), "meter_after": strip_raw(after),
    "n_rate_limit_events": (before or {}).get("n_events", 0),
    "has_result_line": result is not None,
    "subtype": r.get("subtype"), "is_error": r.get("is_error"), "stop_reason": r.get("stop_reason"),
    "num_turns": r.get("num_turns"), "duration_ms": r.get("duration_ms"), "duration_api_ms": r.get("duration_api_ms"),
    "total_cost_usd": r.get("total_cost_usd"), "usage": usage, "tokens": tok, "modelUsage": mu,
    "wall_s": int(rd("wall_s.txt", "0") or 0), "exit_code": rd("exit_code.txt"),
    "final_text": (r.get("result") or "")[-2000:],
    "diff": {"files": len(numstat), "added": added, "deleted": deleted, "changed_by_agent": changed_by_agent},
    "check": {"verdict": verdict_line.split(" ")[0] if verdict_line else None, "reason": verdict_line[5:] if verdict_line else None},
}
s["accepted"] = s["check"]["verdict"] == "PASS"
json.dump(s, open(os.path.join(out, "score.json"), "w"), indent=2)
mb, ma = s["meter_before"] or {}, s["meter_after"] or {}
print(f"{s['run']}: {s['check']['verdict']} turns={s['num_turns']} cost=${(s['total_cost_usd'] or 0):.3f} "
      f"in={tok['input_tokens']} cr={tok['cache_read_input_tokens']} cc={tok['cache_creation_input_tokens']} out={tok['output_tokens']} "
      f"wall={s['wall_s']}s 5h {mb.get('five_hour')}->{ma.get('five_hour')} 7d {mb.get('seven_day')}->{ma.get('seven_day')} "
      f"model_ok={s['model_ok']} | {s['check']['reason']}")

extract.py on GitHub

check.py

#!/usr/bin/env python3
"""Deterministic accept/reject check for one run.

Usage: check.py <task> <workdir> <result.json>
  result.json = the `result` line of the run's stream (only its "result" text is used).
Prints `PASS <reason>` or `FAIL <reason>`; exit 0 on PASS, 1 on FAIL.
"""
import json, os, re, subprocess, sys

HERE = os.path.dirname(os.path.abspath(__file__))
PY = os.path.join(HERE, ".venv", "bin", "python")
RUFF = os.path.join(HERE, ".venv", "bin", "ruff")


def final_text(path):
    try:
        r = json.load(open(path))
    except Exception:
        return ""
    return r.get("result") or "" if isinstance(r, dict) else ""


def sh(args, cwd, env=None):
    p = subprocess.run(args, cwd=cwd, capture_output=True, text=True, env=env)
    return p.returncode, p.stdout + p.stderr


def suite(wd):
    rc, out = sh([PY, "-m", "pytest", "-q", "-p", "no:cacheprovider", "-o", "addopts=", "tests"], wd)
    m = re.search(r"(\d+) passed", out)
    f = re.search(r"(\d+) failed", out)
    e = re.search(r"(\d+) error", out)
    return rc == 0, f"suite {m.group(1) if m else 0} passed, {f.group(1) if f else 0} failed, {e.group(1) if e else 0} errors"


def git_diff(wd, *paths):
    # tracked changes vs the baseline commit + untracked files under paths
    _, d = sh(["git", "diff", "HEAD", "--", *paths], wd)
    _, u = sh(["git", "ls-files", "--others", "--exclude-standard", "--", *paths], wd)
    return d.strip(), u.strip()


def words(s):
    return re.findall(r"[A-Za-z0-9_']+", s)


def strip_fences(s):
    lines = s.strip().split("\n")
    if lines and lines[0].strip().startswith("```"):
        lines = lines[1:]
    if lines and lines[-1].strip().startswith("```"):
        lines = lines[:-1]
    return "\n".join(lines).strip()


def c_explore(wd, txt):
    t = txt.lower()
    miss = []
    if "table.py" not in t:
        miss.append("table.py")
    if "clear_cache" not in t and "_query_cache" not in t:
        miss.append("clear_cache/_query_cache")
    names = ["insert_multiple", "update_multiple", "insert", "update", "upsert", "remove", "truncate"]
    found = {n for n in names if re.search(r"(?<![a-z_])" + n + r"(?![a-z_])", t)}
    if len(found) < 3:
        miss.append(f"only {len(found)} mutating methods {sorted(found)}")
    return (not miss), ("missing " + ", ".join(miss)) if miss else f"table.py + cache method + {len(found)} methods {sorted(found)}"


def c_tests(wd, txt):
    ok, s = suite(wd)
    d, u = git_diff(wd, "tests")
    if d or u:
        return False, f"tests/ modified; {s}"
    return ok, s + "; tests/ untouched"


def c_lint(wd, txt):
    rc, out = sh([RUFF, "check", "tinydb", "--output-format", "concise", "--no-cache"], wd)
    cfg_d, _ = git_diff(wd, "ruff.toml", "pyproject.toml")
    d, _ = git_diff(wd, "tinydb")
    noqa = [l for l in d.split("\n") if l.startswith("+") and "noqa" in l.lower()]
    ok, s = suite(wd)
    why = []
    if rc != 0:
        found = [l for l in out.split("\n") if l.startswith("Found ")]
        why.append("ruff: " + (found[0] if found else out.strip().split("\n")[-1]))
    if cfg_d:
        why.append("lint config changed")
    if noqa:
        why.append(f"{len(noqa)} noqa added")
    if not ok:
        why.append(s)
    return (not why), ("; ".join(why) if why else f"ruff clean; {s}")


DIFF_LINE = re.compile(r"^(diff --git |index [0-9a-f]{6,}\.\.|@@ |\+\+\+ |--- a/|--- /dev/null|[+-] {3,}\S|[+-](def|class|import|from|return) )")


def c_commit(wd, txt):
    msg = strip_fences(txt)
    if not msg:
        return False, "empty answer"
    first = msg.split("\n")[0].strip()
    why = []
    if len(first) > 72:
        why.append(f"first line {len(first)} chars")
    if "push" not in first.lower():
        why.append("no 'push' in first line")
    if "pull" not in first.lower():
        why.append("no 'pull' in first line")
    dl = [l for l in msg.split("\n") if DIFF_LINE.match(l)]
    if dl:
        why.append(f"{len(dl)} diff lines")
    return (not why), ("; ".join(why) if why else f"first line {len(first)} chars: {first!r}")


def c_refactor(wd, txt):
    env = dict(os.environ, TINYDB_TREE=wd)
    rc, out = sh([PY, "-m", "pytest", "-q", "-c", os.path.join(HERE, "hidden", "pytest.ini"),
                  "--rootdir", os.path.join(HERE, "hidden"), os.path.join(HERE, "hidden", "test_T1.py")], wd, env)
    m = re.search(r"(\d+) passed", out)
    hp = int(m.group(1)) if m else 0
    ok, s = suite(wd)
    return (hp == 5 and ok), f"hidden {hp}/5; {s}"


REVIEW_PATTERNS = [
    r"limit\s*=?=\s*0\b",
    r"limit\s+(of\s+)?(0|zero)\b",
    r"\b(0|zero)\s+limit",
    r"zero[- ]limit",
    r"limit.{0,150}\b(falsy|falsey|truthy|truthiness)",
    r"\b(falsy|falsey|truthy|truthiness)\b.{0,150}limit",
    r"if limit:.{0,150}\b(0|zero)\b",
    r"\b(0|zero)\b.{0,150}if limit:",
]


def c_review(wd, txt):
    t = txt.lower().replace("`", "")
    for p in REVIEW_PATTERNS:
        m = re.search(p, t, re.S)
        if m:
            return True, f"matched {p!r}: {m.group(0)[:80]!r}"
    return False, "planted limit=0 defect not mentioned"


def c_summary(wd, txt):
    n = len(words(txt))
    t = txt.lower()
    mods = [m for m in ["database", "table", "queries", "storages", "middlewares", "operations"] if m in t]
    why = []
    if n > 170:
        why.append(f"{n} words > 170")
    if len(mods) < 4:
        why.append(f"only {len(mods)} modules {mods}")
    return (not why), ("; ".join(why) if why else f"{n} words, modules {mods}")


CHECKS = {"explore": c_explore, "tests": c_tests, "lint": c_lint, "commit": c_commit,
          "refactor": c_refactor, "review": c_review, "summary": c_summary}

if __name__ == "__main__":
    task, wd, res = sys.argv[1], os.path.abspath(sys.argv[2]), sys.argv[3]
    ok, why = CHECKS[task](wd, final_text(res))
    print(("PASS " if ok else "FAIL ") + why)
    sys.exit(0 if ok else 1)

check.py on GitHub

analyze.py

#!/usr/bin/env python3
"""Aggregate runs/*/score.json into results.json and results.md.

Per config x task: accept rate, mean tokens, mean cost, mean turns, mean wall.
Per config: measured plan-meter deltas (five_hour, seven_day, in percentage points), computed two ways:
  - per_run_sum: sum over the config's runs of (after - before). Quantized at 1% per run, so
    runs far below 1% mostly read 0: a lower-bound-ish, very noisy number.
  - segments:    runs sorted by start time are cut into maximal contiguous segments of the same
    config; each segment contributes (after of its last run - before of its first run). This
    telescopes the quantization: one +-1 pt error per segment instead of per run. PREFERRED.
  Segments/runs whose window resetsAt changed between before and after are excluded for that window.
Plus the /usage-style local heuristic: weighted = cached*1 + uncached*10 + cache_create*12.5 + output*50,
times the model tier (haiku 1 / sonnet 3 / opus 5). The probe's own Haiku tokens after each run are
counted as overhead in the heuristic (they also move the meter).
Usage: analyze.py [--runs-dir runs] [--out-prefix results]
"""
import glob, json, os, statistics, sys

HERE = os.path.dirname(os.path.abspath(__file__))
RUNS = os.path.join(HERE, sys.argv[sys.argv.index("--runs-dir") + 1]) if "--runs-dir" in sys.argv else os.path.join(HERE, "runs")
OUTP = sys.argv[sys.argv.index("--out-prefix") + 1] if "--out-prefix" in sys.argv else "results"
CFGS = json.load(open(os.path.join(HERE, "configs.json")))
TASKS = list(json.load(open(os.path.join(HERE, "tasks.json"))).keys())
W = {"cache_read_input_tokens": 1, "input_tokens": 10, "cache_creation_input_tokens": 12.5, "output_tokens": 50}
TIER = {c: v["tier"] for c, v in CFGS.items()}


def weighted(tok, tier):
    raw = sum((tok.get(k) or 0) * w for k, w in W.items())
    return raw, raw * tier


MU = {"cacheReadInputTokens": "cache_read_input_tokens", "inputTokens": "input_tokens",
      "cacheCreationInputTokens": "cache_creation_input_tokens", "outputTokens": "output_tokens"}


def model_tier(name, fallback):
    n = (name or "").lower()
    return 5 if "opus" in n else 3 if "sonnet" in n else 1 if "haiku" in n else fallback


def heuristic_run(s):
    """Tiered heuristic per model actually used (modelUsage), so helper calls on another
    model are weighted with their own tier; falls back to `usage` x config tier."""
    mu = s.get("modelUsage") or {}
    if not mu:
        return weighted(s["tokens"], TIER[s["config"]])[1]
    return sum(weighted({MU[k]: v.get(k) or 0 for k in MU}, model_tier(m, TIER[s["config"]]))[1] for m, v in mu.items())


def probe_tokens(run_dir):
    p = os.path.join(run_dir, "probe.jsonl")
    if not os.path.exists(p):
        return {}
    for line in reversed(open(p, errors="replace").read().strip().split("\n")):
        try:
            o = json.loads(line)
        except Exception:
            continue
        if o.get("type") == "result":
            u = o.get("usage") or {}
            return {k: u.get(k) or 0 for k in W}
    return {}


def mean(xs):
    xs = [x for x in xs if x is not None]
    return statistics.mean(xs) if xs else None


def window_delta(b, a, w):
    if not b or not a or b.get(w) is None or a.get(w) is None:
        return None, "missing"
    if b.get(w + "_resets_at") != a.get(w + "_resets_at"):
        return None, "reset"
    return round((a[w] - b[w]) * 100, 2), "ok"


scores = []
for p in sorted(glob.glob(os.path.join(RUNS, "*", "score.json"))):
    s = json.load(open(p))
    if s.get("config") not in CFGS:
        continue
    s["_dir"] = os.path.dirname(p)
    s["_start"] = (s.get("meter_before") or {}).get("ts") or os.path.getmtime(os.path.join(s["_dir"], "stream.jsonl"))
    s["_probe_tok"] = probe_tokens(s["_dir"])
    scores.append(s)
# meter_before.ts is written at extraction time; order by stream file creation instead
for s in scores:
    st = os.stat(os.path.join(s["_dir"], "stream.jsonl"))
    s["_start"] = getattr(st, "st_birthtime", st.st_mtime)
scores.sort(key=lambda s: s["_start"])

cells, per_cfg = {}, {}
for s in scores:
    cells.setdefault((s["config"], s["task"]), []).append(s)
    per_cfg.setdefault(s["config"], []).append(s)

cell_rows = []
for (c, t), ss in sorted(cells.items(), key=lambda kv: (list(CFGS).index(kv[0][0]), TASKS.index(kv[0][1]) if kv[0][1] in TASKS else 99)):
    row = {"config": c, "task": t, "n": len(ss),
           "accept_rate": sum(1 for s in ss if s.get("accepted")) / len(ss),
           "mean_turns": mean([s.get("num_turns") for s in ss]),
           "mean_cost_usd": mean([s.get("total_cost_usd") for s in ss]),
           "mean_wall_s": mean([s.get("wall_s") for s in ss]),
           "mean_tokens": {k: mean([s["tokens"].get(k) for s in ss]) for k in W},
           "mean_weighted_heuristic": mean([heuristic_run(s) for s in ss]),
           "model_ok_all": all(s.get("model_ok") for s in ss)}
    cell_rows.append(row)

# contiguous segments in global start order
segments, cur = [], None
for s in scores:
    if cur and cur["config"] == s["config"]:
        cur["runs"].append(s)
    else:
        cur = {"config": s["config"], "runs": [s]}
        segments.append(cur)

cfg_rows = []
for c, ss in per_cfg.items():
    tok_sum = {k: sum(s["tokens"].get(k) or 0 for s in ss) for k in W}
    probe_sum = {k: sum(s["_probe_tok"].get(k, 0) for s in ss) for k in W}
    raw_w, _ = weighted(tok_sum, TIER[c])
    heur = sum(heuristic_run(s) for s in ss)
    probe_raw, probe_heur = weighted(probe_sum, 1)
    row = {"config": c, "runs": len(ss), "accept_rate": sum(1 for s in ss if s.get("accepted")) / len(ss),
           "tokens_sum": tok_sum, "probe_tokens_sum": probe_sum,
           "cost_sum_usd": round(sum(s.get("total_cost_usd") or 0 for s in ss), 4),
           "weighted_raw_M": raw_w / 1e6, "heuristic_M": (heur + probe_heur) / 1e6}
    for w in ("five_hour", "seven_day"):
        per_run, excl = 0.0, 0
        for s in ss:
            d, why = window_delta(s.get("meter_before"), s.get("meter_after"), w)
            if d is None:
                excl += 1
            else:
                per_run += d
        seg_sum, seg_n, seg_excl = 0.0, 0, 0
        for g in segments:
            if g["config"] != c:
                continue
            d, why = window_delta(g["runs"][0].get("meter_before"), g["runs"][-1].get("meter_after"), w)
            if d is None:
                seg_excl += 1
            else:
                seg_sum += d; seg_n += 1
        row[w] = {"per_run_sum_pts": round(per_run, 2), "per_run_excluded": excl,
                  "segments_sum_pts": round(seg_sum, 2), "segments": seg_n, "segments_excluded": seg_excl,
                  "quantization_error_pts": f"+-{seg_n}",
                  "pts_per_M_heuristic": round(seg_sum / row["heuristic_M"], 4) if row["heuristic_M"] else None,
                  "pts_per_M_weighted_raw": round(seg_sum / row["weighted_raw_M"], 4) if row["weighted_raw_M"] else None}
    cfg_rows.append(row)
cfg_rows.sort(key=lambda r: list(CFGS).index(r["config"]))

out = {"generated_from": len(scores), "cells": cell_rows, "configs": cfg_rows,
       "segments": [{"config": g["config"], "runs": [s["run"] for s in g["runs"]],
                     "before": g["runs"][0].get("meter_before"), "after": g["runs"][-1].get("meter_after")} for g in segments],
       "heuristic_weights": {"tokens": W, "tier": TIER}}
json.dump(out, open(os.path.join(HERE, OUTP + ".json"), "w"), indent=2, default=str)

f = lambda x, d=0: "-" if x is None else (f"{x:,.{d}f}")
md = ["# Results", "", f"{len(scores)} runs. Meter deltas in percentage points of the plan window (1-pt resolution).", "",
      "## Per config x task", "",
      "| config | task | n | accept | turns | input | cache read | cache create | output | cost $ | wall s | heuristic (M) | model ok |",
      "|---|---|---|---|---|---|---|---|---|---|---|---|---|"]
for r in cell_rows:
    t = r["mean_tokens"]
    md.append(f"| {r['config']} | {r['task']} | {r['n']} | {r['accept_rate']:.0%} | {f(r['mean_turns'],1)} | {f(t['input_tokens'])} | "
              f"{f(t['cache_read_input_tokens'])} | {f(t['cache_creation_input_tokens'])} | {f(t['output_tokens'])} | "
              f"{f(r['mean_cost_usd'],4)} | {f(r['mean_wall_s'],1)} | {f(r['mean_weighted_heuristic']/1e6,2)} | {r['model_ok_all']} |")
md += ["", "## Per config: measured meter vs local heuristic", "",
       "| config | runs | accept | cost $ | heuristic (M units) | 5h pts (segments) | 5h pts (per-run sum) | 7d pts (segments) | segments | 5h pts / M heuristic | 7d pts / M heuristic |",
       "|---|---|---|---|---|---|---|---|---|---|---|"]
for r in cfg_rows:
    fh, sd = r["five_hour"], r["seven_day"]
    md.append(f"| {r['config']} | {r['runs']} | {r['accept_rate']:.0%} | {r['cost_sum_usd']} | {r['heuristic_M']:.2f} | "
              f"{fh['segments_sum_pts']} (excl {fh['segments_excluded']}) | {fh['per_run_sum_pts']} | {sd['segments_sum_pts']} (excl {sd['segments_excluded']}) | "
              f"{fh['segments']} | {fh['pts_per_M_heuristic']} | {sd['pts_per_M_heuristic']} |")
md += ["", "If the /usage heuristic matched the real meter, `pts / M heuristic` would be the same for every config.",
       "Each segment carries up to +-1 pt of quantization error; trust a config's number only when its segment sum is several points.", ""]
open(os.path.join(HERE, OUTP + ".md"), "w").write("\n".join(md))
print("\n".join(md))

analyze.py on GitHub

कॉन्फ़िगरेशन:

• configs.json: चार workers, पूरे model ids के साथ और low block के लिए --effort low के साथ।

  configs.json

{
  "haiku-med": {"args": ["--model", "claude-haiku-5-5"], "model": "claude-haiku-5-5", "tier": 1, "note": "default effort"},
  "haiku-low": {"args": ["--model", "claude-haiku-5-5", "--effort", "low"], "model": "claude-haiku-5-5", "tier": 1},
  "sonnet":    {"args": ["--model", "claude-sonnet-5-5"], "model": "claude-sonnet-5-5", "tier": 3, "note": "default effort"},
  "opus":      {"args": ["--model", "claude-opus-5-5"], "model": "claude-opus-5-5", "tier": 5, "note": "default effort"}
}

configs.json on GitHub

tasks.json

{
  "explore":  {"inject": null,                     "mode": null,       "kind": "answer"},
  "tests":    {"inject": "inject/tests.patch",     "mode": "commit",   "kind": "code"},
  "lint":     {"inject": "inject/lint.patch",      "mode": "commit",   "kind": "code"},
  "commit":   {"inject": "reference/T1.patch",     "mode": "stage",    "kind": "answer"},
  "refactor": {"inject": null,                     "mode": null,       "kind": "code"},
  "review":   {"inject": "inject/review.patch",    "mode": "worktree", "kind": "answer"},
  "summary":  {"inject": null,                     "mode": null,       "kind": "answer"}
}

tasks.json on GitHub

ruff.toml

# Pinned lint config for the `lint` task: a small explicit rule set.
# Upstream tinydb @ 18d73a1 is clean under it (the two per-file ignores cover
# pre-existing F401s: the __version__ re-export and middlewares' Optional).
[lint]
select = ["F401", "F541", "F841", "E722"]

[lint.per-file-ignores]
"tinydb/__init__.py" = ["F401"]
"tinydb/middlewares.py" = ["F401"]

ruff.toml on GitHub

Planted बदलाव:

• inject/tests.patch: Query.__ge__ में एक तुलना पलटता है ताकि एक test फ़ेल हो।

  inject/tests.patch

diff --git a/tinydb/queries.py b/tinydb/queries.py
index cdab9fb..db00a2a 100644
--- a/tinydb/queries.py
+++ b/tinydb/queries.py
@@ -322,7 +322,7 @@ class Query(QueryInstance):
         :param rhs: The value to compare against
         """
         return self._generate_test(
-            lambda value: value >= rhs,
+            lambda value: value > rhs,
             ('>=', self._path, rhs)
         )
 

inject/tests.patch on GitHub

inject/lint.patch

diff --git a/ruff.toml b/ruff.toml
new file mode 100644
--- /dev/null
+++ b/ruff.toml
@@ -0,0 +1,6 @@
+[lint]
+select = ["F401", "F541", "F841", "E722"]
+
+[lint.per-file-ignores]
+"tinydb/__init__.py" = ["F401"]
+"tinydb/middlewares.py" = ["F401"]
diff --git a/tinydb/operations.py b/tinydb/operations.py
index 7101a23..f8541fe 100644
--- a/tinydb/operations.py
+++ b/tinydb/operations.py
@@ -10,6 +10,7 @@ This would delete the ``foo`` field from all documents where ``foo`` equals 2.
 
 
 from collections.abc import Callable, MutableMapping
+import sys
 from typing import Any, Union
 
 
@@ -18,7 +19,7 @@ def delete(field: str) -> Callable[[MutableMapping], None]:
     Delete a given field from the document.
     """
     def transform(doc: MutableMapping):
-        del doc[field]
+        removed = doc.pop(field)
 
     return transform
 
diff --git a/tinydb/queries.py b/tinydb/queries.py
index cdab9fb..489a2be 100644
--- a/tinydb/queries.py
+++ b/tinydb/queries.py
@@ -227,7 +227,7 @@ class Query(QueryInstance):
         :return: A :class:`~tinydb.queries.QueryInstance` object
         """
         if not self._path and not allow_empty_path:
-            raise ValueError('Query has no path')
+            raise ValueError(f'Query has no path')
 
         def runner(value):
             try:
diff --git a/tinydb/table.py b/tinydb/table.py
index f4027bf..eb8058c 100644
--- a/tinydb/table.py
+++ b/tinydb/table.py
@@ -818,7 +818,7 @@ class Table:
 
         try:
             raw_table = tables[self.name]
-        except KeyError:
+        except:
             # The table does not exist yet, so it is empty
             raw_table = {}
 
diff --git a/tinydb/utils.py b/tinydb/utils.py
index 43cad4e..07114d5 100644
--- a/tinydb/utils.py
+++ b/tinydb/utils.py
@@ -2,6 +2,7 @@
 Utility functions.
 """
 
+import os
 from collections import OrderedDict, abc
 from collections.abc import Iterator
 from typing import TypeVar, Generic, Union, Optional, TYPE_CHECKING, cast

inject/lint.patch on GitHub

inject/review.patch

diff --git a/tinydb/table.py b/tinydb/table.py
index f4027bf..4e06ba1 100644
--- a/tinydb/table.py
+++ b/tinydb/table.py
@@ -244,10 +244,21 @@ class Table:
 
         return list(iter(self))
 
-    def search(self, cond: QueryLike) -> list[Document]:
+    def search(
+        self,
+        cond: QueryLike,
+        *,
+        sort_by: Optional[str] = None,
+        reverse: bool = False,
+        limit: Optional[int] = None,
+    ) -> list[Document]:
         """
         Search for all documents matching a 'where' cond.
 
+        Results can optionally be sorted by a field (``sort_by``, descending
+        with ``reverse=True``; documents without the field always come last)
+        and truncated to the first ``limit`` results after sorting.
+
         Note: a repeated search with the same ``cond`` is served from an
         internal cache. The returned list is a fresh copy, but its
         documents are the same objects held by the cache -- treat them as
@@ -255,8 +266,34 @@ class Table:
         what a later identical search returns.
 
         :param cond: the condition to check against
+        :param sort_by: optional field name to sort the results by
+        :param reverse: sort in descending order
+        :param limit: optional maximum number of results
         :returns: list of matching documents
         """
+        if limit is not None and limit < 0:
+            raise ValueError('limit must not be negative')
+
+        # The cache always holds the plain, unsorted, untruncated result;
+        # _search returns a fresh list we are free to reorder/slice
+        docs = self._search(cond)
+
+        if sort_by is not None:
+            field = sort_by
+            present = [doc for doc in docs if field in doc]
+            missing = [doc for doc in docs if field not in doc]
+            present.sort(key=lambda doc: doc[field], reverse=reverse)
+            docs = present + missing
+
+        if limit:
+            docs = docs[:limit]
+
+        return docs
+
+    def _search(self, cond: QueryLike) -> list[Document]:
+        """
+        Plain search backed by the query cache. Returns a fresh list.
+        """
 
         # First, we check the query cache to see if it has results for this
         # query

inject/review.patch on GitHub

Prompts और ground truth:

• tasks/explore.md: query cache कहाँ invalidate होता है और कौन-से public methods उसे चलाते हैं।

  tasks/explore.md

Where does TinyDB invalidate its query cache, and which public methods trigger it? Give file paths and function names.

tasks/explore.md on GitHub

tasks/tests.md

pytest fails on this repo. Find the bug in the library code and fix it. Do not change the tests.

tasks/tests.md on GitHub

tasks/lint.md

ruff reports issues in tinydb/ (run `ruff check tinydb`). Fix them all without changing behaviour.

tasks/lint.md on GitHub

tasks/commit.md

Write the commit message for the staged changes. Output only the message.

tasks/commit.md on GitHub

tasks/refactor.md

I keep writing little lambdas to append things to list fields, so can you add two update operations to `tinydb.operations`, next to `increment`, `delete` and friends? `push(field, value)` should append `value` to the list stored in `field`, keeping the existing order, and `pull(field, value)` should remove every occurrence of `value` from that list, not just the first one. If a document doesn't have the field yet, `push` should create it as a one-element list, while `pull` should just leave that document alone without raising. If the field exists but isn't a list, `push` has to raise a `TypeError` instead of doing something weird with it. They should be used exactly like the existing operations, e.g. `db.update(push('tags', 'python'), User.name == 'john')`.

tasks/refactor.md on GitHub

tasks/review.md

Review the uncommitted changes for bugs and edge cases. List findings only.

tasks/review.md on GitHub

tasks/summary.md

Summarize how TinyDB is organised for a new contributor in at most 150 words: the main modules and what each owns.

tasks/summary.md on GitHub

tasks/explore.truth.md

# explore: ground truth (tinydb @ 18d73a1)

- The query cache is `Table._query_cache`, an `LRUCache` (tinydb/utils.py) created in
  `Table.__init__` (tinydb/table.py, `query_cache_class = LRUCache`, capacity
  `default_query_cache_capacity = 10`). It is filled in `Table.search`.
- Invalidation happens in one place: `Table._update_table()` (tinydb/table.py) ends with
  `self.clear_cache()`, and `Table.clear_cache()` does `self._query_cache.clear()`.
- Every public write goes through `_update_table`, so these trigger it:
  `insert`, `insert_multiple`, `update` (all three branches: callable/fields, by doc_ids, by cond),
  `update_multiple`, `upsert` (via `update` or `insert`), `remove` (by cond or doc_ids), `truncate`.
  `Table.__init__(persist_empty=True)` also calls it. `clear_cache()` itself is public.
  `TinyDB` (tinydb/database.py) forwards these via `__getattr__` to the default table;
  `drop_table`/`drop_tables` do not clear a cache, they forget the `Table` instances (`self._tables`).

## Check rule (check.py explore)
PASS iff the final answer
1. contains `table.py`, AND
2. contains `clear_cache` or `_query_cache`, AND
3. names at least 3 distinct methods among
   insert, insert_multiple, update, update_multiple, upsert, remove, truncate (word match, case-insensitive).

tasks/explore.truth.md on GitHub

tasks/review.truth.md

# review: planted defect and matching rule

The uncommitted change is the reference T4 patch (`Table.search(cond, *, sort_by, reverse, limit)`)
with one defect: the truncation guard is `if limit:` instead of `if limit is not None:`, so
`search(cond, limit=0)` returns ALL matching documents instead of `[]` (0 is falsy).
Everything else in the change is correct (missing-field docs last in both directions,
negative limit -> ValueError, cache keeps the plain result).

## Check rule (check.py review)
PASS iff the final answer (case-insensitive) matches at least one of:
- `limit\s*=?=\s*0` (e.g. "limit=0", "limit == 0")
- `limit\s+(of\s+)?(0|zero)\b` (e.g. "limit of 0", "limit zero")
- `\b(0|zero)\s+limit` / `zero[- ]limit`
- `limit` within 150 characters of `falsy|falsey|truthy|truthiness` (either order)
- `if limit:` together with `0` or `zero` within 150 characters
Decoy answers (generic review, or only the sort order) must FAIL.

tasks/review.truth.md on GitHub

सोमवार को यह कीजिए

• Claude Code 2.1.257 या बाद का चलाइए, ताकि force flag मौजूद हो, और 2.1.243 या बाद का, ताकि /tasks हर subagent का मॉडल दिखाए।
• तय कीजिए कि आपको कौन-सी फ़ाइल चाहिए: सिर्फ़ explorer के लिए Explore फ़ाइल, हर subagent के लिए env block।
• Explore.md को .claude/agents/Explore.md के रूप में सहेजिए, या env block को .claude/settings.json में मिला दीजिए।
• पूरा id claude-haiku-5-5 इस्तेमाल कीजिए, haiku alias नहीं, ख़ासकर Bedrock, Vertex या Foundry पर।
• एक session शुरू कीजिए, Explore agent से codebase की खोज करवाइए, फिर /tasks खोलकर पढ़िए कि वह किस मॉडल और effort पर चला।
• जो भी subagent जटिल agentic coding करता है, उस पर Sonnet 5.5 या Opus 5.5 रखिए, और अगर ऐसा agent आपने जानबूझकर पिन किया है तो FORCE सेट मत कीजिए।
• हमारे table पर अपनी repo के लिए भरोसा करने से पहले अपने सात काम harness से दोबारा चलाइए।
• /usage को "कौन-सी skill या agent मेरे local history को खा रहा है" के रूप में पढ़िए, quota के पूर्वानुमान के रूप में नहीं।

माप

plan meter, हर headless run के rate_limit_event से पढ़ा गया, एक point की resolution पर। हर block सतत है, और account को और कुछ नहीं छू रहा था। साप्ताहिक window सभी 182 runs पर 67% पढ़ा गया।

Matrix का सार: 182 runs, पूरा matrix list price में लगभग $4.09, एक five-hour window के दो point और हफ़्ते के शून्य point। हर block में rounding का एक point तक हो सकता है, इसलिए Sonnet का एक point और Opus का एक point अलग नहीं पहचाने जा सकते। इस meter पर किसी भी मॉडल के लिए एक सीमित काम दिखता ही नहीं। Claude Code में तीनों मॉडलों की प्रति-turn सापेक्ष लागत पर support page कोई गुणक नहीं देता s9।

स्थानीय breakdown के सामने, प्रति config:

/usage breakdown आपकी मशीन के local history से निकलता है s3। client binary से पढ़ा गया इसका फ़ॉर्मूला model tier गुणा record weight है। Model tier: Fable 10, Opus 5, Haiku 1, बाकी सब 3। Record weight: cached tokens 1, uncached input 10, cache creation 12.5, output 50। ये client के local attribution weights हैं, billing नहीं, और इस बात का प्रमाण नहीं कि server आपका quota कैसे गिनता है। यह अंतर दिखता है: Haiku medium block का स्कोर 29.48 है और Opus block का 27.30, यानी panel Haiku block को Opus block से ऊपर रखता है, जबकि असली meter ने शून्य बनाम एक point पढ़ा।

एक account के logs के एक महीने ने tokens कहाँ रखे: 2,631 subagent runs, 42 Haiku 4.5 पर, subagents सभी tokens का 48.1%, weighted cost में Haiku 4.5 का 0.27%, और मॉडल कोई भी हो, subagent launch prompt का median 47,117 tokens।

उसी स्वीकार हुए feature (refactor) पर effort भूख बदलता है: Haiku medium ने 13.7 turns में 232,985 cache-read tokens पढ़े, Haiku low ने 7.6 turns में 139,801, Opus ने 8.3 turns में 87,612।

और आगे

• 100k की कीमत की खाई: 100,000 prompt tokens के ऊपर ऊँची दर पूरे request पर लागू होती है, सिर्फ़ अतिरिक्त हिस्से पर नहीं s8। cache tokens 100k में गिने जाते हैं या नहीं, यह उस लेखक की पढ़त है, उद्धृत नियम नहीं। यह API-key की अर्थव्यवस्था के लिए मायने रखता है, subscription के meter के लिए नहीं। Pricing page: s20।
• Tokenizer: वही text Haiku 5.5 पर Haiku 4.5 से लगभग 30% ज़्यादा tokens गिनता है, और API पर adaptive thinking डिफ़ॉल्ट रूप से चालू है, लीवर effort है s12।
• Effort और प्रति काम tokens: एक स्वतंत्र run ने max effort पर प्रति Intelligence Index काम लगभग 162k output tokens मापे, और high effort पर लगभग 55k, स्कोर 38 के लिए s7।
• अपने आँकड़ों और अपनी चेतावनी के साथ एक पूरा practitioner setup, कि बचत छोटे main context से आती है: s4।
• खुला issue #100082, कुछ builds पर subagent frontmatter का मॉडल नज़रअंदाज़: s10।
• खुला issue #100154, subagent transcripts output_tokens कम दर्ज करते हैं, उसी brief के लिए 461 बनाम 15,575, server-side billing पर असर नहीं s11। चूँकि /usage local history से output को 50 से गुणा करता है, यह कमी panel तक पहुँचती है।
• यहाँ जाँचा नहीं गया: main मॉडल के रूप में Haiku (स्रोत Haiku 5.5 को subagent के रूप में पेश करते हैं), gateway के ज़रिए cross-provider मॉडल मिश्रण (Claude Code settings नहीं), 1M-context prompts, और Max plans के साथ आने वाले API credits (Claude Platform, Claude Code quota नहीं)।

FAQ

क्या CLAUDE_CODE_SUBAGENT_MODEL सेट करने से Explore Haiku पर आ जाता है?

नहीं। अकेले यह built-in Explore और Plan subagents को नहीं बदलता s2। Explore agent फ़ाइल जोड़िए, या force flag।

क्या FORCE मेरे Opus subagents को तोड़ देगा?

यह हर subagent पर एक ही मॉडल लागू करता है, इसलिए हाँ: जिस agent को आपने जानबूझकर Opus पर पिन किया था वह भी Haiku पर चलेगा। अगर सिर्फ़ explorer सस्ता होना चाहिए तो Explore फ़ाइल इस्तेमाल कीजिए।

क्या Haiku 5.5 subagent के काम के लिए काफ़ी अच्छा है?

एक छोटी repo पर इन सात सीमित कामों के लिए यह low effort पर 70 में से 70 बार और medium पर 70 में से 70 बार स्वीकार हुआ। Anthropic खुद जटिल agentic coding के लिए Sonnet 5.5 और Opus 5.5 रखता है s1, और Terminal-Bench 4.0 का अंतर 39.2% बनाम 70.6% है।

यह बदलाव plan quota का कितना हिस्सा लौटाता है?

एक सीमित काम के लिए, किसी भी मॉडल पर meter के एक point से कम: पूरे 182-run matrix ने five-hour window को दो point और साप्ताहिक window को शून्य हिलाया। असली हफ़्ता main context और हर subagent के साथ चलने वाले launch prompt में जाता है।

क्या मैं और बचाने के लिए thinking बंद कर सकता हूँ?

Claude Code में इन मॉडलों पर नहीं s21। Effort ही बचा हुआ dial है।