Shopify's 4 Gates In Claude Code: Only The Tests Paid Off

A gate, in the sense Shopify uses it, is a check an AI-written change must pass before it can move on. Rebuilt in Claude Code, only the first one paid off: the acceptance tests. Shopify published four gates and no numbers, so this experiment builds them on a real repository, runs fifty requests with each gate switched off in turn, and counts what each one caught and what it cost.

Helix went public on September 21, 2026. It puts every checkpoint through four gates in order: tests, design review, two reviewers, and an engineer. The post gives no pass rate. The reviewers are the part most people copy, and on these fifty runs the pass rate moved only when the tests were in the loop. The reviewers added nothing, at roughly three times the cost.

Two other findings come out of the same bench. A hook that supposedly cannot be overridden has three exits in its own documentation. And when the tests and the reviewers disagreed, the agent did something nobody had planned for. The counting starts from the case with no gate at all.

The bench repository is tinydb, a real document database, with 226 tests green on the commit used.

Helix is Shopify's internal tool for moving its mobile apps back to native code. Its post states the rule bluntly: each checkpoint goes through four gates, in order.

Gate	What it does in Shopify's post
Behaviour	Runs tests where every relevant case has to pass
Design	Asks a second model to act as a perfectionist design reviewer
Review	Two independent, context-isolated reviewer agents; the loop repeats until both approve
Engineer	A person closes the loop

The sentence that sells the design is "An attempt is allowed to be wrong. It is not allowed to ship until it isn't." The design gate drops out for a library, and the post says a logic-only change simply skips it.

What the post never gives is a number. The Shop app was rebuilt in 12 weeks and the Shopify app has more than 300 screens, and that is the whole ledger. There is no pass rate, no cost and no count of defects that escaped.

The bench is five requests typed the way a developer types them: push and pull operations, a between query, a table rename, a sorted search and a write cache size. Behind each request sit five hidden requirements the agent never sees, the things a careful senior engineer would expect. That makes 25 requirements in total.

The worker is Sonnet 5.5, run headless. Five configurations times five tasks times two repetitions gives 50 runs. One caveat applies: the explicit versions of these requests pass in one shot, so the prompts were shortened to what people actually type.

With no gate at all, 35 of the 50 requirements pass. That is 70%, with three runs out of ten fully clean.

The sort task shows what the misses look like. The agent indexed every document with the sort field, and as soon as one document lacks it the whole search crashes. Five requirements, zero passed, one missing key.

A Stop hook is a script Claude Code runs every time the agent tries to end its turn. Anthropic's own guide names the problem it addresses: Claude stops when the work looks done. Without a check it can run, "looks done" is the only signal available, and you become the verification loop.

The hook lives in the project's settings file, under hooks, Stop, as one command with a timeout. Here it is a single-line wrapper that calls the gate script. The contract is the exit code: 0 lets the turn end, 2 blocks it, and whatever the script printed to standard error goes back to the agent as the reason.

In practice it works like this. The agent sums up its change and stops. The hook runs the acceptance tests, one fails, and the turn does not end. The agent reads the failure as hook feedback, in this case a test that expected a type error and did not get one, fixes the code and stops again. This time the gate is quiet.

A rule in the prompt is advice. A hook is not. Yet the same documentation page lists three ways out.

Exit	What the documentation says
The cap	After stop hooks have continued the turn 8 times in a row, Claude Code overrides the next block and ends the turn. The count resets each time the agent calls a tool, and an environment variable raises it.
The timeout	600 seconds by default for a command hook, and a hook that times out renders no decision. A slow test suite opens the gate on its own.
The flag	The hook input carries stop_hook_active, true when the session is already continuing because of a stop hook. The docs say to check it so you never block on a condition that cannot resolve.

The gate here logs that flag and ignores it. It gives up on its own after six blocks, so the loops stay countable.

There is one more exit that nobody documents. After being blocked once, the agent went looking for the tests: it ran cat on the hook script and searched the whole disk for the test file. Only the allow list stopped it. So the acceptance tests live outside the repository tree, the settings deny reads of the hook and of the hidden tests, and the agent sees the failure but never the file.

The acceptance-test gate is a set of tests, written independently of the implementer, that define what "done" means. Shopify's first gate is not the agent's own tests. A subagent generates integration-style test cases per checkpoint, pushing for edge cases beyond the happy path, and those tests define what proven means.

The writer here is therefore a separate headless Sonnet. It sees the request and the repository before the change, read-only, cannot run code, and is told to write one test per behaviour a careful senior would expect: missing fields, wrong types, boundaries, persistence, the cache, and existing behaviour unchanged. It writes between 23 and 53 tests, 39 on average, in about 40 seconds. The file lands outside the repository, so when a test fails the agent reads the assertion and never the test.

Tests alone take the pass rate from 70% to 84%: 42 of 50 requirements, and six fully clean runs instead of three.

Task	Without tests	With tests
Sorted search	5 of 10	10 of 10
Write cache size	8 of 10	10 of 10
Push and pull	4 of 10	4 of 10

The missing-field crash in the sort task is exactly the kind of case the writer asks about. For the cache task, the writer demanded a value error for a cache size below one.

The push and pull task does not move. It misses three requirements, two of them about a document without the field, and the request never said anything about that. The writer hedges exactly where the prompt is silent: its test accepts a raise or a silent no-op, so the gate cannot recover a requirement nobody stated.

Across the 30 gated runs with tests, the gate sent the agent back 28 times with a real requirement failing, and five times with all of them passing. One of those five was the rename task: the writer wanted names that are not strings rejected, nobody had asked for that, and the agent lost a round to it.

The first version of this gate reran the implementer's own tests. It allowed every run on its first stop while a hidden requirement failed, because an agent's own tests test what it built.

The price of the gate is $0.20 a run instead of $0.085, and 60 seconds instead of 33.

The review gate is a pair of reviewer agents, each working in a fresh context. Shopify's wording: two independent, context-isolated reviewer agents check the new code, every finding has to be fixed, and the loop repeats until both reviewers approve. Anthropic's guide explains why isolation matters: a reviewer in a fresh context sees only the diff and the criteria you give it, not the reasoning that produced the change.

Each reviewer here is a headless Sonnet with read-only tools and at most 15 turns. It returns a verdict as structured JSON: an approve flag and findings, each with a severity of blocking or minor. Two reviewers run in parallel on the request and the full diff, their findings are combined, and the gate blocks when either one finds something blocking. A pair costs between 3 and 6 cents per stop.

Configuration	Requirements passed	Pass rate	Blocks
No gate	35 of 50	70%	none
Reviewers alone	34 of 50	68%	1 in 10 runs
Tests alone	42 of 50	84%	6 in 10 runs (3 right, 3 false)
Reviewers after the tests	41 of 50	82%	2 in 10 runs, both right

Reviewers alone land two points under no gate at all, and reviewers after the tests land two points under the tests alone.

The sort task explains the number. In one run both reviewers blocked: sorting indexes the document, one document without that field raises a key error, and the whole search fails. The agent fixed it and passed five of five.

In two other runs both reviewers saw the exact same crash and rated it minor. A document without the sort field raises a bare key error, and the docstring documents this. Both reviewers approved, and both runs ended at zero of five. Same model, same diff, same bug, two different severities. The reviewers were never wrong when they blocked. The problem is what they wave through.

A paper from August measured the same shape on a public benchmark. Zero-shot scored 77%, one reviewer 77%, and two independent reviewers 75%, all within noise of no review at all.

Setup on LiveCodeBench	Score
Zero-shot	77%
One reviewer	77%
Two independent reviewers	75%
Reviewer plus critic (adversarial review)	87%

Adversarial review is a reviewer paired with a critic that audits the review through structured disagreement before the main agent edits. The same paper found it is the shape that works. It scored 87% where two reviewers scored 75%, and it beat the strongest five-agent baseline at 82%.

Benchmark	Zero-shot	With adversarial review	Cost
SWE-bench	71.6	75.2	about 4.5x the tokens

So the critic went into the gate. Reviewer A reviews. Critic B gets the findings plus the diff and stamps each one upheld or rejected. Only an upheld blocking finding closes the gate.

On this repository the result was 39 of 50, or 78%. The critic pair blocked zero times in ten runs. During validation it did reject a false finding that the two-reviewer shape would have blocked on.

What the critic could not fix is the loop. Three runs hit six blocks on the same acceptance test, and the gate gave up with the real requirement still failing at every stop. In two of them the blocking test was the writer's own mistake, and the agent said so: it would not modify the code for a test it could not read.

The third is the one to remember. The acceptance test wanted a failing update to leave every document untouched, so the agent rewrote the table's update method. The reviewers blocked that, because the change to the update method was not requested and the task was two new functions. The agent reverted. Its last message said the acceptance test and the code review were asking for opposite things, so it had stopped without changing anything. Six blocks later the counter let it out.

The give-up point here sits at six. Claude Code's own cap is eight consecutive blocks and resets on every tool call, so it never fired once in fifty runs.

The fourth gate is the engineer, and Shopify's post promises the gates do not become more lax when nobody is watching. What reaches the engineer is whatever the other gates let through. The ledger below answers which gate paid for itself.

Configuration	Passed of 50	Pass rate	Cost per run	Left for the human
No gate	35	70%	$0.085	15
Acceptance tests	42	84%	$0.20	8
Reviewers alone	34	68%	$0.15	16
Tests with reviewers	41	82%	$0.28	9
Tests with reviewer plus critic	39	78%	$0.28	11

Every gated row takes 54 to 86 seconds per run, against 33 for no gate.

Blocks across all fifty runs break down as follows. The tests sent the agent back 28 times rightly and five times wrongly. The reviewers blocked three times, never wrongly, and three loops gave up. The whole matrix cost $9.95 of list-price API spend as the CLI reports it, and about 3,136 seconds, or 52 minutes, of summed worker time.

The verdict for a repository like this one: keep the test gate, with a writer that is not the implementer and tests the agent cannot edit. Expect about one false block in six, and give the loop a counter. The two-reviewer gate as Shopify wrote it did not move the pass rate here, in either shape. If you keep it, keep it as advice in the feedback, not as a bar.

One trap on the way. The docs describe a bare flag that skips hooks, skills, plugins and memory for scripted calls. On a subscription login it reads only an API key and returns an error, so each reviewer is isolated instead with an empty settings source and hooks disabled.

The limits are real: one repository, one worker model, five tasks, two repetitions per cell. A gap of two points between rows is noise. The fourteen points between no gate and the test gate is not. The design gate was not built, and the engineer was not timed.

Of Shopify's four gates, on this repository, only the tests paid off. The whole kit is in the free pack: the settings file with the hook and the deny list, the stop script, the gate, the writer and reviewer commands with their schemas, and the ledger template.