How it works
The model call is the easy part. Everything below exists because a model can hand you something that looks right and isn't, and something has to catch it. Every number on this page comes from the code it describes.
ticket
│
├─ 1 intake ............ Zod-validated at the boundary
│
├─ 2 requirements ...... every citation checked against the ticket's own words
│
├─ 3 planning .......... steps cite the requirement ids they satisfy
│
├─ 4 checks ............ objective, free, run before any judge
│ └── fail ──► back to 3 (bounded)
│
├─ 5 evaluation ........ rubric judge, evidence required per score
│ └── below threshold ──► back to 3 (bounded)
│
└─ 6 approval .......... stops for a human
└── rejected ──► back to 3, with their note as the instructionA workflow, not an agent. It comes down to one question. Who decides what happens next, your code or the model?
The control flow is TypeScript. The model has no tools, doesn't pick what happens next, and doesn't decide when it's finished. It gets asked three questions, and I check every answer before sending the next one.
Each call starts fresh. Planning only knows about the requirements because my code puts them in the prompt. Nothing is remembered between calls.
| This system | An agent | |
|---|---|---|
| Who sequences the work | Your code, decided before the run starts | The model, decided during the run |
| Tools | None. It returns text and that's it. | Calls them whenever it wants: read, write, search, execute |
| Termination | The state machine hits an end state | The model decides it's done |
| What the trace looks like | A state machine you could draw before it runs | A transcript you read afterward to find out what it did |
Each one covers what it produces, what it stops from going wrong, what I can actually show for it, and the code behind it.
Checks the ticket against a schema before anything else runs.
Why it works this way. Bad input should fail here instead of two model calls later. Acceptance criteria are optional on purpose, since a ticket without them is incomplete rather than invalid, and pointing that out is a later stage's job.
Turns the ticket text into explicit requirements, implied ones, open questions, and what's missing.
Why it works this way. A schema only checks the shape of the response. It can't tell you whether the content is true. A model can return perfectly valid JSON that cites a requirement the ticket never contained. So every explicit requirement has to include a quote, and I check that the quote really appears in the ticket. I ignore whitespace and capitalization, so a quote that runs across a line break still passes, but a reworded one gets rejected. Constrained decoding wouldn't help here, because the JSON was already valid.
Produces an approach, ordered steps, a test strategy, risks, and what's out of scope.
Why it works this way. Each step has to list the requirement ids it covers, which lets me check coverage in normal code instead of asking a model. It does open up a way to cheat, since a model could make up ids, so a step is only allowed to use ids that already exist.
Six checks over the plan that don't need a model: coverage, test strategy, orphan steps, risks without mitigations, whether open questions carried through, and whether there are steps at all.
Why it works this way. If I can check something objectively, I check it in code. Asking a model whether every requirement is covered would be slower, cost money, and sometimes get the counting wrong. These run before the judge, so I never pay to evaluate a plan I already know is incomplete. Warnings don't block anything, since they're judgment calls and belong to the person reviewing.
An LLM judge scores 5 criteria from 1–5 against written anchors, with mandatory evidence per score. Passing threshold: 3.5.
Why it works this way. Only the parts that really need judgment go to the model. The judge has to score every criterion once. Without that rule it can skip the one it would score badly, and the average goes up. The JSON still looks fine, so you wouldn't notice unless you check.
The run stops and waits. A person approves it, or rejects it with a reason.
Why it works this way. This is the part I think matters most. The workflow never approves itself. A model can write something that looks right faster than anyone can read it, so a person has to check it before the work gets built. I tried to make that check quick by handling the boring parts first. The quotes are already verified and the coverage is already counted, so the reviewer can spend their time on what only a person can catch. Rejecting doesn't kill the run either. It goes back to planning, and whatever the reviewer wrote becomes the instruction for the next plan.
Both get called “retry”, but they fire for different reasons, feed back different things, and have different limits. Lumping them together hides what actually happened in a run.
Anything I can check objectively gets checked in code. Requirement coverage is just counting, and asking a model to do it again would be slower, cost more, and sometimes get it wrong. These 5 criteria are the parts that really need judgment.
| Criterion | Question the judge answers |
|---|---|
| Approach soundness | Is the proposed approach a reasonable way to satisfy the requirements? |
| Unsupported assumptions | Does the plan invent constraints, behaviour, or context the ticket and requirements never established? |
| Test strategy quality | Would the stated tests actually catch a regression in the behaviour being changed? |
| Scope discipline | Does the plan stay within what was asked? |
| Risk awareness | Does the plan identify the risks that actually matter for this change, including security and accessibility where relevant? |
Everything above was built against a mock, using responses I’d guessed at. These came out of real runs, and a few of them are mistakes in my own assumptions rather than anything the model did.
One full run on the same ticket. These come from the same pricing table the app uses, so they can't drift from what the run report shows.
| Model | Cost per run | Note |
|---|---|---|
| Haiku 4.5 | $0.017 | Measured ~28s per run. What the recordings and eval numbers use. |
| Sonnet 5 | $0.075 | Measured 4.4x Haiku's cost and ~45s per run. |
| Opus 5 | $0.220 | Measured 13x Haiku and ~98s per run. Thinking is on by default and counts toward output. |
Worth writing down, because knowing what's missing is part of the job. This plans the work. It doesn't write the code.