Brian Canlas

I've shipped AI features in production. The model call was never the hard part. The hard part was knowing whether I could trust what came back, and being able to show someone else why.

So I built the layer that checks it. Every response is validated against a schema, and if it fails, it goes back to the model with the errors so it can try again. Anything I can verify in normal code, I verify in normal code. I only use an LLM to judge the parts that really need judgment. Every run records what it cost and how long it took.

It also stops and waits for a person before anything is approved. A model can write something that looks right faster than anyone can read it, so someone has to check it before the work gets built. I tried to make that check quick by handling the boring parts first.

I picked turning tickets into plans because the output is something you can actually check. The part I wanted to build was the harness around the model call.

You can try it yourself. The example tickets replay real recorded model output, so they cost nothing to run. There's also a walkthrough of how each stage works and why I built it that way.