I spend my days getting models to behave in the real world: agents, voice, retrieval, evals. This is where I think out loud about that craft, and everything around it.
Stop Being the Thing Inside the Loop
A production agent loop needs external checks, durable state, and hard limits. Without those, it is repeated prompting with a budget.
Save an entry, put it in order, and tick it off when you’re done.
Browse the journal
A freight-negotiation agent had become too eager to close, and we could not measure the cost cleanly. An 87-test suite with 19 production cases gave us a release gate for the next model migration.
An LLM request crosses rate limits, tokenization, scheduling, inference, streaming, and billing before your application sees a response. That fuller model makes 429s, latency drift, and surprising bills easier to debug.
Output generation is sequential and keeps cached state in GPU memory while a response runs. That mechanism helps explain the price gap and changes how I set output limits, prompt order, and streaming.
A production agent loop needs external checks, durable state, and hard limits. Without those, it is repeated prompting with a budget.
Our golden cases are anonymized production failures with assertions attached. Keeping the original thread, trace, and expected behavior has made them more useful than cases invented before launch.
In production, the model call sits inside permissions, policy, retries, evals, and traces. Those surrounding systems decide whether a promising demo survives contact with a customer workflow.
Before I build for a new client, I watch the people doing the work: their tabs, workarounds, retyping, and judgment calls. Those first two weeks often change what we build first.
A voice-agent complaint can originate in speech recognition, model decisions, tool execution, or turn timing. I read the transcript and event trace in that order before touching the prompt.
A recruitment voice agent sounded natural but kept missing tool calls. We replaced one fused speech model with a slower, inspectable pipeline and measured whether reliability justified the extra latency.
Occasional notes on building production AI and the craft of shipping systems that hold. No cadence promised; nothing you didn’t ask for.
Atom, so it works in any feed reader. Every entry lands there the moment it publishes.
