Make the model behave
What an LLM request really is, what it costs, and why the model is the smallest problem.
The Index · 19 Entries
Journal
Field notes on agents, voice, retrieval, evals, and the parts of production AI that only show up after software meets real people.
Explore the interactive studies19 experiments
Change an input, inspect the evidence, and follow the result. Every study lives beside the article it explains.
InferenceRequest latency workbench What Actually Happens When You Call an LLM APIInferenceToken work and memory Why Output Tokens Cost More Than InputInferenceProvider contract inspector Running Two Model Providers in ProductionInferenceVoice architecture tradeoffs Trading Voice Quality for ReliabilityInferenceFirst-divergence forensics The Bug Was in the TranscriptData & evaluationRetrieval ranking laboratory Search for Languages the Internet ForgotData & evaluationCaption evidence inspector BLEU LiesData & evaluationMessage replay and lineage Three Thousand WhatsApp Messages a DayData & evaluationEntity resolution bench The Unreasonable Effectiveness of Cleaning DataData & evaluationRelease gates and judge calibration Evals Are the Operating SystemSystemsAgent controller simulator Stop Being the Thing Inside the LoopSystemsConfidence and authority map Where the Human Belongs in the LoopSystemsClaim provenance ledger Writing Down What You Didn't MeasureSystemsWorkflow observation board The First Two Weeks Are for ListeningSystemsProduct boundary inspector The Model Is the Easy PartSystemsIncident to regression workbench Golden Cases Come From FuneralsSystemsException recovery simulator Demos Close Deals. Boring Systems Keep Them.ReflectionA private reflection workspace Own the Failure Before It Becomes a StoryReflectionSix connected project questions Principles I Reread Before Starting Anything New
Reading Paths
Start Here · 4 RoutesEvals as a discipline
Measurement as the operating system: where golden cases come from and what metrics hide.
Agents in the wild
Voice, loops, and the judgment calls about where people belong in automated systems.
The craft
Working philosophy: listening first, owning failures, and shipping boring systems.
RegisterShowing 19 of 19
Lead EntryEngineering6 Min
Stop Being the Thing Inside the Loop
A production agent loop needs external checks, durable state, and hard limits. Without those, it is repeated prompting with a budget.
Engineering4 MIN
Golden Cases Come From Funerals
Our golden cases are anonymized production failures with assertions attached. Keeping the original thread, trace, and expected behavior has made them more useful than cases invented before launch.
Opinion6 MIN
The Model Is the Easy Part
In production, the model call sits inside permissions, policy, retries, evals, and traces. Those surrounding systems decide whether a promising demo survives contact with a customer workflow.
Opinion3 MIN
The First Two Weeks Are for Listening
Before I build for a new client, I watch the people doing the work: their tabs, workarounds, retyping, and judgment calls. Those first two weeks often change what we build first.
Engineering6 MIN
Evals Are the Operating System
A freight-negotiation agent had become too eager to close, and we could not measure the cost cleanly. An 87-test suite with 19 production cases gave us a release gate for the next model migration.
Engineering5 MIN
The Bug Was in the Transcript
A voice-agent complaint can originate in speech recognition, model decisions, tool execution, or turn timing. I read the transcript and event trace in that order before touching the prompt.
Engineering6 MIN
Trading Voice Quality for Reliability
A recruitment voice agent sounded natural but kept missing tool calls. We replaced one fused speech model with a slower, inspectable pipeline and measured whether reliability justified the extra latency.
Engineering5 MIN
Three Thousand WhatsApp Messages a Day
A logistics customer receives about 3,000 WhatsApp messages and 500 emails a day. We turn them into CRM demand records with a queue, an LLM extractor, entity resolution, deduplication, and a review lane.
Opinion3 MIN
Where the Human Belongs in the Loop
Reviewing every AI output wastes the attention human oversight is meant to provide. Route uncertain and irreversible cases to people, then keep their corrections as evaluation data.
Writing3 MIN
Writing Down What You Didn't Measure
I keep a small file of project claims beside the evidence for each one. It preserves the difference between a measured result, an estimate, and something we believe but have not tested.
Opinion3 MIN
Demos Close Deals. Boring Systems Keep Them.
Ninety days after a successful AI demo, clients ask about failed batches, monthly cost, and recovery paths. Renewals depend on the answers.
Engineering7 MIN
What Actually Happens When You Call an LLM API
An LLM request crosses rate limits, tokenization, scheduling, inference, streaming, and billing before your application sees a response. That fuller model makes 429s, latency drift, and surprising bills easier to debug.
Engineering6 MIN
Running Two Model Providers in Production
We run OpenAI and Gemini across client deployments. Prompts need provider-specific work; the eval cases, contracts, and traces are what let us move safely between them.
Writing2 MIN
Own the Failure Before It Becomes a Story
After watching a friend write carefully about a breakup, I started treating personal failures more like postmortems: record what happened before memory makes my role smaller.
Engineering5 MIN
Why Output Tokens Cost More Than Input
Output generation is sequential and keeps cached state in GPU memory while a response runs. That mechanism helps explain the price gap and changes how I set output limits, prompt order, and streaming.
Writing3 MIN
Principles I Reread Before Starting Anything New
Six notes I reread before a new project or commitment. Each comes from a decision that cost more than writing the rule down would have.
Engineering3 MIN
The Unreasonable Effectiveness of Cleaning Data
A logistics customer's transporter table had missing identifiers, aliases, and duplicate companies. Cleaning it improved the downstream systems that depended on those records.
Research3 MIN
BLEU Lies
My image-captioning model scored ROUGE-L 39 and BLEU 10.3 on the same outputs. Reading the captions showed where both numbers were misleading.
Research3 MIN
Search for Languages the Internet Forgot
My Master's work tested dense retrieval where training pairs, benchmarks, and web text were scarce. The hardest part was building evidence that retrieval worked in the target languages.