I watched a team spend a quarter dropping their per-call latency by 40%. The bill kept going up. They were proud of the latency win — the dashboards were beautiful — but they were optimizing the wrong axis. Their bottleneck was not speed. It was that they were using a frontier model to do regex.
There are three axes most agent fleets can be tuned on. In rough order of how often teams pick the wrong one:
- Latency — how fast a single call returns.
- Accuracy — how often the call is right.
- Routing — which model handles the call in the first place.
Teams obsess over latency and accuracy, because they’re the axes the dashboard renders. The axis that actually moves the bill is routing.
The shape of the actual cost
Run a profiler on your agent fleet. Not a latency profiler — a token profiler that tells you how many tokens are flowing through which model at which price tier.
The shape will surprise you. In every fleet I have looked at, somewhere between 60% and 80% of the volume is doing things that an old GPT-3.5-class model could have done for one-twentieth the cost. Format a response. Pick from a list. Confirm a yes/no. Restate a parameter. The frontier model is invoked because the team didn’t want to write a router, and the router is what would have saved them.
What good routing looks like
A good router is boring. It’s a function. The function reads the request and decides which model gets it. Sometimes it’s a small finetune. Sometimes it’s a regex. Sometimes it’s a single line of code that says if intent in {"yes", "no"}: return cheap_model.invoke(prompt).
A bad router is “we’ll add caching later”. Caching is not routing. Caching helps when you ask the same question twice. Routing helps when you ask different questions of different difficulty. The bill is shaped by the second case, not the first.
The diagnostic
Open last month’s API invoice. Divide the total cost by the number of agent invocations. If the number is bigger than half a cent, you are probably calling a frontier model for things that did not need one.
If the number is bigger than five cents, your fleet is on fire and you are paying for the smoke.