I was on a call with a CTO who runs a developer-tools company. They started with GPT-4 for everything. Over the last six months they moved seventy percent of their volume to a local model running on a single H100. The remaining thirty percent is on Claude or Gemini, depending on the task. Their inference bill went from thirty-four thousand dollars a month to twelve hundred.
This is not a story about whether local models match frontier on paper. They don’t, on paper. It’s a story about where the actual call volume goes once a product ships. The frontier handles the hard five percent — the open-ended generation, the genuinely novel reasoning, the user-facing chat. The other ninety-five percent is classification, extraction, formatting, retrieval reranking, summarisation of structured input. A 70B-class model finetuned on the company’s own traffic for an afternoon will outperform a generic frontier model on those tasks, because the eval set is the company’s actual data.
What changed
Three things made the math obvious in the last twelve months:
- Open-source quality. The current generation of open weights closed enough of the quality gap on standard tasks that the cost difference stopped being defensible.
- Inference engines. vLLM and friends made batched throughput on a single GPU competitive with managed inference at lower volumes than people realised.
- Finetuning got cheap. A QLoRA on five thousand of your own examples, on a rented A100 for three hours, beats prompt engineering for almost any narrow task.
The diagnostic
Pull a week of your highest-volume agent endpoint. Categorise the requests by task type. If sixty percent or more fall into one or two narrow categories, you are paying frontier prices for a problem a finetune solves better.
The frontier model will still be in your stack. It just shouldn’t be in your hot path.
The strongest case against thisManaged frontier APIs may keep the long tail
The local-model argument underweights operations, compliance, and opportunity cost; many teams may rationally keep paying frontier prices for boring calls.
- What would make me wrong
- Managed frontier APIs stay cheaper in total cost for narrow production tasks once staffing, reliability, compliance, eval maintenance, and incident response are included.
- Last reviewed
The strongest case against the local-model thesis is that “cost per token” is not total cost. Running a local or fine-tuned model means owning capacity planning, eval drift, GPU availability, model upgrades, prompt compatibility, incident response, and security review. The fully loaded cost can exceed the API bill even when raw inference is cheaper.
There is also a product-risk argument. A frontier provider can improve quality and latency underneath you. A local model can become a forked dependency that someone has to keep alive. For a small team, paying a managed provider may be the right decision because the alternative consumes the same scarce engineers who should be shipping product.
The essay is most vulnerable where the workflow is narrow but not stable. If the task definition changes weekly, finetuning and local evaluation can become churn. In that world, the long tail may stay managed because flexibility beats unit economics.