The Overprovisioning Problem
When a team first integrates an LLM, they pick the best model they know. Usually GPT-5.5 or Claude Opus 4.8. The output is good. The integration is clean. They ship. Then usage grows. The model choice never gets revisited. Every request, whether it is checking if an email address is valid, generating a one-sentence summary, translating a product label, or solving a multi-step reasoning problem, goes to the same endpoint. The model does not know the task is trivial. It was not told. It spends the same compute on a greeting as on a technical analysis. And every token, on every request, is billed at flagship rates. This is overprovisioning. Using a tool far beyond what the task requires. And at scale, the consequences are not abstract.What This Actually Costs
The pricing gap between the cheapest capable model and the most expensive is roughly 200x.| Model | Input (per 1M tokens) | Output (per 1M tokens) |
|---|---|---|
| GLM-4.7 Flash (Z.AI, open-weight) | $0.20 | $0.20 |
| Gemini 3.5 Flash | $1.50 | $9.00 |
| Claude Haiku 4.5 | $1.00 | $5.00 |
| Claude Sonnet 4.6 | $3.00 | $15.00 |
| Claude Opus 4.8 | $5.00 | $25.00 |
| GPT-5.5 | $5.00 | $30.00 |
The “Token Maxxing” Culture
What makes this worse is that a vocal part of the developer community has normalized extreme model usage. “Token maxxing” is the practice of deliberately burning as many tokens as possible in a session, often as a demonstration of what a model can do. Agentic loops run for minutes, calling heavyweight models repeatedly to complete tasks that a lighter model would handle in two seconds. This behavior is presented as exploration. In production, it is a billing disaster waiting to happen. The culture has convinced developers that using more compute is inherently better. It is not. It is just more expensive.Why Manual Routing Does Not Scale
The intuitive fix is simple: figure out which tasks are simple, and send them to cheaper models. Use the heavy model only when the task genuinely needs it. In practice, this is not a decision a team can sustain manually. The model landscape changes every week. New models are released constantly. Benchmarks shift. A model that was the best choice for code generation in January may have been surpassed by three others by March, each with different pricing, context limits, and capability tradeoffs. Keeping up with this requires someone whose entire job is tracking the model ecosystem. Task complexity is not obvious from the prompt. A two-sentence prompt can require deep reasoning. A paragraph-length message can be trivially simple. You cannot classify complexity by token count or a single keyword - it takes a multi-dimensional weighted score over the full prompt to get it right. Routing needs to happen at inference time, per request. A policy document or a set of if-else rules does not hold up in a production system with diverse user behavior. The decision needs to be made dynamically, in the request path, without adding meaningful latency. Most teams cannot staff this. So they default to the expensive model and absorb the cost.Why Existing Solutions Fall Short
Several tools claim to solve this. None of them get it right. The “access to hundreds of models” approach presents a large catalog as a feature. When a classifier has to choose between a hundred options, it stops making decisions and starts approximating. The decision space is too large. More models does not produce better routing. It produces more confusion embedded into the system at the point where precision matters most. The “use an LLM to classify your prompts” approach sounds reasonable until you account for the cost - and the speed. Routor’s own routing decision runs in single-digit milliseconds (p50 4ms, p99 18ms), well under the roughly 300ms window where a delay stops being perceptible to a human. Replace that with an LLM call and routing itself now takes as long as generating the actual answer, roughly doubling time-to-first-token before the real model has even been reached. It gets worse on the failure path: an LLM classifier’s output is a probabilistic guess, not a deterministic decision, so misclassification is a routine outcome, not an edge case - and when that triggers a fallback, you now pay the classification latency, the wrongly-chosen provider’s response time, and the retry’s response time, stacked, before the user sees anything. Latency increases. Cost increases. Reliability gets worse exactly where it matters most. None of that buys a better decision - a fast, deterministic scorer gets the same routing outcome without any of it. Neither approach addresses the actual problem. They reframe it.What’s Been Happening Lately
The last few months have made all of this more urgent, not less. A few things worth knowing if you are deciding how to handle model selection right now. The price gap is widening, not narrowing. DeepSeek’s V4 launch undercut GPT-5.5 by a wide margin while matching or beating it on several published benchmarks, and MiniMax’s M3 release landed in the same range - frontier-level scores for a small fraction of the cost. Every time this happens, the “just use the best model for everything” strategy gets more expensive relative to the alternative. Teams that hardcoded a model name six months ago are now overpaying by a wider margin than they were when they made that choice. Outages are no longer rare. Anthropic reported elevated error rates across multiple Claude models in June, and every major provider has had a comparable incident in the last year. When your app calls one provider directly, an incident on their end is an incident on yours. There is no way to code around a provider being down from inside a single-provider integration. Capacity is getting rationed, not just priced. Coverage this year has pointed to AI providers throttling access and pacing rollouts as compute demand outpaces available capacity. Model availability is becoming as real a constraint as model price. A routing layer that can move traffic to a healthy provider isn’t just a cost optimization anymore, it’s what keeps a product online when one vendor is constrained. Developers are already hedging. A large share of the developer community has started deliberately spreading work across more than one AI vendor rather than betting on one, specifically to avoid being stuck when a single company changes terms, restricts access, or has a bad day. That instinct is correct. It’s just hard to do by hand, for the same reasons manual routing does not scale. The “marketplace of everything” approach is still raising money, and still not solving this. Aggregators that expose hundreds of models through one API keep attracting investment, which says the demand for a single integration point is real. It does not mean dumping every model into one undifferentiated list is the right answer. Choice without a decision engine just moves the overspending problem one layer down, from “which model do I hardcode” to “which of these three hundred models do I hardcode.” None of this changes the shape of the problem described above. It just raises the cost of ignoring it. A model landscape that reshuffles every few weeks, providers that go down without warning, and capacity that gets rationed during demand spikes all point to the same conclusion: routing needs to be automatic, and it needs to happen closer to the request than any team can manage by hand.What the Problem Actually Requires
The solution is not more models. It is not an LLM classifier. It is a precise, fast, lightweight decision engine that understands task complexity from the structure and content of a prompt, maps that complexity to the minimum model tier capable of handling it well, and makes that decision in a few milliseconds without any model calls in the critical path. That decision engine needs to account for:- Task category (coding, math, planning, research, science, reasoning, writing, vision, tool use, multilingual, and more)
- Difficulty - scored across 15 weighted dimensions, not naive keyword matching
- Prompt length and context requirements
- Required capabilities (vision, tool calling, long context, audio, video, files)
- User-defined constraints (cost cap, quality floor, tier range, BFCL minimum)