Shipped
Heuristic routing engine
A 15-dimensional complexity scorer classifies every prompt into one of five tiers (NANO through COMPLEX) without any LLM calls in the decision path. Routing decisions complete in under 10ms at p99. Models are selected by cost within the appropriate tier pool, and reasoning-heavy prompts within COMPLEX are matched to a model built for chain-of-thought work.Automatic fallback and retry
If the selected model fails (timeout, error, rate limit), Routor retries through remaining models in the tier pool, then escalates to the next tier. Provider failures are invisible to the application.Routing profiles
Custom routing rules with dedicated scoped API keys. Each profile enforces a tier range, cost cap, required capabilities, and tool-calling quality floor on every request that uses it.Provider health monitoring
Rolling 5-minute health window for all 13 providers, with p50/p95 latency, success rate, and request volume. Degraded providers are down-weighted in routing automatically.Dashboard
Usage tracking, savings display, transaction history, provider health, model browser, API key management, and playground for testing routing decisions.OpenAI-compatible API
Drop-in replacement. Three lines of code change. No new SDK.In Progress
Conversation storage
Every request and response will be stored in the database (with user opt-in). This is the foundation for everything below. The schema captures prompt, response, routed model, tier, confidence score, latency, and token counts. Without stored conversations, the routing engine cannot learn from real traffic. This ships first.Next
Generation scorer
The current heuristic scorer uses rules to estimate prompt complexity. It works well on common patterns but misclassifies edge cases: highly technical one-line questions that need deep reasoning, or verbose but simple prompts that inflate the complexity estimate. The improvement is a lightweight trained classifier that learns from stored routing decisions. The training signal comes from running offline quality evaluation on stored responses (using a judge model outside the request path), then labeling (prompt, tier) pairs with quality outcomes. This classifier replaces the heuristic scorer as the primary routing signal. The heuristic stays as a fallback for prompts outside the training distribution. Dependency: Conversation storage must ship first.Semantic routing
Once a corpus of routed conversations exists, semantic similarity becomes a viable additional routing signal. The approach: embed every stored prompt using a small embedding model, store embeddings in pgvector (already in the Postgres stack), and at inference time find the nearest neighbor prompts from the corpus. The tier distribution of those neighbors becomes a prior that combines with the generation scorer output. Requests that closely resemble previously handled prompts route with higher confidence. Novel prompt types fall back to the scorer alone. Dependency: Generation scorer and a sufficient corpus of stored conversations.Provider-aware dynamic routing
Current routing ranks candidates by value (quality + cost savings) in the appropriate tier. It does not account for real-time provider performance beyond binary up/down status. This improvement incorporates live signals into the routing weight:- If a provider’s p95 latency is elevated, route away from it even if technically healthy
- Track request volume per provider and predict rate limit proximity before it triggers
- If a provider’s outputs start scoring lower in quality evaluation, reduce its routing weight automatically