Hybrid LLM Routing: Send Each Request to the Cheapest Model That Can Handle It

The price gap is the whole argument

Within a single provider's current lineup the spread is large. As of October 2026, Anthropic lists Haiku 5.5 at $0.10 per million input tokens and $0.50 per million output tokens (for prompts up to 100K tokens), and Opus 5.5 at $4 and $20. That is a 40x difference on both input and output, and Sonnet 5.5 sits in between at $2 and $10. Check current pricing before you plan around these numbers; they change.

If every request goes to the top model, you are paying the top price for password-reset questions, ticket tagging and "summarise this email". Routing is the practice of matching each request to the cheapest model that does that job well enough, and keeping the expensive model for the work that needs it.

• Anthropic pricing (API models)

Three routing patterns, from simple to clever

I almost always start with the first and only add the others when the data says they are worth it.

• Rules by task type. The calling code already knows what it is asking for: classification, extraction, summarisation, drafting or multi-step reasoning. Map each task to a model tier in config. It is boring, predictable and easy to audit.
• A learned router. A small classifier scores each request and picks a strong or weak model. The RouteLLM paper trained routers on human preference data and reported cost reductions of over 2x in some cases without compromising response quality, and found the routers still worked when the strong and weak models were swapped.
• Cascades. Try the cheap model first, run a check on the output (schema validation, a confidence score, a grader model or a rule), and escalate to the stronger model only when the check fails. This works well when a failure is cheap to detect.

• RouteLLM: Learning to Route LLMs with Preference Data (arXiv)

Route on sensitivity, not just cost

Hybrid routing is also how you mix frontier APIs with a private model. Requests that contain regulated data, customer records or anything contractually barred from leaving your network go to a model you host. Everything else can go to an API.

The rule here has to be deterministic. Classify sensitivity with code you can test, such as the data source, tenant settings or a PII detector, and never let a model decide on its own whether something is allowed to leave. Log the routing decision and the reason for every request.

• What Does a Private LLM Actually Cost?

Eval gates decide what each route is allowed to serve

A router without evaluations is a cost cut you cannot defend. Before a cheaper model takes over a task, it should pass the same test set as the model it replaces.

For each task type, keep a golden set of real, anonymised requests with expected outputs or grading criteria. Score every candidate model on it and set a pass threshold per task. A model that is fine for tagging might be unacceptable for drafting customer replies, and the gate should say so.

• Golden sets per task, built from real traffic, refreshed as the product changes.
• A pass threshold per route, agreed with whoever owns the outcome.
• Shadow runs: send a copy of live traffic to the candidate and compare before switching.
• Re-run the gate on every model version change, prompt change or provider update.
• Track escalation rate in a cascade. If the cheap model fails half the time, you are paying for both.

Fallbacks: plan for the provider being down

Every model you depend on will rate-limit you, time out or have an outage. Fallbacks decide what happens next, and they need the same care as the main route.

Gateways such as LiteLLM handle the mechanics. Its proxy retries a model group a set number of times, then falls back to other groups in order, and it separates ordinary failures from context-window errors and content-policy rejections so each can have its own fallback chain. It can also cool down a deployment after repeated failures, and it records in spend logs whether a request was served by a fallback, which is what you want when you later ask why last month's bill moved.

• LiteLLM docs: Fallbacks and reliability

# LiteLLM-style router settings (illustrative model names)
router_settings:
  num_retries: 2
  fallbacks:
    - small-fast: [mid-tier]
    - mid-tier: [frontier]
  context_window_fallbacks:
    - mid-tier: [long-context]
  # Sensitive traffic uses a separate group with no public fallback
  # private-only: []

The fallback trap: breaking your own data rules

The most common mistake I see is a fallback chain that quietly sends private traffic to a public API when the private model is busy. Under load, your data boundary becomes a suggestion.

Keep sensitive traffic in its own route group with fallbacks only to other approved private deployments, or to a clear error. LiteLLM now checks the calling key's model access before trying each fallback target and skips targets the caller could not call directly, which helps, but I would still make the boundary explicit in config and test it by forcing the primary to fail in a staging environment.

What to measure once it is live

A router changes over time even if you never touch it, because providers update models and your traffic mix shifts. A few numbers per route tell you whether it is still doing its job:

• Cost per completed task, not cost per token. A cheap model that needs two retries is not cheap.
• Escalation rate for cascades, and fallback rate for each route, split by reason.
• Latency at the 95th percentile per route, since a slow fallback can hurt more than a failed one.
• Quality signals from users: thumbs down, edits to drafts, tickets reopened.
• The share of traffic on each tier, so a silent shift towards the expensive model shows up before the invoice does.

Where to start

Pull a week of traffic, group it by task, and price it at your current model. Then pick the highest-volume, lowest-risk task, build a golden set for it, and test one cheaper model. That single route often shows whether routing is worth a wider rollout.

If you want one workflow in production with the right model layer behind it, my AI Ops Starter package covers that. If you are designing routing across several products, mixing frontier and private models, or untangling an existing stack, that is a Custom AI Architecture engagement.

• Custom AI Architecture
• AI Ops Starter package
• Talk through your model costs