Should You Route AI Requests to Different Places?
With small, fast models landing everywhere, here is how a tiny team splits AI traffic by difficulty.
Look at the past ten days of AI announcements and one pattern stands out more than any model name: where a request gets processed is now a product decision, not just an infrastructure detail. Perplexity shipped Hybrid Compute on September 1 alongside a local model that runs on the device (PPLX Qwen 3.8 27B). Google DeepMind followed the next day with Gemini 3.8 Flash, and Alibaba had released Qwen3.8 Flash days earlier. Next to the big, expensive models, there is now a dense layer of small, fast ones.
For a small team, this is not a prompt to "upgrade to the better model." It is an invitation to split traffic by difficulty. And to be clear: this has nothing to do with buying GPUs or running your own servers.
Why did execution location suddenly become a design problem?
Because in many products, latency, data boundaries, and network conditions now shape the experience more than raw model quality does. That is the same logic driving the edge and on-device wave: inference closer to the user is faster, keeps more data in place, and keeps working when the connection is weak.
A year ago the practical choice was a single good API. Today each generation ships a Flash-class sibling, and consumer products are starting to bundle local models by default. Users judge your app on whether the answer appears instantly. Routing has quietly become part of perceived quality.
Which requests can move to the lighter path?
Start with the work that is repetitive and low-judgment. The rule is simple: handle easy things cheaply and quickly, escalate only the hard ones.
- Classification and routing: detecting intent, filtering duplicates and spam
- Formatting work: spelling, tone consistency, table cleanup, auto-tagging
- Latency-critical UI: short summaries, autocomplete, query rewriting
- Pre-processing sensitive input: masking phone numbers or order IDs before the cloud call
- Fallback answers for shaky connections
Imagine a booking chatbot for a neighborhood photo studio. Questions about hours, pricing, and parking can be answered instantly by a small model plus document search. Only the layered questions — "is an outdoor shoot open the third weekend, and what happens if it rains?" — go up to the frontier model. Customers get most answers fast, and you spend premium tokens only where they change the outcome.
What should you actually do on Monday?
Read your logs before you change any model. Sorting one week of requests by difficulty and sensitivity usually answers the routing question for you.
- Pull 100 recent requests and label each one easy, sensitive, or hard.
- Point a small model at the "easy" bucket only, and score it against your current answers using the same inputs and the same rubric.
- Write your escalation rules down: low confidence, any mention of refunds, contracts, or money, and any request for a human all go up.
- Build the failure path first. Light path dies, fall back to cloud; cloud dies, fall back to a fixed message.
- Track three numbers only: time to first response, escalation rate, and cost per request.
One warning. Two paths mean two sets of prompts and two chances to drift, so keep a single evaluation set and grade both with it. Local and lightweight paths also vary a lot by device and context, so ship a switch that sends everything back to the big model in one click.
The release calendar will stay noisy. What holds steady is the routing rule underneath: classify the request, serve the easy ones lightly, and reserve accuracy for where it matters. This month simply handed small teams more materials to build that rule with — and only teams with a rule turn new models into lower cost and faster answers.