I Made My 14B Model 40x Faster by Teaching a 0.5B Model to Say "I Got This"

Query routing on a single GPU — what Apple, Google, and Meta are doing at scale, applied to a $400 graphics card

By Corey Ryan · April 13, 2026 · 7 min read

Last week I published findings showing that my GPU spends 83% of its time idle, waiting for data over the PCIe bus. That article explored the hardware bottleneck. This one explores the software solution I built the same week.

The problem was simple: my local AI assistant runs a fine-tuned 14B parameter model on an RTX 5060 Ti. Every query — including "Hi!" and "What time is it?" — loads the full model, generates tokens at ~45 tok/s, and takes 1-2 seconds. For a greeting. On hardware I own.

So I asked: what if most queries never needed the big model at all?

The Pattern: Query Routing

The industry has a name for this: query routing (sometimes called model cascading). A small, fast model classifies incoming queries and decides where to send them — skip the expensive model entirely, retrieve context first, or dispatch to a tool API.

This isn't a new idea. It's how the best AI systems in production work right now:

• Apple Intelligence routes between a ~3B on-device model and cloud models. Simple queries never leave the device.
• Google's Speculative RAG uses a small specialist model to draft RAG-augmented responses before a large model verifies.
• RouteLLM (UC Berkeley/LMSYS) routes between GPT-4 and cheaper models, cutting costs 85% while maintaining 95% of GPT-4 quality.
• Meta's CASCADIA cascade serving system achieves 4x latency reduction across Llama model families.

The pattern is the same everywhere: let a small model decide whether the big model is needed. Escalate only when necessary.

What I Built

I fine-tuned a 0.5B parameter model (Qwen2.5-0.5B-Instruct) as a query router. It classifies every incoming query into one of five categories, and each category maps to a routing action:

The router sits in front of everything. It sees the query first. In ~50 milliseconds, it decides the optimal path. The 14B model only activates when the query genuinely needs reasoning.

The Two-Tier Classification

The router uses a two-tier approach:

Tier 1: Regex (instant, deterministic). A set of pattern-matching rules catches obvious cases. "Hi!" is always a greeting. "Check Ollama status" is always a system command. "Write a poem" is always creative. These fire in microseconds with zero model inference.

Tier 2: Fine-tuned 0.5B LLM (fallback for ambiguous queries). When regex can't confidently classify — "Skye blue?" or "Design my morning routine" — the tiny model makes the call. It was fine-tuned on 222 examples including adversarial cases: conflicting signals, negations, multi-intent queries, typos, and ambiguous short phrases.

Combined accuracy: 100% on a 39-case test battery including edge cases that would confuse most classifiers.

The Numbers

Training the router took 40 seconds. Not minutes. Seconds. On the same GPU that takes 70 minutes to train the 14B model.

Here's the efficiency breakdown:

The key insight: ~35% of queries in typical assistant usage are greetings or system commands. These are being processed by a 14-billion parameter model that was trained for reasoning. That's like hiring a PhD to answer the phone.

Why This Isn't Mixture of Experts

People will confuse this with MoE (Mixture of Experts). They're fundamentally different:

MoE routes tokens within a single model. A gating network decides which internal expert sub-networks activate per token. Mixtral's 8x7B activates ~12B of its 47B parameters per token. The model is one artifact, trained end-to-end.

Query routing selects between entirely separate models. A classifier analyzes the full query and picks which system handles it. The models are trained independently. The router is trained on evaluation data, not jointly with the models it routes to.

MoE is hiring specialists within one organization. Query routing is choosing which organization to call.

The RAG Integration

For personal and factual queries — the ones that need the 14B model — the router doesn't just classify and forward. It retrieves context first.

The pipeline:

1. Router classifies: "What projects am I working on?" → RAG
2. Memory search retrieves relevant context from a vector database (~100-500ms)
3. Context assembled into [CONTEXT]...[/CONTEXT] blocks
4. Enriched prompt forwarded to 14B model

The 14B model receives a grounded prompt with real context, not a bare question it would need to hallucinate an answer to. The router decided this query needed context. A creative writing request ("Write a poem about cats") skips retrieval entirely — the 14B model doesn't need external context for that.

This is the same pattern as Google's RAGRouter, which uses a ~136M parameter classifier to decide whether queries benefit from retrieval-augmented generation or can be handled by long-context LLMs directly.

What This Means for Local AI

The big cloud providers route between models because it saves money at scale. A single RouteLLM deployment saves 85% on API costs. Meta's CASCADIA gets 4x latency improvements.

But query routing might matter more for local systems than cloud:

VRAM is the constraint, not API costs. On a 16GB GPU running a voice synthesis model (4.4GB), a speech-to-text model, and the main LLM, every megabyte matters. A 397MB router that prevents unnecessary 14B model loads is a structural efficiency gain, not a marginal one.

Latency is felt directly. When you're talking to a local assistant through a microphone, the difference between 50ms and 2 seconds for "thanks!" is the difference between conversational and awkward.

The router is free to run. A 0.5B model uses 397MB of VRAM. It classifies in ~50ms. It trains in 40 seconds on 222 examples. The cost-benefit ratio is absurd — you're spending 2.4% of your GPU memory to potentially skip 35% of full-model inferences.

The Failure Mode That Matters

The risk with query routing is misclassification. If the router sends "Hey, how's my son doing?" to the SELF handler (treating it as a greeting), the actual question gets silently dropped. The user gets "Hello!" instead of information about their child.

This is why the two-tier approach matters. Regex handles high-confidence patterns with deterministic accuracy. The fine-tuned LLM handles ambiguous cases where pattern matching fails. And for any query the router isn't confident about, the safe fallback is always to route to the full model.

The failure cost of routing to the 14B model when you didn't need to is: 1-2 extra seconds of latency. The failure cost of NOT routing to the 14B model when you should have is: a wrong or missing answer. The asymmetry means the system should be biased toward escalation, not self-handling.

Try This Yourself

If you're running local models, query routing is one of the highest-leverage optimizations available. The recipe:

1. Pull a tiny model. Qwen2.5-0.5B-Instruct, Phi-3-mini, or similar. Anything under 1B parameters.
2. Fine-tune it as a classifier. 200+ examples, 5 categories, 40 seconds of training. You can use QLoRA to keep VRAM under 2GB.
3. Add a regex tier. Pattern-match the obvious cases (greetings, system keywords). Let the LLM handle the rest.
4. Wire it into your pipeline. The router returns a classification in ~50ms. Use it to decide: skip the big model, retrieve context first, or forward directly.

The total investment is a few hundred training examples and one afternoon. The return is a system that handles a third of its queries 20-40x faster, with zero quality loss on the queries that still hit the large model.

The Broader Point

We've spent the last three years making models bigger. GPT-4, Claude 3.5, Llama 405B. The assumption is that bigger models produce better results, and the cost is justified by quality.

Query routing challenges that assumption — not by replacing big models, but by asking a question that should have been obvious from the start: does this specific query actually need 14 billion parameters to answer?

For "Hello!", the answer is no. For "Explain the implications of quantum decoherence on error correction in topological qubits," the answer is yes. The router's job is to tell the difference. And it turns out a model with 0.5 billion parameters — 28x smaller than the one it's routing for — can do that with 100% accuracy.

The most efficient AI system isn't the one with the best model. It's the one that uses the right model for each query.

This is a follow-up to my previous article, "I Benchmarked My GPU and Found It Was Idle 83% of the Time," which explored the hardware bottleneck in GPU computing. Query routing is one of the software-side solutions to the efficiency problem that article identified.

The implementation described here runs on a single RTX 5060 Ti (16GB) using Ollama, Qwen2.5-0.5B-Instruct (router), and a fine-tuned Qwen2.5-14B-Instruct (reasoner). All code is open and runs locally.

References:

• RouteLLM: A Framework for Serving and Evaluating LLM Routers — UC Berkeley/LMSYS
• Dynamic Model Routing and Cascading for Efficient LLM Inference: A Survey — 2025 survey
• CASCADIA: An Efficient Cascade Serving System for Large Language Models — Meta/CMU, 2025
• Query Routing for Retrieval-Augmented Language Models — Google, 2025
• Apple Intelligence Foundation Language Models — Apple, 2025