I spent six months running everything through one model. Classification. Code generation. The occasional email draft. It was simple, it worked, and my API bill looked like a monthly car payment for something German.
Then I got smart. I designed a multi-model system. I added routers, fallbacks, parallel pipelines, a voting mechanism, a supervisor model that supervised other models, and a planner model that told the supervisor what to supervise. By the end, my architecture diagram had more boxes than a shipping manifest and I was spending more time debugging orchestration than actually shipping features.
What I should have known before I started: choosing models is the easy part.
The Real Problem Is Not the Models
I can pick Qwen2.5-1.5B for classification and Claude Sonnet 4 for creative writing in about fifteen seconds. The hard part is the architecture that connects them.
Five design patterns cover almost everything you would ever need. Pick the right one. Stop before you overthink it.
| Pattern | Complexity | What It Costs You | | --- | --- | --- | | Single Model | None | Everything you cannot do | | Sequential | Low | Your users' patience | | Parallel | Medium | Your API budget | | Hierarchical | High | Your sanity | | Ensemble | Highest | Your savings account |
Pick the simplest one that solves your actual problem. Complexity compounds fast. A two model system is twice as hard to debug as a one model system. A four model system is not four times harder. It is sixteen times harder. There is a curve and you do not want to be past the inflection point.
Sequential: One Model at a Time, Please
The simplest multi-model setup is just chaining models together. Model A does its thing, passes the result to Model B, who hands it to Model C. Everyone stays in their lane.
The Pipeline
You set up a chain where each model specializes in one step. A tiny model classifies the input. A medium one extracts the relevant bits. A big one does the actual reasoning. Each step hands off to the next like an assembly line.
The downside is obvious: latency stacks. If each model takes two seconds and you have three of them, your user waits six seconds before seeing anything. That is an eternity in internet time. Use this pattern only when each step genuinely needs a different capability. Do not pipeline three identical models because it looks clean in a diagram.
The Router
This is the one I use most. A lightweight classifier decides what kind of task this is, then dispatches to the appropriate specialist.
Input arrives
-> Classifier (small model, fast check)
-> Routes to: Code Model / Math Model / Creative Model / General Model
-> Specialist handles it
The router is only as good as the classifier. If the classifier is wrong, the specialist gets a task it was never designed for and the output is garbage. The saving grace is that the classifier can be tiny. A 1.5B param model is often enough to figure out whether something is a code question or a creative writing request, as long as the categories are distinct. When they blur together, everyone suffers.
Parallel: Throw Everything at the Wall
Sometimes you want multiple models to look at the same problem. This is expensive but surprisingly useful.
Fan Out
Fire the same prompt at three different models and collect all the responses. This is great for comparison shopping. When I need a piece of marketing copy, I run it through a few models and pick the one that does not sound like a press release.
The cost multiplies by the number of models. The upside is you get to choose the best output instead of hoping the one model you picked had a good day.
Voting
For classification tasks, you can run the same input through several models and take a majority vote. If three out of four models say this email is spam, it is probably spam.
This breaks down for generation tasks because no two models produce identical sentences. You would need semantic similarity comparison, which pulls you right back into more model calls. I tried voting for creative work exactly once. The models disagreed on everything and the system escalated to a tiebreaker model that was just as indecisive. I ended up with three mediocre paragraphs and a headache.
Hierarchical: Someone Needs to Be in Charge
This is where things get architectural in the bad way. You have a strong model that does not do the work itself. It plans the work and delegates to smaller, cheaper models.
Planner Executor
A big model receives the task and breaks it down into steps. Smaller models execute each step. The big model synthesizes the results into a final answer.
This works beautifully when the planning is the expensive part and the execution is cheap. A 32B model can decompose "write a summary of this financial report" into "extract key numbers, identify trends, check for contradictions, write summary." Each of those subtasks can run on a 7B model. The big model only gets involved at the start and the end.
Supervisor Worker
Same idea, but the supervisor stays involved through the whole process. It assigns work, reviews results, and may send tasks back for revision.
The supervisor becomes the bottleneck. If it takes three seconds to review each worker's output and you have four workers, that is twelve seconds of overhead before you even count the work itself. Keep the supervisor fast or keep the number of workers low. I learned this the expensive way when my supervisor model was a 70B param beast that took eight seconds to say "looks good, proceed."
Ensemble: When You Absolutely Cannot Be Wrong
This is the nuclear option. You run multiple models, compare their outputs, and only return a result if enough of them agree. If they do not agree, you escalate to a bigger model or a fallback.
Weighted Ensemble
Each model gets a confidence weight. You score the outputs and pick the highest weighted result. The weights reflect your real world experience, not benchmark scores. I learned this the hard way when I weighted a model based on its MMLU score and it turned out to be great at multiple choice trivia and terrible at my specific use case.
Consensus Ensemble
Set a threshold. If enough models agree, return that result. If they do not, escalate.
Threshold of 0.7 means roughly two thirds of models need to agree. Lower thresholds make the system faster but less reliable. Higher thresholds mean your system spends more time in the escalation loop. There is no magic number. You have to tune it on your actual data.
I use ensembles for exactly one thing: financial calculations where being wrong costs real money. For everything else, a single good model with a clear prompt beats a committee of confused ones.
When to Actually Do This
Multi-model systems are not the default. They are the upgrade path for when the simple thing stops working. Here is my personal decision tree:
- If all your tasks are roughly the same complexity, use one model.
- If you are prototyping, use one model. Speed matters more than optimization.
- If you would rather keep things simple than squeeze out efficiency, stick with one and pay the tax.
Add a second model only when you hit a real constraint. Maybe the cost is eating you alive on routine tasks. Maybe latency is killing your interactive features. Maybe quality is not cutting it for critical decisions. Those are the signals. Not "this would look cool in an architecture diagram." That is a trap and I have fallen into it multiple times.
Tradeoffs at a Glance
| Pattern | Cost | Latency | Quality | My Pain Level | | --- | --- | --- | --- | --- | | Single Model | Lowest | Lowest | Roll the dice | None | | Sequential | Medium | High | High | Moderate | | Parallel | High | Low | High | Annoying | | Hierarchical | High | High | Highest | Significant | | Ensemble | Highest | Medium | Highest | Consider therapy |
Every pattern gives up something. Pick the one that sacrifices what you have the most of.