⏱️ Lectura: 11 min
Fireworks AI tested model routing in production: an agentic pipeline that chooses, task by task, between an open model and a closed one, reached 93% accuracy, above either model on its own.
📑 En este artículo
- TL;DR
- What happened
- Background: what model routing is
- Technical details and performance
- How to try it today on Fireworks
- Impact and analysis
- What’s next
- Frequently Asked Questions
- What is oracle routing?
- Is Kimi K3 free because it’s an open model?
- Which tasks are best sent to Kimi K3 instead of Fable 5?
- How can K3 be cheaper if it uses ten times more tokens than Fable 5 on SWE?
- Do I need to change my code if I already use the Fireworks API?
- What benchmark did Fireworks use for this comparison?
- References
The experiment pitted Kimi K3, Moonshot AI‘s open model, against Fable 5, Anthropic’s closed model, across 1,030 real agentic tasks: bug fixing in repositories, long terminal sessions, algorithmic problems, implementation across six languages, and a legal benchmark graded by lawyers. In long cycles, routing to K3 can cost up to 50 times less than using Fable 5 alone, according to Fireworks.
TL;DR
- Fireworks evaluated Kimi K3 (open) and Fable 5 (closed) across 1,030 agentic tasks split into 5 categories.
- Model routing between the two reached 93% accuracy, above either one alone.
- Oracle routing, the theoretical ceiling, chose Kimi K3 for 72% to 96% of tasks depending on the category.
- On SWE tasks, K3 tied with Fable 5: 92.4% versus 92.6% of tasks solved.
- K3 used 55 turns and 1.3 million tokens per task on SWE; Fable 5 used only 21 turns and 130,000 tokens.
- On terminal tasks, Fable 5 reached 64 turns and 1.5 million tokens per task, sometimes running until timeout.
- Of 89 terminal tasks, K3 solved 11 that Fable 5 couldn’t, against 7 exclusive to Fable 5, and dominated security and cryptography.
- Fireworks reports that K3 can end up up to 50 times cheaper than Fable 5 in long agentic cycles.
What happened
Fireworks, the inference platform that serves both open and closed models, published the results of its own benchmark comparing Kimi K3 against Fable 5 across five families of agentic tasks. The measurement ran both models with the same harness, on real software development and systems operation tasks.
The test set included 460 SWE-bench-style cases (fixing real bugs in repositories), 89 long terminal tasks (security, cryptography, reverse engineering, systems administration), 100 LeetCode- or AtCoder-style algorithmic problems, 225 implementation tasks across six different languages, and 120 tasks from a legal benchmark graded by lawyers. In total, roughly 1,030 tasks evaluated with the same criteria for both models.
The headline is a tie: K3 solved 92.4% of SWE tasks and Fable 5 solved 92.6%. At first glance, both models look interchangeable. But when Fireworks broke down each benchmark by task type, consistent differences emerged: K3 dominates in symbolic math and dev tooling; Fable 5 wins in web work, data visualization, and breadth of languages (Java, Python, C++). That specialization, not the overall average, is what makes model routing valuable instead of picking a single model.
Background: what model routing is
Model routing means deciding, before or during a task, which AI model will handle it, instead of sending all traffic to a single one. The idea isn’t new: inference providers have offered it as a platform feature for a while, but until now the justification was usually just cost, sending easy work to a cheap model. Fireworks’ study adds a different argument: two models with similar average quality can have opposite strengths beneath that average, and exploiting that gap pushes overall accuracy above either one.
Kimi K3 is the latest version of the Kimi family from Moonshot AI, the Chinese startup that also released K2.6 and splits its assistant into work, code, and browsing variants. Fable 5 is Anthropic’s most recent closed model, part of the same generation as Opus 4.8 and Sonnet 5. Both are served on Fireworks through an OpenAI-compatible API, which in practice lets you route between the two without changing the HTTP client you already use.
Fireworks distinguishes between oracle routing, which runs the task against both models and keeps the cheaper correct answer (the theoretical ceiling), and a real production router, which has to predict which model is better without being able to try both. In this study, the oracle chose K3 for between 72% and 96% of tasks, depending on the category.
Technical details and performance
The cost difference between the two models doesn’t come from a single factor. Fireworks identifies three: each model’s per-token price, the effect of prompt caching, and how much work each model does depending on the task type.
On SWE tasks, K3 works harder: it averages 55 turns and 1.3 million tokens per task, versus 21 turns and 130,000 tokens for Fable 5. On long terminal tasks the pattern flips: Fable 5 is the one that spikes, with 64 turns and 1.5 million tokens per task, sometimes running until timeout. Neither model is more efficient across the board: K3’s extra effort lands on SWE, and Fable 5’s lands on terminal.
Kimi K3 versus Fable 5, family by family
| Aspect | Kimi K3 | Fable 5 |
|---|---|---|
| Model type | Open, downloadable weights | Closed, API only |
| SWE-bench-style (460 tasks) | 92.4% solved | 92.6% solved |
| Main strength | Symbolic math, dev tooling, security and cryptography in terminal | Web, data visualization, multi-language breadth (Java, Python, C++) |
| Turns and tokens on SWE | ~55 turns, ~1.3M tokens | ~21 turns, ~130K tokens |
| Turns and tokens on long terminal tasks | Fewer turns than Fable 5 | ~64 turns, ~1.5M tokens, sometimes timeout |
| Oracle router selection | 72% to 96% of tasks | 4% to 28% of tasks |
Across the 89 terminal cases, K3 solved 11 tasks on its own that Fable 5 never solved, including cracking a 7z hash, FEAL cryptanalysis, detecting leaked secrets, and exploiting an active vulnerability, against 7 tasks exclusive to Fable 5. K3 swept the entire security and cryptography cluster.
💡 Tip: K3’s cost savings depend on the prompt cache hit rate: even though K3 reads ten times more tokens than Fable 5 on an SWE task, with enough prompt caching those repeated tokens come out much cheaper, and the full run still ends up costing less.
A production router has to approximate this decision without seeing both models’ results ahead of time. Here’s the simplified model routing flow that Fireworks describes:
flowchart TD
A["Incoming task"] --> B["Classification router"]
B -->|"terminal, symbolic math, dev tooling"| C["Kimi K3 (open)"]
B -->|"web, multi-language, data"| D["Fable 5 (closed)"]
C --> E["Response and cost per token"]
D --> E
How to try it today on Fireworks
Kimi K3 and Fable 5 are served on Fireworks under the same OpenAI-compatible API, so you can reproduce model routing with the SDK you already have installed.
Client installation, depending on your operating system:
# Linux / macOS
pip install openai
# Windows (PowerShell)
py -m pip install openai
With the package installed, here’s the minimal call against Kimi K3 using the Fireworks endpoint:
from openai import OpenAI
client = OpenAI(
api_key="YOUR_FIREWORKS_API_KEY",
base_url="https://api.fireworks.ai/inference/v1",
)
respuesta = client.chat.completions.create(
model="accounts/fireworks/models/kimi-k3",
messages=[{"role": "user", "content": "Explain what a race condition is"}],
)
print(respuesta.choices[0].message.content)
That first block only confirms that the API key works and that the model responds. To get closer to Fireworks’ experiment you need a minimal router that sends each task category to the model that performed best in the benchmark:
import time
from openai import OpenAI
client = OpenAI(api_key="YOUR_FIREWORKS_API_KEY", base_url="https://api.fireworks.ai/inference/v1")
MODELO_POR_CATEGORIA = {
"terminal": "accounts/fireworks/models/kimi-k3",
"symbolic_math": "accounts/fireworks/models/kimi-k3",
"web_frontend": "accounts/fireworks/models/fable-5",
"multi_lenguaje": "accounts/fireworks/models/fable-5",
}
def enrutar_tarea(categoria, prompt):
modelo = MODELO_POR_CATEGORIA.get(categoria, "accounts/fireworks/models/kimi-k3")
inicio = time.time()
respuesta = client.chat.completions.create(
model=modelo,
messages=[{"role": "user", "content": prompt}],
)
duracion = time.time() - inicio
print(f"model={modelo} time={duracion:.2f}s tokens={respuesta.usage.total_tokens}")
return respuesta.choices[0].message.content
enrutar_tarea("terminal", "Find the vulnerability in this bash script: ...")
To verify that model routing is actually lowering cost, looking at the response isn’t enough: you need to log respuesta.usage.total_tokens from each call, multiply it by the per-token price published on the Fireworks dashboard, and compare the total against what it would have cost to send everything to a single model. That per-task log is the only reliable way to confirm the savings on your own traffic, instead of assuming the number Fireworks reported.
Impact and analysis
The central finding about model routing isn’t that K3 is better or worse than Fable 5: it’s that two models with nearly identical average accuracy can make mistakes in different places, and that gap is exploitable. Fireworks sums it up directly: if you send each task to whoever solves it best, you don’t land at a midpoint between the two models, you end up above both.
That has a practical reading for teams that currently pay a single LLM provider for their entire agentic pipeline: running a long terminal task against a model that tends to spiral (like Fable 5 in this benchmark, with 64 turns and 1.5 million tokens) can end up costing more than solving it with an open model that’s more methodical in that domain.
⚠️ Heads up: the 93% accuracy and the savings of up to 50 times were measured with oracle routing, the theoretical ceiling where you already know which model got it right. A real production router makes a prediction before seeing the outcome, so the number you’ll get with a router trained today will fall below that ceiling.
There’s also a latency trade-off: running more turns means more wall-clock time per task. If you need a response in seconds, that detail matters as much as cost. If you’re running agents in the background at scale, a cost that’s a fraction of the original outweighs a few extra turns.
What’s next
Fireworks argues that a near-perfect router is achievable, but confirming it will take an order of magnitude more routing data and testing on real traffic. The current study used oracle routing to measure the ceiling, not yet a router trained in production.
For Moonshot AI, the result is external validation that K3 competes on equal footing with a frontier closed model in half of the evaluated categories. For Anthropic, the reading is different: Fable 5 still wins on language breadth and web tasks, the type of work with the highest volume in consumer products.
📖 Summary on Telegram: View summary
Try it yourself: create a Fireworks account, generate an API key, and run the second code block against a real task from your backlog to see which model your own router sends it to.
Frequently Asked Questions
What is oracle routing?
It’s a measurement method, not a product: it runs the same task against both models and keeps the cheaper correct answer. It’s used to calculate the theoretical performance ceiling of model routing, it’s not something you can deploy in production as-is.
Is Kimi K3 free because it’s an open model?
No. Being an open model means the weights are downloadable and you can run it on your own infrastructure, but serving it on a platform like Fireworks has a per-token price just like any closed model. What changes is that you can also host it yourself if you have the hardware.
Which tasks are best sent to Kimi K3 instead of Fable 5?
According to Fireworks’ benchmark, K3 has an edge in symbolic math, dev tooling, and terminal tasks related to security and cryptography. Fable 5 wins on web work, data visualization, and the breadth of supported programming languages.
How can K3 be cheaper if it uses ten times more tokens than Fable 5 on SWE?
Most of those extra tokens repeat across turns and fall into the prompt cache, which bills much cheaper than a new token. Fireworks reports that even reading ten times more tokens, K3’s SWE runs still end up costing less than Fable 5’s.
Do I need to change my code if I already use the Fireworks API?
No. Kimi K3 and Fable 5 are served under the same OpenAI-compatible API, so routing between the two is just a matter of changing the model parameter string on each call, not rewriting the HTTP client.
What benchmark did Fireworks use for this comparison?
A custom harness of roughly 1,030 real agentic tasks, split into five families: SWE-bench-style bug fixing, long terminal sessions, algorithmic problems, implementation across six languages, and a legal benchmark graded by lawyers.
References
- Fireworks AI: Kimi K3 is competitive with Fable; Kimi K3 + Fable is SoTA: the original post with the full methodology and benchmark charts.
- Moonshot AI: official site of the Chinese startup that develops the Kimi family of models, including K3.
- Moonshot AI on GitHub: open repositories related to the Kimi models.
- Anthropic: official site of the lab that develops Fable 5, the closed model used in the comparison.
- Fireworks AI: inference platform used to serve both models under an OpenAI-compatible API.
📱 Like this content? Follow @programacion on Telegram for daily tech content in Spanish: quick summaries, fresh content every day.
0 Comments