Fireworks benchmarks Kimi K3 against Fable 5, finds routing beats either model

Fireworks AI published benchmark results on July 21, 2026 comparing Kimi K3, an open model, against Fable 5, a closed model, across about 1,030 real agentic tasks. The company reports that oracle routing between the two models achieved 93% accuracy, and that K3 alone was up to 50 times more cost effective than Fable on long agentic loops, with lower cost across every task family tested.

How the measurement worked

Fireworks ran both models through the same harness, averaging benchmarks that target different kinds of work. The task set included 460 SWE-bench style repository bug fixes, 89 long terminal operations covering security, crypto, reverse engineering and sysadmin work, 100 algorithmic problems in LeetCode or AtCoder style, 225 multi-language implementation tasks across six languages, and 120 lawyer-graded legal agent tasks.

Oracle routing, as defined in the post, is a measurement method that runs each task through both models and picks the cheapest correct answer, establishing a theoretical cost and performance ceiling. A practical router cannot do that, because it must predict which model offers the best tradeoff without running the task twice. In this study, the oracle chose K3 for 72 to 96 percent of tasks, which Fireworks says suggests a near perfect router might be achievable if it learns to separate everyday tasks from the true long tail of frontier work, though it would require an order of magnitude more routing data and real world validation.

K3 and Fable are close but specialized

At a high level the two models look like a tie. On SWE, K3 scored 92.4 percent and Fable 92.6 percent. Across the five task families the models stayed within a few points of each other, with Fable pulling slightly ahead on multi-language coding breadth.

Inside single benchmarks the picture changes. On SWE, K3 was sharpest on symbolic math and dev tooling, while Fable won on web and data visualization work. In the multi-language set, Fable carried Java, Python and C++, while K3 drew even on JavaScript and Rust. On the 89 terminal tasks, K3 had 11 solo wins to Fable's 7, and took the security and crypto cluster outright. K3 cleared tasks Fable never cracked, including a 7z hash, FEAL cryptanalysis, leaked secrets, a live vulnerability and runaway async jobs.

Cost differences come from effort and caching

The price gap is not close even where quality is. On SWE, K3 works much harder than Fable, roughly 55 turns and 1.3 million tokens per task versus 21 turns and 130,000 tokens. On long terminal tasks the pattern reverses, with Fable running up 64 turns and 1.5 million tokens, sometimes straight into a timeout. Neither model is leaner across the board.

Prompt caching does most of the work of turning that effort into K3's price advantage. Even when K3 reads ten times the tokens, cache hits keep SWE runs lower cost than Fable. There is a tradeoff: tasks with extra turns generally mean more wall clock time per run, so a two second response matters for interactive use, while background agents at scale benefit more from a bill that is a fraction of the size.

Routing beats any single model

Per-task routing always outperforms any single model run, according to the post. The oracle router sent 72 to 96 percent of traffic to K3, which means overall quality lands above either model alone at a cost close to using only the cost optimized one. Fireworks plots cost against accuracy for all five families, with K3 to the left of Fable in every case, meaning lower cost, while accuracy trades back and forth.

The practical takeaway from Fireworks is that an open model like K3 should be the default because the oracle sends it most traffic anyway, and the router itself becomes the moat. A router must be tailored to a specific workload, and continuously learning the task and model split is the best way to stay ahead. The post concludes that the single model provider era is ending, and the best AI is a mixture of models rather than the output of one lab.