Fireworks router analysis: model pools beat single models on DeepSWE

Fireworks AI published an analysis of the DeepSWE v1.1 agentic coding benchmark that quantifies how much capability sits unused in a pool of 18 models. The best single model, GPT-6 Astra, passes 74.1% of tasks at $6.52 each. An oracle router that picks the best model for each task reaches 97.6% at $1.88 per task, 23 points higher and under a third of the cost. The oracle is a hindsight calculation: the company ran all 18 models on all 113 tasks and selected the winner per task by measured pass rate, breaking ties on cost. That method biases the result upward because it takes the maximum over 18 noisy estimates, but it still shows the pool contains more capability than any single model exposes.

The analysis also breaks down where the cost savings come from. At the 97.6% point, the oracle sends 94 of 113 tasks to a model costing under $3. The three most expensive models, all above $11.50 per task, are the sole best choice on only three tasks. On 79 tasks, at least one expensive model ties the top score and loses on price alone. A fixed-model policy pays for broad capability on every task, while a router can ask what capability a task actually requires.

A few models cover most of the pool

Fireworks found that the best pair of models adds 13.1 points over the best single model, and the best trio reaches 91.2%. Expanding from three to all 18 models adds another 6.4 points. The company cites LLMRouterBench, which evaluates routing across 33 models and more than 400,000 instances, and finds that a handful of models covers most of what the full set can do. The useful object is a portfolio with complementary coverage, not a catalog of hundreds of nearly interchangeable models.

The open-weight subset also performs well. Restricting the oracle to six open models, DeepSeek V4 Flash and Pro, GLM-5.3 and GLM-5.3 Flash, Kimi K3, and Qwen3.8 Max, still reaches 90.3% at $1.45 per task. That beats every closed model in the field by 16 points while spending under a quarter of what GPT-6 Astra costs. Fireworks says this makes the open versus closed debate less interesting and points to a race toward building a system of models with collectively better intelligence than any single model.

The hard part is predicting before the outcome

A production router has to choose before it sees the result, and a wrong choice costs more than the few dollars it saved. Fireworks measured strictly with pass@1, the probability that a single attempt passes, rather than a rule that counts a model as successful if it ever succeeded across four attempts. The company notes that LLMRouterBench finds several recent routing approaches, including commercial ones, fail to reliably beat simple baselines, and traces much of that to model recall: even when a model with the right capability exists, the router has to recognize when to reach for it.

Fireworks argues that sticking with one known model is rational because a stable error distribution beats a router that unpredictably picks the wrong specialist. The bar for a routing system is to make model specialization predictable enough that changing models improves the system without making its behavior less trustworthy.

FireRouter and production traffic

Fireworks positions FireRouter as a task-level router across open and closed models that is cache aware, so switching models does not discard context already paid for. Over four weeks of the company's own production coding traffic, sessions routed through FireRouter cost $7.42 against $15.81 for Claude Opus 5 alone, a 53% reduction across 2,334 sessions. Fireworks says the selection policy becomes a component of the system alongside context, tools, and tests, and that the experiment measures only the simplest version: pick one model at the start of a task and leave it there.

All cost figures come from the DeepSWE leaderboard's published per-model numbers. The raw cost field in the public trials file does not match the board display, so Fireworks scaled each model's per-task costs to match the published mean. Accuracy comes from four raw rollouts per task-model pair, and the data was refreshed on September 17, 2026.