Fireworks: DeepSeek V4 Pro beats GPT-5.5 on CyberGym, costs quarter per solve

Fireworks AI published results from running DeepSeek V4 Pro 0813 through CyberGym, a benchmark built at UC Berkeley from 1,507 real vulnerabilities across 188 open-source projects. The test asks an agent to locate a flaw in a real codebase, build an input that triggers it, and then patch it. Grading is by execution, not by a judge model: the proof of concept either crashes the target or it does not. Runs averaged 89 turns and 25 minutes per task.

Across 840 traced primary tasks, DeepSeek V4 Pro produced native tool calls on all of them, with no refusals and no output-length truncations. In a matched cohort of 697 tasks, the model entered stage-1 validation on 678 of them, a 97.3% entry rate. For contrast, Fireworks reported that Claude Opus 4.8 entered validation on only 70 of 697 tasks, a 10% entry rate, and finished with a 5.9% reward. The company attributed the low entry rate to checks that blocked the CyberGym utility on newer closed models, and noted that older Opus runs often produced no usable artifact.

Cost and accuracy on the common cohort

On the strict four-model common-valid cohort of 697 tasks, Fireworks reported these results: Kimi K3 had a 68.4% reward rate at $3.363 per task and $4.914 per reward. DeepSeek V4 Pro 0813 had a 53.7% reward rate at $1.343 per task and $2.502 per reward. OpenAI GPT-5.5 had a 47.6% reward rate at $4.592 per task and $9.641 per reward. Claude Opus 4.8 had a 5.9% reward rate at $1.957 per task and $33.275 per reward.

Of the 697 tasks, 38 were solved by DeepSeek V4 Pro and nothing else. Kimi K3 uniquely solved 97, GPT-5.5 got 41, and Opus 4.8 got exactly one. A total of 101 tasks were solved by none of the models.

Fireworks said DeepSeek V4 Pro beat GPT-5.5 outright on both quality and price. On accuracy, the open model was plus 6.03 points with a 95% confidence interval of [+1.29, +10.62] and a McNemar p value of 0.0152. It won 164 tasks that GPT-5.5 lost and lost 122 that it won. The cost per solved vulnerability was a quarter of GPT-5.5's.

Where the open model fails

Kimi K3 won on raw solve rate by 14.8 percentage points, with a paired bootstrap 95% confidence interval of [-19.23, -10.33] and a McNemar p value of 9.43 x 10^-11. Fireworks acknowledged that if success rate is the main criterion, K3 is the best choice. But on identical tasks, DeepSeek V4 Pro cost $2.50 per solved task versus K3's $4.91, a 1.96x cost advantage. Across an 863-task diagnostic, that 2x ratio held.

In a standalone run of 920 tasks, DeepSeek V4 Pro solved 462 of them, a 50.2% rate, costing $1.31 per run and $2.60 per success. Sub-scores were 54.7% on ARVO tasks and 48.7% on OSS-Fuzz tasks. Fireworks noted that once the model completes stage-1 validation, it finishes the full patch 94.7% of the time. The failure point is upstream, at generating the initial exploit, which the company described as an actionable prompt-and-scaffolding fix rather than an unfixable model limitation.

For teams building high-volume security agents, the implication is that an open model can match or beat closed frontier models on accuracy while costing less per solved task, and it avoids the refusal walls that Fireworks said blocked newer closed models from completing the CyberGym utility. The company cited a cybersecurity playbook built on Fireworks by Trilogy, which described the defender's workload as high-volume: auditing every repo, triaging every scan, reviewing every PR, and retesting every fix.