GLM and DeepSeek Are Catching Frontier Models at Finding Vulnerabilities

Intro

It’s becoming obvious that relying on frontier models is getting to be risky business. Either the model gets export controlled, or it gets nerfed with hidden guardrails. The second one is the more annoying problem for defensive cybersecurity work. When a model like Fable outright blocks anything security-related, that’s at least transparent, because we know upfront it doesn’t work for cybersecurity. The real problem is hidden guardrails, a frontier model quietly stops working for a use case and we can’t tell why, and sometimes an older version performs better than the new one (more on that later in the blog).

If open-weight models are catching up, we can de-risk that dependency. Right now, there’s a lot of noise about open-weight models, Chinese ones in particular, catching up to frontier models on cybersecurity. Most of the time these sources are unreliable and potentially testing against poisoned datasets.

We wanted a concrete test of whether they’re good enough to put into our own workflows. This is one of those internal runs to understand open-weight model capabilities. The setup is we isolate the Hacktron orchestration and harness that we use in our Hacktron Pentest and Review, swap in eight different models one at a time, and measure how many real vulnerabilities each one finds on the same codebase. The result is that open-weight models are still trailing the best frontier models, but the gap is narrow and closing fast.

Benchmark and Orchestration

This is one of the new datasets we’re using to test model capabilities. It’s built from Fider, an open-source feedback/feature-voting app written in Go and React, the same target we used to compare Hacktron against XBOW and Aikido. Between v0.33.0 and v0.36.1, the Fider maintainers patched a set of vulnerabilities that XBOW and Aikido had reported. Those patches were published after the training cutoff of the models under test, which limits the risk of data poisoning. The question this benchmark asks is simple, can each model rediscover the bugs a maintainer already fixed?

A few reasons this dataset is a good quick test of model capability:

1. The patches are recent. Every fix we score against was published after the stated training cutoff of the models we tested, so no model should have seen the patch itself.
2. It’s a typical, average application. Not a deliberately-broken teaching app like DVWA, and not a kernel or browser target whose exotic complexity demands a completely different skill set. Fider is the kind of Go + React web app most security teams actually have to review, which makes the results more likely to transfer to real work. One note is, this is still a small app compared to usual apps.
3. The bugs come from strong external teams. These were surfaced by XBOW and Aikido, two big teams building AI security harnesses. In our earlier writeup the Hacktron harness matched and beat both on cost and speed, so the harness is a known quantity. That lets us hold it fixed and isolate the one variable that changes here, the model driving it.

Note: the orchestration has no internet access, so the agents can’t look up the answer online. They’re given restricted tools that only let them read code and write test scripts.

Data: The 15 patched bugs

These are the real patches we score against. We’re particularly interested in high and critical severity vulnerabilities that allow full compromise of user accounts or access to sensitive data.

The results

Here’s how the results look. Frontier models are in muted grey, open-weight in green, and the two native-agent baselines (GPT 5.5 in Codex, Opus 4.8 in Claude Code) in amber:

The numbers we care about most are critical and high severity recall.

And here’s the matrix showing what each model found.

Interpreting the results

Claude Opus 4.8 underperforms open models on this run

We’re not sure why this happens, but on this codebase Opus 4.8 (6/15) landed below two open-weight models, DeepSeek V4 (8/15) and GLM 5.2 (7/15), and below the older Sonnet 4.6, at a fraction of the cost. Our best guesses are that guardrails are getting in Opus’s way, or that the extra reasoning makes it second-guess and drop real findings.

Orchestration and harness matter

“Harness” is a muddled term, so to be clear about what we mean, the harness is the agentic loop wrapper around a model, Claude Code and Codex are harnesses in that sense. Orchestration is the manager on top of it, in our case it does a bunch of deterministic work and drives the harness through different kinds of vulnerability tasks, first to find, then to validate.

A well-designed orchestration on top of the harness beats just launching Claude and asking it to find bugs. We cna look at the results, GPT 5.5 with Hacktron orchestration outperforms GPT 5.5 in Codex, and the same holds for Opus. Concretely, the native harness missed the critical we care about. DeepSeek V4 and GLM 5.2 in Hacktron beat GPT 5.5 in Codex and Opus 4.8.

A cheaper model in a good harness beats a frontier model in a generic one. On top of that, deterministic orchestration gives us confidence we actually covered the whole codebase, when you just ask Codex for vulns, you can never be sure it looked everywhere.

Further, there is also false sense of coverage comes in. If you use Claude Code and it hands you a few real bugs, it’s easy to walk away thinking it found everything. It didn’t, it found the bugs that it looks in limited coverage and it is enough to convince you to trust it, and the unknown unknowns are still sitting there to bite you later. Finding some bugs is not the same as covering the codebase, and a native agent gives you no way to tell the difference.

Frontier still leads on the critical bugs

GPT 5.5 and Gemini 3.5 Flash found almost every critical and high severity bug that XBOW and Aikido reported and the maintainer patched. Sonnet 4.6 and DeepSeek V4 trail behind them. You can find a pile of bugs, but if you miss the one critical that leaks data or compromises the system, the run doesn’t count for much.

The clearest example is the mass-assignment pre-auth ATO (74a26a31), one of the three criticals. Only GPT 5.5 and Gemini 3.5 Flash caught it. Sonnet 4.6, Opus 4.8, and every open-weight model missed it. It’s a subtle bug that needs a bit of reasoning and some Go-specific knowledge like exported struct fields can look server-owned, but unless binding is explicitly restricted, request JSON can still overwrite them.

Open-weight models are catching up

Older open-weight models were much worse at recall and terrible at precision. Now they’re catching up, and on this evidence they look roughly 3 to 6 months behind the current frontier (e.g. GPT 5.5). If that trend holds, we could eventually move most of our workflow onto open-weight models and reserve frontier models for the critical spots, as advisories and orchestrators.

It’s fairly clear that with good orchestration you can compete with frontier models at much lower cost. But some bugs need a level of capability the model simply doesn’t have, and no harness will close that gap. A better harness gets more out of whatever capability the model has, it can’t manufacture capability that isn’t there.

Conclusion

Open models are getting better with around 6 month capability gap, there’s a good chance the next DeepSeek and GLM models reach the threshold we’re looking for, and once they do we’d switch them into parts of our workflow, doing the bulk of our workloads while keeping a frontier model as the orchestrator and advisor for the hardest calls. The one thing we’d still need is a good enough inference provider to support our workload at speed and reliability.

That said, if frontier models keep getting cheaper and ship guardrails that don’t quietly nerf security work, it makes complete sense to keep using them, at the end of the day we would pick whichever model is efficient for us in pareto frontier.