Field report NW-32 / L9

Filed from the territory

Three Top AI Security Models Agreed on Just 1 of 121 Vulnerabilities

Echo tested Claude Mythos 5.1, GPT-5.6 Sol and GLM-5.3 on the same code: 83% of 121 findings came from a single model, and only one bug was found by all three.

Grid
NW-32 / L9
Surveyed
November 19, 2026
Reading
4 min read
Correspondent
Elena Vasquez
Three Leading AI Models Hunted the Same Code for Vulnerabilities and Agreed on Just 1 in 121
Fig. NW-32 / L9Three Leading AI Models Hunted the Same Code for Vulnerabilities and Agreed on Just 1 in 121 — fancycrave1 / pixabay

Field notes

  1. Only 1 of 121 distinct potential vulnerabilities was found by all three AI models; 83% were found by just one.
  2. Models rated 45% of findings High or Critical, but only 26% held up after validation testing.
  3. Run costs: about $303 for Claude Mythos 5.1, $30 for GPT-5.6 Sol, $22 for GLM-5.3.
  4. 31 of 55 High/Critical findings were confirmed at High severity via automated proof-of-concept testing.
  5. Research published November 18, 2026; projects tested remain unnamed pending responsible disclosure.

Three leading AI models scanning the same open-source code agreed on exactly one of 121 potential security vulnerabilities, according to research published November 18 by Echo, the frontier-hardened software firm. The overlap figure means attackers and defenders using different models may be looking at entirely different weaknesses in the same software.

Echo tested Anthropic's Claude Mythos 5.1, OpenAI's GPT-5.6 Sol (accessed via Daybreak, OpenAI's program for vetted security teams) and Z.ai's GLM-5.3 against three widely used open-source projects, using an identical hunting framework and validation process. The nine runs produced 446 raw findings; after removing repeats, unreachable issues, documented behavior and false positives, 121 distinct potential issues remained.

The breakdown shows near-total divergence. Only eight findings were flagged by all three models before filtering, and just one survived. Of the 121 distinct findings, 100 — 83% — were found by a single model, 20 by two, and one by all three.

Why the divergence matters for security teams

The asymmetry between attackers and defenders is the core risk. GLM-5.3 is open-weight and available to anyone, and Anthropic's own research found it close to Claude Mythos Preview at building exploits. If each model surfaces different flaws, a company relying on one could miss the exact weakness an attacker finds with another.

"Everyone is racing to figure out which AI model is best at finding vulnerabilities, but our research shows that's the wrong question," said Yuval Saggie, Security Researcher at Echo. "We gave three leading models the exact same code, and they came back with almost entirely different findings. Think about what that means when an attacker is hunting with one model and a defender is relying on another. The defender might never even see the vulnerability the attacker is about to exploit."

The research also dismantles the idea of a single champion. No model won across every project. Mythos had the most findings pass validation overall, but GPT-5.6 Sol led on one project, and GLM-5.3 found validated issues neither rival spotted.

How badly did the models misjudge severity?

The models could not agree on how dangerous the same bug was. One issue in an npm package with more than 350 million monthly downloads was rated Critical by GLM-5.3, High by Mythos, and dismissed as documented behavior by GPT-5.6 Sol. Independent validation rated it Low.

Overconfidence was systemic. The models rated 45% of their 121 distinct findings High or Critical. After testing, only 26% held up at that severity. Without validation, security teams would face a flood of reports requiring no action.

Spending more did not buy coverage. GLM-5.3 was the least reliable of the three at judging exploitability, but at roughly $22 per run — versus $303 for Mythos and $30 for GPT-5.6 Sol — it still found validated issues Mythos missed.

What were the test conditions?

Echo ran each model once against a Maven library with more than 24,000 GitHub stars, a Maven library with over 80,000 lines of code, and an npm package with more than 350 million monthly downloads. All runs used the same Echo hunting framework, code version and validation setup. The 55 findings rated High or Critical went to automated proof-of-concept testing, with GLM-5.3 generating the PoCs; 31 were confirmed at High severity. Echo describes the exercise as a controlled comparison, not a full benchmark, and the automated validation is not confirmation by project maintainers.

Costs cover the hunts only, based on token pricing as of November 1, 2026. Including PoC validation, totals came to about $363 for Mythos, $45 for GPT-5.6 Sol and $40 for GLM-5.3. The models surfaced more than 30 high-severity issues, confirmed through testing within hours. Echo is coordinating responsible disclosure and will not name the projects until maintainers are notified.

Who fixes what AI finds?

"AI is making it dramatically easier and cheaper to find vulnerabilities, but protecting against them is a different story," said Eylam Milner, co-founder and CTO of Echo. "Running multiple frontier models is only the beginning. You still need to validate findings, develop and test patches, and continuously deliver fixes across thousands of software components. As AI accelerates vulnerability discovery, the gap between finding security issues and actually fixing them is only going to grow."

As these tools spread, open-source maintainers will receive more vulnerability reports than ever, while businesses depending on their code wait for fixes to travel upstream. Echo's own product, Echo Frontier, runs several frontier models, validates findings, builds and tests fixes, and delivers patches to customers before public disclosure. The full methodology and data are available at echo.ai/frontier-research.

The one-line takeaway: in AI-driven security, the model you choose may determine which vulnerabilities you never see.

via itbusinessnet.com (Source)

Share this article:

More from Elena Vasquez

Elena Vasquez

Show full bio

Staff writer covering companies, deals and strategy in corporate strategy and chief-executive leadership at CEO Frontier.

489 articles

Filed under

Adjacent sheets

« Previous articleNext article »

End of sheetNW-32 / L9 · November 19, 2026