Three Leading AI Models Hunted the Same Code for Vulnerabilities and Agreed on Just 1 in 121

AI is changing how quickly vulnerabilities can be discovered in software, with frontier models now analyzing large codebases, identifying potential security weaknesses, and helping researchers uncover issues that might otherwise take significantly longer to find.
But as these capabilities become more powerful and accessible, an important question is emerging: Which AI model is actually best at finding vulnerabilities?
New research from Echo suggests the answer isn’t simple.
Same code, dramatically different findings
Echo's security research team put three leading frontier AI models to the test: Anthropic's Claude Mythos 5.1, OpenAI's GPT-5.6 Sol (accessed through Daybreak), and Z.ai's GLM-5.3.
Each model was tasked with hunting for vulnerabilities in the exact same code across three widely used open-source projects, using the same vulnerability-hunting framework. And the results revealed a striking lack of agreement.
Across 121 distinct potential security issues, only one was identified by all three models. In fact, 83% of the findings were unique to a single model.
That means three highly capable AI systems, examining the same software, uncovered largely different sets of potential vulnerabilities. And the differences went beyond discovery, as the models also varied in their performance across projects, how they assessed the severity of potential vulnerabilities, and how much time and money they required to complete their hunts.
What happens when attackers and defenders use different AI models?
The findings raise an important concern for the future of cybersecurity.
As AI-powered vulnerability discovery becomes more accessible, attackers and defenders can use similar capabilities to search for weaknesses in widely used software.
But if different models uncover different vulnerabilities, an organization relying on a single model could miss the exact weakness an attacker discovers using another.
The research also challenges the growing focus on identifying the single best AI model for security research. Strong performance on one codebase doesn't necessarily translate to another, and finding more potential vulnerabilities doesn't automatically mean finding the most important ones.
Broader coverage may require combining the strengths of multiple models. But that introduces another challenge: the cost and complexity of running frontier models, validating their findings, and determining which vulnerabilities actually pose a risk.
Finding vulnerabilities is only the beginning
AI can dramatically accelerate vulnerability discovery, but identifying a potential weakness doesn't mean it's exploitable, or that a fix is available. AI-generated findings still need to be investigated, reproduced, assessed for severity, and ultimately remediated. As discovery accelerates, this process becomes increasingly important.
That's the challenge Echo Frontier is designed to address. By combining multiple frontier AI models with automated validation, security research, and continuous remediation, Echo identifies previously unknown vulnerabilities, develops and tests patches, and delivers fixes to customers through the software components they already use, including ahead of public disclosure.
The broader takeaway from the research is clear: As AI models become increasingly capable of finding vulnerabilities, relying on a single model can leave organizations with dangerous blind spots. And discovering more vulnerabilities only creates meaningful protection when those findings can be validated and fixed.
Read the full research
How did Mythos, GPT-5.6 Sol, and GLM-5.3 compare across vulnerability discovery, severity accuracy, speed, and cost?
Echo's full research report breaks down the results, methodology, and implications for organizations navigating the next era of AI-powered cybersecurity.
Frontier models can now find and exploit zero days in your code
Find out what to do when there's no CVE, scanner alert or fix available yet.
What are the 7 blind spots in your vulnerability scans?
Discover when "0 vulnerabilities" doesn't actually mean you're clean.
.png)




.avif)
.avif)