Mythos vs. GPT-5.6 Sol vs. GLM-5.3

Executive summary
Frontier AI models are changing the speed and scale of vulnerability discovery, making it possible to uncover security weaknesses in widely used software faster and more cheaply than ever before. As these capabilities become increasingly accessible to both attackers and defenders, the race to stay ahead is seriously intensifying.
So naturally, everyone wants to know: which AI model is best at finding vulnerabilities?
Echo researchers decided to put the three leading frontier models – Anthropic's Claude Mythos 5.1, OpenAI's GPT-5.6 Sol (with Daybreak Blue), and Z.ai's GLM-5.3 – to the test against the exact same code across three widely used open-source projects. Together, they produced 121 distinct security findings, and only one was identified by all three.
83% of the findings were unique to a single model, revealing just how differently these systems approach vulnerability discovery, even when given the same task, and no single model consistently came out ahead across all three projects. Faster and cheaper models uncovered validated findings that more expensive models missed, while the models often reached dramatically different conclusions about the severity of the same issue.
The research also exposed how difficult it is to separate meaningful security issues from the massive volume of potential vulnerabilities AI can generate. Across nine vulnerability-hunting runs, the models initially produced 446 findings, but only 42 of them passed automated validation, meaning ~91% of initial findings were filtered out before reaching a validated result.
For security teams, the implications go well beyond choosing the right model. Different models can uncover entirely different vulnerabilities in the same software, and combining their capabilities creates a growing need to validate, prioritize, and remediate what they find. Thus, the real advantage may not come from finding the single best model, but from combining their strengths to turn AI-powered discovery into actual protection.

Model discoveries: Overlap in just 1 of 121 findings
One of the most striking results was how little agreement there was across the discovered vulnerabilities. Echo gave Mythos 5.1, GPT-5.6 Sol, and GLM-5.3 the same vulnerability-hunting task against the same exact code across three widely used open-source projects:
- A Maven library with more than 24,000 GitHub stars.
- A Maven library containing more than 80,000 lines of code.
- An npm package with more than 350 million monthly downloads.
Despite analyzing the same code, the models returned remarkably different findings. Of 121 distinct security findings, 100 (83%) were identified by just one single model, 20 (17%) by two, and only one (0.8%) by all three. This challenges the assumption that frontier models are interchangeable in security research.
For organizations evaluating AI-powered security tools, the implications are significant given that a high number of findings doesn't guarantee comprehensive coverage, and strong performance on one codebase doesn't necessarily translate to another. And achieving broader coverage may require running multiple frontier models, adding substantial compute costs on top of the validation and remediation work that follows.

The larger Maven library produced the most distinct findings, but this shouldn't be interpreted as a measure of relative security risk across projects. What matters is the variation between models. Even with the same code and hunting infrastructure, they explored different weaknesses and reached different conclusions.
Model performance: Mythos vs. GPT-5.6 Sol vs. GLM-5.3
If the goal is to identify the best model for vulnerability discovery, which one performed best? The answer depends on the project and how performance is measured. Across all three projects, Mythos led with 25 model-attributed validation passes, followed by GLM-5.3 with nine and GPT-5.6 Sol with eight. But the results varied significantly by project.
Maven library – 80K+ lines of code
Mythos led with 21 of 29 validation passes, followed by GLM-5.3 with six and GPT-5.6 Sol with two. Notably, GLM-5.3 uncovered six validated findings the other models missed, showing why Mythos's lead didn't necessarily mean better overall coverage.
Maven library – 24K+ GitHub stars
GPT-5.6 Sol dominated with six of seven validation passes, while GLM-5.3 accounted for one. Mythos had none because it classified all five of its final findings as Medium, below the High/Critical threshold for testing. This highlights how severity assessments can determine whether a potentially legitimate vulnerability is ever validated.
npm package – 350M+ monthly downloads
Six findings passed automated validation: four attributed to Mythos and two to GLM-5.3. GPT-5.6 Sol had none, as all its findings fell below the High/Critical threshold.
Importantly, findings identified by multiple models weren't necessarily tested more than once. If an issue was already in the validation pipeline, duplicate submissions were filtered rather than retested.

These are model-attributed validation passes, not necessarily 42 distinct vulnerabilities. Duplicate issues can be attributed to more than one model.
Speed and cost comparison
The models also differed significantly in how long they took to complete their vulnerability hunts and how much those hunts cost.
Across all three projects, GPT-5.6 Sol was by far the fastest, completing its hunts in approximately 24 minutes. Mythos required approximately 198 minutes, while GLM-5.3 took approximately 174 minutes. The cost differences were equally substantial.

Despite these differences, the research didn't find a simple relationship between speed, cost, and security coverage.
- GPT-5.6 Sol completed its hunts in a fraction of the time required by the other models, yet accounted for six of the seven validation passes on one Maven library.
- GLM-5.3 was the least expensive model overall and still uncovered validated findings that Mythos missed.
- Mythos's longer runtime and higher cost coincided with the largest total number of model-attributed validation passes.
These results challenge the assumption that the most expensive or computationally intensive model will necessarily uncover the most meaningful security issues in every environment. For organizations evaluating frontier models, efficiency should not be separated from the types of vulnerabilities a model identifies, the quality of its severity assessments, and the extent to which its findings complement those of other models.
Severity assessment disagreements
Even when the models identified the same security issue, they often disagreed on how serious it was. In one example involving the npm package, GLM-5.3 classified a finding as Critical, Mythos rated it High, and GPT-5.6 Sol dismissed it as documented behavior. Automated validation ultimately assessed the issue as Low severity.
These differences have real consequences for security teams, as a model's severity assessment can determine which potential vulnerabilities receive further investigation and which are overlooked.
In Echo's testing, only findings initially rated High or Critical advanced to further filtering and proof-of-concept validation. On one Maven library, Mythos identified a previously known security issue but rated it Medium, meaning it never reached that stage.
The opposite happened as well, with some findings initially rated High or Critical turning out to be less severe than expected, unreachable, or impossible to reproduce. Of the 152 final model-attributed findings, 74 initially met the High/Critical threshold, but only 42 model-attributed findings ultimately passed automated validation after additional filtering and testing.
This creates a difficult tradeoff. Overestimating severity can waste valuable resources on issues that don't warrant immediate attention, while underestimating it risks leaving meaningful vulnerabilities uninvestigated.
As organizations increasingly rely on AI to prioritize security findings, the accuracy of those assessments could become just as consequential as the vulnerabilities the models fail to discover. Independent validation is essential to distinguishing genuine security risks from AI-generated noise.
Implications for attackers and defenders
Frontier AI models are making vulnerability discovery faster, cheaper, and more accessible, changing the balance between attackers and defenders.
For defenders, these capabilities create opportunities to identify security weaknesses earlier and reduce the time between discovery and remediation. For attackers, they create new ways to search widely used software for vulnerabilities that have not yet been publicly disclosed.
Echo's findings suggest that organizations relying on a single AI-powered security tool may be operating with significant blind spots. An attacker using one model could uncover vulnerabilities that a defender relying on another model never identifies.
While the research doesn't measure real-world attacker success, it demonstrates why strong performance from one model cannot be treated as comprehensive coverage.
Combining multiple models may broaden discovery coverage, but it also creates a growing operational challenge. More findings must be deduplicated, validated, prioritized, and ultimately remediated.
Without an effective process for separating meaningful security issues from noise, the volume of AI-generated findings could quickly become a burden rather than an advantage.

Beyond discovery: Validation and remediation
One of the most important findings in the research came from examining what happens after frontier models identify potential vulnerabilities.
Across nine vulnerability-hunting runs, the models initially generated 446 findings. After each hunt's cleanup, which removed repeated reports, false positives, and multiple descriptions of the same issue, 152 final model-attributed findings remained. Once overlapping findings across models were consolidated, those 152 reports represented 121 distinct potential bugs.
Of the final findings, 74 were classified as High or Critical and eligible for further assessment. Nineteen submissions were filtered because the underlying issue had already entered the validation pipeline, usually through an earlier model run. That left 55 findings to undergo proof-of-concept testing, of which 42 passed automated validation.
Approximately 91% of initial findings did not reach a validated result, including duplicates, false positives, lower-severity issues, and findings that couldn't be reproduced. This doesn't mean 404 confirmed vulnerabilities were discarded – the initial count includes unverified findings and repeated reports, while the 42 validation passes aren't necessarily unique bugs.
From AI discovery to actual protection
The research highlights just how much work goes into separating meaningful vulnerabilities from the hundreds of potential findings AI models can generate. And even after a finding is validated, it still needs to be investigated, patched, tested, and delivered to the software that depends on it.
Echo Frontier: Moving protection ahead of disclosure
Echo Frontier brings these capabilities together in one continuous process. With access to all leading frontier AI models, Echo can combine their different strengths and incorporate new capabilities as models evolve, expanding discovery coverage beyond what any single model can provide. That's particularly important given that 83% of the research findings were unique to one model.
From there, Echo Frontier's automated pipeline filters out duplicates, false positives, and findings that can't be reproduced, allowing researchers to focus on meaningful security issues. It then develops, tests, and delivers patches through its continuously maintained software components, including backports for older versions.
Customers receive these protections through the software they already use, without having to run vulnerability hunts, investigate findings, or manage patches themselves. And as a CVE Numbering Authority, Echo also coordinates responsible disclosure with maintainers and supports upstream remediation.
By bringing together frontier models, validation infrastructure, and continuous remediation, Echo Frontier turns AI-powered vulnerability discovery into protection that can reach customers before vulnerabilities are publicly disclosed.

Research methodology and scope
Research scope
Echo evaluated three frontier AI models:
- Anthropic Claude Mythos 5.1
- OpenAI GPT-5.6 Sol
- Z.ai GLM-5.3
GPT-5.6 Sol was run with Daybreak Blue enabled, OpenAI's access tier for approved defenders.
The models were tested against three widely used open-source projects, described without identifying their names or versions while responsible disclosure and remediation work continues:
- A Maven library with more than 24,000 GitHub stars.
- A Maven library containing more than 80,000 lines of code.
- An npm package with more than 350 million monthly downloads.
Each model completed one vulnerability-hunting run per project, resulting in nine runs. All three models analyzed the same target code version for each project using Echo's production vulnerability-hunting framework.
Findings and validation
The nine runs produced 446 initial findings, reduced to 152 after each hunt's cleanup. Consolidating overlapping reports across models resulted in 121 distinct potential bugs.
Only findings initially classified as High or Critical advanced to further filtering and proof-of-concept testing. Of the 74 findings meeting that threshold, 19 submissions were excluded because the same issue had already entered the validation pipeline. The remaining 55 underwent proof-of-concept testing, with 42 passing automated validation. These 42 results represent model-attributed validation passes, rather than necessarily distinct, independently confirmed vulnerabilities.
Time and cost
Hunt duration was measured for each model and project. Estimated costs are based on the pricing available to Echo for each model as of November 1, 2026, and exclude downstream validation, patch development, testing, and engineering expenses. Pricing for GPT-5.6 Sol and GLM-5.3 reflects Echo's access terms rather than a public rate card, and GPT-5.6 Sol's estimate reflects promotional pricing, so costs may differ for other organizations.
Research limitations
This research is a controlled comparison rather than a comprehensive benchmark of each model's cybersecurity capabilities. Each model was tested only once per project, and results may vary across additional runs, configurations, or software targets.
The threat-modeling step covers up to 250 source files. The two smaller libraries fit within that limit and were covered in full. For the larger Maven library, it covered 250 of roughly 420 in-scope files, so that project was only partly covered.
The High/Critical severity threshold also influenced validation results, as findings rated Medium or Low did not advance to proof-of-concept testing, even when they might have represented legitimate security issues.
Finally, the 121 distinct findings include issues that were not independently validated, and automated validation does not constitute confirmation by project maintainers. Echo is coordinating responsible disclosure and thus withholding technical details that could expose unpatched vulnerabilities.
Research team
This research was led by Yuval Saggie, Security Researcher at Echo, with key support from Laiza Angrest, Engineer at Echo, and Mor Weinberger, Architect at Echo. Using Echo's vulnerability-hunting and validation infrastructure, the team evaluated three leading frontier AI models, examining differences in vulnerability discovery, severity assessment, and validation results across three widely used open-source projects.
Key takeaways
- There is no single clear winner. Mythos produced the most validation passes overall, but GPT-5.6 Sol led on one Maven library, while GLM-5.3 uncovered validated findings the other models missed. Meaning, performance varied significantly by project.
- Different models find different vulnerabilities. Of 121 distinct findings, 100 (83%) were unique to a single model. Just one was shared by all three, highlighting the potential blind spots of relying on a single model.
- Speed and cost don't tell the whole story. GPT-5.6 Sol completed its hunts far faster than the other models, while GLM-5.3 had the lowest estimated cost. Both uncovered validated findings that more expensive or slower models missed.
- Severity assessments can change what gets investigated. Models disagreed on how serious the same security issues were. Because only High/Critical findings advanced to validation, these differences directly influenced which potential vulnerabilities received further investigation.
- AI-generated findings require substantial filtering. Of 446 initial findings, only 42 model-attributed findings passed automated validation. This highlights the importance of deduplication, severity screening, and reproducibility testing.
- Discovery alone doesn't equal protection. Finding a potential vulnerability is only the beginning. Real protection requires validation, prioritization, patch development, testing, and reliable delivery.
About Echo
Echo delivers frontier-hardened software across the software supply chain. From containers and libraries to VMs, OS packages, and more, Echo software is continuously vetted for threats and malware, hardened, and patched against vulnerabilities, including those discovered by frontier models before public disclosure.
Rather than leaving security teams to chase vulnerabilities and manage fixes, Echo delivers secure-by-default software that's ready to deploy, without requiring enterprises to change the tools, versions, or workflows they already use.
To date, Echo has screened more than 26 million package versions, blocked over 4,000 malicious packages before reaching customers, and eliminated more than 2.4 million known vulnerabilities across the artifacts it maintains.
Learn more at echo.ai.
Frontier models can now find and exploit zero days in your code
Find out what to do when there's no CVE, scanner alert or fix available yet.
What are the 7 blind spots in your vulnerability scans?
Discover when "0 vulnerabilities" doesn't actually mean you're clean.
.png)




.avif)
.avif)