AI vs. Open Source Legend: Anthropic "Mythos" Takes on Curl, but Daniel Stenberg Remains UnfazedFollowing the launch of Claude Mythos, Anthropic’s latest flagship (and highly expensive) AI model, the company showcased its prowess in cybersecurity by hunting for software vulnerabilities. Today, Daniel Stenberg, the legendary creator of curl, shared his firsthand experience collaborating with Anthropic to put Mythos to the test.
The Evaluation Process
Invited via the Linux Foundation, Stenberg originally expected direct access to the model. However, Anthropic shifted the approach, running the analysis internally on curl’s 176,000 lines of code.
The final report from Mythos acknowledged a significant challenge: curl is one of the most heavily scrutinized projects in history, already protected by a gauntlet of tools like CodeQL, Coverity, and OSS-Fuzz. Finding a critical flaw in such a hardened codebase was never going to be easy.
The Results: 5 Flaws, Only 1 Real Vulnerability
The Mythos report identified five primary issues, but a human review revealed a more nuanced reality:
1 Genuine Vulnerability: Classified as "Low Risk" and scheduled for a fix in version 8.21.0.
3 Documentation Errors: Issues with how features were described, not how they functioned.
1 Functional Bug: A standard coding error that did not pose a security threat.
Beyond these five, Mythos flagged several other bugs that the AI itself admitted were not security-related. The curl team is currently reviewing these findings and will implement fixes where they see fit.
Stenberg’s Verdict: Better, but Not a "Cyber-Weapon"
Stenberg praised Mythos as an impressive leap over traditional static analysis tools. However, he maintained a grounded perspective, stating that in his personal view, Mythos did not feel "exceptionally dangerous" or superior to existing tools in a way that would fundamentally change the threat landscape. It is, at best, a refined evolution of what came before.
Even though Mythos is a GPT-5.5-Cyber compliant AI, the case of curl demonstrates that AI still suffers from "false positives." The fact that only one of the five reported vulnerabilities is a genuine vulnerability reflects that in the world of critical infrastructure security, humans remain the most crucial last line of defense in distinguishing between "annoying bugs" and "system-breaking vulnerabilities."
The article should raise the question of ROI (Return on Investment). Considering Mythos's very high API cost, yet only one low-level vulnerability was found in a curl-level project, the question is: "Is it worth it for typical companies to spend enormous sums of money on AI, or should they instead hire experts to better configure OSS-Fuzz?"
Why curl? Because curl permeates everything from cars to spacecraft. Anthropic's choice to test curl was an attempt to prove that "if we can find vulnerabilities in curl, we can find them everywhere." However, Stenberg's results proved the strength of open-source standards more than the capabilities of AI.
Microsoft Tests Low Latency Profile to Kill Windows 11 Lag for Good.
Source: Daniel Stenberg
AI vs. Open Source Legend: Anthropic "Mythos" Takes on Curl, but Daniel Stenberg Remains UnfazedFollowing the launch of Claude Mythos, Anthropic’s latest flagship (and highly expensive) AI model, the company showcased its prowess in cybersecurity by hunting for software vulnerabilities. Today, Daniel Stenberg, the legendary creator of curl, shared his firsthand experience collaborating with Anthropic to put Mythos to the test.
The Evaluation Process
Invited via the Linux Foundation, Stenberg originally expected direct access to the model. However, Anthropic shifted the approach, running the analysis internally on curl’s 176,000 lines of code.
The final report from Mythos acknowledged a significant challenge: curl is one of the most heavily scrutinized projects in history, already protected by a gauntlet of tools like CodeQL, Coverity, and OSS-Fuzz. Finding a critical flaw in such a hardened codebase was never going to be easy.
The Results: 5 Flaws, Only 1 Real Vulnerability
The Mythos report identified five primary issues, but a human review revealed a more nuanced reality:
1 Genuine Vulnerability: Classified as "Low Risk" and scheduled for a fix in version 8.21.0.
3 Documentation Errors: Issues with how features were described, not how they functioned.
1 Functional Bug: A standard coding error that did not pose a security threat.
Beyond these five, Mythos flagged several other bugs that the AI itself admitted were not security-related. The curl team is currently reviewing these findings and will implement fixes where they see fit.
Stenberg’s Verdict: Better, but Not a "Cyber-Weapon"
Stenberg praised Mythos as an impressive leap over traditional static analysis tools. However, he maintained a grounded perspective, stating that in his personal view, Mythos did not feel "exceptionally dangerous" or superior to existing tools in a way that would fundamentally change the threat landscape. It is, at best, a refined evolution of what came before.
Even though Mythos is a GPT-5.5-Cyber compliant AI, the case of curl demonstrates that AI still suffers from "false positives." The fact that only one of the five reported vulnerabilities is a genuine vulnerability reflects that in the world of critical infrastructure security, humans remain the most crucial last line of defense in distinguishing between "annoying bugs" and "system-breaking vulnerabilities."
The article should raise the question of ROI (Return on Investment). Considering Mythos's very high API cost, yet only one low-level vulnerability was found in a curl-level project, the question is: "Is it worth it for typical companies to spend enormous sums of money on AI, or should they instead hire experts to better configure OSS-Fuzz?"
Why curl? Because curl permeates everything from cars to spacecraft. Anthropic's choice to test curl was an attempt to prove that "if we can find vulnerabilities in curl, we can find them everywhere." However, Stenberg's results proved the strength of open-source standards more than the capabilities of AI.
Microsoft Tests Low Latency Profile to Kill Windows 11 Lag for Good.
Source: Daniel Stenberg
Comments
Post a Comment