The Single Bug Mythos Found in curl — Marketing and Reality in AI Security Analysis

Mythos, which Anthropic called “dangerously capable,” found just one vulnerability in one of the most rigorously tested C codebases on Earth. Does that point to the model’s incompetence, or does it point to the fact that we have been measuring the value of AI code analysis on the wrong axis?

Introduction — The Model That Was Supposed to Bring a “Tsunami of Disaster”

In April 2026, Anthropic announced that its new model Mythos was “dangerously” good at finding security flaws in source code. So good, in fact, that it could not be released to the general public immediately; the company would first hand it to a curated set of enterprises and projects to give defenders a head start. The press dutifully ran with the framing. Some security executives reported to their boards that a “tsunami of vulnerabilities” was about to land, and pulled budget forward. The CISO of one Dutch quasi-public agency wrote on Hacker News: “Thanks to my boss being slightly panicked by Mythos, our security budget went up. Never waste a good marketing scare.”

On May 6, the model was finally pointed at the curl source tree. curl is a C project that has been refined continuously since 1996 and now spans roughly 178,000 lines of source. It is installed across some 20 billion instances, running on 110 operating systems and 28 CPU architectures. With 188 cumulative CVEs published over its lifetime, it is not a bug-free codebase — but it has been worked over by OSS-Fuzz, Coverity, CodeQL, paid audits, and, over the past 8–10 months, by AI-based security tools including AISLE, Zeropath, and OpenAI Codex Security. AI tools alone produced 200–300 merged bug fixes during that window. By the time Mythos arrived, curl was effectively “hard mode.” What the new model could find in such a codebase was the small but real test of May.

Lead maintainer Daniel Stenberg posted the results to his blog on May 11. The title says it all: “Mythos finds a curl vulnerability — yes, as in singular one.” That one line punctured the marketing balloon Anthropic had inflated in April. But on the same day, the 685-point Hacker News thread did not simply ratify Stenberg’s conclusion. The top-voted comments pushed back, arguing not “Mythos isn’t dangerous” but “curl is the exception.” That asymmetry is the starting point for this essay.

Section 1 — How Five Became One

The report Stenberg received from Mythos covered 178K LOC and listed five “Confirmed security vulnerabilities.” He noted dryly that “I think using the term confirmed is a little amusing when the AI says it confidently by itself.” After several hours of review by his security team, only one of the five turned out to be a real vulnerability. The other four broke down as follows. Three were false positives — limitations already documented in the API spec being mislabeled as flaws. The last was, in Stenberg’s words, “just a bug” — a behavioral inconsistency with no security impact.

The remaining one is a severity-low CVE to be disclosed alongside the curl 8.21.0 release planned for late June. In Stenberg’s phrasing, it is a flaw “not going to make anyone grasp for breath.” A single low-severity CVE. That was the substance of the “tsunami.” The distance between the April marketing message and the May report was more than the usual expectations adjustment.

That said, the report was not entirely without value. Roughly twenty findings were classified as ordinary bugs rather than vulnerabilities, and Stenberg explicitly praised this portion. “Barely any false positives, so I presume they have had a rather high threshold for certainty.” The curl team is going through the twenty one by one and merging patches for the ones they agree with. In other words, Mythos came up short as a “security tool” against the hype, but earned its keep as a “code reviewer.”

Read without context, the result makes the model look weak. But the first sentence of Mythos’s own report — which Stenberg quoted — is decisive. “curl is one of the most fuzzed and audited C codebases in existence (OSS-Fuzz, Coverity, CodeQL, multiple paid audits). The likelihood of finding something on the hot path (HTTP/1, TLS, URL parsing core) is low.” And indeed, the model found nothing on those hot paths. The model accurately read the difficulty of the environment it was entering, and its output matched that read. That kind of meta-awareness was rare in earlier tools.

The numerical comparison is also worth doing. Over the prior 8–10 months, AISLE, Zeropath, Codex Security, and similar AI tools had collectively driven 200–300 bug fixes into curl, of which “several dozen” became actual published CVEs. Mythos, arriving in the same codebase, produced one low CVE and around twenty ordinary bugs. By simple arithmetic, that is less than one month of output from the earlier tools. But the curl that those earlier tools encountered and the curl that Mythos encountered are not the same codebase. The latter is the residue left after the former. That is the precise point at which Stenberg’s conclusion and the majority HN view diverge.

Stenberg’s conclusion is two lines. “The big hype around this model so far was primarily marketing.” And “I see no evidence that this setup finds issues to any particular higher or more advanced degree than the other tools have done before Mythos.” He did add a caveat that evaluating a whole model on the results of a single codebase is unfair. But the headline congealed around “marketing hype.”

Section 2 — Stenberg vs. HN — The Trap of “Well-Hardened Code”

An interesting asymmetry emerged here. Stenberg’s post hit 685 points and drew 281 comments. The top-voted comments did not endorse Stenberg 100 percent. patrickmeenan staked out the opposing side clearly. “Mythos’s message is that it makes top security expertise, top language/protocol/code expertise, available to anyone. The risk was releasing that access to the world before defenders could access that level of expertise. curl, as a foundational tool, has been picked over by security, protocol, and language experts for years. That Mythos found anything is interesting but not a sign that it’s been marketing hype and isn’t dangerous. You can bet 99.99% of projects are not as secure as curl. It doesn’t matter whether they’re open or closed — LLMs will decompile and explore closed source too. If a project hasn’t been through fuzzing, prior AI tools, and expert review, you should already assume it’s hackable.”

srcreigh reached the same conclusion from another angle. “curl is essentially a relatively simple, well-isolated tool. Compare it to operating systems, web browsers, databases, or the codebases of multi-billion-dollar companies. It makes sense that Mythos/ChatGPT 5.5 is much better at the kind of complexity curl doesn’t have. curl is clearly full-featured as a ‘client for anything,’ but it’s orders of magnitude less complex than other software we rely on.”

EMM_386’s comment is more direct. “If an AI agent finds zero bugs in a given software utility, how does that mean ‘AI is not very good at finding bugs’? What if there really are zero bugs? Saying ‘5 felt like nothing compared to the broad list we expected’ just means expectations didn’t match reality — not that Mythos is less capable than claimed.” yjftsjthsd-h directly quoted Stenberg’s phrase “not particularly dangerous” and pushed back: “I don’t think that follows. As the post itself notes, curl was already analyzed to death with every tool available; most software isn’t at that level.”

These objections compress into a single line: “Results against well-hardened code cannot be used to conclude anything about the tool’s general performance.” Stenberg himself buried a caveat in the body — “This is just one source code repository and maybe it is much better on other things.” But his headline conclusion was “marketing hype,” and the caveat lived under it. The HN majority effectively lifted that buried caveat to the top.

That said, Stenberg’s conclusion carries empirical weight too. He is not a casual outside observer; he is the person who actually triaged five down to one. He saw what kinds of false positives the four bad ones were, and he judged that, qualitatively, they were not different from the false positives of earlier AI tools. When he says “I see no evidence that this setup finds issues to any particular higher or more advanced degree,” that is the comparative testimony of someone who has run five different tools across a year. Even if patrickmeenan and srcreigh are right, there is no answer to “Then on what other codebase would you validate Mythos?” Anthropic’s marketing said “dangerously good,” and that danger was implicitly general. If the curl result partially refutes that generality, the refutation has meaning in itself.

rzmmm’s first comment captures the ambiguity most cleanly. “It’s a good reminder for us all that the competition in this space is rough and lots of more or less subtle marketing is involved.” The conclusion that Mythos is incompetent and the conclusion that well-hardened code is immune are both only partly correct. To take only the partly correct portions of each, we need a new evaluation axis.

Section 3 — A New Evaluation Axis for AI Code Analysis

To date, AI code analysis tools have been graded almost exclusively on the metric “how many CVEs did you find in benchmark codebase X.” curl was effectively one of those standard benchmarks. But the May results expose the fact that this benchmark no longer discriminates. Against curl, any tool is now likely to produce single-digit results, because there are almost no novel flaws left to find. To distinguish between tools in this state, the axis itself has to change.

The first new axis is signal-to-noise ratio in reported findings. The most striking part of Mythos’s scorecard is not that one of five security findings was real (20% precision), but that the report had “barely any false positives” among the roughly twenty non-security bugs. That is the part Stenberg himself praised, and the basis for the inference that Mythos set its certainty threshold high. One reading: that threshold was too loose for the security domain and appropriate for the general bug domain. The fact that all four security false positives were “limits documented in the API spec” reveals a structural weakness — the model analyzed the code but did not read the docs. That points to what the next generation of tools needs to augment: combined analysis of code + spec/documentation.

The second new axis is self-awareness about codebase difficulty. The fact that the Mythos report itself wrote, “curl is one of the most fuzzed/audited C codebases. The likelihood of finding something on the hot path is low,” is something rarely seen in previous static analyzers. The model estimates “what should be expected from this environment” before producing results, and then produces results consistent with that estimate. If that self-awareness is honest — and from Stenberg’s report it appears to be — then what the tool reports to its user is no longer just a list of findings but an evaluation of the security maturity of the codebase. For a CISO, that evaluation may be more valuable than the findings list.

The third axis is the one patrickmeenan and srcreigh pointed to: a function of complexity and unaudited surface area. 99.99% of projects have not been reviewed to curl’s depth, and operating systems, browsers, databases, and in-house monolithic codebases are orders of magnitude more complex than curl. The real test of AI code analysis is over there, not here. If Anthropic’s next disclosure includes flaw statistics from private enterprise codebases — anonymized — then the May “marketing hype” critique may be partially recovered. But at present, no such data exists. The only verifiable basis for the “dangerously good” claim is the curl result, and the curl result is weak.

The fourth axis is the most operationally meaningful: value as a PR-review assistant. As Stenberg notes in his post, the curl team already uses GitHub Copilot and Augment as PR-review bots, and these augment rather than replace human review (“They help us, they don’t replace us”). The twenty general bugs from Mythos effectively fall into this category. It is less exciting than the security marketing message but more durable. The compounding effect of an AI tool reinforcing a human reviewer is cumulative, measurable, and has a low false-positive cost. Security hype creates a large single-announcement impact but collapses in one go on a result like “one out of five.” Which side the market chooses to value is the next chapter of this story.

Conclusion — What to Measure

Back to the lead question. Does the fact that Mythos found exactly one low-severity CVE in curl point to the model’s incompetence, or to the fact that our evaluation axis is wrong? The answer is that both are partly correct. The model did not, on curl, demonstrate the general capability implied by the “dangerously good” marketing message. To that extent, Stenberg’s “marketing hype” is accurate. At the same time, curl is a codebase that has been picked over for eight years by every kind of tool, and a single-digit result in such a codebase says more about the state of the code than about the model. To that extent, patrickmeenan’s rebuttal is accurate.

The problem is that when both statements are simultaneously correct, the marketing only takes one side. Anthropic claimed generality with the word “dangerously”; Stenberg refuted that generality with the result of a single codebase. HN responded that the single codebase is not a representative sample of generality. What is missing from this triangle is data. Anonymized statistics from generic enterprise codebases about what Mythos finds, false-positive rates, signal-to-noise ratios. Without that data, no side can close the argument.

But one thing this incident did make clear: the era of evaluating AI code analysis purely by CVE count is over. Well-hardened code yields almost no CVEs. The value of a new model is decided not by flaw counts but by a multi-axis evaluation — precision of the flaw signal, meta-evaluation of codebase maturity, compounding utility as a PR-review aid, and discovery rate in unaudited territory. The fact that Mythos found only one in curl is the kind of headline that fits on a single slide; the fact that it had almost no false positives is the slide that next quarter’s tool-selection meeting will turn on. When the market learns to look at the second slide instead of the headline, the AI security analysis industry will move to the next stage of its hype cycle. Stenberg’s May 11 post became the first textbook for that lesson.