A Lot of Misdiagnosis

The experiment was detailed in a blog post published July 9 by Nikos Baxevanis of the foundation’s protocol security team, under a title that doubled as the firm’s thesis, i.e. “The triage is the product.” The findings drew wide attention as the most flagged issues turned out to be false positives (even though there were real bugs in the mix).

The headline discovery is real enough, as the agents helped surface a remotely-triggerable panic in gossipsub, part of the libp2p peer-to-peer networking layer that Ethereum consensus clients run on. The flaw was fixed and disclosed as CVE-2026-34219 (the kind of bug that, if found first by an attacker, could have been used to disrupt nodes across the network).

Finding Bugs Was the Easy Part

The surprise, the foundation wrote, was not that AI agents could find bugs but “how little of the work went into finding them, and how much went into telling the real bugs from the ones that just looked real.”

The team catalogued the recurring shapes of those imposters, à la crashes that only occur in debug builds and never in production, reproducers that rely on unreachable internal values no attacker could actually supply, and formal-verification proofs that are technically true but so unconstrained they demonstrate nothing.

The foundation’s answer was a hard evidentiary standard it summarized as “reproducible or it didn’t happen.” To elaborate, every candidate finding is henceforth required to ship with a self-contained artifact that reproduces the failure against the actual code, independent of how confident the reporting agent claims to be.

Agents, in this context, can be viewed as hypothesis generators (search tools, not decision-makers) organized into recon, hunting, gap-filling, and validation stages, with humans making the final call.

The Numbers Behind the Hype

The post also offered a rare benchmark for how well the current generation of tools performs. A property-based testing agent generated roughly 1,000 candidate findings, and after expert review, about 86% of its top-tier recommendations survived scrutiny (strong for a machine, but a rate that still demands a human filter before anything touches production code).

The tools are clearly finding real vulnerabilities in critical infrastructure, thereby undermining the dismissal that AI-generated bug reports are pure noise. Yet the workload has not disappeared but simply moved downstream to triage, where experienced engineers separate signal from simulation. For a network securing hundreds of billions of dollars in value, that filter is important.

The foundation is now pushing the work forward rather than treating it as a one-off. Its Ecosystem Support Program, for instance, is funding a dedicated grant round for AI-powered protocol security, covering research, auditing, and vulnerability detection.