Anthropic Aims to Automate Software Verification

The gap between writing software and proving it is safe is closing.

Formal verification, the mathematical proof that code will run without error, has traditionally been a prohibitively expensive undertaking. A team of specialists required 20 person-years to formally verify the seL4 operating system kernel, a process reserved for high-stakes applications like avionics software for Airbus. Anthropic's Frontier Red Team, in conjunction with partner Theorem, has now deployed its Claude model to take on this work, producing roughly 180 million lines of proof code in just a few weeks.

The effort focused on 123 open-source libraries that are part of Google's OSS-Fuzz security project. Rather than generating new code, Claude was tasked with verifying existing software. The model translated crash tests into mathematical problems, analyzed the compiled machine code, and generated extensive proofs of correctness. An independent verification program called Rocq then checked these proofs, a process Anthropic says requires almost no human oversight. The company describes this as "fully autonomous" proof generation, though it notes the tools and formal foundations were constructed by humans at Theorem.

During this process, Claude uncovered hundreds of bugs that had eluded conventional testing. Many of these issues only manifested under specific conditions, such as with unusually large input files or a particular sequence of memory accesses. In one instance, while verifying a widely used PDF reader, the team discovered as many security bugs in a single week as are typically reported for that software over an entire year. In another case, Claude identified a logical error that had gone undetected for more than a decade. Anthropic states that even when a complete proof ultimately fails, the attempt can still surface real bugs.

The company acknowledges that "formally verified" does not mean "error-free." The generated proofs are contingent on strict conditions, including a specific processor architecture, single-threaded programs, and inputs under 64 MB, whereas the libraries in question are used on various hardware and in different configurations. The compute cost is also significant, with individual proofs consuming up to hundreds of millions of tokens, and Anthropic does not report the project's total cost. However, the company notes that Opus 5.5, a newer model, can solve about half the proofs with approximately 40 percent fewer tokens than Opus 5, and that proofs are reusable.

If artificial intelligence can both write code and mathematically prove its safety under defined conditions, it could significantly reduce the time developers spend reviewing AI-generated software. Anthropic anticipates that formal verification with AI will soon be as practical as standard testing. Theorem's roadmap already targets critical infrastructure libraries such as glibc, OpenSSL, and cURL.

The development shifts the central question in software development from whether AI can program to whether it can prove that the code it writes is safe. If that proof can be delivered reliably, the implications for software automation could be profound, extending beyond the capabilities of current coding tools.