How does the Claude Code team think about and implement validation, and how do they verify their work based on evidence using end-to-end testing, property testing, and diff testing?



At the development team for the AI coding tool 'Claude Code,' Claude writes and reviews approximately 90% of the code. Anthropic engineer Adi Osmani explains the verification mechanisms the Claude Code development team has built to maintain code quality while entrusting a large amount of development work to AI.

How the Claude Code team thinks about verification / claude.dev Blog
https://claude.dev/blog/how-the-claude-code-team-thinks-about-verification/

In software development using AI agents, if humans need to review all deliverables, only one or two agents can be run simultaneously. However, by establishing a system that allows AI to verify its own deliverables, it becomes possible to run dozens or even hundreds of agents in parallel.



In internal tests conducted by Anthropic, having a separate evaluation agent validate files generated by Claude improved the task success rate by up to 10 points compared to iterative processing using normal prompts. Using AI, tests and benchmarks can be created quickly, and once created, tests can be run repeatedly without consuming many tokens.

Osmani identifies three types of tests that are particularly effective in AI agent development: 'end-to-end (E2E) testing,' 'property testing,' and 'diff testing.'

E2E testing is a test that replicates actual user actions on a shopping website, such as adding items to a cart and completing a payment. It allows you to verify not only that individual functions work correctly, but also that the entire system operates as intended.



Property testing is a technique for defining and verifying conditions that a program should always satisfy. For example, by setting a condition such as 'encoding and then decoding data returns the original data,' and automatically generating thousands of inputs for testing, bugs that were not anticipated during development can be discovered.

Differential testing is a technique that involves inputting the same data into both old and new systems and comparing the results. It is useful when changing internal implementations while maintaining functionality, such as during system migration or code rewriting.

The development team at Claude Code emphasizes not only passing tests but also actually running the modified application. They use web browsers and smartphone simulators to have Claude perform the same level of functionality testing as a human tester. The application startup method and verification procedure are summarized in a skill called 'verify,' which Claude uses before committing code.

If a test fails, Claude investigates the cause and fixes the code, but if the problem is not resolved after three attempts at fixing and verifying, it calls on a human to intervene. This eliminates the need for humans to review all the code, allowing them to focus on problems that AI alone cannot solve.



Furthermore, the Claude Code development team has created a reliability checklist consisting of 39 items based on past issues and other factors. For example, if the session saving format is changed, they check whether old conversation history can be loaded correctly. Claude verifies only the necessary items depending on the code differences, so they usually only check 3 to 10 items in practice.

For code reviews, we've adopted a strategy of improving the accuracy of a single review agent rather than having multiple AI agents review the same code. By documenting points to pay attention to during the review in a file called 'REVIEW.md' and utilizing past review results, we are increasing the bug detection rate while keeping costs down.

It's also crucial to prevent the AI from rewriting the rules that evaluate its own work. The Claude Code development team references a separate REVIEW.md file, which is managed independently of the changes being made, and fixes the review target to specific commits. They've also implemented a mechanism to prevent commits from being made if code reviews and functionality checks haven't been performed.



Furthermore, we regularly evaluate the performance of the AI agents responsible for verification. We check whether the AI can find bugs by reproducing code that has caused failures in the past, while also checking whether excessive detections are occurring using code that is running without problems. Problems that the AI overlooks are used to create new verification rules.

Osmani points out that continuously improving the verification mechanism itself is crucial to enhancing the quality of development by AI agents. With a proper verification system in place, the human role shifts from checking each line of code to defining what constitutes completion of the work and verifying that those conditions have been met. This, he says, is key to running a large number of AI agents simultaneously.

in AI,   Software, Posted by log1i_yk