SlopCodeBench is a benchmark tool for measuring the performance degradation of AI coding agents in long-term, iterative tasks.

While having AI create programs can yield working code quickly, repeatedly adding features can lead to inconsistencies with older processes or make the code more complex. To investigate these issues, SlopCodeBench is a benchmark developed to measure the maintenance of functionality and changes in quality while gradually adding new specifications to the code created by the AI itself.
SlopCodeBench

SprocketLab/slop-code-bench: SlopCodeBench: Measuring Code Erosion Under Iterative Specification Refinement
advanced-context-engineering-for-coding-agents/benchmarking-new-models-on-slop-code-bench.md at main · humanlayer/advanced-context-engineering-for-coding-agents
https://github.com/humanlayer/advanced-context-engineering-for-coding-agents/blob/main/benchmarking-new-models-on-slop-code-bench.md
◆What is SlopCodeBench?
SlopCodeBench measures the behavior and code quality of an AI-generated program as it iteratively adds new specifications and makes corrections.

◆How SlopCodeBench works
• Checkpoints for repeated specification changes
The official website's 'Problems' section allows users to view 36 different challenges, including processing configuration files and evaluating digital circuits, with multiple checkpoints provided for each challenge.

For example, the 'cfgpipe' assignment has six checkpoints, and initially, the specifications for a program that analyzes a configuration file and outputs the results are provided.

Because AI extends its functionality by building upon previous code each time a new request arrives, the initial program design will also influence later modifications.

◆Verification results of the AI coding agent
- While it may be possible to support new features, maintaining older features is difficult.
When evaluating the system separately into 'Core,' which verifies new features at each stage, and 'Strict,' which determines whether all tests, including those for past features, have been passed, the Core pass rate was 65.82-73.98%, while the Strict pass rate was 12.24-16.33%, highlighting the difficulty of continuing development while maintaining past features.

The highest-performing Strict GPT, 'GPT-6 Astra XHigh,' passed 32 out of 196 checkpoints, achieving a score of 16.33%, while the next highest-performing GPT, 'GPT-6 SOL Medium,' passed 30 checkpoints, achieving a score of 15.31%.

Increasing the amount of reasoning doesn't necessarily improve the pass rate.
Opus 5.5 passed 28 inferences at Medium, but this dropped to 25 at XHigh, and the estimated execution cost increased by approximately 4.22 times, from $193.47 (approximately 30,900 yen) to $817.04 (approximately 130,500 yen).

However, in Astra, XHigh performed better, and the number of executions for each condition was only once, so we cannot definitively conclude that the increase in inference amount is the cause of the performance decrease.

- The pass rate of the final completed program
When focusing on the final stages of the 36 tasks, a maximum of three tasks passed all tests, indicating that even AIs that showed results along the way found it difficult to maintain functionality until completion.

◆ Quality of generated code
SlopCodeBench examines not only the success or failure of functionality, but also 'structural degradation' due to an increase in complex functions and 'redundancy' due to an increase in redundant processing. In the October verification, we examined the final code of 28 common challenges, and the percentage of lines in which Opus 5.5 problem patterns were detected was 77.75% for XHigh and 94.25% for Medium.

This represents the percentage of lines of code that matched the detection rules, not the actual percentage of bugs, and demonstrates that a higher test pass rate doesn't necessarily mean more concise code.

The results from SlopCodeBench demonstrate that the ability of AI to create new features and the ability to maintain existing features and code quality even after repeated changes are two different capabilities.
◆Check the results on the official website
On the official website's '

The benchmark software itself can be downloaded and run for free from the official GitHub repository , but it has many test items and consumes a lot of credits, so careful cost management is necessary.
◆SlopCodeBench evaluation
On the social news site Hacker News, there has been a discussion about SlopCodeBench, with some arguing that 'overly defensive programming with LLMs leads to complex code,' while others have posted evaluations stating that 'it is an excellent benchmark and low inference settings can sometimes be a wise choice.' In response, Dex Horthy, the author of the verification article, explains that 'the pass rate for Fable was 33% when tested with only 6 problems, compared to around 14-15% for all problems.'
◆Summary
SlopCodeBench allows you to check not only the model's pass rate but also its execution cost and code degradation, making it a useful evaluation method when entrusting long-term development to AI.
Related Posts:







