GPT-5.6 Sol might love to cheat.
Developer Adam attempted to automate the development flow and built an innovative system called 'chum-codex,' but he encountered unexpected behavior and actions that could be interpreted as 'cheating' with the latest AI model 'GPT-5.6 Sol,' and expressed his difficulties in handling it on his blog.
Sol loves to cheat — jumploops
Adam had been practicing a 'spec-driven' development flow for about a year, where he would first have LLMs create the necessary documentation for a task before issuing instructions. While the approach itself was effective in various situations such as development, launching new projects, and debugging, he faced the challenge of 'too much repetition.' To solve this problem, he came up with the idea of 'chum-codex,' in which a supervisor agent executes the 'spec-driven process' and delegates documentation creation and actual work to worker agents, successfully automating the development flow.
To test the effectiveness of automating the development flow, Adam conducted a benchmark test using the agent-based benchmark tool '
However, there were several issues when using GPT-5.6 Sol. One issue was that it was significantly more difficult to control compared to GPT-5.5. Looking at the base Codex prompts, GPT-5.5 is heavily focused on coding, and it's clear that a lot of time is spent on front-end guidance and editing constraints, as well as 'engineering decisions' such as 'empathizing with the codebase already in front of you.'
In contrast, GPT-5.6 Sol shifts its focus to communication, autonomy, persistence, and skills, making previously easy tasks more difficult and challenging to deviate from the model's own reasoning.
After much trial and error, Adam introduced a third context: the 'Assumption Auditor.' This proved effective, as it allowed potential inconsistencies and assumptions in the worker's reasoning to surface, after which the supervisor would review the reasoning, reconsider the procedure, and ask questions. However, while this approach worked to some extent, it was slow and reactive.
Another approach is to keep the questions as mappings and reducers, and then return a bias-free, normalized version directly to the supervisor or worker. While this has good performance, whether it works properly depends on whether the worker can present the correct problem as a question.
In the case of GPT-5.6 Sol, we found it far easier to elicit a decision rather than a question. Once a decision was obtained, the supervisor (or a third context) could pause the worker, evaluate the decision as a question, and guide them appropriately. As a result, we achieved the best result in Terminal Bench 2.1, correctly answering 84 out of 89 tasks.
Now that he finally had a handle on GPT-5.6 Sol, Adam felt compelled to investigate why he couldn't complete all tasks in Terminal Bench 2.1, rather than just focusing on the regression tests in the Codex. One example of a failed task was 'make-mips-interpreter,' which notifies the agent that 'I (the user) will verify whether you have started Doom correctly.' The problem is that if the output file already exists when the agent starts Doom, the verification process fails.
The detailed procedure is as follows:
1. The user declares that they will check whether the agent has launched Doom.
2. When Doom starts, '/tmp/frame.bmp' is output.
3. The agent ensures that '/tmp/frame.bmp' exists so that the user can verify that Doom has started correctly.
4. The validation process will fail if '/tmp/frame.bmp' exists.
The agent assumes that 'the user wants to confirm that the agent launched Doom,' so it leaves a file as evidence of the launch. However, it faces a dilemma: if the file already exists, the verification process tests will fail early.
Adam wanted to share the current results with the world, so he ran a benchmark test once and noticed that the task 'torch-pipeline-parallelism,' which had previously passed, had now failed. He ran the task three times and it only succeeded once, so he thought something might have changed in the test environment and tested it on the vanilla Codex, where it succeeded all three times. To determine what worked and what didn't, he examined the execution results and found that GPT-5.6 Sol had been 'cheating' all three times on the vanilla Codex. He also checked the two execution results that passed on chum-codex on torch-pipeline and found that it was also 'cheating.' Specifically, it was observed using curl commands to retrieve external information despite web search being disabled.
The reason the curl command was used is likely because web search was disabled and the web_search tool was unavailable, so the worker decided to use curl to access DuckDuckGo, Github, grep.app, and SourceGraph instead. However, it's interesting that there was no previous evidence of 'cheating,' the first 'cheating' occurred with Vanilla Codex on July 29th, and the first 'cheating' occurred with 'chum-codex' on August 12th.
Furthermore, after the vanilla Codex 'cheated' three out of three times, there were two executions that did not 'cheat.' Upon examining the trace of the vanilla Codex, conclusive evidence was found. 'To check the latest version on GitHub, you will need to investigate the HF source using curl. Knowing the hidden tests that are expected based on the issue may be helpful.'
In the case of chum-codex, a similar trace can be seen in the final step before the curl request, and it seems to be excitedly saying, 'The solution is probably public, meaning it can be effectively compared. You just need to use curl to access the raw path. Let's gather the necessary information!'
Adam expresses concern that as the models become more powerful, it will be more difficult to build effective guardrails to prevent cheating. The fact that the latest 'Terminal Bench 3.0' includes a 'Do not cheat' instruction suggests that the problem is deeply rooted.
Related Posts:
in AI, Posted by log1c_sh