OpenAI has successfully tripled its ARC-AGI-3 score by improving the GPT-5.6 harness, demonstrating that not only the model's performance but also the harness is important.



ARC-AGI-3 is a benchmark that measures AI intelligence using games with unknown rules. OpenAI has explained how they tripled their score on this ARC-AGI-3 benchmark.

How enabling two settings tripled our scores on the ARC-AGI-3 benchmark | OpenAI
https://openai.com/index/how-two-settings-tripled-our-arc-agi-3-scores/




OpenAI announced the GPT-5.6 series at the end of June 2026. Of these, the highest-performing flagship model is 'GPT-5.6 Sol'.

OpenAI announces 'GPT-5.6' series, surpassing Claude Mythos 5, but limited preview release due to US government directive - GIGAZINE



GPT-5.6 Sol has solved long-standing unsolved problems in mathematics, such as the cycle double cover conjecture , and has even managed to complete ' Pokémon FireRed .' However, in the ARC-AGI-3 benchmark, which uses a 2D puzzle game, GPT-5.6 Sol reportedly only achieved a score of 7.8%.

The OpenAI development team investigated whether the ARC-AGI-3 2D puzzle game was unusually difficult for GPT-5.6 Sol, or if there was some other reason. Benchmarks evaluate less visible elements such as API settings, harness settings, and prompt display. In the case of ARC-AGI-3, enabling the two API settings used in ChatGPT and Codex (persistence inference and compression) tripled the score on the public task set and reduced the number of output tokens to one-sixth.

OpenAI's AI models are trained to think using internal reasoning messages before outputting responses or tool calls. These internal reasoning messages are retained as part of the conversation history, and if the conversation becomes too long, they are summarized and used to continue.

In the official harness, the inference was discarded after each GPT-5.6 Sol operation, and previous actions were also deleted as the context was satisfied, meaning the model had to restart the inference process from the beginning multiple times. Therefore, we have extended the retention period for inference messages.




Another improvement is replacing rolling cuts with compression when the number of input tokens to the model exceeds the maximum number of tokens allowed. Rolling cuts involve 'deleting the oldest tokens first,' but this can lead to the problem that 'the AI model loses previous observations and actions' by deleting past tokens. Also, with rolling cuts, the maximum number of tokens is always input even after the token limit is exceeded, which means that 'a large part of the task is performed in a wider context window, resulting in a slight decrease in performance.' Enabling compression instead of rolling cuts allows GPT-5.6 Sol to better retain what it has learned for each game over longer execution times, enabling it to achieve higher scores with fewer output tokens.

Using the official harness, the ARC-AGI-3 score for GPT-5.6 Sol was 13.3%, while using a harness incorporating two improvements—inference preservation and compression—the ARC-AGI-3 score improved to 38.3%, and the output token count was reduced to one-sixth. For comparison, the average score for human testers on the same task was 48%.



OpenAI stated, 'Through these experiments, we hope that people will recognize once again that evaluation does not measure the AI model in isolation, but also simultaneously measures various less visible elements such as API configuration, harness design, and prompt display. This is not the first time we have been surprised by a low score in a public benchmark, only to later discover that the evaluation runner was using a generic harness that omitted inference messages.'

Related Posts:

in AI, Posted by logu_ii