NVIDIA's AI agent 'AVO' achieved 100% in ARC-AGI-3, while the same base model achieved approximately 30% under different conditions, highlighting the importance of the 'execution infrastructure (harness).'

NVIDIA's AI agent system, 'Agentic Variation Operators (AVO),' has achieved a 100% score in the publicly available set of the 'ARC-AGI-3' benchmark, which measures an AI's ability to reason while interacting with unknown environments. The Claude Opus 5 used as the base model was reported by the ARC Prize, the developer of ARC-AGI-3, to have scored approximately 30% in model evaluations under different conditions, demonstrating the importance of peripheral systems that handle memory and tool utilization, in addition to the AI model itself, for long-term tasks.

NVIDIA AVO Reaches 100% on ARC-AGI-3, Demonstrating a Frontier-Level General-Purpose Architecture for Long-Horizon Autonomous Agents | NVIDIA Technical Blog

https://developer.nvidia.com/blog/nvidia-avo-reaches-100-on-arc-agi-3-demonstrating-a-frontier-level-general-purpose-architecture-for-long-horizon-autonomous-agents/

Unlike typical benchmarks where you read a problem statement and answer, ARC-AGI-3 has the AI operate in a game-like environment where it doesn't know the rules or objectives. The AI must actually take action, observe the results, and deduce what to do to get closer to the goal. The public set includes 25 different environments and a total of 183 levels.

'ARC-AGI-3,' a system that measures AI intelligence using games with unknown rules, has been released. It allows users to actually play games that AIs cannot yet clear but humans can 100% clear - GIGAZINE

ARC-AGI-3 uses an index called 'Relative Human Action Efficiency (RHAE)' to evaluate performance. It evaluates not only whether the game can be completed, but also how many attempts it takes to complete the game compared to a standard calculated for a first-time player. Therefore, to achieve 100% RHAE, it is necessary to complete all games and all levels while maintaining an operational efficiency equal to or greater than that of a human.

In tasks that involve long periods of trial and error, simply calling upon a high-performance AI model may not be enough to fully utilize past failures and discoveries. This is where the 'execution infrastructure (harness)' becomes crucial, as it handles memory, tool utilization, state management, and recovery from failures around the AI model.

AVO (Autonomous Voice Optimization) is equipped with persistent memory to store past trials and evaluation results, and a supervisory function to guide the process towards an alternative approach when it encounters difficulties, enabling it to continue autonomous work for extended periods. The main agent progresses through a cycle of situational awareness, planning, execution, and evaluation.

NVIDIA has added an interface for manipulating ARC-AGI-3 while maintaining the basic structure of AVO, which was developed for GPU kernel optimization. When using Claude Opus 5 as the base AI model for AVO, it reportedly cleared all 183 levels of the public set and achieved 100% on RHAE.

In an evaluation of ARC-AGI-3 on Claude Opus 5 under different conditions, conducted by ARC Prize, the developer of ARC-AGI-3, the RHAE (Real-Hour Exposure Achieved) was approximately 30%. The fact that AVO achieved 100% RHAE while using the same model as its base demonstrates that evaluating only the AI model does not represent the overall performance of the agent system.

NVIDIA explains that what's important with ARC-AGI-3 isn't just the 100% figure, but the fact that the same agent structure designed for GPU code optimization could also be applied to conquering unfamiliar games. They further state that 'not only the performance of the AI model itself, but the entire system, including memory, tools, feedback, and recovery from failure, influences long-term autonomous work,' and that further verification using a wider range of models is needed to determine if this trend can be seen in other models.