What is an AI agent's 'harness'? Four mechanisms for making AI models actually work.
The term 'harness' is increasingly being used in relation to AI agents. Earendil, a developer of AI-related software, has posted an article explaining harnesses.
What is a Harness? | EARENDIL
Large-scale language models used in ChatGPT and Claude can understand input text and generate responses. However, to perform a series of tasks such as 'searching the web to gather the latest information,' 'editing files,' 'checking calculation results and correcting errors,' and 'sending completed documents via email,' additional mechanisms are required outside of the AI model.
That's where the harness comes in. Earendil introduces the idea of 'agent = model + harness' as a simplified expression. If the AI model is responsible for the reasoning part, then the harness is the part that prepares the environment so that the AI model can actually carry out its tasks.
According to Earendil, a typical harness serves four main purposes.
1: System prompt
System prompts are instructions that tell the AI model its role and course of action, providing assumptions such as 'what kind of work will you be responsible for?' and 'what rules will you follow?' Earendil likens them to 'the instructions a new employee receives on their first day of work.'
2: Tools
To enable AI models to perform tasks that go beyond simply generating text, we will allow them to invoke functions such as web searches, code execution, and email composition. The key is not just providing the tools, but ensuring the AI model can select the necessary tools based on the situation. This will allow it to make decisions such as performing another search if the search results are insufficient.
3: Agent Loop
This system involves an AI model performing a task, reviewing the results, and then deciding on the next action, repeating this process. For example, if an AI agent is asked to 'research nearby schools and send the comparison results,' it would first perform a web search, and if the information is insufficient, it would conduct additional searches. Once sufficient data is gathered, it would create a spreadsheet file, review its contents, and then draft an email. This iterative process of deciding the next action based on intermediate results is one of the key differences between a system that generates a one-off answer and an AI agent.
4: Transformation Layer
AI models from multiple providers, such as OpenAI and Anthropic, may have different methods for retrieving them and different data formats. Using a conversion layer makes it easier to use multiple AI models from the same harness. This allows for applications such as changing models depending on the type of work, or comparing costs and results.
Earendil's open-source harness, ' Pi ,' is also designed to allow switching between multiple AI providers and numerous models. Furthermore, Pi allows you to change the behavior and available functions of the AI agent to suit your work style by modifying extensions, skills, prompts, and more.
Why is the AI coding tool 'Pi,' which has limited functionality, so high-performing? Its four basic functions help reduce AI costs - GIGAZINE
An example of how harnesses significantly impact the capabilities of an AI agent is NVIDIA's AI agent system, 'Agentic Variation Operators (AVO).' In the ARC-AGI-3 benchmark, which measures an AI's ability to infer while interacting with an unknown environment, the base system, Claude Opus 5, scored approximately 30% under different conditions in the ARC Prize. However, by combining persistent memory, tools, and features to monitor the progress of the work, AVO cleared all 183 levels of the publicly available set and achieved 100%.
NVIDIA's AI agent 'AVO' achieves 100% in ARC-AGI-3, while evaluation of the same base model under different conditions yields approximately 30%, highlighting the importance of the 'execution infrastructure (harness)' - GIGAZINE
NVIDIA prefaced its statement by saying that 'the two evaluations, one at approximately 30% and the other at 100%, had different conditions such as inference settings and agent systems, so this is not a direct measurement of performance improvement solely due to AVO,' but it is an example of how evaluating a single model alone cannot represent the performance of the entire agent system.
In other words, when considering the capabilities of an AI agent, in addition to the AI model itself, the design of the harness is also important, including what kind of instructions it will receive, what tools it can use, how it will utilize past results within the agent loop, and how it will try again when it fails.
Earendil stated that in order for users to easily select AI models and tools according to their needs and adjust them to suit their own work environment, it is important for open-source harnesses that are not dependent on specific AI models to become widespread.
Related Posts:
in AI, Posted by log1d_ts