An AI agent doesn't necessarily become smarter the more 'memory' it has, IBM Research proves with 8 models.
An experiment by IBM Research showed that simply providing AI agents with a large amount of past experience as 'memories' does not necessarily improve their performance. Depending on the AI model, selecting only the necessary information rather than providing all of it can lead to better performance.
How Much Memory Does Your Agent Actually Need?
AI agents are systems that combine large-scale language models with the ability to manipulate external tools, enabling them to perform tasks that require multiple steps. However, they may not be able to effectively reuse past successes and failures, and may repeat the same mistakes.
ALTK-Evolve, developed by IBM Research, is a system that extracts successful strategies and lessons learned from failures from past work records as guidelines to be used in future work. In other words, using ALTK-Evolve allows you to leverage past experience without retraining the AI model.
The research team used guidelines created from training tasks in 'AppWorld,' where ALTK-Evolve operates nine fictional apps such as calendars and messaging apps, as 'memory,' and had eight different AI models perform 585 tasks. The experiment compared three methods: 'no memory,' 'providing all guidelines,' and 'selecting only the necessary guidelines depending on the task.'
The following graph compares the 'no memory' approach with the best-performing memory method for 168 typical tasks across 5 representative models out of 8. In particular, with gpt-oss-120b, selecting only the necessary guidelines increased the task completion rate from 39.9% to 56.0%. The increase in token consumption was also limited to about 5%, while the method of providing all guidelines increased token consumption by about 51%, yet the performance improvement was less than that of the method of selecting only the necessary guidelines. This suggests that simply providing a large amount of information is not enough; the way memory is provided needs to be adjusted to suit the model.
On the other hand, DeepSeek-V3.2, Claude Opus 4.6, and GPT-5.5 showed the highest performance when all guidelines were provided. The research team explains that the optimal way to provide memory differs from model to model and is not determined solely by the size of the model.
The research team plans to study a mechanism that selects useful memories not only based on the semantic similarity of the text, but also on the success or failure of the actual task. Furthermore, since these results are based on experiments using AppWorld, they plan to conduct further testing in a wider range of environments.
Related Posts:
in AI, Posted by log1d_ts