Apr 02, 2025 12:00:00

Research reveals big differences between human and AI 'thinking,' and AI may not be good at inference

In recent years, AI research has progressed to the point where it is possible to have real-time conversations with an accuracy comparable to that of humans. However, a research team from the University of Amsterdam and other institutions has discovered that there is a large gap between the way humans think and the way AI thinks.

Evaluating the Robustness of Analogical Reasoning in GPT Models
(PDF file)

https://openreview.net/pdf?id=t5cy5v9wph

Scientists discover major differences in how humans and AI 'think' — and the implications could be significant | Live Science
https://www.livescience.com/technology/artificial-intelligence/scientists-discover-major-differences-in-how-humans-and-ai-think-and-the-implications-could-be-significant

A research team led by Martha Lewis, an assistant professor of

neurosymbolic AI at the University of Amsterdam, first presented OpenAI's large-scale language model GPT-3 with the problem of 'identifying the characters that follow a given topic by referring to examples presented.'

Below is an example of an actual problem.
・When 'abcd' becomes 'abce', what happens to '1 2 3 4'?
・When 'abcd' becomes 'abce', what happens to 'ijkl'?
・When 'abbcd' becomes 'abcd', what happens to 'ijkkl'?

The results of the experiment showed that humans were able to quickly provide the correct answers to these questions. On the other hand, the AI was able to correctly answer questions such as 'When 'abcd' becomes 'abce', what happens to 'ijkl'?' but was prone to making mistakes on relatively complex questions such as 'When 'abbcd' becomes 'abcd', what happens to 'ijkkl'?'

In addition, the research team presented GPT-3, GPT-3.5, and GPT-4 with anagram problems in which the alphabet was rearranged, and questions such as 'fill in the '?' in the table below.'
[code]2 2 2
5 5 5
6 6 ?[/code]

It has been reported that large-scale language models are more likely to make mistakes on these problems than humans. 'AI is good at identifying and matching patterns, but it seems not to be very good at using those patterns to generalize,' Lewis said.

Most AI trains on a large number of data sets, and the more training data used to train it, the more patterns it will identify. However, Lewis said, 'It's not what's in the data that counts, it's how the data is used.' In addition, overseas media LiveScience said, 'AI is becoming more widely used in research, case law analysis, sentencing estimation, and other fields, but if it has poor analogical reasoning ability, it may not be able to handle cases where precedents are slightly different.'

Through this study, the research team concluded that 'AI models lack the ability for 'zero-shot' learning, where they observe samples of data that were not present during training and predict the answer according to a question.'