Experiments have shown that humans are prone to betraying AI even when they expect it to cooperate.

A research team at the University of Jaume I in Spain has experimentally revealed that when people realize that the other party is a program and not a human, they tend to prioritize their own gain over cooperation, even if they know the other party will cooperate. This tendency remained unchanged even when the benefits of the artificial agent controlled by the program went to real humans.

Exploiting the machine: Human cooperation with artificial agents in prisoner's dilemma - ScienceDirect
https://www.sciencedirect.com/science/article/pii/S0167487026000498

A lab experiment reveals a hidden bias against artificial agents
https://www.scienceofmoney.org/when-the-other-driver-is-a-robot-why-people-exploit-machines-they-expect-to-coop-767/

The research team conducted an experiment using the ' Prisoner's Dilemma ,' a widely used game theory problem, to investigate cooperative behavior between humans and programs. In the experiment, two participants each chose to 'cooperate' or 'not cooperate.' If both cooperated, they would receive a stable benefit, but if only one of them cooperated, the non-cooperating participant would receive the greatest benefit.

346 Spanish university students participated in the experiment, divided into two groups: one playing games with other humans, and the other playing against artificial intelligence agents. It's important to note that these artificial intelligence agents were not generative AI-powered agents, but rather computer programs that probabilistically determined choices based on past human behavior. Furthermore, the experiment was conducted in April 2022, before generative AI became widespread, and the research team emphasizes that the participants were not very familiar with it.

The artificial agent was designed to probabilistically replicate human behavior based on data from games played between humans beforehand. Furthermore, since the profits earned by the artificial agent were given to other participants in the same laboratory, choosing not to cooperate with the artificial agent indirectly reduced the profits of real humans.

Participants first participated in 10 one-time rounds where they chose to 'cooperate' or 'not cooperate' with a different opponent each time. Following this, they played three sets of 10 rounds each against the same opponent, with each round preceded by a response indicating whether they expected the opponent to cooperate. Participants in the group facing an artificial agent were clearly informed that their opponent was an artificial agent. Furthermore, they were told that the artificial agent was designed to probabilistically mimic human choices based on experimental data from human interactions, and that any profits earned by the artificial agent would be shared with other participants.

In experiments involving a single round with an opponent, no statistically significant difference in cooperation rates was observed depending on whether the opponent was a human or an artificial agent. In both cases, the cooperation rate was low from the start and decreased to almost zero with each subsequent round.

On the other hand, in experiments where participants repeatedly faced the same opponent, the rate of cooperation was lower when facing an artificial agent than when facing a human. In the second and third sets, the probability of participants cooperating when facing an artificial agent was about 15 percentage points lower than when facing a human.

In games played by humans, the rate of cooperation in the early stages increased from approximately 65% to approximately 80% as rounds progressed, indicating that participants learned the benefits of cooperation. In contrast, participants playing against artificial agents did not see an increase in the rate of cooperation in the early stages even after repeated rounds, and there was no tendency for them to carry cooperative behavior into subsequent rounds.

The research team points out that 'the differences observed in the experimental results were not caused by a lack of trust in artificial agents.'

When facing a new opponent, participants expected the artificial intelligence (AI) to cooperate to the same extent as humans, and in some situations, they believed the AI would be even more cooperative than humans. Nevertheless, the behavior of 'using the opponent'—expecting cooperation from the opponent but refraining from cooperating oneself—was more frequent when dealing with the AI.

When participants were asked about the reasons for their decisions, 88.8% of the group interacting with artificial agents responded that they based their decisions on 'reason,' which was higher than the 83% of the group interacting with humans. It is possible that emotions such as empathy are less likely to come into play when interacting with artificial agents, making it easier to perceive actions that exploit the other party as rational choices.

The research team suggests that participants may have thought they were less likely to face retaliation if they betrayed the artificial agent because it acts probabilistically. However, the experiment was conducted with students from a single university and before the widespread adoption of ChatGPT, so further verification is needed to determine if the same results would be obtained now that opportunities to interact with AI have increased.