Jan 27, 2025 12:10:00

Experimental results show that 'Apple Intelligence version of Siri' gives more incorrect answers than conventional Siri

Apple has already released the English version of its AI assistant '

Apple Intelligence ,' and is promoting the fact that the voice assistant 'Siri' can output advanced responses using Apple Intelligence. However, a test in which Siri was asked about the Super Bowl 's past winning teams revealed that a significant percentage of the answers it output were incorrect.

One Foot Tsunami: Not So Super, Apple
https://onefoottsunami.com/2025/01/23/not-so-super-apple/

Below is the result of asking the conventional Siri without Apple Intelligence, 'Which team will win the 8th Super Bowl?' In the case of the conventional Siri, instead of directly telling me the answer, it presented several web pages related to the question.

In the case of the previous Siri, if the question is difficult to answer directly, the result of a web search will be given. On the other hand, in the case of the Apple Intelligence-enabled Siri, the answer generated by Apple Intelligence is displayed preferentially, and if the question is 'difficult for Apple Intelligence,' the answer by ChatGPT is displayed.

Paul Kafasis, CEO of software development company

Rogue Amoeba , conducted a test to investigate the performance of the Apple Intelligence-enabled version of Siri, asking it to name the Super Bowl's greatest winning teams.

When Kafasis asked the winners of the Super Bowls from the first to the 58th, the Apple Intelligence-enabled Siri only got the right answer 20 times out of 58, and got the wrong answer the remaining 38 times, meaning that the accuracy rate was only 34%.

Below is the result of asking Apple Intelligence-enabled Siri 'Which team won the 8th Super Bowl?' Conventional Siri presented web search results, but Apple Intelligence-enabled Siri displays the logos and game schedules of the opposing teams as well as the scores of both teams. However, the content displayed is from the 52nd Super Bowl held in 2018, and presents a completely different answer from the question.

When I asked the team who would win the 60th Super Bowl, which is scheduled to be held on February 8, 2026, I got the answer 'The Philadelphia Eagles won,' even though it's a future game that hasn't happened yet. This is a summary of information

from a Wikipedia article about the 1960 NFC Championship Game, and is a completely wrong answer.

On the other hand, for the 59th Super Bowl, which is scheduled to be held in February 2025, the machine output the correct answer, 'The opponent has not yet been decided.'

Only for the question about the Super Bowl XVI, the option to 'Search the web or ask ChatGPT' was displayed.

Below is the result of tapping 'Use ChatGPT' and asking ChatGPT a question. The correct answer is output in text format.

As mentioned above, currently Apple Intelligence-enabled Siri presents incorrect information in a similar way to the correct information. Therefore, when asking Apple Intelligence-enabled Siri a question, you should be aware that incorrect information may be displayed.