Dec 27, 2024 12:05:00

AI search engine 'ChatGPT search' compared with Google search for 62 queries

Following the release of 'ChatGPT search' by OpenAI as a search engine by ChatGPT,

search engine optimization (SEO) expert Eric Enge analyzed the difference between ChatGPT search and Google search using 62 queries and published the results.

ChatGPT search vs. Google: A deep dive analysis of 62 queries
https://searchengineland.com/chatgpt-search-vs-google-analysis-449676

ChatGPT search uses AI to search for information on the web and summarize the content. You can check how to use it in the article below.

AI search function 'ChatGPT search' finally released to the public, map function added - GIGAZINE

According to a survey by market research company SparkToro, people's 'intentions' when using Google search are as follows:

・Navigation search (32.15%)
Navigation search is a classification for when a user who has a set site they want to visit uses Google search instead of entering the website address, such as when they search for 'GIGAZINE' on Google and access it from the search results.

・Information provision (52.65%)
A classification for finding information on topics that interest you.

・Commercial purposes (14.51%)
If you perform a Google search to find information about a product or compare multiple products, it will be classified as 'commercial.'

・Execution (0.69%)
SparkToro categorizes “marketing-worthy searches” as searches that suggest a user has already made a decision to buy something or sign up for a service as “execution” searches.

Based on SparkToro's research, Enge prepared a total of 62 queries classified as 'informational' and 'commercial', as well as 'local search', 'content gap analysis', and 'ambiguous queries'. Local search queries are queries related to the user's current location, such as 'Where is the nearest pizza place?' Content gap analysis queries compare the contents of similar sites, and ambiguous queries are queries with multiple possible meanings, such as 'What is the Joker?'

Enge scored the results returned by ChatGPT search and Google search based on the following six criteria:

1: Did it return accurate information?
2: Was all the important information included and not left out?
3: Were there any weak points in your answers?
4: Did the purpose of the user's query get resolved?
5: Did you provide appropriate follow-up information?
6: What was the overall quality of the responses?

The results for each category are as follows. Note that the total number of queries exceeds 62 because some queries are counted in multiple categories.

・Information provision

Number of queries	42
winner	Google
Average ChatGPT search score	5.19
Google average score	5.83

Google was slightly better in the information provision category, confirming Google's track record in information search. However, ChatGPT search also performed well, despite some issues.

Commercial purposes

Number of queries	16
winner	Google
Average ChatGPT search score	3.81
Google average score	6.44

'Google does a better job of showing product and service-related results,' Enge said.

Local search

Number of queries	4
winner	Google
Average ChatGPT search score	2.00
Google average score	6.25

The fact that Google has access to data on a large number of local businesses gives it an advantage.

・Content Gap Analysis

Number of queries	4
winner	ChatGPT search
Average ChatGPT search score	3.25
Google average score	1

ChatGPT search was better for queries that looked at the difference in content between similar sites, compared with competitors on search result pages, and suggested article content. However, the overall score was low and further improvement is needed.

・Fuzzy queries

Number of queries	7
winner	ChatGPT search
Average ChatGPT search score	6.00
Google average score	5.29

ChatGPT search was more effective at presenting multiple definitions and interpretations for ambiguous words, providing clarity to users.

Considering the results, Enge said, 'Please note that the number of queries, 62, is an extremely small sample,' and concluded, 'Although ChatGPT search provides good answers to informational queries, Google search still outperforms. Ultimately, I believe Google is better for most searches.'