Jun 12, 2024 20:00:00

The ARC Prize is a competition that will award $1 million to researchers who develop general artificial intelligence with human-level intelligence.

Zapier co-founder Mike Knoop and Google software engineer François Cholet are running the ARC Prize, a competition that offers up to $500,000 in prize money to researchers who develop intelligent, open-source, general-purpose artificial intelligence (AGI).

Announcing ARC Prize
https://arcprize.org/blog/launch

Announcing ARC Prize - YouTube


ARC Prize - Competition Details
https://arcprize.org/competition

Recent large-scale language models, including OpenAI's GPT-4, memorize patterns in the data they are trained on in detail and apply those patterns to different contexts, making it appear as if the AI is making inferences . However, this is merely memorization by the large-scale language model, not actually inference. In addition, general large-scale language models cannot generate new inferences for new situations that they have not been trained on.

On the other hand, general artificial intelligence can efficiently acquire and adapt to new skills in unknown situations, just like humans. 'Without AGI, AI cannot surpass human intelligence. We want AGI that can discover, invent, and move humanity forward together with humans,' Knoop and his colleagues said.

Knoop and his colleagues also raised the issue that since the release of GPT-4, many high-performance, large-scale language models have been developed in closed silos, arguing that 'instead of developing AI in closed silos and slowing down the invention of new ideas by other companies, we should strive to incentivize companies to develop in open source. This will make it easier for new ideas to emerge, and as these ideas become more widespread, it will establish a more level playing field among AI companies of all sizes.'

To make this ideal a reality, Knoop and his team have announced the launch of the ARC Prize, a competition that will award researchers developing open source, high-performance AGI with a total prize of $1 million (approximately 150 million yen).

The key to measuring the performance of the submitted AGI is

the 'ARC-AGI ' index introduced in Cholet's paper ' On the Measure of Intelligence .' ARC-AGI tests performance other than the AI's memorization ability. The following example is a task that asks the user to 'correctly fill in the blanks in the grid based on the two legends above.'

Even the AI that had the highest score on ARC-AGI to date will only achieve a 34% ARC-AGI score by 2024. On the other hand, if a human child performs the ARC-AGI task, they can achieve a score of 85% to 100%. In other words, Knoop and his colleagues are offering a cash prize to researchers who develop AGIs that can achieve results close to the human ARC-AGI score.

Specifically, up to five researchers who develop an AGI that achieves a score of 85% or higher (the average human ARC-AGI score) among those submitted between the start of submissions on June 11, 2024 and the deadline on November 10, 2024, will be awarded a grand prize of $500,000.

In addition, among the AGIs with the highest ARC-AGI scores, the first place AGI will receive $25,000 (approximately 3.9 million yen), the second place AGI will receive $10,000 (approximately 1.5 million yen), and the third to fifth places will each receive $5,000 (approximately 780,000 yen).

In addition, a $45,000 paper prize will be awarded to the paper that best advances our understanding of how to achieve high performance with ARC-AGI.

The results of the ARC Prize will be announced on December 3, 2024. To enter the competition, participants must submit reproducible code or methods into the public domain.