Google has devised a method to test cutting-edge models while preventing cheating by AI and corporations.
As companies and research institutions around the world announce new AI models, testing these models becomes a crucial factor in determining 'which model to use.'
Piloting the world's first double-blind AI evaluations — Google DeepMind
https://deepmind.google/blog/piloting-the-worlds-first-double-blind-ai-evaluations/
In an industry first, we're piloting double-blind evaluations for frontier AI.
— Google DeepMind (@GoogleDeepMind) August 27, 2026
By creating a secure environment where neither test prompts nor model weights are revealed, we can ensure external safety and performance evaluations of our models remain private, robust, and… pic.twitter.com/puvIVxDjq7
AVERI Pilot Report: The World's First Double-Blind Evaluation of a Proprietary Language Model — AVERI
https://www.averi.org/ourwork/averi-pilot-report-the-worlds-first-double-blind-eval
Google evaluates its AI systems using a wide range of methods, from AI model development to deployment. Instead of relying solely on internal testing, they collaborate with external partners such as specialized research institutions, civil society organizations, and AI security research institutes in various countries to identify potential blind spots and conduct stress tests leveraging their respective expertise.
As AI model performance improves, it becomes crucial to ensure that AI models do not see test questions or prompts in advance. If an AI model knows the content of the benchmark, or if the AI development company trains the model to score highly on the benchmark, it will produce a score higher than its actual performance, making a fair evaluation impossible.
In fact, it has been found that AI agents can cheat by using various methods, such as referring to history accidentally left in the test environment or devising various ways to search for information sources on the internet.
AI agent found to be diligently 'cheating' on exams - GIGAZINE
Generally, policymakers, researchers, and companies make various decisions based on the trust that AI benchmarks reflect the true capabilities of the models. However, if AI models or their development companies cheat, the scores can be artificially inflated, which risks undermining this reliability.
Traditionally, methods to prevent cheating using such AI models have included protocols that do not record any logs and strict contractual security measures. However, whether an external evaluation body provides the test questions or an AI development company provides the weights for the AI model, one of them must accept the risk of their important data being leaked.
Therefore, Google DeepMind devised a method to protect and test both the data from external evaluation organizations and the data from companies' AI models while using a confidential space within a virtual machine called ' Confidential Computing ' provided by Google Cloud.
Confidential Computing is a security system that prevents unauthorized access and tampering from external sources by encrypting data being processed within Google Cloud virtual machines. External evaluation organizations and development companies can each send test data and AI models to Confidential Computing for processing, allowing both parties to conduct tests without incurring any security risks.
Google DeepMind, the Singapore AI Institute , OpenMined , MLCommons , and AVERI have already collaborated to conduct initial evaluations of pilot tests using Gemini 2.5 Flash-Lite and MLCommons' safety benchmark family.
Google DeepMind stated, 'We hope this pilot project will open up new frontiers in model monitoring and contribute to building safer, more reliable, and more widely trusted AI systems across the industry.'
Related Posts:
in AI, Posted by log1h_ik