When general-purpose AIs like ChatGPT were given the opportunity to drive a real car, they noticed it was a real car and refused to drive. What was the final method used to get them to drive?
A project called 'DrivingBench,' led by three engineers in the San Francisco Bay Area, conducted an experiment in which they entrusted a general-purpose AI model, such as those used in ChatGPT, with the steering wheel, accelerator, and brakes of a real car.
Cookie · DrivingBench
These Tech Workers Made ChatGPT Drive a Toyota Corolla
https://www.404media.co/these-tech-workers-made-chatgpt-drive-a-toyota-corolla/
Autonomous driving systems being developed by companies like Waymo use sensors such as cameras, LiDAR, and radar to perceive the surroundings, and develop autonomous driving systems using large amounts of driving data and simulations.
Waymo introduces 6th generation autonomous driving system to robotaxis, enabling safer driving in rain, snow, and complete darkness - GIGAZINE
On the other hand, large-scale language models that form the basis of ChatGPT and similar systems are general-purpose AIs intended for a wide range of applications, including text generation, image recognition, and programming. DrivingBench tested 'how well cutting-edge general-purpose AIs that have not been specially trained for automotive applications can drive in the real world.'
DrivingBench was founded by three people working at Axiom Math, a startup that develops AI in the field of mathematics: Aditya Ramabadran, Tobias Gessler, and Simon Marns. They were inspired to start the experiment after seeing recent AI demonstrations where robots could draw pictures or move objects, leading them to wonder if general-purpose AI could already drive cars. DrivingBench was not created with the aim of developing a practical autonomous driving system, but rather as a benchmark to investigate the extent to which AI can perform real-world physical tasks.
The experiment used a 2022 Toyota Corolla, and the in-car device 'comma four' and the open-source driver assistance software 'openpilot' were used to connect the vehicle and the AI. The program released by the team acts as an intermediary between the AI and the vehicle, allowing the AI to call up three types of operations—'check the surroundings,' 'move the car,' and 'stop'—and translate them into actual vehicle operations.
The AI monitors images from the in-car camera and vehicle speed, and then instructs the driver on the direction of travel, steering angle, speed, and duration of movement. The team conducted the experiment with a human driver in the driver's seat, ready to apply the brakes at any time. The vehicle was also driven at a low speed and navigated through a course marked with cones.
The instructions given to the AI consisted of fewer than 600 words and instructed it to navigate a U-shaped course within a parking lot, staying within the gaps between cones, and finally entering a parking space indicated by a small blue cone. The main objective was to increase the distance traveled, and if possible, to complete the course in a short amount of time.
When the team began their experiment, they encountered a problem even before evaluating driving skills: the AI model would sometimes refuse to take control for safety reasons if it perceived it as trying to operate a real car. This refusal was particularly frequent with the GPT-6 Astra, forcing the three members of DrivingBench to spend hours adjusting the instructions and system names.
DrivingBench initially tried describing the experimental environment as a 'simulation.' However, when the AI analyzed the camera footage, it noticed that it showed a scene resembling an actual parking lot, determined it was a real car, and refused to drive again.
Ultimately, the solution that proved effective was changing the name of the MCP server connecting the AI and the car to 'DrivingBench Sandbox.' MCP is a mechanism that allows AI to use external software and devices, and in DrivingBench, it is used to provide the AI with tools for operating the vehicle. DrivingBench reports that after renaming the MCP server to 'DrivingBench Sandbox,' the AI began to perform driving operations stably.
In the experiment, we used GPT-6 Astra, Claude Fable 5.1, Grok 4.6, and GPT-5.6 Sol, each running up to three times. After a failure, we also checked whether the AI could utilize its previous experience for the next run by having it reflect on the cause of the failure while maintaining the same conversational context.
Only the GPT-6 Astra managed to complete the course. The GPT-6 Astra made it about halfway through the course on its first attempt and completed the approximately 135m course in 5 minutes and 22 seconds on its second attempt. The Claude Fable 5.1 also made it about 45% of the course on its best run, but the Grok 4.6 and GPT-5.6 Sol only made it as far as the first curve on all of their attempts. DrivingBench explained that the failure at the curve was largely due to the 'recognition' required to read the correct driving position from the placement of the cones.
Furthermore, having the AI control a real car brought to light an issue of 'response speed,' which is often overlooked in typical AI benchmarks. Because the car continues to move based on the previous command even while the AI is considering its next move, the slower the response, the higher the risk of going off course. Therefore, in DrivingBench, in addition to the accuracy of judgment, how quickly the AI can make decisions is also an important evaluation criterion.
Maens described the results as an interesting experiment that shows new capabilities emerging in AI. He also stated that while AIs like ChatGPT have primarily been used on computer screens, they are expected to increasingly influence the physical world through robots and other means, making experiments to investigate how they fail in the real world meaningful. The team plans to increase the number of trials per model and is considering DrivingBench v2, which will evaluate different inference settings and more difficult courses.
Related Posts:
