Google's research team announces 'GameNGen', an AI model that functions as a game engine, can draw DOOM at 20 fps according to input and has actual play movies

A team of four Google researchers has published a paper on ' GameNGen ,' an AI model that functions as a game engine, and has also released a video of it actually playing the FPS game

GameNGen
https://gamengen.github.io/

[2408.14837] Diffusion Models Are Real-Time Game Engines
https://arxiv.org/abs/2408.14837

According to the research team, GameNGen was able to simulate Doom on a single TPU and achieved a frame rate of over 20 frames per second. You can see how it works in the following movie.

GameNGen - YouTube


Although there are some scenes that feel like generated AI, such as enemies and bullets appearing out of nowhere and destroyed objects resurrecting, the game still feels like a proper game, with the remaining bullets decreasing when you fire a gun and the armor value increasing when you obtain armor placed on the map.

The model structure looks like this. Basically, we have a reinforcement learning agent play a game to collect a large amount of 'input & gameplay' data, and then condition it based on the input and frame sequence using Stable Diffusion v1.4.

By intentionally corrupting the context frames by adding Gaussian noise to the encoded frames during training, the model is able to correct the information sampled in the previous frame to maintain visual stability over time.

The GameNGen model trained in this way achieved a PSNR of 29.4, which is equivalent to lossy JPEG compression, in predicting the 'next frame'. In addition, we randomly cut out 3.2 seconds of a play movie using GameNGen and a play movie of an actual game, and when a human evaluator guessed 'which is the actual game', the accuracy rate was only 60%, which shows that GameNGen can generate highly accurate images.

There are five full gameplay videos uploaded on the GameNGen project page , so if you're interested, check them out.