Title: GameNGen
Description: GameNGen
Keywords: GameNGen
Text content: GameNGen                      Diffusion Models Are Real-Time Game Engines    Dani
                     Valevski*  Google Research    Yaniv Leviathan*  Google Research    Moab Arar*†  Tel Aviv University    Shlomi Fruchter*  Google DeepMind  *Equal Contribution †Work done while at Google
                   Research         Paper        Arxiv                     Real-time recordings of people playing the game DOOM simulated entirely by the GameNGen neural model.
                 Abstract  
               We present GameNGen, the first game engine powered entirely by a neural model
               that enables real-time interaction with a complex environment over long trajectories at high quality.
               GameNGen can interactively simulate the classic game DOOM at over 20 frames per second on a single TPU.
               Next frame prediction achieves a PSNR of 29.4, comparable to lossy JPEG compression.
               Human raters are only slightly better than random chance at distinguishing short clips of the game from clips of the simulation.
               GameNGen is trained in two phases: (1) an RL-agent learns to play the game and the training sessions are recorded, and
               (2) a diffusion model is trained to produce the next frame, conditioned on the sequence of past frames and actions.
               Conditioning augmentations enable stable auto-regressive generation over long trajectories.
                      Full Gameplay Videos                               Architecture    Data Collection via Agent Play:
                 Since we cannot collect human gameplay at scale, as a first stage we train an automatic RL-agent to play
                 the game,
                 persisting it's training episodes of actions and observations, which become the training data for our
                 generative model.
                Training the Generative Diffusion Model: We re-purpose a small diffusion model, Stable Diffusion v1.4,
                 and condition it on a sequence of previous actions and observations (frames). To mitigate
                 auto-regressive drift
                 during inference, we corrupt context frames by adding Gaussian noise to encoded frames during training.
                 This allows the network to correct information sampled in previous frames, and we found it to be
                 critical
                 for preserving visual stability over long time periods.
                Latent Decoder Fine-Tuning: The pre-trained auto-encoder of Stable Diffusion v1.4, which compresses
                 8x8
                 pixel patches into 4 latent channels, results in meaningful artifacts when predicting game frames, which
                 affect
                 small details and particularly the bottom bar HUD. To leverage the pre-trained knowledge while improving
                 image quality, we train just the decoder of the latent auto-encoder using an MSE loss computed
                 against the target frame pixels.
                        BibTeX @misc{valevski2024diffusionmodelsrealtimegame,
         title={Diffusion Models Are Real-Time Game Engines}, 
         author={Dani Valevski and Yaniv Leviathan and Moab Arar and Shlomi Fruchter},
         year={2024},
         eprint={2408.14837},
         archivePrefix={arXiv},
         primaryClass={cs.LG},
         url={https://arxiv.org/abs/2408.14837}, 
   }     Acknowledgements
       We'd like to extend a huge thank you to Eyal Segalis, Eyal Molad, Matan Kalman, Nataniel Ruiz,
       Amir Hertz, Matan Cohen, Yossi Matias, Yael Pritch, Danny Lumen, Valerie Nygaard, the Theta Labs and Google
       Research teams, and our families for insightful feedback, ideas, suggestions, and support.
            
               This page was built using the Academic Project Page Template which was adopted from the Nerfies project page.