GAUDI: A Neural Architect for Immersive 3D Scene Generation, Arxiv.
Samples from GAUDI (Allow a couple minutes of loading time for videos.)
Miguel Angel Bautista*,
Pengsheng Guo*,
Samira Abnar,
Walter Talbott,
Alexander Toshev,
Zhuoyuan Chen,
Laurent Dinh,
Shuangfei Zhai,
Hanlin Goh,
Daniel Ulbricht,
Afshin Dehghan,
Joshua M. Susskind
Apple (*equal contribution)
Summary
- We introduce GAUDI, a generative model that captures the distribution of 3D scenes parametrized as radiance fields.
- We decompose generative model in two steps: (i) Optimizing a latent representation of 3D radiance fields and corresponding camera poses. (ii) Learning a powerful score based generative model on latent space.
- GAUDI obtains state-of-the-art performance accross multiple datasets for unconditional generation and enables conditional generation of 3D scenes from different modalities like text or RGB images.
We introduce GAUDI, a generative model capable of capturing the distribution of complex and realistic 3D scenes that can be rendered immersively from a moving camera. We tackle this challenging problem with a scalable yet powerful approach, where we first optimize a latent representation that disentangles radiance fields and camera poses. This latent representation is then used to learn a generative model that enables both unconditional and conditional generation of 3D scenes. Our model generalizes previous works that focus on single objects by removing the assumption that the camera pose distribution can be shared across samples. We show that GAUDI obtains state-of-the-art performance in the unconditional generative setting across multiple datasets and allows for conditional generation of 3D scenes given conditioning variables like sparse image observations or text that describes the scene.
Model
Our model is composed of two stages: latent representation optimization and generative modeling. Finding powerful latent representation for scene radiance fields and camera poses is critical to obtain good performance. To achieve this, we design a decoder with three modules:
- A scene decoder