Webpage (with video results) | Paper

This is the offical code release for ZeroNVS: Zero-Shot 360-Degree View Synthesis from a Single Real Image.

What is in this repository: 3D SDS distillation code, evaluation code, trained models

In this repository, we currently provide code to reproduce our main evaluations and also to run ZeroNVS to distill NeRFs from your own images. This includes scripts to reproduce the main metrics on DTU and Mip-NeRF 360 datasets.

How do I train my own diffusion models?

Check out the companion repository, https://github.com/kylesargent/zeronvs_diffusion.

Acknowledgement

This codebase is heavily built off existing codebases for 3D-aware diffusion model training and 3D SDS distillation, namely Zero-1-to-3 and threestudio. If you use ZeroNVS, please consider also citing these great contributions.

Requirements

The code has been tested on an A100 GPU with 40GB of memory.

To get the code:

git clone https://github.com/kylesargent/zeronvs.git
cd zeronvs

To set up the environment, use the following sequence of commands. The exact setup that will work for you might be platform dependent. Note: it's normal for installing tiny-cuda-nn to take a long time.

conda create -n zeronvs python=3.8 pip
conda activate zeronvs
pip install torch==2.0.1+cu118 torchvision==0.15.2+cu118 --extra-index-url https://download.pytorch.org/whl/cu118
conda install -c "nvidia/label/cuda-11.8.0" cuda-toolkit
pip install ninja git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch
pip install -r requirements-zeronvs.txt
pip install nerfacc -f https://nerfacc-bucket.s3.us-west-2.amazonaws.com/whl/torch-2.0.0_cu118.html

Finally, be sure to initialize and pull the code in the zeronvs_diffusion submodule.

cd zeronvs_diffusion
git submodule init
git submodule update
cd zero123
pip install -e .
cd ..
cd ..

Data and models

Since we have experimented with a variety of datasets in ZeroNVS, the codebase consumes a few different types of data formats.

To download all the relevant data and models, you can run the following commands within the zeronvs conda environment

gdown --fuzzy https://drive.google.com/file/d/1q0oMpp2Vy09-0LA-JXpo_ZoX2PH5j8oP/view?usp=sharing
gdown --fuzzy https://drive.google.com/file/d/1aTSmJa8Oo2qCc2Ce2kT90MHEA6UTSBKj/view?usp=drive_link
gdown --fuzzy https://drive.google.com/file/d/17WEMfs2HABJcdf4JmuIM3ti0uz37lSZg/view?usp=sharing
unzip dtu_dataset.zip

MipNeRF360 dataset

You can download it here. Be sure to set the appropriate path in resources.py

DTU dataset

Download it here (hosted by the PixelNeRF authors). Be sure to unzip it and then set the approriate path in resources.py

Your own images

Store them as 256x256 png images and pass them to launch_inference.sh (details below).

Models

We release our main model, trained with our