Feed-forward 3D reconstruction of signed distance fields from unposed multi-view images

Laura Fink, Linus Franke, George Kopanas, Marc Stamminger, Peter Hedman

Official repository | Link to project page | My home page

Abstract

We propose a feed-forward method for dense Signed Distance Field (SDF) regression from unstructured image collections in less than three seconds, without camera calibration or post-hoc fusion. Our key insight is that the intermediate feature space of pretrained multi-view feed-forward geometry transformers already encodes a powerful joint world representation; yet, existing pipelines discard it, routing features through per-view prediction heads before assembling 3D geometry post-hoc, which discards valuable completeness information and accumulates inaccuracies.

We instead perform 3D extraction directly from features of feed-forward geometry transformer via learned volumetric extraction: voxelized canonical embeddings that progressively absorb multi-view geometry information through interleaved cross- and self-attention into a structured volumetric latent grid. A simple convolutional decoder then maps this grid to a dense SDF. We additionally propose a scalable, validity-aware supervision scheme directly using SDFs derived from depth maps or 3D assets, tackling practical issues like non-watertight meshes. Our approach yields complete and well-defined distance values across sparse- and dense-view settings and demonstrates geometrically plausible completions.

Citation

📋 Prerequisites

• Linux with an NVIDIA GPU. The GPU has to be visible during setup: the native extensions are compiled for the architecture of the card that is present.
• The released Objaverse evaluation configuration requires 7-17 GiB of VRAM (depending on nvblox_enabled).
• Miniconda or Anaconda.
• CUDA Toolkit 12.8. If you do not have it system-wide, the optional step below installs it into the environment.

⚙️ Setup Instructions

Clone the repository (with submodules)

git clone --recurse-submodules https://github.com/lorafib/fus3d.git
cd fus3d

If you already cloned without --recurse-submodules:

git submodule update --init --recursive

Create a conda environment

conda create -n fus3d python=3.10 -y -c conda-forge
conda activate fus3d

(Optional) CUDA 12.8 and compilers from conda

Skip this if CUDA 12.8, the NVTX development headers, and a GCC 13 toolchain are already installed. Otherwise install the complete CUDA 12.8.1 development toolkit and matching compiler into the environment:

conda install -y \
  -c nvidia/label/cuda-12.8.1 \
  -c conda-forge \
  cuda=12.8.1 cuda-nvtx-dev=12.8.90 \
  gcc_linux-64=13.4.0 gxx_linux-64=13.4.0 \
  cmake ninja

Install dependencies

pip install --upgrade pip
pip install -r requirements.txt

Build the native extensions

All three compile against the GPU that is present, so run this on the machine you will train on.

pip install --no-build-isolation -e src/loss/earth_mover_dist
(cd src/loss/chamfer/chamfer3D && python setup.py install)

nvblox provides the TSDF/SDF fusion path and needs its own CMake build. It fetches gflags, glog, sqlite3 and stdgpu at configure time, so this step needs network access.

NVBlox (optional)

With test.nvblox_enabled: true and nvblox not installed, a warning will be displayed but continues without it. train.df_loss_from: "depth_gt" needs it installed though.

export CONDA_ENV=$(conda info --base)/envs/fus3d # your conda env path
export NVBLOX_DIR="$PWD/dep/nvblox"
export CUDACXX="${CONDA_ENV}/bin/nvcc"
export CUDAHOSTCXX="${CONDA_ENV}/bin/x86_64-conda-linux-gnu-g++"
export CC="${CONDA_ENV}/bin/x86_64-conda-linux-gnu-gcc"
export CXX="${CUDAHOSTCXX}"
export CUDA_INC_PATH="${CONDA_ENV}/targets/x86_64-linux/include"
export PATH="${CONDA_ENV}/bin:${PATH}"
export LD_LIBRARY_PATH="${CONDA_ENV}/lib${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}"
# TODO replace with your CUDA_ARCHITECTURES, "native" is not always supported
cmake --fresh -S "${NVBLOX_DIR}" -B "${NVBLOX_DIR}/build" \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_CUDA_ARCHITECTURES=86 \
  -DCMAKE_POLICY_VERSION_MINIMUM=3.5 \
  -DBUILD_PYTORCH_WRAPPER=ON \
  -DBUILD_RENDERER=OFF \
  -DBUILD_TESTING=OFF \
  -DBUILD_BENCHMARKS=OFF \
  -DCMAKE_PREFIX_PATH="${CONDA_PREFIX}/lib/python3.10/site-packages/torch/;${CONDA_PREFIX};${CONDA_PREFIX}/lib"
cmake --build "${NVBLOX_DIR}/build" --target py_nvblox -j
pip install -e ${NVBLOX_DIR}/nvblox_torch

Verify the install

python -c "import torch, nvblox_torch, pyrender; print(torch.version.cuda, torch.cuda.is_available())"
python src/visualization/renderer_offscreen.py   # offscreen EGL test

🧪 Run Examples

Prepare data

Example data and pretrained checkpoints are available here:

⬇️ Download example data and checkpoints

Download example_data_objaverse and point the dataset config at it:

# config/dataset/objaverse.yaml
roots: [/path/to/example_data_objaverse]

The example set ships 15 scenes with rendered views, depth maps, camera transforms.json, ground-truth meshes, and the train/test split indices.

Run

Put the pretrained checkpoints into checkpoints/, then test inference / eval with:

python src/main.py +experiment=test_config_sdf_only_final_objaverse

Training runs the two stages in order:

# starting from VGGT 
python src/main.py +experiment=train_config_objaverse_stage1
# starting from stage1 checkpoint
python src/main.py +experiment=train_config_objaverse_stage2

Every config lives in config/experiment/. train_config_* trains, test_config_* evaluates, and any field can be overridden on the command line, e.g. trainer.max_steps=100.

If you use Slurm and/or your compute node does not have no network access you can check out: See INSTALL_SLURM.md.

🗂️ Dataset Preparation

dataset_toolkits/ is adapted from the TRELLIS dataset toolkit: the latent- and feature-encoding stages were dropped, depth maps are written, and extended with visibility filtering.

See dataset_toolkits/DATASET.md for how to build the full dataset.

To train on a new dataset, register the loader in src/dataset/data_module.py, src/dataset/__init__.py, and add a matching config/dataset/*.yaml.

Color Fus3D via Differentiable Rendering

Addtional to the orignal Fus3D paper, I added preliminary support for color prediction and differentiable rendering in the frame of my doctoral thesis. The differentiable rendering is based on NeuS and Neuralangelo.

Inference / evaluation:

python src/main.py +experiment=test_config_sdf_objaverse_color

Training can be done with:

# starting from stage1 checkpoint
python src/main.py +experiment=train_config_objaverse_color_stage1
# starting from color stage1 checkpoint
python src/main.py +experiment=train_config_objaverse_color_stage2

✅ TODO

• Initial code release
  • Example data
  • Checkpoint release
  • Dataset toolkit
• Preliminary support of colored SDFs
• Hugging Face hosting
• Gradio demo
• (Support free camera trajectories)

🙏 Acknowledgements

Thanks to TRELLIS, AnySplat, VGGT, CUT3R, NoPoSplat, NeuS, Neuralangelo, SparseNeuS, nvblox, ChamferDistancePytorch, and PyTorchEMD for their great code.

📄 License

MIT — see LICENSE, except for these vendored trees, which keep their upstream terms:

src/model/encoder/vggt/ is not MIT: the VGGT License requires that derivative works be distributed under its own terms, and that a copy travel with them.