ZeroMSF is a model for monocular scene flow that jointly estimates geometry and motion in a zero-shot feedforward fashion. It leverages a joint geometry-motion estimation architecture and a scale-adaptive optimization strategy. The model exhibits strong generalization abilities, benefiting from a diverse training set compiled from six data sources.

Installation

Testing environment:

Hardware: Single NVIDIA RTX 3090 GPU
System: Ubuntu 24.04, CUDA 12.6

Pull code with all submodules:

git clone --recurse-submodules git@github.com:NVlabs/zero-msf.git

Install with conda:

# conda environment
conda create -y --name zero_msf python=3.10
conda activate zero_msf
# install pytorch
# here showing pytorch 2.6 and cuda 12.6
# see https://pytorch.org/get-started/locally/ for other versions
pip install torch==2.6.0 torchvision==0.21.0 --index-url https://download.pytorch.org/whl/cu126
python -c "import torch; print(torch.cuda.is_available())" # make sure can see CUDA by printing True
# install most dependencies
pip install -r requirements.txt
# (optional) install curope
conda install -y -c conda-forge libstdcxx-ng
cd zmsf/mast3r/dust3r/croco/models/curope
python setup.py build_ext --inplace
cd -

Demo

First, download model checkpoint from Google Drive and put it under checkpoints/.

Input sample should be provided as a folder containing two images. If more than two images are found within the input folder, only the first two are used. Some samples can be downloaded from Google Drive.

Run demo:

model_config=zmsf_shift+dynamicreplica+vkitti+kubric+pointodyssey+spring_adap_241106
python demo.py \
     --model zmsf/configs/${model_config}.yaml \
     --data PATH_INPUT_FOLDER \
     --output PATH_OUTPUT_FOLDER

Estimated point clouds and scene flow can be found under PATH_OUTPUT_FOLDER.

Visualize results with viser:

python visualize_viser.py \
     --steps 50 \
     --data PATH_OUTPUT_FOLDER

Citation

@InProceedings{liang2025zeroshot,
    author    = {Liang, Yiqing and Badki, Abhishek and Su, Hang and Tompkin, James and Gallo, Orazio},
    title     = {Zero-Shot Monocular Scene Flow Estimation in the Wild},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2025},
    pages     = {21031-21044}
}