Abstract

New, powerful 3D representations such as DUSt3R’s invariant point maps, which encode 3D shape and camera parameters, have significantly advanced feed-forward 3D reconstruction. While point maps assume static scenes, Dynamic Point Maps (DPMs) extend this concept to dynamic 3D content by also representing scene motion. However, DPMs have so far been limited to image pairs and, like DUSt3R, require post-processing through optimisation when more than two views are involved. We argue that DPMs are much more meaningful when applied to videos and introduce V-DPM to demonstrate this. First, we show how to design DPMs for videos in a way that optimises their representational power, simplifies neural prediction, and enables reuse of pretrained models. Second, we implement these ideas on top of VGGT, a recent state-of-the-art 3D reconstruction model. Although VGGT was trained on static scenes, we show that a small amount of synthetic data is sufficient to adapt it into an effective V-DPM predictor. This approach yields state-of-the-art 3D and 4D reconstruction in dynamic settings. In particular, unlike recent dynamic extensions of VGGT such as π3, DPMs reconstruct not only dynamic depth but also the full 3D motion of every point in the scene.

Method

V-DPM first processes a multi-view input using the VGGT network as a backbone, it appends learnable camera and time tokens at input for later prediction. A DPT head is used to decode time-variant point maps, where the 3D reconstruction is at the respective time frame for each image. A further time-invariant decoder is used to compute time-synchronised point maps, where the learned time token is used for conditioning through AdaLN transformer blocks.

Qualitative Comparison

V-DPM outperforms all other methods across dynamic reconstruction tasks. Please refer to our paper for quantitative results. Here we provide qualitative comparisons.