OmniVGGT, a model that can effectively benefit from an arbitrary number of auxiliary geometric modalities during both training and inference.
Key Contributions
Overview of OmniVGGT. Our goal is to design an enhanced spatial foundation model that not only reconstructs scenes from images but also accepts an arbitrary number of corresponding camera parameters and depth information as auxiliary inputs to guide more accurate 3D scene reconstruction.
🔧 GeoAdapter for Stochastic Multimodal Inputs
OmniVGGT uses a light-weight GeoAdapter that supports arbitrary numbers of input images, auxiliary modalities, and their combinations.
📊 Stochastic Multimodal Fusion
We adopt a rule-based stochastic training strategy for OmniVGGT, ensuring its robustness and generalization when receiving varying numbers and combinations of modalities at test time.
🔌 Enhancing VLA Model Performance
We integrate OmniVGGT with vision-language-action (VLA) models. Additional auxiliary information provides the VLA model with richer spatial understanding, thereby improving its success rate across various VLA tasks.
Auxiliary Information Guidance
Compare the reconstruction quality with auxiliary information guidance.
💡 Drag the slider to compare the two models
Input Images
3D Reconstruction Comparison
More Interactive Visualization
OmniVGGT reconstruction on diverse scenes. Click any thumbnail below to explore its 3D reconstruction and point cloud visualization.
💡 Hover over thumbnails to preview their image sequences
Experimental Results
OmniVGGT flexibly accommodates casual inputs, including arbitrary numbers of images, auxiliary information, and input combinations. It achieves state-of-the-art performance across multiple tasks, such as dense reconstruction, multi-view depth estimation, camera pose estimation, and vision-language-action, while maintaining high inference efficiency.
Performance improvement with different ratios of auxiliary information injection.
OmniVGGT consistently outperforms VGGT even without auxiliary inputs, and all relevant metrics improve when different percentages of camera and depth information are provided.
Visual Results of OmniVGGT with Different Auxiliary Information.
By exploiting auxiliary information, OmniVGGT is capable of accurately reconstructing complex and highly challenging geometric environments.
(Top) Camera pose and intrinsic information help correct challenging views with little or no overlap.
(Middle) Providing depth information leads to more accurate local geometry, such as on door surfaces.
(Bottom) When both depth and camera information are provided, the relative distances and viewing angles are properly corrected.
Visual Comparisons on 7-Scenes, NRGBD, and ETH3D datasets.
OmniVGGT exhibits accurate spatial relationships and geometric consistency, even in extremely challenging cases.
More Visualizations on Image Pair Input.
VLA Performance Comparison
Compare the performance of Ours and Kosmos-VLA on different tasks.
💡 Side-by-side comparison of the two methods
Scene 1
Scene 2
Performance Table on CALVIN Benchmark
| Method | Tasks Completed in a Row (%) | Avg. Len. ↑ | |||||
|---|---|---|---|---|---|---|---|
| 1 | 2 | 3 | 4 | 5 | |||
| Task: ABCD→D | |||||||
| Kosmos-VLA (w/ rgb) | 92.9 | 85.4 | 79.4 | 74.4 | 68.1 | 4.00 | |
| Kosmos-VLA (w/ rgb-d) | 93.4 | 85.8 | 80.5 | 75.3 | 69.2 | 4.04 | |
| Ours (w/ rgb) | 93.8 | 86.6 | 81.0 | 75.5 | 69.5 | 4.07 | |
| Ours (w/ rgb-d) | 93.7 | 86.8 | 81.4 | 76.7 | 70.2 | 4.08 | |
| Task: ABC→D | |||||||
| Kosmos-VLA (w/ rgb) | 90.1 | 79.1 | 69.2 | 59.6 | 50.9 | 3.49 | |
| Kosmos-VLA (w/ rgb-d) | 93.6 | 86.0 | 78.6 | 72.9 | 64.8 | 3.97 | |
| Ours (w/ rgb) | 93.8 | 86.9 | 77.9 | 70.3 | 62.2 | 3.92 | |
| Ours (w/ rgb-d) | 95.1 | 87.7 | 79.2 | 70.8 | 63.0 | 3.96 | |