GitHub

This repository is the official implementation of Feat2GS, a unified framework to probe “texturel and geometry awareness” of visual foundation models. Novel view synthesis serves as an effective proxy for 3D evaluation.


Interactive_Demo.mp4

Updates

  • [July 10, 2025] Add a new evaluation of VGGT encoder and decoder features. See the results here.

Get Started

Installation

  1. Clone Feat2GS and download pre-trained model from DUSt3R/MASt3R.
git clone https://github.com/fanegg/Feat2GS.git
cd Feat2GS/submodules/mast3r/
mkdir -p checkpoints/
wget https://download.europe.naverlabs.com/ComputerVision/DUSt3R/DUSt3R_ViTLarge_BaseDecoder_512_dpt.pth -P checkpoints/
wget https://download.europe.naverlabs.com/ComputerVision/MASt3R/MASt3R_ViTLarge_BaseDecoder_512_catmlpdpt_metric.pth -P checkpoints/
cd ../../
  1. Create the environment, here we show an example using conda.
conda create -n feat2gs python=3.11 cmake=3.14.0
conda activate feat2gs
pip install "torch==2.5.1" "torchvision==0.20.1" "numpy<2" --index-url https://download.pytorch.org/whl/cu121  # use the correct version of cuda for your system
cd Feat2GS/
pip install -r requirements.txt
pip install submodules/simple-knn
  1. Optional but highly suggested, compile the cuda kernels for RoPE (as in CroCo v2).
# DUST3R relies on RoPE positional embeddings for which you can compile some cuda kernels for faster runtime.
cd submodules/mast3r/dust3r/croco/models/curope/
python setup.py build_ext --inplace
cd ../../../../../../
  1. (Optional) follow this instruction to install requirements for probing Zero123.

Usage

  1. Data preparation (We provide our evaluation and inference datasets: link)
  cd <data_root>/Feat2GS/

If you want to build custom datasets, please follow and edit:

build_dataset/0_create_json.py ## create dataset_split.json to split train/test set
build_dataset/1_create_feat2gs_dataset.py ## use dataset_split.json to create dataset
  1. Evaluate Visual Foundation Models:
Step Description (link to command)
(1) DUSt3R initialization & Feature extraction
(2) Readout 3DGS from features & Jointly optimize pose
(3) Test pose initialization
(4) Render test view for evaluation
(5) Metric
(Optional) Render video with generated trajectory 
  # Run evaluation for all datasets, all VFM features, all probing modes
  bash scripts/run_feat2gs_eval_parallel.sh
  # (Example) Run evaluation for a single scene, DINO feature, Geometry mode
  bash scripts/run_feat2gs_eval.sh

Note

To run experiments in parallel, we added a GPU lock feature to ensure only one evaluation experiment runs per GPU. Once an experiment finishes, the GPU is automatically unlocked. If interrupted by Ctrl+C, the GPU won’t be unlocked automatically. To fix this, manually delete the .lock files in your LOCK_DIR. To disable this feature, comment out these lines in the script: L4-L5, L9-L22, L223-L233, L330-L331.

Config Operation
GPU Edit in <AVAILABLE_GPUS>
Dataset Edit in <SCENES[$Dataset]>
Scene Edit in <SCENES_$Dataset>
Visual Foundation Model Edit in <FEATURES>
Probing Mode Edit in <MODELS>
Inference-only Mode Comment out STEP (3)(4)(5) in execute_command 
  # Evaluate Visual Foundation Models on DTU dataset
  bash scripts/run_feat2gs_eval_dtu_parallel.sh
  # Run InstantSplat for evaluation
  bash scripts/run_instantsplat_eval_parallel.sh
  1. After training, render RGB/depth/normal video with generated trajectory.
  # If render depth/normal, set RENDER_DEPTH_NORMAL=true
  # Set type of generated trjectory by editing <TRAJ_SCENES>
  bash scripts/run_video_render.sh
  # Render video on DTU dataset
  bash scripts/run_video_render_dtu.sh

🎮 Interactive demo

🚀 Quickstart

  1. Input Images
  • Upload 2 or more images of the same scene from different views
  • For best results, ensure images have good overlap
  1. Step 1: DUSt3R Initialization & Feature Extraction
  • Click "RUN Step 1" to process your images
  • This step estimates initial DUSt3R point cloud and camera poses, and extracts DUSt3R features for each pixel
  1. Step 2: Readout 3DGS from Features
  • Set the number of training iterations, larger number leads to better quality but longer time (default: 2000, max: 8000)
  • Click "RUN Step 2" to optimize the 3D model
  1. Step 3: Video Rendering
  • Choose a camera trajectory
  • Click "RUN Step 3" to generate a video of your 3D model
gradio demo.py

💡 Tips

  • Processing time depends on image resolution and quantity
  • For optimal performance, test on high-end GPUs (A100/4090)
  • Use the mouse to interact with 3D models:
    • Left button: Rotate
    • Scroll wheel: Zoom
    • Right button: Pan

Acknowledgement

This work is built on many amazing research works and open-source projects, thanks a lot to all the authors for sharing!

Citation

If you find our work useful in your research, please consider giving a star ⭐ and citing the following paper 📝.

@inproceedings{chen2025feat2gs,
  title={Feat2gs: Probing visual foundation models with gaussian splatting},
  author={Chen, Yue and Chen, Xingyu and Chen, Anpei and Pons-Moll, Gerard and Xiu, Yuliang},
  booktitle={Proceedings of the Computer Vision and Pattern Recognition Conference},
  pages={6348--6361},
  year={2025}
}

Contact

For feedback, questions, or press inquiries please contact Yue Chen and Xingyu Chen.

Read the original on github.com ↗