(PE3R reconstructs 3D scenes using only 2D images and enables semantic understanding through language.)

PE3R: Perception-Efficient 3D Reconstruction
Jie Hu, Shizun Wang, Xinchao Wang
xML Lab, National University of Singapore
📔 [paper] 🎥 [video] 🤗 [demo]

Why PE3R

• 🚀 Input efficiency: Operate solely with 2D images.
• 🚀 Time efficiency: Accelerated 3D semantic reconstruction.
• 🚀 Generalizability: Zero-shot generalization across scenes and objects.

Quick Start

Install

conda create --name pe3r
conda activate pe3r
git clone https://github.com/hujiecpp/PE3R.git
cd PE3R
pip install -r requirements.txt

Checkpoints

MASt3R, SAM 2, SAM and SigLIP are pulled from the Hugging Face Hub on first run. The two remaining weights are published under the checkpoints release and have to be downloaded manually:

mkdir -p checkpoints
curl -L -o checkpoints/ObjectAwareModel.pt \
  https://github.com/hujiecpp/PE3R/releases/download/checkpoints/ObjectAwareModel.pt
curl -L -o checkpoints/Prompt_guided_Mask_Decoder.pt \
  https://github.com/hujiecpp/PE3R/releases/download/checkpoints/Prompt_guided_Mask_Decoder.pt

Usage

Run from the repository root — the checkpoint paths are relative to the working directory.

python pe3r_demo.py

Acknowledgements

• DUSt3R / MASt3R
• SAM / SAM2 / MobileSAM
• SigLIP

BibTeX

@article{hu2025pe3r,
  title={PE3R: Perception-Efficient 3D Reconstruction},
  author={Hu, Jie and Wang, Shizun and Wang, Xinchao},
  journal={arXiv preprint arXiv:2503.07507},
  year={2025}
}