Understanding multimodal signals in egocentric vision, such as RGB video, depth, camera poses, and gaze, is essential for applications in augmented reality, robotics, and human-computer interaction, enabling systems to better interpret the camera wearer’s actions, intentions, and surrounding environment. However, building large-scale egocentric multimodal and multitask models presents unique challenges. Egocentric data are inherently heterogeneous, with large variations in modality coverage across devices and settings. Generating pseudo-labels for missing modalities, such as gaze or head-mounted camera trajectories, is often infeasible, making standard supervised learning approaches difficult to scale. Furthermore, dynamic camera motion and the complex temporal and spatial structure of first-person video pose additional challenges for the direct application of existing multimodal foundation models. To address these challenges, we introduce a set of efficient temporal tokenizers and propose EgoM2P, a masked modeling framework that learns from temporally-aware multimodal tokens to train a large, general-purpose model for egocentric 4D understanding. This unified design supports multitasking across diverse egocentric perception and synthesis tasks, including gaze prediction, egocentric camera tracking, and monocular depth estimation from egocentric video, and also serves as a generative model for conditional egocentric video synthesis. Across these tasks, EgoM2P matches or outperforms specialist models while being an order of magnitude faster. We will fully open-source EgoM2P to support the community and advance egocentric vision research.
We benchmark EgoM2P's multitasking abilities with SOTA specialist models in downstream tasks, including egocentric perception and synthesis. We also benchmark it on unseen datasets without any fine-tuning to show the strong generalization ability of the pretrained feature.
Compared to specialist SOTA models that require geometry-based test-time optimization, our feed-forward EgoM2P predicts camera trajectories with better translation and rotation accuracy, achieving an inference speed of 300+ FPS
EgoM2P can also predict smooth and plausible camera trajectories in the sense that it learns to capture the uniqueness of egocentric head motion, while baselines suffer from temporal jittering
EgoM2P effectively predicts realistic egocentric camera trajectories, even on unseen dataset ADT
EgoM2P can better understand human intentions
Baselines requires sequence-level optimization which is time-consuming. Our model achieves at least 30 times faster inference speed, while ensuring temporal consistency
EgoM2P has strong generalization abilities to unseen datasets
EgoM2P demonstrates superior performance to generate depth-aligned egocentric RGB videos, minimizing hallucinations and produces realistic finger motions
Given ground-truth camera intrinsics and an egocentric video, we compare EgoM2P with the SOTA baseline MegaSAM (CVPR 2025) for 4D reconstruction. Unlike MegaSAM, which relies on SOTA monocular depth estimators and expensive geometry optimization, EgoM2P efficiently reconstructs dynamic egocentric scenes. For a 2-second video at 8 FPS, EgoM2P completes the reconstruction in less than 1 second, whereas MegaSAM requires 71 seconds.
@InProceedings{li2025egom2p,
author = {Li, Gen and Chen, Yutong and Wu, Yiqian and Zhao, Kaifeng and Pollefeys, Marc and Tang, Siyu},
title = {EgoM2P: Egocentric Multimodal Multitask Pretraining},
booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
month = {October},
year = {2025},
pages = {10830-10843}
}
@InProceedings{li2024egogen,
author = {Li, Gen and Zhao, Kaifeng and Zhang, Siwei and Lyu, Xiaozhong and Dusmanu, Mihai and Zhang, Yan and Pollefeys, Marc and Tang, Siyu},
title = {EgoGen: An Egocentric Synthetic Data Generator},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2024},
pages = {14497-14509}
}