Beyond 3D VQAs: Injecting 3D Spatial Priors into Vision-Language Models for Enhanced Geometric Reasoning

Chun-Hsiao Yeh^1,2, Shengyi Qian¹, Manchen Wang¹, Yi Ma^2,3, Joseph Tighe¹, Fanyi Xiao¹

¹FAIR at Meta   ²UC Berkeley   ³HKU

CVPR 2026 (Main Track)  

|  Project Page  |  Paper  |  arXiv  |

TL;DR

GASP teaches VLMs fundamental geometry through point correspondence and depth consistency supervision at every transformer layer. The training head is discarded at inference, so there is zero overhead, and no 3D VQA data is required.

Key Results

• Internal layer-wise correspondence: below 5% → over 70%
• +18.2% on All-Angles Bench, +29.0% on VSI-Bench, +15.0% on BLINK Multi-View
• Zero inference overhead

Method

GASP attaches a lightweight correspondence head to every LLM transformer layer. The head is initialized via SVD decomposition of the pretrained query projection weights. During training, it receives a dual geometric supervision signal:

1. Contrastive Correspondence Loss. An InfoNCE loss on ground-truth point correspondences from large-scale video scenes (DL3DV) enforces 2D view-invariance across frames.
2. Depth Consistency Loss. A soft-argmax depth prediction using the correspondence distribution acts as a discriminative geometric regularizer, forcing the model to distinguish visually similar objects at different depths.

At inference, the correspondence head is discarded. The geometric priors are permanently embedded in the LLM's learned attention weights, enabling robust spatial reasoning without auxiliary inputs or additional parameters.

Correspondence Analysis

PCK, confidence-accuracy correlation, and temporal robustness across all transformer layers. GASP exceeds 70% peak PCK while baselines stay below 5%.

Visual Correspondence

(a) Patch-wise correspondence with optical flow encoding. (b) Attention heatmap for a query point.

Code Release

Coming soon. Please star or watch this repo to get notified.

Citation

@inproceedings{yeh2026gasp,
  title     = {Beyond 3D VQAs: Injecting 3D Spatial Priors into Vision-Language Models for Enhanced Geometric Reasoning},
  author    = {Yeh, Chun-Hsiao and Qian, Shengyi and Wang, Manchen and Ma, Yi and Tighe, Joseph and Xiao, Fanyi},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year      = {2026}
}

Acknowledgements

We thank the authors of DL3DV, LLaVA-NeXT-Video, and Qwen2.5-VL for their open-source datasets and models.

Contact

For questions, please contact Chun-Hsiao Yeh.