Overview

This paper studies VLA initialization as a controlled representation-design problem along three axes: capability-level embodied VQA supervision, parameter-update strategy, and robot-data pretraining. [arXiv]

Key findings:

• The original pretrained VLM representation is a major source of action performance (>20% drop when training from scratch).
• Embodied VQA adaptation is conditionally useful: its benefit depends on the downstream bottleneck.
• LoRA provides a more reliable initialization than Full Finetune — overly reshaping the pretrained representation weakens VLA initialization.
• The best initialization follows a staged route: adapt with {Grounding + Egocentric Understanding}, then continue with LoRA-based robot-data pretraining.

Main Results

Table 1: Single-Domain VQA Adaptation

Effect of adapting the VLM with a single embodied VQA domain before VLA training.

Table 2: Domain Composition

Combining multiple VQA domains.

Table 3: Robot-Data Pretraining

Quick Start

The experiments follow a two-stage pipeline. Stage 1 adapts the VLM with embodied VQA supervision. Stage 2 trains the VLA policy from the adapted checkpoint.

Stage 1: VLM Adaptation with Qwen3-VL

We use Qwen3-VL with LoRA to adapt the base VLM on embodied VQA data. The high-level workflow:

1. Clone the Qwen3-VL repo and install its environment following the qwen-vl-finetune instructions.
2. Prepare your embodied VQA data in the Qwen-VL conversation format (JSON/JSONL with image + conversations fields), and register it in data/__init__.py within the finetune module. See Data Sources below for what datasets to download per domain.
3. Launch LoRA training via torchrun with --lora_enable True. Our paper uses rank 16, alpha 32, lr 5e-5, 1 epoch, 800K samples per domain, and unfreezes the last 25% of vision encoder layers.

Stage 2: VLA Training with StarVLA

We use StarVLA to train the VLA policy from the Stage-1 adapted checkpoint.

1. Clone the StarVLA repo (use the stable starVLA branch) and install its environment following the Quick Start Guide.
2. Point framework.qwenvl.base_vlm in the training config to your Stage-1 LoRA-merged checkpoint directory (HuggingFace format).
3. Launch VLA training with either the QwenOFT (MLP action head) or QwenPI (flow-matching diffusion action expert, pi0-style) framework.

For training commands, dataset preparation (LeRobot format), evaluation (client-server architecture), and full configuration details, refer to the documentation.

Embodied VQA Data Sources

We organize Stage-1 data into seven capability-oriented domains.

Citation

If you find this work useful, please cite:

@article{lin2026rethinking,
  title={Rethinking VLM Representation for VLA Initialization},
  author={Lin, Weifeng and Huang, Siyuan and Li, Hao and Chen, Tingwei and An, Ruichuan and Wei, Xinyu and Liu, Jianbo and Li, Hongsheng},
  journal={arXiv preprint arXiv:2605.25802},
  year={2026}
}