🌐 Project Page  •  📄 Paper (arXiv)  •  🤗 Checkpoints (HuggingFace)  •  🎬 Videos  •  🔬 Reproduce

TL;DR — Standard behavior-cloning finetuning of a vision-language model on robot demos silently erases the pretrained VLM's semantics and decouples its language output from its actions. Anchor-Align adds two lightweight losses on top of action prediction that (i) anchor the trainable VLA to a frozen copy of the pretrained VLM (preserving vision-language reasoning) and (ii) align the pre-action hidden state with a discrete motion-direction label derived from the executed motion. The result: 22.6% on LIBERO-PRO position swap (vs 2.3% for standard BC), 90.3% LIBERO-Plus average (vs 85.1%), and near-doubled real-world success (28.3% → 54.2%) on a UFactory xArm7 setup.

Table of Contents

• Motivation
• Method: Anchor-Align
• Results
• Real-World Experiments
• Pretrained Checkpoints
• Installation
• Data Preparation
• Training Code
• Evaluation
• Diagnostic Framework
• Repository Layout
• Key Files
• Acknowledgments
• Citation

Motivation

Failure mode 1 — the pretrained VLM's semantics are silently erased

Consider a tabletop kitchen scene with a green mug, a pink mug, a plate, and distractors. A policy is trained only to "pick up the green mug and place it on the plate" across many spatial arrangements. At test time we prompt "pick up the pink mug." The underlying VLM already understands color — a good finetune should preserve that.

Standard behavior-cloning ignores the new instruction and picks the green mug anyway. Anchor-Align follows the instruction correctly, because the anchoring loss keeps the VLM's color semantics intact throughout finetuning.

Failure mode 2 — the model's language and actions disagree on the same observation

Standard BC supervises only the action head. The language head is free to drift, so the same VLA can report "left" as its next-move direction while its action head simultaneously commands an upward motion on the same observation. Anchor-Align derives a discrete motion-direction label from each ground-truth trajectory (six direction words: up, down, left, right, forward, backward) and trains the pre-action hidden state to predict it jointly with the continuous action.

Method: Anchor-Align

The training loss is:

L_total  =  L1_action  +  0.1 × L_anchor  +  0.02 × L_align

Vision-Language Anchoring (Anchor loss)

A frozen copy of the pretrained VLM serves as the anchor. At each training step we take the layer-wise hidden states from the student (LoRA-finetuned) VLA and the frozen teacher, then compute per-layer MSE on the non-action token positions (BOS + vision patches + text prompt). Action-token positions are masked out so anchoring never fights the action objective.

L_anchor = (1 / 2σ²) · mean_over_24_layers[ MSE( student_h, teacher_h ) ]

Applied to all 24 transformer layers of Qwen2.5-0.5B, with σ = 1.0. This is the single most important design choice — anchoring the full stack (not just the last layer) is what preserves compositional visual grounding.

Language-Action Alignment (Align loss)

At the last text token position (immediately before the action tokens), the pre-action hidden state is projected through a small learned head and dotted with the frozen lm_head. This yields logits over the Qwen2.5 vocabulary; we take the cross-entropy against a six-way motion-direction label (up, down, left, right, forward, backward) derived from the chunk-averaged action delta.

L_align = CrossEntropy( proj(pre_action_hidden) @ frozen_lm_head , direction_word )

Direction words are verified to be single-token in the Qwen2.5 vocabulary at startup. Samples with near-zero motion (L2 norm of XYZ delta < 0.15) are ignored so the loss only fires on decisive motions. align_version=7 uses the pre-action hidden state (after the LLM's self-attention), which preserves the vision patches and therefore does not degrade spatial reasoning.

Architecture

Input Images (2x224x224)
    |
    +-- DINOv2 ViT-L ----> (B, 256, 1024) --+
    |                                         +-- concat on dim=2 --> (B, 256, 2176) per image
    +-- SigLIP SO400M ---> (B, 256, 1152) --+
                                              |
                              concat on dim=1 (2 images) --> (B, 512, 2176)
                                              |
                                      Projector MLP
                                  (2176 -> 8704 -> 896)
                                              |
                                      (B, 512, 896) vision tokens
                                              |
    [BOS] --- concat ----- [vision tokens] ---- [text/action tokens] ---- [EOS]
              |                                        |
              +---------- Qwen2.5-0.5B (24 layers) ----+
                           LoRA rank-64 finetuned
                           output_hidden_states=True
                                              |
                  +--------------------------+
                  |                          |
          Action Head                  Anchor loss  (24 layers)
       (MLPResNet, 24 blocks)         on non-action positions
       --> predicted_actions               |
           (B, 8, 7)               Align loss
                                   pre-action hidden --> CE(direction)

Results

Tables below are reproduced from the paper. Reproduction targets and tolerances for the released checkpoints are in REPRODUCE.md.

Standard LIBERO suites

Success rates on the four standard (unperturbed) LIBERO suites. Anchor-Align achieves the highest success rate on every suite, surpassing methods with substantially larger backbones and large-scale robot-action pretraining.

Robustness and generalization — LIBERO-PRO and LIBERO-Plus

Success rates under perturbation on the LIBERO-Spatial suite (paper Table 1). Bold = best, underline = second best.

*Our implementation of knowledge insulation adapted to VLA-Adapter. Position swap is the hardest axis: MolmoAct and OpenVLA-OFT score 0%, standard BC reaches 2.3%, while Anchor-Align reaches 22.6%.

Qualitative — generalization to semantic perturbations

Same task, different phrasing / different object identity / shuffled positions. Anchor-Align retains the VLM's semantic understanding of what the instruction refers to, while the baseline latches onto memorized appearance shortcuts.

Per-suite robustness — Object, Goal, and Long

The same gains carry over to the remaining three LIBERO suites. Each radar plot compares Anchor-Align (orange) against the standard BC VLA-Adapter baseline (gray) across nine evaluation axes: two from LIBERO-PRO (Language Rephrase, Object Swap) and seven from LIBERO-Plus.

Largest gains: Robot Init State +18.6 on Object; Language Instruction +11.9 on Goal; Lighting Condition +20.8, Object Layout +18.6, and Camera Viewpoint +17.7 on Long.

Long-horizon generalization — CALVIN ABC→D

Each rollout chains five language instructions; k/5 is the fraction of rollouts completing the first k, and Len is the average number of consecutively completed tasks.

Multi-seed significance

Mean ± standard deviation over 5 training seeds on LIBERO-Spatial; the method gaps are far larger than seed-to-seed variability.

Real-World Experiments

We evaluate on a UFactory xArm7 across four held-out perturbation regimes: spatial rearrangement, cluttered scene, compositional object-layout, and semantic perturbation (pink-mug OOD).

Each row is a held-out perturbation regime; green borders = Anchor-Align succeeds, red borders = baseline fails on the same setup. The layout seen during training (left) differs from the layout at test (right). Anchor-Align generalizes across all three regimes on both VLA backbones we tested; the baseline fails on every held-out configuration.

Real-world rollouts (appendix): broccoli pick-and-place

Six evenly-spaced keyframes per row. The broccoli's position and the surrounding distractors are simultaneously swapped across rollouts — each episode is a unique scene configuration. All six succeed.

Videos

See many more demonstrations on the project page. Direct links to representative clips:

Pretrained Checkpoints

The four release checkpoints live on HuggingFace at Dwipz/Anchor-Align; this GitHub repo contains the inference and evaluation code for them. The two repos are paired: HF hosts the weights, GitHub hosts the code.

Each subfolder is a self-contained inference bundle: merged base VLM (model.safetensors) + LoRA adapter (lora_adapter/adapter_model.safetensors) + action head (action_head--<step>_checkpoint.pt) + alignment projector (align_dir_proj--<step>_checkpoint.pt) + proprio projector (proprio_projector--<step>_checkpoint.pt) + tokenizer + config. See each subfolder's MODEL_CARD.md for full per-metric breakdowns.

Download a checkpoint

from huggingface_hub import snapshot_download
# Grab a single checkpoint into a local directory
local_dir = snapshot_download(
    repo_id="Dwipz/Anchor-Align",
    allow_patterns=["config.json", "libero-spatial/*"],   # or libero-object/*, libero-goal/*, libero-long/*
    local_dir="./checkpoints",
)
print("Downloaded to:", local_dir)

Run inference on the downloaded checkpoint

Point the corresponding eval script (from this repo) at the local path:

# LIBERO Standard evaluation (uses the Spatial checkpoint above)
CUDA_VISIBLE_DEVICES=0 python experiments/robot/libero/run_libero_eval.py \
  --pretrained_checkpoint ./checkpoints/libero-spatial \
  --task_suite_name libero_spatial \
  --use_proprio True --num_images_in_input 2 --use_pro_version True
# LIBERO-PRO perturbation evaluation
CUDA_VISIBLE_DEVICES=0 python experiments/robot/libero_pro/run_libero_pro_eval.py \
  --pretrained_checkpoint ./checkpoints/libero-spatial \
  --base_suite_name libero_spatial \
  --perturbation_type lan  \
  --use_proprio True --num_images_in_input 2 --use_pro_version True

The inference flags shown here (--use_proprio True --num_images_in_input 2 --use_pro_version True --use_l1_regression True --center_crop True --num_open_loop_steps 8) are the same flags used in the paper — see the "Evaluation" section below for LIBERO-Plus and batched variants.

Installation

Environment Setup

conda create -n anchor-align python=3.10.16 -y
conda activate anchor-align

Install Dependencies

# Install PyTorch (use a command specific to your machine: https://pytorch.org/get-started/locally/)
pip install torch==2.2.0 torchvision==0.17.0 torchaudio==2.2.0
# Clone and install
git clone https://github.com/dwipddalal/Anchor-Align.git
cd Anchor-Align
pip install -e .

Data Preparation

LIBERO benchmark variants

We report numbers on three LIBERO benchmarks: the standard suite (LIBERO), the paraphrase / swap / object perturbation suite (LIBERO-PRO), and the 7-category robustness suite (LIBERO-Plus). All three install as the libero Python package, so they are mutually exclusive in a single environment — install whichever variant matches the eval you want to run, or use separate conda envs.

The helper script installs any single variant or clones all three side-by-side:

# Standard LIBERO only (LIBERO Standard eval)
bash setup/install_libero_variants.sh libero --dest ../libero-variants
# LIBERO-PRO (Standard + PRO evals)
bash setup/install_libero_variants.sh libero-pro --dest ../libero-variants
export LIBERO_PRO_ROOT="../libero-variants/LIBERO-PRO"
# LIBERO-Plus (Standard + Plus evals)
bash setup/install_libero_variants.sh libero-plus --dest ../libero-variants
export LIBERO_PLUS_ROOT="../libero-variants/LIBERO-plus"
# All three side-by-side (switch between them with `pip install -e .`)
bash setup/install_libero_variants.sh all --dest ../libero-variants

Then install the extra requirements this repo needs on top:

pip install -r experiments/robot/libero/libero_requirements.txt

The Plus eval script (run_libero_plus_eval*.py) reads LIBERO_PLUS_ROOT to find the perturbation BDDL / init-state files, and the PRO SLURM templates add LIBERO_PRO_ROOT to PYTHONPATH. See REPRODUCE.md for the full per-benchmark reproduction recipe.

LIBERO RLDS datasets (only needed for the alignment diagnostic, not inference)

None of the benchmark evals need these. They are only required by the alignment diagnostic (experiments/robot/libero/run_alignment_test.py), which samples frames from the RLDS pipeline. Download the modified LIBERO RLDS datasets (~10 GB total):

git clone git@hf.co:datasets/openvla/modified_libero_rlds

Note: Rename the downloaded directory to remove the modified_ prefix so paths match the expected structure below.

Important — --remap-axes when running the diagnostic. The libero-spatial, libero-object, and libero-goal checkpoints use a transposed X/Y direction-label convention, so you must pass --remap-axes to reproduce their alignment numbers; the libero-long checkpoint uses the LIBERO-frame convention, so run it without the flag. (--dataset selects the RLDS frames to probe on and accepts libero_spatial_no_noops or libero_object_no_noops; the remap depends on the checkpoint, not the dataset. Use --phase full for a real run.) See the header comment in run_alignment_test.py for the per-checkpoint rule.

If you encounter AttributeError: 'NoneType' object has no attribute 'eglQueryString':

sudo apt-get update
sudo apt-get install libgl1-mesa-dev libegl1-mesa-dev libgles2-mesa-dev libglew-dev

Directory Structure

.
├── data
│   └── libero
│       ├── libero_spatial_no_noops/1.0.0/
│       ├── libero_object_no_noops/1.0.0/
│       ├── libero_goal_no_noops/1.0.0/
│       └── libero_10_no_noops/1.0.0/
└── pretrained_models
    ├── configs/
    └── prism-qwen25-extra-dinosiglip-224px-0_5b/

VLM backbone (optional, not needed for inference)

The four HuggingFace checkpoints already ship with a merged model.safetensors that includes the backbone weights, so you do NOT need to download the backbone separately to run inference / eval. It's only needed if you want to re-apply the released LoRA adapters to the base VLM yourself (vla-scripts/merge_lora_weights_and_save.py).

If you do need it, download the Prismatic VLM (Qwen2.5-0.5B + DINOv2 + SigLIP) into pretrained_models/:

# Requires huggingface_hub CLI (installed by `pip install -e .` above)
huggingface-cli download Stanford-ILIAD/prism-qwen25-extra-dinosiglip-224px-0_5b \
  --local-dir pretrained_models/prism-qwen25-extra-dinosiglip-224px-0_5b \
  --local-dir-use-symlinks False
# Or with Python:
python -c "
from huggingface_hub import snapshot_download
snapshot_download(
    repo_id='Stanford-ILIAD/prism-qwen25-extra-dinosiglip-224px-0_5b',
    local_dir='pretrained_models/prism-qwen25-extra-dinosiglip-224px-0_5b',
    local_dir_use_symlinks=False,
)
"

The backbone is ~2.5 GB. pretrained_models/configs/ is already in the repo and doesn't need to be downloaded.

Training Code

Everything needed to run and evaluate the released checkpoints ships in this repo, and the full training configuration of each checkpoint is documented in its model card on HuggingFace.

Evaluation

LIBERO Standard

CUDA_VISIBLE_DEVICES=0 python experiments/robot/libero/run_libero_eval.py \
  --use_proprio True \
  --num_images_in_input 2 \
  --pretrained_checkpoint ./checkpoints/libero-spatial \
  --task_suite_name libero_spatial \
  --use_pro_version True

Swap libero_spatial for libero_object, libero_goal, or libero_10, and point --pretrained_checkpoint at the matching checkpoint.

LIBERO-PRO (perturbation robustness)

# Language perturbation (paraphrased instructions)
CUDA_VISIBLE_DEVICES=0 python experiments/robot/libero_pro/run_libero_pro_eval.py \
  --use_proprio True \
  --num_images_in_input 2 \
  --pretrained_checkpoint ./checkpoints/libero-spatial \
  --base_suite_name libero_spatial \
  --perturbation_type lan \
  --use_pro_version True
# Object perturbation      →  --perturbation_type object
# Position-swap perturbation →  --perturbation_type swap

LIBERO-Plus (7-category perturbation suite, batched)

CUDA_VISIBLE_DEVICES=0 python experiments/robot/libero_plus/run_libero_plus_eval_batched.py \
  --use_proprio True \
  --num_images_in_input 2 \
  --pretrained_checkpoint ./checkpoints/libero-spatial \
  --base_suite_name libero_spatial \
  --batch_size 48 \
  --use_pro_version True

All release evals in this repo use the same inference flags: use_proprio=True, num_images_in_input=2, use_film=False, use_l1_regression=True, center_crop=True, num_open_loop_steps=8. Baseline and Anchor-Align are compared with identical flags — the only difference is the checkpoint being evaluated.

Diagnostic Framework

To measure whether a VLA's language head agrees with its action head, we introduce a four-axis diagnostic pipeline that operates on any robot trajectory without additional human annotation.

Each episode is segmented into four short windows corresponding to distinct behaviors — Motion Direction, Orientation, Grasp, and Task Completion. Ground-truth labels for each window are derived programmatically from the demonstration (e.g. the sign of the XYZ delta gives the direction label; the gripper state gives the grasp label). Templated questions are then posed to the VLA on the same observations, and its language accuracy on each axis is compared against its action success on that same axis. This gives us a per-axis measure of language-action alignment. Prior co-trained VLAs are poorly aligned on every axis; Anchor-Align raises alignment from 16.8% to 78.4% on LIBERO-PRO rollouts and turns the alignment-success correlation strongly positive (+0.51).

Repository Layout

Key Files

Acknowledgments

This codebase builds on VLA-Adapter (Wang et al., 2025), which provides the base model architecture (Prismatic VLM + action head) and training infrastructure. We also thank OpenVLA-OFT, MiniVLA, and starVLA for their open-source contributions.

Citation

@article{dalal2026anchoralign,
  title   = {Generalizable VLA Finetuning via Representation Anchoring and Language-Action Alignment},
  author  = {Dalal, Dwip and Patel, Shivansh and Jain, Chahit and Kim, Jeonghwan and Mishra, Utkarsh and Baratian, Alex and Ha, Hyeonjeong and Ji, Heng and Lazebnik, Svetlana and Jain, Unnat},
  journal = {arXiv preprint arXiv:2607.13429},
  year    = {2026}
}