🧭 Overview

ST-LLM is a Spatio-Temporal Large Language Model designed to reason jointly over egocentric video, 3D point clouds, and natural language. It is evaluated on the newly proposed REA (Reasoning about Environments and Actions) dataset, covering five fine-grained tasks:

• Relative Direction
• Relative Distance
• Find My Item
• Furniture Affordance Prediction
• Action Planning

Our method introduces a cross-modal alignment module and positional encoding to fuse local temporal cues with global spatial scene context, significantly improving task performance over existing MLLMs.

🚧 Project Status

We are actively improving this repository. Below is a summary of what is already released and what’s still in progress.

✅ Done

• Improved REA dataset
• Released training and inference code
• Uploaded REA question-answer dataset
• Uploaded 3D point cloud data
• Released arXiv preprint

🔜 Upcoming

• Write complete training and usage instructions
• Upload inference code
• Build Dataset Visualizer
• Upload a small batch of example data
• Refactor and clean up the repository
• Reconstruct video-level point clouds with updated pipeline

⚙️ Quick Start

This guide walks through setting up the environment for training and inference with ST-LLM, including dependencies like FlashAttention, LLaVA-NeXT, MinkowskiEngine, Openscene, PointNet++, Mask2Former, Semantic-SAM, and Deformable DETR.

Step 1. Create Conda Environment

export Main=$(pwd)
conda create -n stllm python=3.9
conda activate stllm

Step 2. Install PyTorch (2.4.1) with CUDA (11.8)

MinkowskiEngine requires CUDA version < 12.0.

conda install pytorch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1 pytorch-cuda=11.8 -c pytorch -c nvidia

Step 3. Install Build Tools

pip install ninja setuptools==69.5.1 

✅ Ensure ninja is in your PATH. You can check this with:

which ninja

Step 4. Install FlashAttention (v2.5.7)

TMPDIR=/tmp \
PIP_CACHE_DIR=/tmp/pip-cache \
TORCH_EXTENSIONS_DIR=/tmp/torch-extensions \
TRITON_CACHE_DIR=/tmp/triton-cache \
MAX_JOBS=4 \
pip install -v flash-attn==2.5.7 --no-build-isolation

Step 5. Install LLaVA-NeXT

cd LLaVA-NeXT
pip install -e ".[train]"

Step 6. Install OpenScene

conda install conda-forge::openexr
conda install openblas-devel -c anaconda # Please find a way to install openblas
# Install MinkowskiEngine
git clone https://github.com/NVIDIA/MinkowskiEngine.git
cd MinkowskiEngine
python setup.py install --blas=openblas

Step 7. Install PointNet++ and accelerated giou from source:

cd LLaVA-NeXT/llava/model/openscene/third_party/pointnet2
python setup.py install
cd ../utils
python cython_compile.py build_ext --inplace

Step 8. Install Python Dependencies

cd $Main
pip install -r requirements.txt

Step 9. Install Deformable DETR

git clone https://github.com/fundamentalvision/Deformable-DETR.git
cd Deformable-DETR
cd ./models/ops
sh ./make.sh
cd $Main

Step 10. Install Semantic-SAM and Mask2Former Please install Semantic-SAM follow their instructions on their repo.

# Install Mask2Former
cd Semantic-SAM
git clone https://github.com/facebookresearch/Mask2Former.git
cd Mask2Former/mask2former/modeling/pixel_decoder/ops
sh make.sh

Step 11. Substitute modeling_utils.py in Transformers package.

# Find the path to Tranformers
tf_path=$(python -c "import transformers; import os; print(os.path.dirname(transformers.__file__))")
echo "$tf_path"
cd $Main
mv modeling_utils.py "$tf_path"

We provide a simple script to run inference on a sample REA QA example. Make sure the pretrained weights and sample data are properly downloaded.

(coming soon, currently eval code)
python LLaVA-NeXT/llava/train/inference.py

📊 Dataset: REA

The Reasoning about Environments and Actions (REA) dataset contains five types of spatio-temporal reasoning tasks:

• Relative Direction
• Relative Distance
• Find My Item
• Furniture Affordance Prediction
• Action Planning

Each QA sample in the dataset consists of:

• A short egocentric action video (sampled from EPIC-KITCHENS)
• A 3D point cloud of the environment (REA 3D Data, see below)
• A Question-Answer pair (under REA_dataset)

For more details, refer to our project page or see Section 3 of our paper. Note: Currently, the point clouds are reconstructed per scene, rather than per video. This provides more accurate geometry, as the reconstructions are manually verified and annotated by humans. We will also provide the corresponding reconstruction image names used to generate the 3D point cloud. These images can be used for 2D-LLM-based inference, and are available in the Google Drive.

Before using our data or running any code, please download the EPIC-KITCHENS dataset (RGB video frames).
We use the downsampled version of the videos for all processing.

• Official website: https://epic-kitchens.github.io/epic-fields/
• Download the RGB frames (downsampled version) following their instructions.

📁 REA 3D Data

We release the 3D data on Google Drive. This package contains:

• Point clouds reconstructed for each scene
• Camera poses for the egocentric action video
  (32 uniformly sampled frames per clip)

Instructions to place the data: To be provided.

🏗️ Model Architecture

ST-LLM combines three modalities:

• Egocentric Video: Captures local temporal context
• 3D Point Cloud: Encodes the global spatial layout
• Text Instruction: The QA prompt to be answered

We use a Q-Former-like cross-modal alignment module with 3D positional encoding to merge these modalities before feeding them into an LLM decoder.

📦 Training

bash LLaVA-NeXT/scripts/video/train/stllm_rea_train.sh

🔍 Evaluation

cd baseline_inference

To Evaluate LLaVA-Video-7B-Qwen2

python llava_video_qwen2_inference.py --start 0 --end -1 --cuda 0 \
    --json_path ../REA_dataset/qa_val_1757_v20.json \
    --rgb_dir /path/to/EPIC-KITCHENS/rgb \
    --pretrained lmms-lab/LLaVA-Video-7B-Qwen2 \
    --model_name llava_qwen \
    --scene_level_recon True \
    --pcd_folder /path/to/epic-kitchens-vggt-anyloc-val-scene

To evaluate LLaVA-OV-Qwen2-7B

python llava_video_qwen2_inference.py --start 0 --end -1 --cuda 0 \
    --json_path ../REA_dataset/qa_val_1757_v20.json \
    --rgb_dir /path/to/EPIC-KITCHENS/rgb \
    --pretrained lmms-lab/llava-onevision-qwen2-7b-ov \
    --model_name llava_qwen \
    --scene_level_recon True \
    --pcd_folder /path/to/epic-kitchens-vggt-anyloc-val-scene

To evaluate Qwen2-VL-7B-Instruct
Modify the paths in the main function and run

python qwen2vl7binstruct_inference.py

🙏 Acknowledgments

We thank the authors of EPIC-KITCHENS, VISOR, EPIC-FIELDS, and COLMAP for their foundational work.
This project also builds on frameworks like LL3DA, VGGT, FlashAttention, LLaVA-NeXT, MinkowskiEngine, Openscene, PointNet++, Mask2Former, Semantic-SAM, and Deformable DETR.
We appreciate the compute support from the Delta GPU cluster and funding from the National Science Foundation (NSF).

📜 License

This project is released under the MIT License.

🔗 Citation

If you use our work, please cite:

@misc{zheng2025spatiotemporalllmreasoningenvironments,
      title={Spatio-Temporal LLM: Reasoning about Environments and Actions},
      author={Haozhen Zheng and Beitong Tian and Mingyuan Wu and Zhenggang Tang and Klara Nahrstedt and Alex Schwing},
      year={2025},
      eprint={2507.05258},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2507.05258},
}