Overview

To properly benchmark token merging in vision transformers for video, we patch it spatio-temporally into existing models and test against Kinetics-400, Something-Something v2 and EPIC-KITCHENS-100. As well as this, we qualitatively investigate the different merging decisions made by different layers and the degree to which merging relies on semantics. We compare against other existant token reduction methods and briefly introduce a hybrid dropout/merging method that displays a slight edge over the two.

Results

Install

To get the SlowFast backend working, you'll need to set up the following environment:

conda create -n vhdytm

conda install python=3.10

pip install torch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1 --index-url https://download.pytorch.org/whl/cu118

pip install simplejson

conda install av -c conda-forge

pip install opencv-python

cd ../
git clone https://github.com/facebookresearch/pytorchvideo.git;
cd pytorchvideo;
pip install -e .;
cd ../video-how-do-your-tokens-merge

pip install 'git+https://github.com/facebookresearch/fairscale'

pip install opencv-python

pip install psutil

pip install scipy

pip install pandas

pip install tensorboard

pip install pillow==9

pip install matplotlib

pip install scikit-learn

pip install setuptools==59.5.0

For further packages:

pip install einops

pip install timm==0.4.9

pip install transformers

pip install wandb

Datasets

Kinetics-400

Download the videos (here is a version available online).
The files for the training and validation splits are bundled, otherwise they should be formatted as the filename of each video followed by the ground truth class (space-separated).
Update the <KINETICS-400-VIDEO> placeholder to be the path to the dataset split files.
Update the <KINETICS-400-PATH-PREFIX> placeholder to be the path that precedes video filenames.

Something-Something v2

Download the videos from here.
Download the frame lists from here.
Extract frames (something like this should work).
Update the <SOMETHING-SOMETHING-V2-VIDEO> placeholder to be the path to the dataset split files.
Update the <SOMETHING-SOMETHING-V2-PATH-PREFIX> placeholder to be the path that precedes video filenames.

EPIC-KITCHENS-100

Download the frames directly with this downloader.
Clone the annotations from here.
Update the <EPIC-KITCHENS-100-VIDEO> placeholder to be the path to the videos.
Update the <EPIC-KITCHENS-100-ANNOTATIONS> placeholder to be the path to the annotations repo.
Using the official downloader should give you a folder structure like PXX/PXX_XX/rgb_frames, but if you somehow end up with it ordered differently you may need to change the EPICKITCHENS.PARTICIPANT_FIRST option or edit this.

Checkpoints

Model	K-400	SSv2	EK-100
TimeSformer	Checkpoint	Checkpoint	Finetuned
Motionformer	Checkpoint	Checkpoint	Checkpoint
VideoMAE	Checkpoint	Checkpoint	Finetuned
ViViT	Checkpoint	Finetuned	Finetuned

Examples

We have notebooks for basic demonstrations and visualisations.

Acknowledgments

Our code is based upon the following prior works: ToMe, SlowFast, TimeSformer, MotionFormer, VideoMAE and timm.

Citation

@inproceedings{pollard2025video,
    author    = {Pollard, Sam and Wray, Michael},
    title     = {Video, How Do Your Tokens Merge?},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
    month     = {June},
    year      = {2025},
    pages     = {3347-3356}
}