Verbs in Action: Improving verb understanding in video-language models

JAX implementation for Verb-Focused Contrastive (VFC) learning of video-text models. For details, see arXiv.

Training

Like other projects in Scenic, all model parameters, training sets and datasets are specified using configuration files.

An example command-line to train our Verb-Focused Contrastive (VFC) pre-training on the Spoken Moments in Time dataset using this config file is:

$ python -m scenic.projects.verbs_in_action.main \
  --config=scenic/projects/verbs_in_action/configs/vfc.py \
  --workdir=verb_focused_contrastive/

Likewise, you can train a baseline model on the Spoken Moments in Time dataset. Oue baseline is a standard contrastive video-text model and corresponds to the run coined as Baseline for example in tables 2, 3 or 6 of our paper. We follow this config file and run:

$ python -m scenic.projects.verbs_in_action.main \
  --config=scenic/projects/verbs_in_action/configs/baseline.py \
  --workdir=baseline_contrastive/

Model Zoo

Note that we are not planning to open-source the PaLM generated captions of the Spoken Moments in Time dataset. However, all the details to reproduce the process to generate the hard negative captions with Large Language Models are included in our paper.

Kinetics-verb

We introduce the Kinetics-verb split which consists in isolating classes from the Kinetics-400 dataset that share a common noun with another class, but have a different verb. We use this rule to create a subset of 97 classes from the Kinetics-400 test set.

Citation

If you use the verbs in action project, please cite the following BibTeX entry:

@inproceedings{momeni2023verbs,
  title={Verbs in Action: Improving verb understanding in video-language models},
  author={Momeni, Liliane and Caron, Mathilde and Nagrani, Arsha and Zisserman, Andrew and Schmid, Cordelia},
  booktitle={International Conference on Computer Vision (ICCV)},
  year={2023}
}