GitHub

Paper Website License: MIT

Shirui Chen, Cole Harrison, Ying-Chun Lee, Angela Jin Yang, Zhongzheng Ren, Lillian J. Ratliff, Jiafei Duan*, Dieter Fox*, Ranjay Krishna*

*Equal advising

TOPReward is a reward modeling method that uses token probabilities from vision-language models (VLMs) as zero-shot reward signals for robotics. By computing the log-likelihood that a model assigns to a task instruction given a video trajectory, TOPReward provides scalable, annotation-free reward estimation for robot learning and data curation.

The codebase supports two prediction methods:

  • TOPReward: Computes log-likelihood rewards for instruction matching on video trajectories (the TOPReward method)
  • GVL: Generative Value Learning - predicts task completion percentages (0-100%) from shuffled video frames

News

  • 02-22-2026: TOPReward is available on arXiv.

Table of Contents

Quick Start

After setup (see Getting Started), run either prediction mode:

GVL Prediction

HYDRA_FULL_ERROR=1 PYTHONPATH=. uv run python3 -m topreward.scripts.predict \
  --config-dir configs/experiments \
  --config-name predict_gvl

TOPReward Prediction

HYDRA_FULL_ERROR=1 PYTHONPATH=. uv run python3 -m topreward.scripts.predict \
  --config-dir configs/experiments \
  --config-name predict_topreward \
  model=qwen

Single Runner Script

If you prefer one shell entrypoint, use:

topreward/scripts/run_predict.sh --config-name predict_gvl dataset=nyudoor model=gemini
topreward/scripts/run_predict.sh --config-name predict_topreward dataset=austin_sirius_dataset model=qwen prediction.add_chat_template=true

The script selects an experiment config by name and forwards all remaining arguments as Hydra overrides.

Common override types:

  • Group selection: dataset=... model=... data_loader=... mapper=... prompts=...
  • Scalar values: prediction.num_examples=20 prediction.output_dir=./results model.model_name=...
  • Booleans: prediction.add_chat_template=true prediction.use_video_description=false

Results are saved under outputs/DATE_TIME/ with predictions, raw outputs, and metrics.

Tip: you can override any config at the CLI, e.g. model.temperature=0.5.

Getting Started

Prerequisites

  • Python 3.11+
  • uv for environment and dependency management
  • ffmpeg available on your system PATH

Installation

  1. Clone the repository:

    git clone https://github.com/TOPReward/TOPReward.git
cd TOPReward

  2. Install ffmpeg (if not already installed):

    # macOS (Homebrew)
brew install ffmpeg
# Ubuntu / Debian
sudo apt-get update && sudo apt-get install -y ffmpeg

  3. Set up a uv virtual environment and install dependencies:

    uv venv
source .venv/bin/activate
uv sync

Environment Variables

Create a .env file in the project root:

cp .env.example .env

Then edit .env with your credentials:

OPENAI_API_KEY="your-openai-api-key"
GOOGLE_API_KEY="your-google-api-key"
HUGGING_FACE_HUB_TOKEN="your-hugging-face-token"

Configuration (Hydra)

Configuration lives in configs/:

  • configs/model/: model configs (e.g., gemini.yaml, gemma.yaml, openai.yaml)
  • configs/dataset/: dataset configs
  • configs/data_loader/: data loader configs (e.g., huggingface.yaml, local.yaml)
  • configs/prompts/: prompt styles
  • configs/experiments/: complete experiment presets (e.g., predict_gvl.yaml)

Override parameters from the command line. Examples:

# Run with explicit experiment config
PYTHONPATH=. uv run python3 -m topreward.scripts.predict --config-dir configs/experiments --config-name predict_gvl
# Override individual fields
PYTHONPATH=. uv run python3 -m topreward.scripts.predict --config-dir configs/experiments --config-name predict_gvl \
  model=gemini dataset=berkeleymvp data_loader=huggingface model.temperature=0.5

Run TOPReward on a single local video directly (no frame extraction step):

topreward/scripts/run_predict.sh --config-name predict_topreward \
  data_loader=local \
  dataset=local_video \
  dataset.video_path=/absolute/path/to/video.mp4 \
  dataset.instruction="open the drawer" \
  prediction.num_examples=1 \
  prediction.output_dir=./results/local_video_topreward

Extending TOPReward

Adding a New Model

TOPReward clients inherit from topreward.clients.base.BaseModelClient. You only need to implement _generate_from_events(self, events: list[Event]) -> str, which receives a provider-agnostic sequence of text/image events already assembled by the framework. See topreward/clients/gemini.py for a complete reference implementation.

  1. Implement a client in topreward/clients/my_model.py:
# topreward/clients/my_model.py (concise example)
import os
from typing import cast, List
from loguru import logger
from topreward.clients.base import BaseModelClient
from topreward.utils.aliases import Event, ImageEvent, ImageT, TextEvent
from topreward.utils.images import encode_image
class MyModelClient(BaseModelClient):
  def __init__(self, *, rpm: float = 0.0, model_name: str):
    super().__init__(rpm=rpm)
    if not os.getenv("MY_MODEL_API_KEY"):
      raise OSError("Missing MY_MODEL_API_KEY")
    self.model_name = model_name
    logger.info(f"Using MyModel '{self.model_name}'")
  def _generate_from_events(self, events: List[Event]) -> str:
    parts: List[bytes | str] = []
    for ev in events:
      if isinstance(ev, TextEvent):
        parts.append(ev.text)
      elif isinstance(ev, ImageEvent):
        parts.append(encode_image(cast(ImageT, ev.image)))
    # Call your provider with `parts` and return the provider's text response.
    # Placeholder response for docs/tests:
    return "Frame 1: Task Completion: 50%\nFrame 2: Task Completion: 100%"
  1. Add a Hydra config at configs/model/my_model.yaml:
_target_: topreward.clients.my_model.MyModelClient
model_name: my-model-name
rpm: 15  # requests per minute (rate limiter)
  1. Use your model via CLI or experiment config:
PYTHONPATH=. uv run python3 -m topreward.scripts.predict \
  --config-dir configs/experiments \
  --config-name predict_gvl \
  model=my_model

Adding a New Dataset

Create a dataset config that matches the keys used by our HuggingFace loader (configs/data_loader/huggingface.yaml). Example:

# configs/dataset/my_dataset.yaml
name: my_dataset
dataset_name: "org-or-user/my_dataset_on_hub"
camera_index: 0
max_episodes: 100
num_frames: 15
num_context_episodes: 2

Then choose a loader (e.g., Hugging Face) in your experiment or via CLI:

PYTHONPATH=. uv run python3 -m topreward.scripts.predict \
  --config-dir configs/experiments \
  --config-name predict_gvl \
  dataset=my_dataset data_loader=huggingface

Troubleshooting

  • macOS library path: 
    export DYLD_FALLBACK_LIBRARY_PATH=/opt/homebrew/lib
  • GPU OOM (CUDA): reduce batch_size or image resolution in the model config (e.g., configs/model/gemini.yaml).
  • Hugging Face authentication: ensure HUGGING_FACE_HUB_TOKEN is set in .env for gated models/private datasets.
  • API rate limits: consider lowering concurrency or increasing TQDM_MININTERVAL when applicable.

Citation

If you use TOPReward in your research, please cite:

@article{chen2026topreward,
  title={TOPReward: Token Probabilities as Hidden Zero-Shot Rewards for Robotics},
  author={Chen, Shirui and Harrison, Cole and Lee, Ying-Chun and Yang, Angela Jin and Ren, Zhongzheng and Ratliff, Lillian J and Duan, Jiafei and Fox, Dieter and Krishna, Ranjay},
  journal={arXiv preprint arXiv:2602.19313},
  year={2026}
}

Acknowledgements

TOPReward builds on OpenGVL / GVL (Generative Value Learning). We reuse and adapt substantial portions of that implementation throughout this repository, and thank the OpenGVL authors: Paweł Budzianowski, Emilia Wiśnios, Gracjan Góral, Michał Tyrolski, Igor Kulakov, Viktor Petrenko, and Krzysztof Walas.

Video processing utilities in topreward/utils/video_utils.py are adapted from LeRobot (HuggingFace), licensed under Apache 2.0.

License

This project is licensed under the MIT License. See LICENSE for details.

Read the original on github.com ↗