🖋 Kairui Hu, Penghao Wu, Fanyi Pu, Wang Xiao, Xiang Yue, Yuanhan Zhang, Bo Li, and Ziwei Liu

🔥 News

• [2025-3] 🎉🎉 We update Video-MMMU leaderboard to include Kimi-k1.6-preview-20250308, VideoLLaMA3-7B. Thanks for the acknowledgement to Video-MMMU!
• [2025-2] 🎉🎉 We update Video-MMMU leaderboard to include Qwen-2.5-VL-72B, Qwen-2.5-VL-7B, mPLUG-Owl3-7B, InternVideo2.5-Chat-8B, VideoChat-Flash-7B@448. Thanks for the acknowledgement to Video-MMMU!
• [2025-1] 🎉🎉 We introduce VideoMMMU, a multi-modal, multi-disciplinary video benchmark that evaluates the knowledge acquisition capability from educational videos.

🧠 Overview

Video-MMMU is the first benchmark to assess knowledge acquisition from educational videos, evaluating how well LMMs learn new knowledge from videos and apply what they learn in practice.

1) Knowledge-Intensive Video Collection

Video-MMMU features 300 lecture-style videos covering 6 professional disciplines—Art, Business, Science, Medicine, Humanities, and Engineering, spanning 30 subjects.

2) Knowledge Acquisition-Based Question Design

Each video is accompanied by 3 QA pairs, designed to evaluate video-based learning at different cognitive levels:

• Perception – Identifying key information.
• Comprehension – Understanding underlying concepts.
• Adaptation – Applying knowledge to new scenarios.

This results in 900 question-answer pairs (300 videos × 3 QA pairs per video), systematically measuring a model's ability to acquire and apply knowledge from educational videos.

❓QA Design

Perception

• ASR (Automatic Speech Recognition): The Art category (top left).
• OCR (Optical Character Recognition): The Business category (bottom left).

Comprehension

• Concept Comprehension: The Humanities category (top center).
• Problem-Solving Strategy Comprehension: The Science category (bottom center).

Adaptation

• Case Study Analysis: The Medicine category (top right).
• Problem-Solving Strategy Adaptation: The Engineering category (bottom right).

🔍 A New Perspective on VideoQA

Videos as a Knowledge Source

Traditional VideoQA benchmarks focus primarily on evaluating how well models interpret visual content. Video-MMMU is the first to treat videos as a source of knowledge, assessing how effectively LMMs acquire knowledge from educational videos.

Measuring Knowledge Gain: The Δknowledge Metric

A key novelty of Video-MMMU is that it evaluates not just a model’s absolute accuracy but also its delta accuracy—the improvement in performance after learning from a video. A model may initially fail to solve an exam question, but we give the model a video where a human could learn to solve the question by watching this video. Video-MMMU evaluates how well LMMs improve their performance after watching the relevant video. Video-MMMU introduces Δknowledge to quantify the model's learning gain on the Adaptation track questions. Δknowledge is defined as the normalized performance gain: