arXiv / Demo / Models / Subscribe

💡 Introduction

VILA is a family of open VLMs designed to optimize both efficiency and accuracy for efficient video understanding and multi-image understanding.

💡 News

• [2025/7] We release OmniVinci, a state-of-the-art visual-audio joint understanding omni-modal LLM built upon VILA codebase!
• [2025/7] We release Long-RL that supports RL training on VILA/LongVILA/NVILA models with long videos.
• [2025/6] We release PS3 and VILA-HD. PS3 is a vision encoder that scales up vision pre-training to 4K resolution. VILA-HD is VILA with PS3 as the vision encoder and shows superior performance and efficiency in understanding high-resolution detail-rich images.
• [2025/1] As of January 6, 2025 VILA is now part of the new Cosmos Nemotron vision language models.
• [2024/12] We release NVILA (a.k.a VILA2.0) that explores the full stack efficiency of multi-modal design, achieving cheaper training, faster deployment and better performance.
• [2024/12] We release LongVILA that supports long video understanding, with long-context VLM with more than 1M context length and multi-modal sequence parallel system.
• [2024/10] VILA-M3, a SOTA medical VLM finetuned on VILA1.5 is released! VILA-M3 significantly outperforms Llava-Med and on par w/ Med-Gemini and is fully opensourced! code model
• [2024/10] We release VILA-U: a Unified foundation model that integrates Video, Image, Language understanding and generation.
• [2024/07] VILA1.5 also ranks 1st place (OSS model) on MLVU test leaderboard.
• [2024/06] VILA1.5 is now the best open sourced VLM on MMMU leaderboard and Video-MME leaderboard!
• [2024/05] We release VILA-1.5, which offers video understanding capability. VILA-1.5 comes with four model sizes: 3B/8B/13B/40B.

• [2024/05] We release AWQ-quantized 4bit VILA-1.5 models. VILA-1.5 is efficiently deployable on diverse NVIDIA GPUs (A100, 4090, 4070 Laptop, Orin, Orin Nano) by TinyChat and TensorRT-LLM backends.
• [2024/03] VILA has been accepted by CVPR 2024!
• [2024/02] We release AWQ-quantized 4bit VILA models, deployable on Jetson Orin and laptops through TinyChat and TinyChatEngine.
• [2024/02] VILA is released. We propose interleaved image-text pretraining that enables multi-image VLM. VILA comes with impressive in-context learning capabilities. We open source everything: including training code, evaluation code, datasets, model ckpts.
• [2023/12] Paper is on Arxiv!

Performance

Image Benchmarks

Video Benchmarks

Efficient Deployments

^{NOTE: Measured using the TinyChat backend at batch size = 1.}

Inference Performance

Decoding Throughput ( Token/sec )