LLaVA-NeXT: Open Large Multimodal Models

Release Notes

📢 Announcement: The training pipeline in this repository is now considered legacy. For the latest training pipeline with support for LLaVA-OneVision and LLaVA-OneVision-2, please refer to lmms-engine.

• [2025/08/29] 🔥 LLaVA-Critic-R1 We release LLaVA-Critic-R1, a family of generative critic VLM trained through GRPO using pairwise critic data. LLaVA-Critic-R1 not only demonstrates strong critic capability, but also achieves state-of-the-art policy performance at the 7B scale. Refer to LLaVA-Critic-R1 for more training details.

📄 Explore more: - LLaVA-Critic-GRPO Dataset: Download the dataset. - LLaVA-Critic-R1-7B: LLaVA-Critic-R1 trained based on Qwen-2.5-VL-7B. - LLaVA-Critic-R1-7B-Plus-Qwen: LLaVA-Critic-R1+ trained based on ThinkLite-VL-7B. - LLaVA-Critic-R1-7B-Plus-Mimo: LLaVA-Critic-R1+ trained based on MiMo-VL-7B-RL-2508. - LLaVA-Critic-R1-7B-Plus-LLaMA32v: LLaVA-Critic-R1+ trained based on Llama-3.2-11B-Vision-Instruct. - Paper: Detailed information about LLaVA-Critic-R1.

• [2024/10/04] 🔥 LLaVA-Video (formerly LLaVA-NeXT-Video) has undergone a major upgrade! We are excited to release LLaVA-Video-178K, a high-quality synthetic dataset for video instruction tuning. This dataset includes:

• 178,510 caption entries

• 960,792 open-ended Q&A pairs
• 196,198 multiple-choice Q&A items

Along with this, we’re also releasing the LLaVA-Video 7B/72B models, which deliver competitive performance on the latest video benchmarks, including Video-MME, LongVideoBench, and Dream-1K.

📄 Explore more: - LLaVA-Video-178K Dataset: Download the dataset. - LLaVA-Video Models: Access model checkpoints. - Paper: Detailed information about LLaVA-Video. - LLaVA-Video Documentation: Guidance on training, inference and evaluation.

• [2024/09/13] 🔥 🚀 LLaVA-OneVision-Chat. The new LLaVA-OV-Chat (7B/72B) significantly improves the chat experience of LLaVA-OV. 📄

• [2024/08/06] 🔥 🚀 LLaVA-OneVision (OV)! The new LLaVA-OV models (0.5B/7B/72B) achieve new state-of-the-art performance across single-image, multi-image, and video benchmarks, sometimes rivaling top commercial models on 47 diverse benchmarks. 📄 Explore More:
• [Paper]: In-depth insights, new emegerging scenarios, ie, strong video understadning through task transfer from images.
• [LLaVA-OV Doc]: Model inference and evaluation guidance.

• [Scripts]: Start training models on your single-image/multi-image/video data.

• [2024/07/16] 🔥 LLaVA-NeXT-Video has been upgraded. The new 32B model achieves the best open-source performance on several video benchmarks, including Video-MME. Please refer to this page for details, refer to llava_next-video_demo for demo.

• [2024/06/23] 🔥 LLaVA-NeXT-Interleave is released. We utilize image-text interleaved format to unify multi-image, video, and 3D tasks in one LLM and achieve SoTA performance on a wide range of benchmarks. Check out paper, blog, and checkpoints to see new capabilities and improved performance! We have released 0.5b, 7b, and 7b-dpo models.

• An all-round LLM for multi-image, video, and 3D with strong performance [demo]
• Construct interleave training data M4-Instruct

• Construct multi-image benchmark LLaVA-Interleave Bench

• [2024/05/25] 🔥 Wondering "What Else Influences Visual Instruction Tuning Beyond Data?" Our new blog summarizes empirical explorations to ablate the various design choices in improving LMMs except instruct data itself. Meanwhile, open-source the recapioned high-quality data using LLaVA-NeXT-34B on [COCO] [LCS] [CC3M].

• Architectures (LMM & Vision Encoder)
• Visual Representations (Resolution & # Tokens)

• Training Strategies (High-quality data & Trainable modules)

• [2024/05/10] 🔥 LLaVA-NeXT (Stronger) models are released, with support of stronger LMM inlcuding LLama-3 (8B) and Qwen-1.5 (72B/110B) Check out [blog] and [checkpoints] to see improved performance!

• [2024/05/10] 🔥 LLaVA-NeXT (Video) is released. The image-only-trained LLaVA-NeXT model is surprisingly strong on video tasks with zero-shot modality transfer. DPO training with AI feedback on videos can yield significant improvement. [Blog], [checkpoints] and [sglang]
• [2024/01/30] 🔥 LLaVA-NeXT is out! With additional scaling to LLaVA-1.5, LLaVA-NeXT-34B outperforms Gemini Pro on some benchmarks. It can now process 4x more pixels and perform more tasks/applications than before. Check out the blog post, and explore the demo! Models are available in Model Zoo. Training/eval data and scripts coming soon.

More

• [2024/03/10] 🔥 Releasing LMMs-Eval, a highly efficient evaluation pipeline we used when developing LLaVA-NeXT. It supports the evaluation of LMMs on dozens of public datasets and allows new dataset onboarding, making the dev of new LMMs much faster. [Blog] [Codebase]

• [2023/11/10] LLaVA-Plus is released: Learning to Use Tools for Creating Multimodal Agents, with LLaVA-Plus (LLaVA that Plug and Learn to Use Skills). [Project Page] [Demo] [Code] [Paper]

• [2023/11/02] LLaVA-Interactive is released: Experience the future of human-AI multimodal interaction with an all-in-one demo for Image Chat, Segmentation, Generation and Editing. [Project Page] [Demo] [Code] [Paper]
• [2023/10/26] 🔥 LLaVA-1.5 with LoRA achieves comparable performance as full-model finetuning, with a reduced GPU RAM requirement (ckpts, script). We also provide a doc on how to finetune LLaVA-1.5 on your own dataset with LoRA.
• [2023/10/12] Check out the Korean LLaVA (Ko-LLaVA), created by ETRI, who has generously supported our research! [🤗 Demo]
• [2023/10/05] 🔥 LLaVA-1.5 is out! Achieving SoTA on 11 benchmarks, with just simple modifications to the original LLaVA, utilizes all public data, completes training in ~1 day on a single 8-A100 node, and surpasses methods like Qwen-VL-Chat that use billion-scale data. Check out the technical report, and explore the demo! Models are available in Model Zoo. The training data and scripts of LLaVA-1.5 are released here, and evaluation scripts are released here!
• [2023/09/26] LLaVA is improved with reinforcement learning from human feedback (RLHF) to improve fact grounding and reduce hallucination. Check out the new SFT and RLHF checkpoints at project [LLavA-RLHF]
• [2023/09/22] LLaVA is accepted by NeurIPS 2023 as oral presentation, and LLaVA-Med is accepted by NeurIPS 2023 Datasets and Benchmarks Track as spotlight presentation.
• [2023/11/06] Support Intel dGPU and CPU platforms. More details here.
• [2023/10/12] LLaVA is now supported in llama.cpp with 4-bit / 5-bit quantization support!
• [2023/10/11] The training data and scripts of LLaVA-1.5 are released here, and evaluation scripts are released here!
• [2023/10/10] Roboflow Deep Dive: First Impressions with LLaVA-1.5.
• [2023/09/20] We summarize our empirical study of training 33B and 65B LLaVA models in a note. Further, if you are interested in the comprehensive review, evolution and trend of multimodal foundation models, please check out our recent survey paper ``Multimodal Foundation Models: From Specialists to General-Purpose Assistants''.

• [2023/07/19] 🔥 We release a major upgrade, including support for LLaMA-2, LoRA training, 4-/8-bit inference, higher resolution (336x336), and a lot more. We release LLaVA Bench for benchmarking open-ended visual chat with results from Bard and Bing-Chat. We also support and verify training with RTX 3090 and RTX A6000. Check out LLaVA-from-LLaMA-2, and our model zoo!
• [2023/06/26] CVPR 2023 Tutorial on Large Multimodal Models: Towards Building and Surpassing Multimodal GPT-4! Please check out [[Slides](https://datarelease.blob.core.windows.net/tutorial/vision_foundation_models_2023/slides/Chuny