Sa2VA：图像和视频的密集理解的统一模型

来自加州大学默塞德分校、字节跳动种子公司、武汉大学和北京大学的研究人员提出了 Sa2VA，这是一种突破性的统一模型，旨在对图像和视频进行深入的理解。

Multi-Modal Large Language Models (MLLMs) have seen rapid advancements in handling various image and video-related tasks, including visual question answering, narrative generation, and interactive editing. However, achieving fine-grained video content understanding, such as pixel-level segmentation, tracking with language descriptions, and performing visual question answering on specific video prompts, still poses a critical challenge in this field. State-of-the-art video perception models excel at tasks like segmentation and tracking but lack open-ended language understanding and conversation capabilities. At the same time, video MLLMs demonstrate strong performance in video comprehension and question answering but fall short in handling perception tasks and visual prompts.

多模态大型语言模型 (MLLM) 在处理各种图像和视频相关任务方面取得了快速进步，包括视觉问答、叙述生成和交互式编辑。然而，实现细粒度的视频内容理解，例如像素级分割、语言描述跟踪以及对特定视频提示进行视觉问答，仍然是该领域的关键挑战。最先进的视频感知模型擅长分割和跟踪等任务，但缺乏开放式语言理解和对话能力。与此同时，视频 MLLM 在视频理解和问答方面表现出强大的性能，但在处理感知任务和视觉提示方面表现不佳。

Existing attempts to address video understanding challenges have followed two main approaches: MLLMs and Referring Segmentation systems. Initially, MLLMs focused on developing improved multi-modal fusion methods and feature extractors, eventually evolving towards instruction tuning on LLMs with frameworks like LLaVA. Recent developments have attempted to unify image, video, and multi-image analysis in single frameworks, such as LLaVA-OneVision. In parallel, Referring Segmentation systems have progressed from basic fusion modules to transformer-based methods that integrate segmentation and tracking within videos. However, these solutions lack a comprehensive integration of perception and language understanding capabilities.

解决视频理解挑战的现有尝试遵循两种主要方法：MLLM 和参考分割系统。最初，MLLM 专注于开发改进的多模态融合方法和特征提取器，最终发展到使用 LLaVA 等框架对 LLM 进行指令调整。最近的发展尝试将图像、视频和多图像分析统一到单一框架中，例如 LLaVA-OneVision。与此同时，参考分割系统已经从基本的融​​合模块发展到基于变压器的方法，在视频中集成了分割和跟踪。然而，这些解决方案缺乏感知和语言理解能力的全面整合。

To overcome this limitation, researchers from UC Merced, Bytedance Seed, Wuhan University, and Peking University have proposed Sa2VA, a groundbreaking unified model for a dense grounded understanding of images and videos. The model differentiates itself by supporting a comprehensive range of image and video tasks through minimal one-shot instruction tuning, addressing the limitations of existing multi-modal large language models. Sa2VA’s innovative approach integrates SAM-2 with LLaVA, unifying text, image, and video in a shared LLM token space. The researchers have also introduced Ref-SAV, an extensive auto-labeled dataset containing over 72K object expressions in complex video scenes, with 2K manually validated video objects to ensure robust benchmarking capabilities.

为了克服这一限制，来自加州大学默塞德分校、字节跳动种子公司、武汉大学和北京大学的研究人员提出了 Sa2VA，这是一种突破性的统一模型，用于对图像和视频进行密集的理解。该模型的独特之处在于，通过最少的一次性指令调整来支持全面的图像和视频任务，解决了现有多模态大语言模型的局限性。 Sa2VA 的创新方法将 SAM-2 与 LLaVA 集成，在共享的 LLM 代币空间中统一文本、图像和视频。研究人员还推出了 Ref-SAV，这是一个广泛的自动标记数据集，包含复杂视频场景中超过 72K 的对象表达，以及 2K 手动验证的视频对象，以确保强大的基准测试功能。

Sa2VA’s architecture integrates two main components: a LLaVA-like model and SAM-2, connected through a novel decoupled design. The LLaVA-like component consists of a visual encoder processing images and videos, a visual projection layer, and an LLM for text token prediction. The system employs a unique decoupled approach where SAM-2 operates alongside the pre-trained LLaVA model without direct token exchange, maintaining computational efficiency and enabling plug-and-play functionality with various pre-trained MLLMs. The key innovation lies in the connection mechanism using a special “[SEG]” token, allowing SAM-2 to generate segmentation masks while enabling gradient backpropagation through the “[SEG]” token to optimize the MLLM’s prompt generation capabilities.

Sa2VA 的架构集成了两个主要组件：类 LLaVA 模型和 SAM-2，通过新颖的解耦设计连接。类似 LLaVA 的组件由处理图像和视频的视觉编码器、视觉投影层和用于文本标记预测的 LLM 组成。该系统采用独特的解耦方法，其中 SAM-2 与预训练的 LLaVA 模型一起运行，无需直接进行令牌交换，从而保持计算效率并通过各种预训练的 MLLM 实现即插即用功能。关键的创新在于使用特殊“[SEG]”令牌的连接机制，允许SAM-2生成分段掩码，同时通过“[SEG]”令牌实现梯度反向传播，以优化MLLM的提示生成能力。

The Sa2VA model achieves state-of-the-art results on referring segmentation tasks, with Sa2VA-8B scoring 81.6, 76.2, and 78.9 cIoU on RefCOCO, RefCOCO+, and RefCOCOg respectively, outperforming previous systems like GLaMM-7B. In conversational capabilities, Sa2VA shows strong performance with scores of 2128 on MME, 81.6 on MMbench, and 75.1 on SEED-Bench. The model excels in video benchmarks, surpassing previous state-of-the-art VISA-13B by substantial margins on MeVIS, RefDAVIS17, and ReVOS. Moreover, Sa2VA’s performance is noteworthy considering its smaller model size compared to competitors, showing its efficiency and effectiveness across both image and video understanding tasks.

Sa2VA 模型在参考分割任务上取得了最先进的结果，Sa2VA-8B 在 RefCOCO、RefCOCO+ 和 RefCOCOg 上的得分分别为 81.6、76.2 和 78.9 cIoU，优于 GLaMM-7B 等之前的系统。在对话能力方面，Sa2VA 表现强劲，在 MME 上得分为 2128，在 MMbench 上得分为 81.6，在 SEED-Bench 上得分为 75.1。该模型在视频基准测试中表现出色，在 MeVIS、RefDAVIS17 和 ReVOS 上大幅超越了之前最先进的 VISA-13B。此外，考虑到与竞争对手相比，Sa2VA 的模型尺寸更小，其性能值得注意，显示了其在图像和视频理解任务中的效率和有效性。

In this paper, researchers introduced Sa2VA which represents a significant advancement in multi-modal understanding by successfully integrating SAM-2’s video segmentation capabilities with LLaVA’s language processing abilities. The framework's versatility is shown through its ability to handle diverse image and video understanding tasks with minimal one-shot instruction tuning, addressing the long-standing challenge of combining perception and language understanding. Sa2VA’s strong performance across multiple benchmarks, from referring segmentation to conversational tasks, validates its effectiveness as a unified solution for a dense, grounded understanding of visual content, marking a significant step forward in the multi-modal AI systems field.

在本文中，研究人员介绍了 Sa2VA，它成功地将 SAM-2 的视频分割功能与 LLaVA 的语言处理能力集成在一起，代表了多模态理解方面的重大进步。该框架的多功能性体现在它能够通过最少的一次性指令调整来处理不同的图像和视频理解任务，解决将感知和语言理解相结合的长期挑战。 Sa2VA 在从指代分割到对话任务等多个基准测试中的强劲表现，验证了其作为统一解决方案的有效性，可对视觉内容进行密集、扎实的理解，标志着多模式人工智能系统领域向前迈出了重要一步。

Check out the Paper and Model on Hugging Face. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter and join our Telegram Channel and LinkedIn Group. Don’t Forget to join our 65k+ ML SubReddit.

查看拥抱脸上的纸张和模型。这项研究的所有功劳都归功于该项目的研究人员。另外，不要忘记在 Twitter 上关注我们并加入我们的 Telegram 频道和 LinkedIn 群组。不要忘记加入我们 65k+ ML SubReddit。

FREE UPCOMING AI WEBINAR (JAN 15, 2025): Boost LLM Accuracy with Synthetic Data and Evaluation Intelligence

即将举行的免费人工智能网络研讨会（2025 年 1 月 15 日）：利用综合数据和评估智能提高 LLM 准确性

Join this webinar to gain actionable insights into boosting LLM model performance and accuracy while safeguarding data privacy.output

参加本次网络研讨会，获得可操作的见解，以提高 LLM 模型的性能和准确性，同时保护数据隐私。输出