RelationLMM: Large Multimodal Model as Open and Versatile Visual Relationship Generalist.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Visual relationships are fundamental to perception and reasoning, underpinning tasks like scene graph generation and human-object interaction.
• Current approaches often lack task generalization, employ rigid formulations, and underutilize language-based reasoning.
• Large Multimodal Models (LMMs) show promise for unification but struggle with visual relationship tasks due to task conflicts and insufficient instance detail.

Purpose of the Study:

• To develop a unified framework, RelationLMM, capable of handling diverse visual relationship tasks using Large Multimodal Models (LMMs).
• To overcome limitations in existing methods by creating a single, flexible model for multiple visual relationship tasks.
• To leverage the language-in, language-out capabilities of LMMs for improved visual reasoning.

Main Methods:

• Reforming data for LMMs, focusing on task unification and instance-level information rather than architectural changes.
• Disassembling complex tasks into simpler, common sub-tasks.
• Verbally estimating instance confidence and augmenting instance diversity without additional modules.

Main Results:

• RelationLMM demonstrates strong performance, generalizability, and flexibility across various visual relationship tasks.
• The proposed data-centric approach effectively addresses task conflicts and insufficient instance information within LMMs.
• A single model with a unified set of weights achieves proficiency in multiple specialized tasks.

Conclusions:

• RelationLMM establishes a novel, data-driven approach to enhance LMMs for visual relationship understanding.
• The framework offers a flexible and generalizable solution, unifying diverse tasks under a single model.
• This work highlights the potential of data reformation in LMMs for advancing visual reasoning capabilities.