Indoor Scene Change Captioning Based on Multimodality Data

Area of Science:

• Computer Vision
• Robotics
• Natural Language Processing

Background:

• Scene change recognition is crucial for applications like anomaly detection.
• Existing methods often focus on static scenes or lack 3D understanding.
• Previous 3D scene change captioning used simulated data, limiting real-world applicability.

Purpose of the Study:

• To develop a framework for describing indoor scene changes using natural language text.
• To address limitations of existing methods by incorporating 3D structures and real-world data.
• To generate large-scale indoor scene change caption datasets.

Main Methods:

• Proposed an end-to-end framework for scene change description.
• Utilized multiple input modalities: RGB images, depth images, and point cloud data.
• Generated large-scale indoor scene change caption datasets.

Main Results:

• Models combining RGB images and point cloud data showed high performance in sentence generation and caption correctness.
• The framework demonstrated robustness in understanding change types, even in complex datasets.
• Evaluated model performance across various input modalities and dataset complexities.

Conclusions:

• The developed datasets and framework advance the field of indoor scene change understanding.
• Multi-modal input, particularly RGB and point cloud data, is effective for scene change captioning.
• The approach is suitable for real-world robotic applications requiring scene understanding.