OSCaR: Object State Captioning and State Change Representation

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Understanding object state changes in dynamic visual environments is key for AI, especially for human-AI interaction.
• Traditional methods for object captioning and state change detection are limited in scope and expressiveness.
• Existing language representations for object changes are often restricted to a small set of symbolic words.

Purpose of the Study:

• To introduce a new dataset and benchmark, Object State Captioning and State Change Representation (OSCaR), for evaluating AI models.
• To assess the capabilities of Multimodal Large Language Models (MLLMs) in comprehending object state changes in video.
• To provide a resource for advancing research in multimodal understanding of dynamic environments.

Main Methods:

• Developed the OSCaR dataset with 14,084 annotated video segments featuring nearly 1,000 unique objects from egocentric videos.
• Established a benchmark for evaluating MLLMs on object state captioning and state change representation.
• Conducted experiments using a fine-tuned model to assess current MLLM performance on the OSCaR benchmark.

Main Results:

• Experiments revealed that current MLLMs demonstrate some ability but lack a comprehensive understanding of object state changes.
• The fine-tuned model showed initial capabilities but requires significant enhancements in accuracy and generalization.
• The OSCaR benchmark highlights the need for more robust AI models for real-world dynamic scene understanding.

Conclusions:

• The OSCaR dataset and benchmark provide a critical resource for advancing AI's ability to interpret dynamic visual information.
• Significant improvements are needed in MLLMs to accurately and reliably understand object state transitions.
• Future research should focus on enhancing model accuracy and generalization for complex real-world scenarios.