Talking with Actionbits-A Part-Enhanced VLM for Action and Interaction Recognition in Animals

Area of Science:

• Computer Vision
• Animal Behavior Analysis
• Machine Learning

Background:

• Understanding animal behavior is crucial for ecological monitoring and scientific research.
• Current methods for animal action recognition face limitations with fine-grained motions, spatial relationships, and multi-individual interactions in wild settings.

Purpose of the Study:

• To introduce AIRA (Action and Interaction Recognition in Animals), a unified framework designed to overcome existing challenges in recognizing animal actions and interactions.
• To improve the robustness and generalization capabilities of animal behavior recognition models.

Main Methods:

• AIRA utilizes a vision-language model (VLM) with an action-centered representation space focusing on body parts and motion.
• Introduced Actionbit tokens, generated by a large language model (LLM), to encode part-specific movements.
• Employed Part-Enhanced Prompt Fine-tuning (PEPF) with Action-actionbit Alignment (AbA) and Part-Vision Prompting (PVP) to enhance VLM sensitivity to pose and motion cues.

Main Results:

• AIRA demonstrated consistent improvements in both action and interaction recognition across multiple benchmarks.
• The action-centered approach and relational reasoning proved vital for analyzing wild animal behavior.
• Enhanced recognition of fine-grained motions and interactions was observed.

Conclusions:

• AIRA offers a significant advancement in animal behavior analysis, particularly for in-the-wild scenarios.
• The proposed Actionbit tokens and PEPF method effectively capture fine-grained part-motion semantics.
• This framework paves the way for more accurate and comprehensive ecological monitoring through improved animal behavior understanding.