Goal-Guided Prompting With Adaptive Modality Selection for Efficient Assembly Activity Anticipation in Egocentric...

Area of Science:

• Computer Vision
• Artificial Intelligence
• Human-Computer Interaction

Background:

• Augmented reality (AR) devices with egocentric observation and multimodal perception offer potential for smart assistants in assembly tasks.
• Anticipating future activities in egocentric videos is crucial for human-robot collaboration but faces challenges in accuracy and computational efficiency.

Purpose of the Study:

• To develop an efficient and accurate method for anticipating near-future assembly activities from egocentric videos.
• To address the trade-off between accuracy and computational cost in existing egocentric activity anticipation models.

Main Methods:

• Proposed a goal-guided prompting framework with adaptive modality selection (GP-AMS).
• Injected high-level goal clues into prompts to guide a vision-language (V-L) model for future activity prediction.
• Employed a mask-and-predict strategy with casual masking and probabilistic token-dropping.
• Developed an adaptive modality selection strategy to dynamically choose modalities, optimizing computational load.

Main Results:

• The GP-AMS framework demonstrated consistent improvements in anticipation accuracy on public datasets.
• Achieved significant savings in computation budgets compared to existing methods.
• Validated the feasibility of the approach for real-world AR devices.

Conclusions:

• The proposed GP-AMS method effectively bridges the semantic gap for future activity anticipation.
• Achieved a superior balance between accuracy and computational efficiency in egocentric activity anticipation.
• Paved the way for practical deployment of advanced smart assistants in assembly tasks.