Social Reasoning-Aware Trajectory Prediction via Multimodal Language Model.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Natural Language Processing

Background:

• Advancements in large language models (LLMs) offer powerful context understanding and generative capabilities.
• Existing trajectory prediction methods often struggle with complex social interactions and scene context.

Purpose of the Study:

• To propose VLMTraj, a novel multimodal trajectory predictor that utilizes the knowledge from multimodal large language models.
• To reframe trajectory prediction as a visual question answering task for enhanced reasoning.

Main Methods:

• VLMTraj converts historical trajectories and scene images into natural language prompts and visual tokens.
• It employs an auxiliary multi-task question-answering module to improve understanding of scene context and social relationships.
• Training involves optimizing a numerical tokenizer and then training the language model on visual question answering prompts.

Main Results:

• VLMTraj demonstrates superior performance compared to existing numerical-based trajectory predictors.
• The model successfully interprets social relationships between pedestrians.
• It accurately predicts multimodal future trajectories on public benchmarks.

Conclusions:

• Language-based models, like VLMTraj, are effective for pedestrian trajectory prediction.
• The proposed method showcases the potential of leveraging LLMs for complex spatiotemporal reasoning tasks.