CAPformer: Pedestrian Crossing Action Prediction Using Transformer.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Robotics

Background:

• Predicting pedestrian crossing behavior is crucial for autonomous vehicle safety in urban environments.
• Existing state-of-the-art temporal models often rely on recurrent architectures.
• Recent benchmarks highlight the performance of various methods on JAAD and PIE datasets.

Purpose of the Study:

• To propose the first self-attention alternative, based on transformer architecture, for anticipating pedestrian crossing behavior.
• To fuse video and kinematic data using a multi-branch architecture.
• To evaluate the model's performance against leading methods on benchmark datasets.

Main Methods:

• Developed a novel self-attention model utilizing transformer architecture.
• Implemented a multi-branch network fusing video (RubiksNet, TimeSformer) and kinematic data (transformer encoder).
• Conducted experiments focusing on input data pre-processing, including pose keypoints and ego-vehicle speed.

Main Results:

• The proposed model achieved performance comparable to the top-ranked PCPA model (F1 Score ~0.78).
• Using only bounding box coordinates and image data, the model surpassed PCPA (F1=0.75 vs. F1=0.72).
• Identified and addressed issues with kinematic data sources like pose keypoints and ego-vehicle speed.

Conclusions:

• The transformer-based self-attention model is a viable alternative to recurrent architectures for pedestrian behavior prediction.
• The model offers advantages in parallelization and processing entire sequences, capturing relationships missed by recurrent models.
• This approach enhances the safety and reliability of autonomous vehicles in complex urban scenarios.