Reliable object tracking by multimodal hybrid feature extraction and transformer-based fusion.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Robotics

Background:

• Visual object tracking using visible light faces challenges like low light, high dynamic ranges, and background clutter.
• Existing multimodal approaches often use local feature interactions, limiting feature modeling potential.
• Reliable object tracking requires robust feature extraction and fusion across diverse visual cues.

Purpose of the Study:

• To propose a novel multimodal hybrid tracker (MMHT) for robust single object tracking.
• To enhance feature modeling by integrating frame-event data and advanced neural networks.
• To improve tracking performance in complex visual environments.

Main Methods:

• Utilized a hybrid backbone combining an artificial neural network (ANN) and a spiking neural network (SNN) for feature extraction.
• Employed a unified encoder to align features from different visual modalities.
• Developed an enhanced transformer-based module with attention mechanisms for multimodal feature fusion.

Main Results:

• The MMHT model effectively constructs a multiscale and multidimensional visual feature space.
• Achieved discriminative feature modeling, leading to enhanced tracking accuracy.
• Demonstrated competitive performance against state-of-the-art methods in extensive experiments.

Conclusions:

• The proposed MMHT model effectively addresses challenges in visual object tracking.
• Integrating frame-event data with hybrid ANN-SNN architectures improves tracking reliability.
• The study highlights the potential of multimodal fusion for robust computer vision tasks.