Accurate Long-Term Multiple People Tracking using Video and Body-Worn IMUs

Area of Science:

• Computer Vision
• Robotics
• Sensor Fusion

Background:

• Current video-based multi-person tracking methods heavily rely on human appearance, leading to reduced accuracy with non-discriminative features or apparel changes.
• Visual occlusions and appearance variations significantly challenge existing person tracking systems.

Purpose of the Study:

• To develop a robust multi-person tracking method by fusing video data with motion signals from body-worn inertial measurement units (IMUs).
• To overcome the limitations of appearance-based tracking by incorporating motion dynamics independent of visual cues.

Main Methods:

• A neural network was proposed to correlate person detections from video with IMU orientations.
• A graph labeling problem was formulated to achieve globally consistent tracking solutions integrating both video and inertial data.
• A new dataset with synchronized video and IMU recordings, including ground-truth annotations, was created and released.

Main Results:

• The proposed fusion method demonstrated robustness against appearance variations and visual occlusions.
• The system achieved an average IDF1 score of 91.2% on the newly introduced challenging dataset.
• Reconstruction of intermediate positions during occlusions was significantly more stable using fused sensor data.

Conclusions:

• Fusing video and IMU data offers a more reliable approach to multi-person tracking compared to video-only methods.
• The method's reliance on motion signals makes it applicable in diverse scenarios where individuals can be equipped with inertial sensors.
• The release of the new dataset will facilitate further research in appearance-invariant, multi-modal person tracking.