Vehicle Detection and Tracking Using Thermal Cameras in Adverse Visibility Conditions

Area of Science:

• Autonomous Systems
• Sensor Fusion
• Computer Vision

Background:

• Level 5 autonomy requires vehicles to operate in all conditions, posing significant sensing challenges.
• Existing sensors like cameras, lidar, and radar have limitations in adverse weather (fog, snow, rain, smoke).
• No single sensor reliably perceives the environment under all visibility conditions.

Purpose of the Study:

• To demonstrate the benefit of Long Wave Infrared (LWIR)/thermal cameras in self-driving vehicle sensor stacks.
• To address the sensory gap in adverse visibility conditions for autonomous driving.
• To improve vehicle detection and tracking performance in challenging environments.

Main Methods:

• Trained a machine learning-based image detector on LWIR thermal image data for vehicle detection.
• Explored Joint Probabilistic Data Association and Multiple Hypothesis Tracking for vehicle tracking.
• Fused thermal camera data with front-facing radar information.
• Implemented algorithms on a 2017 Lincoln MKZ using FLIR thermal cameras.
• Validated tracking algorithm performance through Unreal Engine simulations.

Main Results:

• LWIR thermal cameras enhance perception capabilities during adverse visibility.
• Fusion of thermal data with radar improves vehicle tracking accuracy.
• Machine learning detection on thermal imagery is effective for identifying vehicles.
• Simulations confirmed the robustness of the tracking algorithms.

Conclusions:

• LWIR thermal cameras are a valuable addition to autonomous vehicle sensor suites.
• Sensor fusion, particularly with thermal imaging, is crucial for achieving robust Level 5 autonomy.
• The study provides a viable approach for improving self-driving car performance in challenging weather and visibility scenarios.