Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Concept Videos

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

Autoregressive and Residual Index Convolution Model for Point Cloud Geometry Compression.

Sensors (Basel, Switzerland)·2026
Same author

Optimizing Automated Optical Inspection: An Adaptive Fusion and Semi-Supervised Self-Learning Approach for Elevated Accuracy and Efficiency in Scenarios with Scarce Labeled Data.

Sensors (Basel, Switzerland)·2024
Same author

Object Detection, Recognition, and Tracking Algorithms for ADASs-A Study on Recent Trends.

Sensors (Basel, Switzerland)·2024
Same author

360° Map Establishment and Real-Time Simultaneous Localization and Mapping Based on Equirectangular Projection for Autonomous Driving Vehicles.

Sensors (Basel, Switzerland)·2023
Same author

ConcentrateNet: Multi-Scale Object Detection Model for Advanced Driving Assistance System Using Real-Time Distant Region Locating Technique.

Sensors (Basel, Switzerland)·2022
Same author

DoA Estimation for FMCW Radar by 3D-CNN.

Sensors (Basel, Switzerland)·2021

Related Experiment Video

Updated: Aug 7, 2025

Tracking Infiltration Front Depth Using Time-lapse Multi-offset Gathers Collected with Array Antenna Ground Penetrating Radar
07:14

Tracking Infiltration Front Depth Using Time-lapse Multi-offset Gathers Collected with Array Antenna Ground Penetrating Radar

Published on: May 1, 2018

7.8K

Deep Learning Derived Object Detection and Tracking Technology Based on Sensor Fusion of Millimeter-Wave Radar/Video

Jia-Jheng Lin1, Jiun-In Guo1,2,3, Vinay Malligere Shivanna1

  • 1Institute of Electronics, Nation Yang Ming Chiao Tung University, Hsinchu 30010, Taiwan.

Sensors (Basel, Switzerland)
|March 11, 2023
PubMed
Summary

This study introduces a deep learning method fusing mmWave radar and RGB cameras for advanced driver-assistance systems (ADAS). This sensor fusion enhances object detection and tracking in adverse weather, improving road safety.

Keywords:
deep learningdepth sensorearly fusionmillimeter-wave radarobject detection and trackingsensor fusion

More Related Videos

Harmonic Radar Tags for Insect Tracking: Lightweight, Low-cost, and Accessible
14:44

Harmonic Radar Tags for Insect Tracking: Lightweight, Low-cost, and Accessible

Published on: May 13, 2025

829
Microfluidic Platform with Multiplexed Electronic Detection for Spatial Tracking of Particles
11:54

Microfluidic Platform with Multiplexed Electronic Detection for Spatial Tracking of Particles

Published on: March 13, 2017

9.4K

Related Experiment Videos

Last Updated: Aug 7, 2025

Tracking Infiltration Front Depth Using Time-lapse Multi-offset Gathers Collected with Array Antenna Ground Penetrating Radar
07:14

Tracking Infiltration Front Depth Using Time-lapse Multi-offset Gathers Collected with Array Antenna Ground Penetrating Radar

Published on: May 1, 2018

7.8K
Harmonic Radar Tags for Insect Tracking: Lightweight, Low-cost, and Accessible
14:44

Harmonic Radar Tags for Insect Tracking: Lightweight, Low-cost, and Accessible

Published on: May 13, 2025

829
Microfluidic Platform with Multiplexed Electronic Detection for Spatial Tracking of Particles
11:54

Microfluidic Platform with Multiplexed Electronic Detection for Spatial Tracking of Particles

Published on: March 13, 2017

9.4K

Area of Science:

  • Computer Vision
  • Sensor Fusion
  • Artificial Intelligence

Background:

  • Object detection and tracking are critical for Advanced Driver-Assistance Systems (ADAS).
  • RGB cameras struggle with performance in adverse weather and lighting conditions.
  • mmWave radar offers robust performance irrespective of weather and lighting.

Purpose of the Study:

  • To propose a deep learning-based early sensor fusion method combining mmWave radar and RGB camera data.
  • To realize an embedded system for object detection and tracking applicable to ADAS and intelligent transportation systems.
  • To enhance the reliability and performance of object detection and tracking systems in diverse environmental conditions.

Main Methods:

  • An early fusion deep neural network architecture is developed to integrate mmWave radar and RGB camera features.
  • The system is trained end-to-end to directly output detection and tracking results.
  • The method is implemented on embedded systems, including NVIDIA Jetson Xavier.

Main Results:

  • The proposed early fusion method significantly improves object detection and tracking performance, especially in adverse weather and lighting.
  • The system demonstrates efficient operation, achieving 17.39 frames per second on an embedded platform.
  • The fused sensor approach compensates for the limitations of RGB cameras in challenging conditions.

Conclusions:

  • The deep learning-based early fusion of mmWave radar and RGB cameras provides a robust solution for object detection and tracking in ADAS.
  • The developed system is suitable for real-time applications in both vehicles and smart Road Side Units (RSUs) for traffic monitoring.
  • This approach enhances safety and efficiency in intelligent transportation systems by ensuring reliable performance across various environmental conditions.