Multimodal Large Language Model-Enabled Machine Intelligent Fault Diagnosis Method with Non-Contact Dynamic Vision...

Area of Science:

• Engineering
• Artificial Intelligence
• Sensor Technology

Background:

• Smart manufacturing requires high equipment reliability and availability.
• Traditional fault diagnosis methods using vibration sensors have limitations in adaptability, maintenance, and preprocessing.
• There is a need for advanced, non-contact fault diagnosis techniques.

Purpose of the Study:

• To pioneer the use of event camera data for bearing fault classification.
• To fine-tune a multimodal large model (Qwen2.5-VL-7B) using dynamic visual information for fault diagnosis.
• To establish a novel, end-to-end intelligent analysis paradigm for non-contact fault detection.

Main Methods:

• Utilized high-temporal-resolution dynamic visual information from an event camera.
• Processed sparse pulse events into event frames via time surface processing.
• Reconstructed event frames into high-temporal-resolution video using spatiotemporal denoising and ROI definition.
• Employed two LoRA fine-tuning strategies (Strategy A: OpenCV frame extraction; Strategy B: built-in video pipeline) with Qwen2.5-VL-7B for bearing fault classification.

Main Results:

• Achieved classification accuracies of 0.9247 (Strategy A) and 0.9540 (Strategy B).
• Demonstrated successful bearing fault classification under varying operating conditions and rotational speeds.
• Validated the effectiveness of non-contact sensing and end-to-end intelligent analysis for fault diagnosis.

Conclusions:

• Event camera-based dynamic visual information is effective for bearing fault diagnosis.
• The multimodal large model fine-tuned with event data offers a promising new paradigm for smart manufacturing.
• Non-contact sensing combined with advanced AI enables robust and efficient equipment health monitoring.