High accurate and explainable multi-pill detection framework with graph neural network-assisted multimodal data...

Area of Science:

• Computer Science
• Artificial Intelligence
• Medical Informatics

Background:

• Pill misuse is a global health crisis, causing one-third of worldwide deaths.
• Current pill identification methods often fail with visually similar pills or in real-world settings.
• Existing datasets lack diversity, featuring only single pills in controlled environments.

Purpose of the Study:

• To address the challenge of multi-pill detection and identification in unconstrained, real-world scenarios.
• To develop a robust AI framework capable of distinguishing hard-to-identify pills.
• To introduce a novel dataset of multi-pill images captured under realistic conditions.

Main Methods:

• Proposed a novel method for constructing heterogeneous a priori graphs, integrating co-occurrence, relative size, and visual semantic correlations.
• Developed a framework to combine a priori information with visual features for enhanced pill detection.
• Created and utilized a new multi-pill image dataset captured in unconstrained environments.

Main Results:

• The proposed framework demonstrated superior robustness, reliability, and explainability.
• Achieved significant improvements in COCO mAP: 9.4% over Faster R-CNN and 12.0% over YOLOv5.
• Outperformed all existing detection benchmarks across all evaluation metrics.

Conclusions:

• The AI-based pill identification solution offers a promising approach to reduce medication errors.
• The developed framework effectively tackles multi-pill detection in real-world settings.
• This research opens new avenues for patient safety through advanced AI.