Related Experiment Videos
A lightweight hybrid deep learning framework for multi-pill detection, multi-attribute recognition, OCR-based imprint
1School of Computer Science Engineering and Information Systems, Vellore Institute of Technology, Vellore, Tamil Nadu, India.
Introduction:
Adverse drug events (ADEs) remain a major cause of preventable healthcare complications due to incorrect pill identification, dosage errors, and confusion between visually identical pills, particularly among older adults, visually impaired individuals, and people with limited health literacy. Recent advances in artificial intelligence and computer vision have enabled automated pill recognition systems. However, many existing methods address detection, classification, and imprint recognition as isolated tasks without providing a lightweight unified framework suitable for real-time healthcare deployment.
Methods:
This study presents a lightweight hybrid framework for multi-pill and multi-attribute recognition to support pharmacovigilance and artificial intelligence-driven clinical decision support. The proposed framework integrates pill detection using YOLOv26n, MobileNetV4-ConvSmall-based classification, EasyOCR-based imprint recognition, and metadata retrieval into a unified prediction pipeline. A customized benchmark dataset containing 400 pill classes, each with 50 images, for a total of 20,000 images, was used for classification. Each image has two pills, comprising 40,000 manually labeled pill instances used for detection. Comparative experiments, functional comparisons with existing pill recognition systems, and cross-device qualitative evaluations were conducted to assess framework performance, robustness, and deployment feasibility.
Results:
YOLOv26n achieved a precision of 0.963, a recall of 0.982, mAP@50 of 0.989, and mAP@50-95 of 0.981, while MobileNetV4-ConvSmall achieved a class accuracy of 99.60% with higher performance in pill shape and color accuracy. Comparative optical character recognition (OCR) analysis shows that EasyOCR achieved more reliable imprint recognition performance than TesseractOCR and TrOCR. Comparative framework evaluation demonstrated that the YOLOv26n with MobileNetV4-ConvSmall combination achieved the high overall recognition performance, obtaining a Top-1 accuracy of 89.83% and a Top-5 accuracy of 99.49%. Cross-device qualitative evaluation using smartphone cameras and direct laptop uploads demonstrated robust performance under varying image noise conditions.
Discussion:
The proposed framework demonstrates the potential of lightweight deep learning architectures for scalable medication identification and AI-assisted pharmacovigilance applications. By integrating detection, classification, imprint recognition, and metadata retrieval into a unified framework, the proposed system has the potential to support medication identification and improve assistive healthcare workflows. The final clinical judgment and medication verification should remain under the supervision of a healthcare professional.