DLI-IT: a deep learning approach to drug label identification through image and text embedding

Area of Science:

• Pharmaceutical Sciences
• Computer Science
• Artificial Intelligence

Background:

• Drug labels are crucial for tracking pharmaceuticals from production to consumers.
• Label images can identify illegal, unapproved, or dangerous drugs.
• Manual drug identification is time-consuming and costly, necessitating AI solutions.

Purpose of the Study:

• To develop an AI model for fast and accurate drug label identification.
• To leverage both image and text data for enhanced drug detection.
• To create a model that identifies suspicious pharmaceutical products.

Main Methods:

• Developed the Drug Label Identification through Image and Text embedding (DLI-IT) model.
• Utilized Connectionist Text Proposal Network (CTPN) for text-based image cropping.
• Employed Tesseract OCR for text recognition and universal sentence embedding for vectorization.

Main Results:

• Trained on 1749 opioid and 2365 non-opioid drug labels.
• Tested on 300 external opioid drug labels, achieving 88% precision.
• Outperformed previous methods by up to 35% in drug label identification.

Conclusions:

• The DLI-IT model effectively combines image and text embedding for drug identification.
• Deep learning framework enhances the accuracy of pharmaceutical analysis.
• This approach offers a competitive solution for advancing drug label identification.