Assessing Multimodal AI for Visual Information Extraction of Pharmacology

Area of Science:

• Pharmacology
• Computational Biology
• Artificial Intelligence

Background:

• Increasing use of herbal dietary supplements alongside prescription medications raises concerns about potential harmful interactions.
• Effective pharmacovigilance for natural products necessitates expert analysis of diverse evidence, including visual data from scientific literature.
• Previous research established the utility of a Natural Product Knowledge Graph (NP-KG) for natural product safety assessments.

Purpose of the Study:

• To evaluate the accuracy and resilience of multimodal AI models for computer-assisted data extraction from visual elements (tables and figures) in pharmacology literature.
• To assess the feasibility of scaling the NP-KG to encompass the thousands of natural products available by automating data extraction.

Main Methods:

• Evaluation of 3 open-weight and 7 closed-weight multimodal models.
• Performance assessment based on visual information extraction from select tables and images within pharmacology literature.
• Analysis of extraction accuracy using a modified relative error rate.

Main Results:

• The top-performing multimodal models achieved 90% accuracy in extracting tabular data.
• These models successfully extracted 45% of data from figures, with a modified relative error rate of 0.05.
• Image resolution and information density were identified as key limitations impacting extraction performance.

Conclusions:

• Multimodal AI models demonstrate significant potential for automating data extraction from visual content in pharmacology literature, crucial for natural product safety.
• Accurate data extraction from figures and tables is essential for scaling knowledge graphs and enhancing natural product pharmacovigilance.
• Further improvements in model performance are needed to overcome challenges related to image quality and data complexity.