Assessing Multimodal AI for Visual Information Extraction of Pharmacology

While Americans are using herbal dietary supplements (natural products) more than ever, the consumption of natural products with prescription drugs can lead to harmful interactions. Pharmacovigilance of natural products depends on careful expert review and interpretation of a wide variety of evidence. In prior work, we demonstrated the value of knowledge graph (NP-KG) for assisting with natural product safety investigations. However, scaling the NP-KG from 33 natural products to the thousands on the market requires computer-assisted data extraction, particularly from visual elements (figures or tables) of pharmacology literature. We evaluated the accuracy and resilience of 8 open- and closed-source multimodal models by performing visual information extraction from select tables and images. The best performing models could accurately extract 90% of tabular data and 45% of data reported figures with a modified relative error rate of 0.05. Image resolution and information density were primary hindrances to better extraction performance.