A deep neural network model for classifying pharmacy practice publications into research domains

Area of Science:

• Bibliometrics
• Pharmacy Practice Research
• Artificial Intelligence in Academia

Background:

• Pharmacy practice research encompasses clinical, social, and administrative domains, but lacks standardized categorization.
• The Granada Statements highlight the need for consensus on defining research domains.
• Four key domains are proposed: clinical, education, social & administrative, and basic & translational.

Purpose of the Study:

• To develop a machine learning classifier for categorizing pharmacy practice faculty publications into four defined research domains.
• To evaluate the classifier's performance against state-of-the-art general-purpose large language models (gpLLMs) using a zero-shot approach.

Main Methods:

• A Bidirectional Encoder Representations from Transformers (BERT) model was finetuned using 1000 abstracts (2018-2021) from pharmacy practice faculty publications.
• The developed model, Pharmacy Practice Research Domain Classifier (PPRDC), was compared with 7 leading gpLLMs on 80 randomly selected abstracts (2023).
• Performance was assessed using F1, recall, precision, accuracy, and Cohen's kappa for reproducibility.

Main Results:

• The PPRDC achieved high 5-fold cross-validation metrics: 89.4% F1, 90.2% recall, 89.0% precision, and 95.5% accuracy.
• PPRDC demonstrated perfect reproducibility (Cohen's kappa = 1.0) and significantly outperformed all tested gpLLMs in zero-shot classification.
• Specific F1 scores for PPRDC were: Education (96.2%), Clinical (92.7%), Social (85.8%), and Translational (83.1%).

Conclusions:

• The PPRDC offers superior performance compared to gpLLMs for classifying pharmacy practice research abstracts.
• This tool advances bibliometric studies and supports the Granada Statements' goals by aiding authors and editors in publication-related decisions.
• PPRDC represents a significant step forward in the systematic analysis and categorization of academic research in pharmacy practice.