Off-the-Shelf Large Language Models for Guiding Pharmacoepidemiological Study Design

Area of Science:

• Pharmacoepidemiology
• Artificial Intelligence in Healthcare
• Health Informatics

Background:

• Pharmacoepidemiological studies are crucial for understanding drug safety and effectiveness.
• Large language models (LLMs) offer potential tools for research design support.
• Evaluating LLM capabilities in specialized scientific fields is essential.

Purpose of the Study:

• To assess the utility of ChatGPT and Gemini in supporting pharmacoepidemiological study design.
• To evaluate the coherence, relevance, and coding accuracy of LLM-generated responses for study protocols.
• To identify strengths and limitations of LLMs in this domain.

Main Methods:

• Analysis of 48 pharmacoepidemiological study protocols (2018-2024).
• Human expert evaluation of LLM responses across seven key study design components.
• Assessment of coding accuracy using ATC, CPT, and ICD systems.

Main Results:

• Both LLMs demonstrated high coherence (>90%) in most study design components.
• ChatGPT excelled in 'Index date' and 'Study design' coherence.
• Gemini led in 'Study outcome' and 'Study exposure' coherence.
• Relevance varied, with lower agreement on covariates and follow-up.
• Coding accuracy showed low agreement, highest with ATC (50%).

Conclusions:

• LLMs show promise for supporting pharmacoepidemiological study design, particularly in coherence.
• Limitations in relevance and coding accuracy necessitate critical expert oversight.
• Further development is needed to enhance LLM utility in complex research design.