Understanding contraceptive switching rationales from real world clinical notes using large language models.

Area of Science:

• Natural Language Processing
• Health Informatics
• Pharmacovigilance

Background:

• Extracting reasons for treatment switching from unstructured clinical notes is challenging.
• Understanding these reasons is crucial for medical research and patient care.

Purpose of the Study:

• To evaluate the zero-shot capabilities of GPT-4 and open-source large language models (LLMs) in extracting contraceptive switching information.
• To identify key factors influencing contraceptive switching using advanced AI techniques.

Main Methods:

• Zero-shot evaluation of GPT-4 and eight open-source LLMs on 1964 clinical notes.
• Utilizing transformer-based topic modeling to identify reasons for switching.
• Clinical expert evaluation of extracted information for accuracy and hallucination rates.

Main Results:

• GPT-4 achieved high microF1 scores (0.85 for started, 0.88 for stopped contraceptives) outperforming the best open-source model.
• GPT-4 demonstrated 91.4% accuracy in extracting reasons for switching with a low 2.2% hallucination rate.
• Key switching reasons identified include patient preference, adverse events, and insurance coverage.

Conclusions:

• LLMs, particularly GPT-4, are valuable tools for extracting complex treatment factors from clinical notes.
• These findings offer insights into real-world contraceptive switching behaviors.
• AI-driven analysis can enhance understanding of treatment adherence and patient-centered care.