Extracting TNFi switching reasons and trajectories from real-world data using large language models

Area of Science:

• Health Informatics
• Artificial Intelligence in Medicine
• Real-World Evidence Generation

Background:

• Tumor necrosis factor inhibitors (TNFi) are crucial for treating inflammatory conditions.
• Understanding TNFi switching patterns is vital for optimizing patient care.
• Manual chart review for treatment changes is time-consuming and resource-intensive.

Purpose of the Study:

• To evaluate the capability of large language models (LLMs) to automate the identification of TNFi treatment switching.
• To determine the reasons behind TNFi switches using LLM-based chart review.
• To compare the performance of advanced LLMs against open-source models and traditional methods.

Main Methods:

• An observational study utilized de-identified electronic health record (EHR) data from 2012-2023.
• GPT-4 was employed to extract TNFi treatment switches and reasons from clinical notes.
• Performance was benchmarked against eight open-source LLMs, structured EHR data, and expert annotations.

Main Results:

• GPT-4 demonstrated strong performance in identifying stopped drugs (micro-F1=0.75), started drugs (0.80), and reasons for switching (0.83).
• Open-source models like Starling-7B-beta and Llama-3-8B showed competitive results.
• The primary reasons for TNFi switching were lack of efficacy (56.9%), adverse events (13.5%), and insurance/cost (10.8%).

Conclusions:

• LLMs, including GPT-4 and deployable open-source models, can effectively extract complex treatment information from clinical notes.
• Automated chart review using LLMs facilitates scalable analysis of real-world data.
• This technology supports enhanced real-world evidence generation for treatment patterns.