Large Language Models to Help Appeal Denied Radiotherapy Services

Area of Science:

• Artificial Intelligence in Medicine
• Oncology
• Health Informatics

Background:

• Physician burnout and delayed patient care are significant issues.
• Appealing insurer denials for medical services is a time-consuming administrative task.
• Large language models (LLMs) show potential for automating administrative tasks in healthcare.

Purpose of the Study:

• To evaluate the performance of different large language models (LLMs) in generating appeal letters for denied radiotherapy services.
• To assess the utility of LLM-generated letters in expediting the insurance appeal process.

Main Methods:

• Evaluated GPT-3.5, GPT-4, and GPT-4 with internet search (GPT-4web).
• Developed a fine-tuned version of GPT-3.5 (GPT-3.5ft) using 53 physician-written appeal letters.
• Presented 20 simulated patient histories to LLMs and scored generated letters by blinded radiation oncologists on clarity, clinical detail, reasoning, citations, and submission readiness.

Main Results:

• GPT-4 and GPT-4web generated superior letters compared to GPT-3.5, with better clinical reasoning and readiness for submission.
• All evaluated LLMs produced linguistically clear summaries without confabulation and were deemed useful for expediting appeals.
• Fine-tuning GPT-3.5 (GPT-3.5ft) with a small dataset compromised performance across all evaluated domains.
• LLMs demonstrated weaknesses in supporting clinical assertions with relevant, cited literature.

Conclusions:

• Commercially available LLMs can draft effective appeal letters for denied radiotherapy services when prompted correctly.
• LLMs have the potential to reduce the administrative burden on healthcare providers.
• Task-specific fine-tuning with small datasets may negatively impact LLM performance.