Related Experiment Video
Updated: Jun 25, 2026
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
Decreasing Administrative Effort Related to Non-Approval of Image-guidED Procedures Using Large Language Models - The
Colin J McCarthy1, Vijay Ramalingam1, Yiftach Barash1
1Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts (C.J.M., V.R., Y.B., A.S.).
Rationale And Objectives:
To evaluate whether large language models (LLMs) can generate accurate, clinically valid, and usable letters to appeal insurance denials for radiology procedures.
Materials And Methods:
This pilot study generated insurance appeal letters for a simulated clinical scenario. Four LLMs (Claude 3.5, Nova Pro, Llama-3.1-70B, ChatGPT-4o) were used with zero-shot, few-shot, and retrieval-augmented generation (RAG) techniques. Four board-certified interventional radiologists, blinded to model and technique, scored letters for content (accuracy, personalization, references), grammar and structure (readability, tone, persuasiveness), and usability (estimated editing time, usefulness as a template). References were verified for accuracy, and outputs were carefully examined for hallucinations. Statistical analyses included ANOVA, Chi-square, and Fleiss' Kappa for interrater reliability.
Results:
Mean content and grammar scores were 3.9 ± 0.95 and 4.3 ± 0.9 (out of 5), with no significant differences by model or technique (p >.05). Reviewer agreement was poor (Fleiss' Kappa -0.18 for content, -0.085 for grammar). Hallucinations were flagged by reviewers in 16/48 assessments, significantly more often with the online model (ChatGPT-4o: 58% vs offline 25%; p =.03). Of 44 references, 80% from the offline models were fabricated compared with 29% from ChatGPT-4o (p <.001). Estimated editing time was less than 10 min in 71% of responses, and the reviewers felt the letters would be useful as templates in 73% of cases.
Conclusion:
LLM-generated appeal letters for insurance denials were generally well received, with high usability and adequate quality. However, fabricated references and hallucinations remain prevalent, necessitating careful human review before clinical use.
