Radiology Report Annotation Using Generative Large Language Models: Comparative Analysis.

Area of Science:

• Artificial Intelligence in Medicine
• Natural Language Processing for Healthcare
• Radiology Informatics

Background:

• Large language models (LLMs) demonstrate increasing capabilities applicable to medical tasks.
• The potential of LLMs, specifically GPT-3.5 and GPT-4, for medical documentation requires thorough evaluation.
• Automating the generation of radiology report impressions can alleviate healthcare professional workload.

Purpose of the Study:

• To comparatively analyze the performance of GPT-3.5 and GPT-4 in annotating radiology reports and generating impressions from chest CT scans.
• To assess the effectiveness of in-context learning (ICL) and retrieval-augmented generation (RAG) for impression generation.
• To investigate the impact of prompt design on LLM performance in medical summarization.

Main Methods:

• Comparative analysis of GPT-3.5 and GPT-4 using zero-shot and few-shot learning scenarios.
• Application of in-context learning (ICL) and retrieval-augmented generation (RAG) techniques.
• Evaluation using ROUGE, Instructor Similarity, and BERTScore metrics to assess n-gram, contextual, and semantic similarity.

Main Results:

• GPT-4 demonstrated distinct performance differences compared to GPT-3.5 across learning scenarios.
• Retrieval-augmented generation (RAG) achieved a superior BERTScore of 0.92, indicating high semantic accuracy.
• Both GPT-3.5 and GPT-4 maintained high language tone fidelity (Instructor Similarity ~0.92), highlighting prompt influence.

Conclusions:

• Prompt design is a critical factor in optimizing LLM performance for generating accurate medical impressions.
• Retrieval-augmented generation shows significant potential for generating semantically rich and contextually relevant radiology report impressions.
• Standardized integration of advanced LLMs like GPT-4 in healthcare practices could enhance documentation efficiency and accuracy.