Bosniak classification of renal cysts using large language models: a comparative study

Area of Science:

• Artificial Intelligence in Radiology
• Medical Imaging Analysis
• Natural Language Processing in Healthcare

Background:

• The Bosniak classification system aids in assessing renal cyst malignancy risk.
• Inter-observer variability in Bosniak classification presents a diagnostic challenge.
• Large language models (LLMs) offer potential for standardized classification of renal cysts using textual data.

Purpose of the Study:

• To evaluate the performance of five LLMs in classifying renal cysts based on synthetic CT report descriptions.
• To compare zero-shot and few-shot prompting strategies for LLM-based renal cyst classification.
• To assess the accuracy, sensitivity, and specificity of LLMs in differentiating Bosniak categories.

Main Methods:

• A synthetic dataset of 100 renal cyst cases (20 per Bosniak category) was created.
• Five LLMs (GPT-4, Gemini, Copilot, Perplexity, NotebookLM) were tested.
• Zero-shot, few-shot, and retrieval-augmented generation (RAG) prompting strategies were employed.

Main Results:

• GPT-4 achieved the highest accuracy (99% with few-shot learning).
• Few-shot prompting significantly improved LLM performance across models (p < 0.05).
• Bosniak IIF lesions proved challenging for all evaluated LLMs.

Conclusions:

• LLMs can accurately classify renal cysts when provided with structured textual descriptions.
• Few-shot learning substantially enhances LLM classification accuracy for renal cysts.
• Further research is needed to address challenges in classifying borderline Bosniak IIF lesions.