Benchmarking Large Language Models and Prompt Engineering Strategies in Microsatellite Instability Cancers:...

Area of Science:

• Artificial Intelligence in Oncology
• Clinical Decision Support Systems
• Biomedical Natural Language Processing

Background:

• General-purpose large language models (LLMs) have uncharacterized reliability for complex clinical tasks in specialized domains like microsatellite instability (MSI) cancers.
• The lack of a domain-specific benchmark for evaluating LLM capabilities in MSI oncology poses risks to patient safety.

Purpose of the Study:

• To develop and validate the Microsatellite Instability Cancer Benchmark (MSIC-Bench) for evaluating LLMs in MSI oncology.
• To systematically assess LLM performance across prompting strategies and identify areas for improvement.

Main Methods:

• Developed MSIC-Bench, a 511-question benchmark from clinical guidelines and curated knowledge.
• Evaluated three state-of-the-art LLMs (GPT-4o, Gemini 2.5 Pro, Claude Opus 4) using four prompting strategies (vanilla, chain-of-thought, reflection of thoughts, RAG).
• Assessed performance based on accuracy, safety, error composition, and token usage across multiple-choice and open-ended modalities.

Main Results:

• LLMs exhibited a 'scaffolding effect,' with accuracy decreasing in open-ended scenarios.
• Retrieval-augmented generation (RAG) was the most effective intervention, shifting bottlenecks from knowledge deficits to retrieval failures.
• RAG improved accuracy and safety by reducing fabrications, though it introduced a trade-off with false refusals. Hybrid-RAG showed robust performance.

Conclusions:

• Current LLMs lack specialized knowledge for MSI oncology; RAG is crucial for addressing this gap.
• Optimizing RAG requires focusing on retrieval precision and high-quality knowledge bases for trustworthy clinical AI.
• MSIC-Bench provides a framework to guide future development of clinical AI in MSI oncology.