Open-source offline-deployable retrieval-augmented large language model for assisting pancreatic cancer staging.

Area of Science:

• Radiology and Artificial Intelligence
• Medical Informatics
• Oncology

Background:

• Large language models (LLMs) show promise in radiology but face challenges like data leakage and limited transparency.
• Developing secure and reliable AI tools for medical applications is crucial.

Purpose of the Study:

• To develop an open-source, offline-deployable retrieval-augmented LLM (RA-LLM) system for pancreatic cancer staging.
• To address data leakage, improve accuracy, and enhance reasoning transparency using reliable external knowledge (REK).

Main Methods:

• Utilized local LLMs (Llama-3.2 11B, Gemma-3 27B) and a cloud-based comparator (GPT-4o mini).
• Employed the Japanese pancreatic cancer guideline as REK for retrieval-augmented generation (RAG).
• Evaluated system performance on 100 simulated pancreatic cancer cases, comparing RAG vs. non-RAG LLMs.

Main Results:

• RAG significantly improved TNM staging accuracy across all tested LLMs (e.g., GPT-4o mini 61% to 90%, p < 0.001).
• Gemma-3 27B demonstrated performance comparable to GPT-4o mini, with high retrieval metrics and comparable execution times.
• Resectability classification also showed improvement with RAG, particularly for GPT-4o mini and Llama-3.2 11B.

Conclusions:

• An offline-deployable RA-LLM system for pancreatic cancer staging was successfully developed and open-sourced.
• RA-LLMs offer superior performance over baseline LLMs, enhancing accuracy and transparency in radiological applications.
• The offline-capable Gemma-3 27B provides a viable alternative to cloud-based LLMs, ensuring data security and comparable efficacy.