Application of NotebookLM, a large language model with retrieval-augmented generation, for lung cancer staging

Area of Science:

• Artificial Intelligence in Radiology
• Medical Imaging Analysis
• Clinical Decision Support Systems

Background:

• Large language models (LLMs) are gaining traction in radiology but face challenges with reliability and hallucinations.
• Retrieval-augmented generation (RAG) offers a solution by enabling LLMs to reference reliable external knowledge (REK).

Purpose of the Study:

• To evaluate the utility and reliability of NotebookLM, a RAG-equipped LLM, for lung cancer staging.
• To compare the performance of NotebookLM against GPT-4 Omni (GPT-4o) in lung cancer staging.

Main Methods:

• Summarized Japanese lung cancer staging guidelines as REK for NotebookLM.
• Tasked NotebookLM with staging 100 fictional lung cancer cases using CT findings.
• Compared NotebookLM's accuracy with GPT-4o (with and without REK provided directly in prompt).

Main Results:

• NotebookLM achieved 86% diagnostic accuracy in lung cancer staging.
• GPT-4o achieved 39% accuracy with REK and 25% without.
• NotebookLM demonstrated 95% accuracy in locating references within the REK.

Conclusions:

• NotebookLM, a RAG-LLM, demonstrated superior performance in lung cancer staging compared to GPT-4o.
• Accurate reference location retrieval by NotebookLM aids in evaluating response reliability and detecting hallucinations.
• RAG-LLMs like NotebookLM show significant potential for applications in medical image diagnosis.