Optimizing theranostics chatbots with context-augmented large language models.

Area of Science:

• Nuclear Medicine
• Theranostics
• Artificial Intelligence in Healthcare

Background:

• Nuclear medicine theranostics is an emerging interdisciplinary field requiring extensive knowledge.
• Healthcare professionals need efficient tools for in-depth research on complex therapies.
• Large Language Models (LLMs) show potential for educating professionals but face domain-specific performance limitations.

Purpose of the Study:

• To develop and evaluate a framework for improving medical theranostic chatbots using contextual augmentation.
• To create the first theranostic chatbot by enhancing LLM performance with relevant information.
• To assess the impact of Retrieval Augmented Generation (RAG) techniques on chatbot accuracy.

Main Methods:

• Framework development for contextual augmentation in LLMs.
• Evaluation of five state-of-the-art LLMs using translated English and German questions.
• Comparison of chatbot responses with and without contextual augmentation via Naïve RAG and Advanced RAG.
• User study and LLM-based evaluation of answer quality.

Main Results:

• Advanced RAG techniques substantially enhance LLM performance in theranostics.
• CLAUDE 3 OPUS and GPT-4O demonstrated superior performance, effectively integrating contextual information.
• Significant improvements were noted with Advanced RAG over Naïve RAG in GEMINI 1.5 and COMMAND R+ variants.

Conclusions:

• Contextual augmentation effectively addresses the complexities of nuclear medicine theranostics.
• Advanced RAG is crucial for improving the reliability of LLM-based theranostic tools.
• Future work should broaden question scope, explore more RAG methods, and refine evaluation metrics.