Retrieval Augmented Thought Process for Private Data Handling in Healthcare

Area of Science:

• Artificial Intelligence
• Medical Informatics
• Natural Language Processing

Background:

• Large Language Models (LLMs) offer significant potential in healthcare but face limitations.
• Privacy concerns restrict the use of sensitive patient data in LLM training.
• Current LLM integration with information retrieval is not robust to imperfect data access.

Purpose of the Study:

• To introduce a novel method, Retrieval-Augmented Thought Process (RATP), to enhance LLM performance in healthcare.
• To address the challenges of privacy and imperfect information retrieval in LLM applications.
• To improve the accuracy and reliability of LLMs using external, up-to-date knowledge.

Main Methods:

• RATP formulates LLM thought generation as a multi-step decision process using external knowledge.
• Monte-Carlo Tree Search is employed to optimize the thought process.
• A proxy reward function is learned for cost-efficient inference.

Main Results:

• RATP demonstrated a 35% increase in accuracy for question-answering tasks.
• Performance was evaluated on a private dataset of electronic medical records excluded from LLM training.
• RATP outperformed in-context retrieval-augmented generation methods.

Conclusions:

• RATP offers a robust solution for integrating LLMs with external knowledge in healthcare.
• The method enhances LLM accuracy and reliability, particularly with sensitive, private data.
• RATP represents a significant advancement in applying LLMs to complex medical question-answering scenarios.