Improving TCM question answering through tree-organized self-reflective retrieval with LLMs.

Area of Science:

• Artificial Intelligence in Medicine
• Knowledge Representation
• Natural Language Processing

Background:

• Large language models (LLMs) show promise for healthcare question answering (Q&A).
• Traditional knowledge representation struggles with the complexity of Traditional Chinese Medicine (TCM).
• Existing retrieval-augmented generation (RAG) frameworks are not optimized for TCM's unique structure.

Purpose of the Study:

• To evaluate a novel Tree-Organized Self-Reflective Retrieval (TOSRR) framework.
• To enhance LLM performance on TCM Q&A tasks using innovative knowledge organization and self-correction.

Main Methods:

• Developed a hierarchical knowledge system structuring TCM as subject-predicate-object-text (SPO-T) units.
• Implemented an iterative self-reflection mechanism for dynamic knowledge retrieval and validation.
• Evaluated performance using TCM Medical Licensing Examination (MLE) and Classics Course Exam (CCE) questions.

Main Results:

• TOSRR integrated with GPT-4 improved TCM MLE accuracy by 19.85% and CCE recall from 27% to 38%.
• Expert evaluation showed an 18.64-point improvement in safety, consistency, explainability, compliance, and coherence.
• Retrieval-Augmented Generation Assessment (RAGAs) metrics confirmed superior knowledge utilization and precision.

Conclusions:

• The TOSRR framework effectively enhances LLM performance in TCM knowledge tasks.
• Hierarchical knowledge representation and self-reflective retrieval are key to its success.
• The framework shows potential for application in TCM education and teaching.