RiTeK: A Dataset for Large Language Models Complex Reasoning over Textual Knowledge Graphs in Medicine.

Area of Science:

• Medical Informatics
• Artificial Intelligence in Medicine
• Knowledge Representation

Background:

• Answering complex medical questions requires accurate retrieval from medical Textual Knowledge Graphs (TKGs).
• Large Language Models (LLMs) can benefit from relational path information in TKGs for enhanced inference.
• Existing medical TKGs are scarce, possess limited expressiveness, and lack comprehensive evaluation benchmarks for retrieval systems.

Purpose of the Study:

• To address the limitations in medical TKG retrieval by developing a novel dataset and benchmark.
• To create a dataset (RiTeK) for evaluating LLMs' complex reasoning capabilities over medical TKGs.
• To assess the performance of current LLM-driven retrieval systems on medical TKGs.

Main Methods:

• Developed the Dataset for LLMs Complex Reasoning over medical Textual Knowledge Graphs (RiTeK).
• Synthesized realistic user queries integrating diverse topological structures, relational information, and complex textual descriptions.
• Conducted rigorous medical expert evaluation for query validation and assessed 11 representative LLM-driven retrievers.

Main Results:

• Existing LLM-driven retrieval methods demonstrate significant limitations when evaluated on the RiTeK benchmark.
• Current retrievers struggle to effectively handle the complexity and topological structures present in medical TKGs.
• The study reveals a notable gap in the performance of current retrieval systems for semi-structured medical data.

Conclusions:

• The RiTeK dataset serves as a comprehensive benchmark for evaluating LLM-based retrieval systems in the medical domain.
• There is a pressing need for the development of more effective retrieval systems tailored for semi-structured medical data.
• Findings underscore the challenges and limitations of current approaches in leveraging medical TKGs for complex question answering.