Hierarchical context enhancement for long-tail entity retrieval augmented generation.

Area of Science:

• Natural Language Processing
• Information Retrieval
• Artificial Intelligence

Background:

• Domain-specific question answering (DSQA) using Retrieval-Augmented Generation (RAG) suffers from performance degradation.
• Semantic drift, particularly with low-frequency terms, is a primary cause of this degradation.

Purpose of the Study:

• To propose a novel method, Hierarchical Context Enhancement Retrieval-Augmented Generation (HCE-RAG), to address the challenge of low-frequency terms in DSQA.
• To enhance the accuracy and robustness of RAG systems in handling domain-specific queries with rare entities.

Main Methods:

• Offline entity-sensitive contextual tagging during indexing to anchor low-frequency entities.
• Constrained query reflection for entity-focused query clarification during query processing.
• Hybrid retrieval with Random Reciprocal Rank Fusion (RRF) to balance context and exact matching for robust entity identification.

Main Results:

• Achieved a 29 percentage point gain in Recall@10 on low-frequency entities in experiments.
• Demonstrated strong performance on a dedicated domain-specific QA benchmark.

Conclusions:

• The proposed HCE-RAG method effectively handles low-frequency terms, significantly improving DSQA performance.
• HCE-RAG is a plug-and-play solution that can enhance existing state-of-the-art RAG algorithms for long-tail entity question answering.