Optimising clinical information extraction: a comparative study of retrieval-augmented generation techniques in...

Area of Science:

• Natural Language Processing
• Clinical Informatics
• Artificial Intelligence

Background:

• Extracting clinical information from unstructured aged care notes is challenging due to text heterogeneity.
• Retrieval-augmented generation (RAG) can enhance large language model (LLM) outputs, but retrieval strategies need study for clinical information extraction (IE).

Purpose of the Study:

• To systematically compare six retrieval methods within a RAG pipeline for clinical IE.
• To evaluate retrieval strategies on named entity recognition tasks in Australian aged care notes.

Main Methods:

• Compared sparse retrieval (BM25), dense retrieval, dense reranking, dynamic linear fusion, reciprocal rank fusion (RRF), and hybrid reranking.
• Evaluated strategies on extracting dementia agitation symptoms and malnutrition risk factors using metrics like context relevance, answer quality, and item-level accuracy.

Main Results:

• Reranking and hybrid ensemble strategies significantly outperformed standalone sparse and dense retrieval.
• Dense reranking achieved top scores for agitation extraction (Answer F1: 0.946, Item-level Accuracy: 0.963).
• Ensemble methods led malnutrition risk factor identification (Item-level Accuracy: 0.944), with dense reranking close behind.

Conclusions:

• Reranking-based retrieval substantially enhances RAG performance for clinical IE.
• This study offers a practical approach for automated analysis of unstructured clinical text.
• The proposed four-stage workflow provides a replicable framework for future clinical IE research.