Test collections for electronic health record-based clinical information retrieval

Area of Science:

• Medical Informatics
• Information Retrieval
• Clinical Research

Background:

• Clinical information retrieval (IR) systems are crucial for accessing patient data within electronic health records (EHRs).
• Developing robust evaluation methods for these systems is essential for advancing clinical IR research.
• Existing methods may not fully capture the nuances of clinical narrative data.

Purpose of the Study:

• To develop and evaluate test collections for assessing clinical IR systems.
• To advance the field of clinical IR research through systematic evaluation.
• To explore the performance of different retrieval models on clinical data.

Main Methods:

• Utilized a large dataset of 45,000 patients' EHR data from Mayo Clinic Biobank.
• Indexed 42 million free-text EHR documents using a clinical IR system.
• Developed 56 clinical topics and created test collections with human assessment guidelines, evaluating five retrieval models.

Main Results:

• Achieved moderate inter-judge agreement (Kappa=0.49) in relevance judgments.
• Identified that conventional retrieval models performed best overall, while concept-based models excelled in conceptual retrieval tasks.
• Demonstrated the feasibility of creating test collections, noting challenges in relevance judgment and model performance.

Conclusions:

• Clinical IR offers a valuable approach to patient information discovery from clinical narratives, with reduced semantic dependency.
• The developed test collections show promise for evaluating clinical IR systems, though further investigation into challenges is needed.
• This work provides a foundation for more rigorous evaluation of clinical IR tools.