Evaluating semantic similarity methods for comparison of text-derived phenotype profiles

Area of Science:

• Biomedical informatics
• Natural Language Processing
• Clinical Data Analysis

Background:

• Semantic similarity is crucial for biomedical analysis, enabling patient-like me studies, automated coding, differential diagnosis, and outcome prediction.
• Existing research on semantic similarity primarily focuses on tasks like protein interaction and rare disease diagnosis, with less exploration in comparing patient phenotype profiles for clinical applications.
• There is a lack of experimental studies investigating optimal parameters and superior methods for semantic similarity in clinical phenotype profile comparisons.

Purpose of the Study:

• To develop a platform for reproducible benchmarking and comparison of experimental conditions for patient phenotype similarity.
• To evaluate the task of ranking shared primary diagnoses from uncurated phenotype profiles using clinical text data.

Main Methods:

• Developed a platform for benchmarking patient phenotype similarity.
• Evaluated 300 semantic similarity configurations and one embedding-based approach.
• Utilized phenotype profiles derived from all text narratives in the Medical Information Mart for Intensive Care (MIMIC-III) database.

Main Results:

• Evaluated 300 semantic similarity configurations and one embedding-based approach.
• Measures not using external information content performed slightly better on average.
• The best-performing configurations for diagnosis classification utilized term-specificity and annotation-frequency measures, achieving high area under the receiver operating characteristic curve and Top Ten Accuracy.

Conclusions:

• Identified and interpreted the performance of numerous semantic similarity configurations for classifying diagnoses from text-derived phenotype profiles.
• The study provides a foundation for future research in diverse clinical settings and related tasks.
• Term-specificity and annotation-frequency measures are highlighted as effective for clinical phenotype profile analysis.