Towards a framework for developing semantic relatedness reference standards.

Area of Science:

• Medical Informatics
• Computational Linguistics
• Biomedical Natural Language Processing

Background:

• Current research on computerized semantic relatedness lacks standardized, reproducible reference datasets.
• Existing reference standards are often purpose-specific, not publicly available, and lack sufficient methodological detail.
• This hinders the evaluation and comparison of computational approaches for biomedical concept relatedness.

Purpose of the Study:

• To develop a robust framework for creating reference standards for functional testing of computerized semantic relatedness measures.
• To address practical and theoretical challenges in generating reliable reference standards for biomedical terminologies.
• To facilitate reproducible research and comparative analysis in the field of semantic relatedness.

Main Methods:

• Developed a framework integrating experiences from medical informatics and computational linguistics.
• Applied the framework to create a pilot reference standard for 101 medical term pairs rated by 13 experts.
• Utilized clustering and factor analyses to examine inter-rater variability and identify potential outliers.

Main Results:

• The pilot reference standard demonstrated moderate reliability.
• Data-driven analyses successfully identified systematic differences among raters and outlier groups.
• Two ontology-based relatedness measures were evaluated using the developed standard.

Conclusions:

• The proposed framework provides a structured approach to developing reference standards for semantic relatedness.
• The pilot study demonstrates the framework's utility in analyzing rater performance and standardizing evaluations.
• The framework and associated resources (reference standard, R program) are open-sourced to promote reproducibility and future research in medical informatics.