Semantic Relations in Compound Nouns: Perspectives from Inter-Annotator Agreement

Area of Science:

• Biomedical Natural Language Processing
• Computational Linguistics
• Ontology Engineering

Background:

• Defining a consensus set of semantic relations has been a long-standing challenge.
• Biomedical language processing and ontologies necessitate robust methods for evaluating semantic relations.

Purpose of the Study:

• To investigate the role of inter-annotator agreement in selecting appropriate sets of semantic relations.
• To evaluate the suitability of competing semantic relation sets for biomedical text.

Main Methods:

• Two judges annotated a biomedical dataset of noun-noun compounds.
• Semantic relations between compound elements were labeled using two different sets of relations.
• Inter-annotator agreement was analyzed for each set of relations.

Main Results:

• High inter-annotator agreement was observed despite minimal training and documentation.
• Disagreements between the judges were consistent across the annotation task.
• The results indicate the effectiveness of the relation-based approach.

Conclusions:

• Inter-annotator agreement is a valuable metric for choosing between semantic relation sets.
• The relation-based approach demonstrates utility in semantic representation for biomedical text.
• Further research can refine semantic relation identification in biomedical NLP.