Building and evaluating annotated corpora for medical NLP systems.

Area of Science:

• Clinical informatics
• Natural language processing in healthcare
• Medical data annotation

Background:

• Accurate annotation of clinical notes is crucial for developing reliable clinical decision support systems.
• Existing annotation schemas may require refinement for specific clinical tasks.
• Pilot studies are essential for validating annotation methodologies before large-scale implementation.

Purpose of the Study:

• To assess the feasibility and reliability of a pilot annotation project for disorder mentions in clinical notes.
• To evaluate inter-annotator agreement among retrieval experts using a defined annotation schema.
• To determine the potential for scaling the annotation process based on initial results.

Main Methods:

• Sixty clinical notes were selected for the pilot annotation.
• Four retrieval experts independently annotated the same set of clinical notes.
• Annotation focused on identifying and marking disorder mentions.
• Inter-annotator agreement was calculated using a four-way agreement metric.

Main Results:

• The pilot project successfully annotated sixty clinical notes for disorder mentions.
• Four retrieval experts achieved encouraging levels of inter-annotator agreement.
• The results indicate consistency in applying the annotation guidelines and schema.

Conclusions:

• The pilot annotation project demonstrates a promising approach for extracting disorder mentions from clinical text.
• Encouraging inter-annotator agreement supports the scalability of this annotation methodology.
• The established guidelines and schema are suitable for larger-scale clinical note annotation projects.