Building gold standard corpora for medical natural language processing tasks

Area of Science:

• Medical Natural Language Processing (NLP)
• Clinical Informatics
• Data Annotation

Background:

• High-quality annotated corpora are essential for developing and evaluating medical NLP tools.
• Existing resources may lack comprehensive coverage or standardized annotation schemas for diverse clinical text types.

Purpose of the Study:

• To construct three annotated corpora serving as gold standards for medical NLP.
• To facilitate de-identification and information extraction tasks using clinical notes, trial announcements, and drug labels.
• To enable translational NLP research through interoperable annotation schemas.

Main Methods:

• Annotation of clinical notes, clinical trial announcements, and FDA drug labels.
• Development of annotation schemas for Personal Health Information (PHI), medications, diseases/disorders, and signs/symptoms.
• Calculation of inter-annotator agreement using F-measures.

Main Results:

• Achieved high inter-annotator agreement (F-measures 0.8467–0.9176) for PHI de-identification.
• Demonstrated high agreement for information extraction tasks including medications, diseases, and symptoms.
• Public release of annotated clinical trial and FDA drug label corpora planned.

Conclusions:

• The constructed annotated corpora provide reliable gold standards for medical NLP.
• Aligned annotation schemas promote cross-corpora interoperability for translational research, such as clinical trial eligibility screening.
• These resources will advance the development of robust medical NLP applications.