Machine Learning in Psychiatric Health Records: A Gold Standard Approach to Trauma Annotation.

Area of Science:

• Computational Linguistics
• Psychiatric Healthcare
• Machine Learning

Background:

• Psychiatric electronic health records (EHRs) are complex and unstructured, posing challenges for machine learning (ML) applications.
• Accurate identification of clinical information, such as trauma, is crucial for understanding disease heterogeneity and treatment in psychiatric disorders.

Purpose of the Study:

• To develop a gold-standard, publicly available dataset of annotated psychiatric EHRs.
• To establish clinically-informed guidelines for annotating traumatic events.
• To demonstrate the dataset's utility for training ML models to detect symptoms, substance use, and trauma.

Main Methods:

• Compiled a corpus of 200 narrative-heavy psychiatric EHRs.
• Developed a detailed annotation scheme with clinical experts and computational linguists.
• Annotated the corpus for trauma-related events and clinical information, achieving high inter-annotator agreement (0.715 for spans, 0.874 for attributes).

Main Results:

• Created the first gold-standard dataset for labeling traumatic features in psychiatric EHRs.
• Achieved high inter-annotator agreement, indicating reliable annotation.
• Developed ML models with micro F1 scores of 0.76 (spans) and 0.82 (attributes), demonstrating predictive reliability.

Conclusions:

• The established gold-standard dataset is suitable for training ML models in psychiatric healthcare.
• The dataset facilitates the detection of symptoms, substance use, and trauma in EHRs.
• This resource will advance ML applications for understanding psychiatric disease heterogeneity and treatment implications.