Tracking medical students' clinical experiences using natural language processing.

Area of Science:

• Medical Education
• Natural Language Processing
• Clinical Informatics

Background:

• Current methods for tracking graduate medical student clinical skills rely on manual or basic electronic entry.
• There is a need for more efficient and accurate methods to assess clinical competency.

Purpose of the Study:

• To evaluate automated methods for identifying core clinical problems within medical students' clinical notes.
• To assess the performance of natural language processing techniques in this task.

Main Methods:

• Utilized section header identification algorithms and the KnowledgeMap concept identifier to process 290 clinical notes.
• Located Unified Medical Language System (UMLS) concepts to identify 10 institution-defined core clinical problems.
• Analyzed the predictive relevance of concepts within different note sections.

Main Results:

• The best automated search strategies achieved an area under the receiver operator characteristic curve of 0.90-0.94 for classifying discussions of core clinical problems.
• Unified Medical Language System (UMLS) concept identification demonstrated high recall (0.91) and precision (0.92).
• Concepts from the chief complaint, history of present illness, and assessment and plan sections were strongest predictors of relevance.

Conclusions:

• Automated tracking of clinical experience provides detailed reports without increasing trainee workload.
• This approach shows promise for improving clinical skills assessment and has potential applications in clinical research and phenotype identification.