CLEAR: Comparative Letter Examination and Analysis for Red Flags

Area of Science:

• Medical Education
• Artificial Intelligence in Healthcare
• Natural Language Processing

Background:

• Letters of Recommendation (LORs) are crucial for fellowship selection.
• Screening LORs for "red flags" is time-consuming for program directors.
• Identifying struggling learners or professionalism concerns is key.

Purpose of the Study:

• To compare the speed and consistency of Microsoft Copilot against human reviewers in screening LORs.
• To evaluate AI's effectiveness in identifying red flags in fellowship applications.
• To assess the efficiency of NLP models in medical education.

Main Methods:

• Retrospective analysis of 195 de-identified LORs from a neonatal-perinatal medicine fellowship.
• Independent screening by two human reviewers and a rule-based NLP model (Microsoft Copilot).
• Comparison of time to completion and red flag detection accuracy.

Main Results:

• The NLP model screened LORs in 25 minutes versus 554 minutes for humans.
• The AI model achieved 76% agreement with human reviewers in detecting red flags.
• AI demonstrated consistent identification of key terms, unlike variable human performance.

Conclusions:

• A rule-based NLP model provides an efficient and consistent method for initial LOR screening.
• AI tools can streamline the fellowship application review process.
• This technology supports faster, more reliable identification of potential issues in LORs.