CLEAR: Comparative Letter Examination and Analysis for Red Flags

Narrative letters of recommendation (LORs) remain a central element in fellowship selection. Programs screen these letters for language signifying a struggling learner or professionalism concerns, also known as "red flags," when determining interview offers. However, thorough screening is time consuming for program directors.

To compare the speed and consistency of Microsoft Copilot to human reviewers in screening LORs for red flags.

A retrospective analysis was conducted using de-identified LORs submitted during the 2024-2025 neonatal-perinatal medicine fellowship application cycle at a single fellowship site. Two reviewers independently screened each letter for predefined red flags. Disagreements were resolved by consensus or third-party adjudication. A rule-based natural language processing (NLP) model, refined through prompt adjustments, screened the same letters. Time to completion and red flag detection were compared.

A total of 195 LORs were reviewed. Following adjudication, red flags were confirmed in 21 letters. The NLP model flagged 16 letters and showed 76% (16 out of 21) agreement with the final adjudicated review. It processed all letters in 25 minutes, compared to the 554 minutes required by human reviewers. The model reliably identified terms "solid" and "good" with sentence-level context and showed consistency across the dataset, while humans varied more in detection, particularly with vague or indirect phrasing.

A rule-based NLP model offers an efficient and consistent method for initial LOR screening.