Implementation of Artificial Intelligence in Writing Letters of Recommendation

Area of Science:

• Medical Education
• Artificial Intelligence in Medicine
• Residency Admissions

Background:

• Letters of recommendation (LoRs) are crucial for residency applications but suffer from subjectivity and reliability concerns.
• Artificial intelligence (AI) presents a potential solution for standardizing LoR quality.
• This study evaluated the comparative quality of AI-generated versus traditional LoRs.

Purpose of the Study:

• To compare the quality of AI-generated letters of recommendation (LoRs) against traditional, human-written LoRs.
• To assess the impact of AI-generated LoRs on residency candidate evaluations.
• To determine if AI can enhance the standardization and reliability of LoRs.

Main Methods:

• Faculty reviewers assessed anonymized LoRs for residency candidates, unaware of AI generation.
• Candidate quality was standardized using pre-interview Thalamus scores.
• Independent sample t tests and Kendall's W tests were used for statistical analysis.

Main Results:

• AI-generated LoRs received significantly higher ratings than traditional LoRs (4.14 vs. 3.29, p < 0.0001).
• AI LoRs outperformed traditional LoRs for both lower-quality (4.17 vs. 2.85, p < 0.0001) and higher-quality candidates (4.13 vs. 3.63, p = 0.006).
• High interrater agreement was observed for all assessed LoRs (Kendall's W, p < 0.05).

Conclusions:

• AI-generated LoRs are rated as superior to traditional LoRs, especially for lower-quality candidates.
• AI holds promise as a tool to improve the quality and standardization of letters of recommendation in residency applications.
• Further investigation into the ethical implications and broader applications of AI in LoR generation is warranted.