Should my recommendation letter be written by artificial intelligence?

Area of Science:

• Medical Education
• Artificial Intelligence in Healthcare
• Surgical Residency Admissions

Background:

• Letters of recommendation are critical components of the residency application process.
• The increasing volume of applications necessitates efficient and effective evaluation methods.
• The potential role of artificial intelligence (AI) in academic writing support is under exploration.

Purpose of the Study:

• To evaluate the efficacy of an AI tool in generating letters of recommendation for orthopedic residency applicants.
• To compare AI-generated letters with human-written letters based on program director preferences.
• To assess the ability of program directors to distinguish between AI-generated and human-written letters.

Main Methods:

• Three academic staff members and an AI tool generated recommendation letters for 13 applicants.
• Blinded orthopedic program directors evaluated and selected preferred letters based on predefined criteria.
• Program directors' ability to identify AI-generated letters was recorded.

Main Results:

• Human-written letters were selected more frequently than AI-generated letters by all program directors (p < 0.05).
• AI-generated letters were preferred for 31-38% of applicants, depending on the reviewer.
• Program directors' accuracy in identifying AI-generated letters varied significantly (15% to 92%, p < 0.05).

Conclusions:

• While AI tools can assist in drafting recommendation letters, human-written versions remain preferred in orthopedic residency admissions.
• The detectability of AI-generated letters by experienced reviewers is variable.
• Further research is needed to optimize AI tools for academic writing support in medical education.