Perceptibility of AI-revised content in pharmacy residency letters of intent

Generative artificial intelligence (AI) may be impacting the creation of, and decisions regarding acceptance of, pharmacy residency applications. Currently, studies assessing the detectability of or potential advantages conferred by AI revisions on letters of intent (LOIs) are lacking. This study assesses the perceptibility and quality impact of AI on pharmacy residency LOIs.

Six LOIs originally submitted by applicants to a postgraduate year 1 residency program between 2015 and 2021 (prior to the widespread availability of consumer AI programs) served as baseline letters and were labeled as 0 (ie, without AI revisions). Investigators revised the 6 baseline letters with OpenAI's ChatGPT app; each letter received 2, 4, and 6 AI revisions, yielding a total of 24 LOIs for evaluation. Twelve blinded, paired pharmacists evaluated the 24 unique letters using a 5-category evaluation rubric for quality and responded if AI-generated content was suspected, resulting in a total of 48 responses. The primary outcome was the detectability of AI utilization to create or revise content within the LOIs, as measured by reviewers' dichotomous responses. The secondary outcome was the impact of AI revision, as measured by reviewers' numeric scores on the 5-category evaluation rubric.

Overall, the rate of correct determination of AI revision was 35%. Reviewers were less accurate in identifying authorship for letters with AI revision than for original letters. Letters with 2 and 4 AI revisions (OR for both, 0.61; P = 0.005) were equally difficult for reviewers to detect. Detectability lessened for letters with 6 AI revisions (OR, 0.55; P < 0.0001). Letters with 6 AI-generated revisions received higher rubric scores (a mean of 15.9 points out of 20 possible points) than original letters (mean, 10.0 points; P = 0.008) and those with 2 AI-generated revisions (mean, 10.2 points; P = 0.01).

The use of generative AI to revise pharmacy residency LOIs was not readily detected by reviewers and conferred a scoring advantage upon the sixth revision. These findings suggest the need for further research regarding the detection and influence of AI in LOIs for postgraduate training positions.