Testing the Implementation and Acceptance of Generative Artificial Intelligence to Augment Vascular Surgery Journal...

Medical applications of generative artificial intelligence (GAI) have received much attention, however, it remains unclear whether GAI can be a valuable tool to enhance medical education. This study aimed to evaluate the effectiveness of GAI in vascular surgery journal club and to assess the attitudes of participants at various stages of medical training.

We conducted a series of vascular surgery journal clubs augmented by a commercially available large language model (LLM), ChatGPT 4o. Trainees presented preselected manuscripts, followed by summaries from the LLM. We posed structured and unstructured questions to trainees and ChatGPT and gathered real-time survey data to evaluate LLM accuracy and participant attitudes. Survey responses were analyzed using Fisher's exact test for proportions and analysis of variance for means. We also assessed the validity of 5 alternative citations generated by the LLM for each manuscript (N = 9) and checked for potential fabrications and research relevancy to subject matter content.

rural, academic medical center.

Participants included medical students (N = 10), surgical residents and fellows (N = 14), and vascular surgery attendings (N = 4).

Among 90 survey responses from 28 respondents on multiple peer-reviewed articles, 96% agreed that LLM provided a factually correct summary of each article, though 36% believed some information was omitted. 76% thought the LLM provided a thorough and factually correct explanation to questions asked about the article. Overall, 92% found the integration of LLM beneficial in summarizing and answering questions about the article, provided it aims to supplement rather than substitute for trainee presentations. The accuracy of the summary of an article by LLM, its responses to article-related questions, and the usefulness of LLM to foster learning were significantly associated with learner type (p = 0.02, p < 0.001, p = 0.04), respectively. Medical students and vascular surgery attendings specifically indicated additional value in introducing GAI into a traditional vascular surgery journal club format. Among 45 peer-reviewed journal article citations for the 9 tested manuscripts, 87% were scholarly verified citations, of which 97% contained research content relevant to the tested manuscript. The remaining 6 citations (13%) contained errors including nonexistent journal article titles and mismatch between journal article title and associated hyperlink. Technical issues with the LLM were uncommon.

Overall, participants found GAI to be a valuable resource for summarizing key research findings in peer-reviewed journal articles, answering pertinent questions using supplemental resources, and promoting educational growth for all types of learners. We recommend training programs consider adoption of GAI for vascular surgery journal club.