The Validity, Accuracy, and Comprehensiveness of Artificial Intelligence for Generating Resident Educational Material...

The current education of plastic surgery residents is hamstrung by reliance on outmoded teacher-learner methods including the incorporation of a passive system of lectures and talks into the formal conferences. The hypothesis was that artificial intelligence (AI), if trained, could generate satisfactory topical, specific subject, faculty-resident conference discussions or guides.

Three topics, namely, breast reduction, frontal sinus fracture, and flexor tendon injury, were selected, and an AI platform was trained with a vetted bibliography to generate discussion guides. A set of consensual cardinal teaching points were established by 3 faculty, and the AI-generated discussion guides were assessed by 6 residents with the criteria of completeness, specificity, and absence of hallucinations on a scale of 1 to 10. The same set of residents assessed faculty-generated discussion guides with the same criteria.

The interrater reliability was 0.85. The mean and standard deviations of the rating values for breast reduction, frontal sinus, and flexor tendon repair were 4.39 ± 1.91, 3.39 ± 2.55, and 2.56 ± 1.62, respectively. The resident assessment of faculty-generated discussion guides yielded means and standard deviations of 8.94 ± 0.73, 8.56 ± 1.2, and 8.94 ± 0.9, respectively. A statistical comparison of the 2 groups of means and standard deviations revealed a significant difference.

The platform generated pages of text and language that lacked content or, worse, was nonsensical. These results call into question the utility value of AI in the current development to contribute to the more sophisticated realm of plastic surgery resident education.