Medical students' appraisal of curated AI answers in a cardiovascular system course

The purpose of this study is to evaluate medical student perspective of AI-curated responses through qualitative feedback and assessment of accuracy and helpfulness.

First, students submitted questions about course content. Next, these questions were inputted into OpenEvidence. Finally, generated responses were edited by investigators and evaluated by students and faculty.

The primary outcome of this study was the accuracy, helpfulness, and qualitative feedback for AI-generated responses to medical student questions.

Overall, students found most AI responses to be helpful. Qualitative feedback revealed AI responses were often too detailed and lacked visual explanations. Students also noted difficulty in grading the accuracy of AI responses.

Although students generally found the AI-curated responses to be helpful, significant shortcomings exist. Students struggled to assess the accuracy of the responses, found responses to be too detailed, and believed responses lacked beneficial visual diagrams.