Enhancing the Objective Structured Clinical Examination Using Artificial Intelligence

Background: Artificial intelligence (AI) offers promising solutions for nurse practitioner education, especially in addressing challenges related to evaluating Objective Structured Clinical Examinations (OSCEs), such as examiner bias and delayed feedback. AI tools employing natural language processing and generative AI have the potential to enhance the accuracy and efficiency of clinical assessments. Objective: This product evaluation was conducted to determine whether AI-generated OSCE assessments align with faculty evaluations. Methods: A descriptive correlational design was used to assess product acceptability, feasibility, and agreement between AI and faculty assessments. A convenience sample of 13 nurse practitioner students was randomly divided into either a traditional evaluation group or an AI-assisted group. The AI-generated transcripts were scored using the same rubric used by faculty, and agreement was measured with Spearman's correlation, Cohen's Kappa, and interrater reliability percent agreement (IRR%). Results: Spearman's correlations ranged from negligible to moderate, with the highest in the physical/mental health exams category (r = .54, p < .05). However, Cohen's Kappa (.14-.41) and IRR% (31%-54%) showed weak agreement. Conclusions: These results suggest that AI feedback was inconsistent with faculty assessments, possibly due to technical issues and limitations in the rubric. Despite these limitations, this product evaluation demonstrated that the AI tool was easy to use and that faculty believed it could improve feedback quality. Implications for Nursing: These findings underscore both the promise and the current limitations of AI-supported clinical assessment. With thoughtful attention to these shortcomings, the ease of integration and capacity for enhanced learning offered by AI tools can help advance clinical competence and foster excellence in nurse practitioner and doctoral nursing education.