Enhancing the Objective Structured Clinical Examination Using Artificial Intelligence.

Area of Science:

• Nursing Education
• Artificial Intelligence in Healthcare
• Clinical Assessment Technologies

Background:

• Artificial intelligence (AI) presents potential solutions for nurse practitioner education, particularly in overcoming challenges in Objective Structured Clinical Examinations (OSCEs) like examiner bias and feedback delays.
• Natural language processing and generative AI tools can improve the accuracy and efficiency of clinical assessments in nursing education.
• AI tools offer a promising avenue to enhance the evaluation process for Objective Structured Clinical Examinations (OSCEs) in nurse practitioner programs.

Purpose of the Study:

• To evaluate an AI tool for its ability to generate OSCE assessments that align with faculty evaluations.
• To determine the product acceptability and feasibility of using AI for clinical assessments in nurse practitioner education.
• To assess the agreement between AI-generated and faculty-led assessments of Objective Structured Clinical Examinations (OSCEs).

Main Methods:

• A descriptive correlational design was employed to assess product acceptability, feasibility, and agreement.
• A convenience sample of 13 nurse practitioner students was divided into traditional and AI-assisted evaluation groups.
• AI-generated transcripts were scored using the same rubric as faculty, with agreement measured by Spearman's correlation, Cohen's Kappa, and interrater reliability percent agreement (IRR%).

Main Results:

• Spearman's correlations between AI and faculty assessments ranged from negligible to moderate (highest r = .54 in physical/mental health exams).
• Cohen's Kappa (.14-.41) and IRR% (31%-54%) indicated weak agreement between AI and faculty evaluations.
• AI feedback was inconsistent with faculty assessments, suggesting potential technical issues or rubric limitations.

Conclusions:

• Current AI feedback in OSCEs demonstrates inconsistency with faculty assessments, highlighting areas for improvement.
• Despite limitations, the AI tool was found to be user-friendly, and faculty perceived its potential to enhance feedback quality.
• AI in clinical assessment holds promise for advancing nursing education, but careful consideration of current limitations is necessary for effective integration.