Beyond the Bot: A Dual-Phase Framework for Evaluating AI Chatbot Simulations in Nursing Education

Area of Science:

• Nursing Education
• Artificial Intelligence
• Simulation-Based Learning

Background:

• Rapid integration of AI chatbots in nursing education, especially for simulation-based learning.
• Lack of structured evaluation models for AI-generated simulations.
• Need for frameworks to assess prompt design and chatbot performance.

Purpose of the Study:

• Introduce the AI-Integrated Method for Simulation (AIMS) evaluation framework.
• Adapt the FAITA model for dual-phase evaluation (prompt design and chatbot performance).
• Assess AI chatbot utility in nursing education simulations.

Main Methods:

• Developed and applied the AIMS framework (Phase 1: prompt domains, Phase 2: performance criteria).
• Utilized a scoring rubric and feedback loop for refinement.
• Demonstrated application using an AI tool (Eval-Bot v1) with expert review.

Main Results:

• AI chatbot showed strengths in prompt alignment and interactive learning for emergency planning.
• Challenges identified in personalization, inclusivity, tone, and complexity adaptation.
• Concerns flagged regarding safety prompts and inclusive language, impacting Phase 2 performance.

Conclusions:

• AIMS provides a practical, replicable method for evaluating AI chatbots in simulation-based education.
• Framework aids in identifying strengths and areas for improvement in AI integration.
• Emphasizes intentional design, safety, and inclusivity for effective and ethical AI use in health education.