Faculty perceptions of AI-versus human-summarized narrative exit survey data across three nursing programs

Area of Science:

• Nursing Education
• Educational Technology
• Program Evaluation

Background:

• Generative AI tools are increasingly adopted in higher education for data analysis.
• Nursing education utilizes student evaluations for insights, but manual summarization is time-intensive.
• AI tools present efficiency gains but require careful consideration of reliability, bias, and pedagogical impact.

Purpose of the Study:

• To compare faculty perceptions of AI-generated versus human-generated summaries of narrative exit survey data.
• To assess the feasibility of integrating artificial intelligence (AI) into program evaluation processes.
• To evaluate the quality of AI-generated summaries in the context of nursing education.

Main Methods:

• A cross-sectional, descriptive pilot study design was employed.
• Five faculty members rated summaries generated by Microsoft Copilot and human analysis.
• A 7-point Likert scale assessed accuracy, clarity, bias, and relevance.

Main Results:

• AI-generated summaries received higher quality ratings (mean=5.9) compared to human-generated summaries (mean=5.0).
• Faculty perceptions indicated a preference for AI-generated content in this pilot study.
• The study provides initial data on AI performance in summarizing qualitative educational data.

Conclusions:

• AI can be a valuable supportive tool in program evaluation, not a complete replacement for human review.
• Integrating AI alongside human oversight can enhance efficiency in program evaluations.
• Maintaining fidelity to student voices and context is crucial when implementing AI in educational assessments.