Comparing AI-generated and traditional textbook multiple-choice questions in nursing education: A prompt...

To explore the potential and challenges of GLM-4 model in nursing test design, and to evaluate its effectiveness in generating multiple-choice questions under the theme of management of surgical shock patients.

The Delphi method was employed, inviting 8 nursing education experts with master's degrees or higher, or associate professor titles and above, to review 25 MCQs generated by GLM-4. Various prompting techniques (e.g. zero-shot prompting, few-shot prompting, Chain-of-Thought, Chain of Thought with Self Consistency, and Tree-of-Thought) were used to guide the model in generating questions, which were assessed for content coverage, difficulty appropriateness, and language quality. Experts rated the questions using a 4-point Likert scale, with consistency analyzed through Kendall's W, ICC3, and coefficient of variation.

AI excelled in assessing foundational knowledge and clinical judgment, aligning with undergraduate needs. Expert ratings averaged 3-4, with an authority coefficient of 0.86 ± 0.10 and ICC3 > 0.90, indicating high reliability. Chain-of-Thought scored highest (content 3.85, difficulty 3.85, language 3.40) but had poor consistency (Kendall's W ≈ 0.05); Few-shot showed strong consistency (Kendall's W > 0.75) but lower quality (2.0-2.2). Issues included inaccurate language and unrealistic scenarios.

Generative AI offers efficiency and diversity in nursing test design, reducing expert reliance. However, limitations include low difficulty, weak reasoning, and flawed distractors. Human review and iterative prompting are recommended to optimize complex scenarios and enhance AI's role in nursing education.