Development of vignette-based evaluation with generative artificial intelligence (VEGA) for nursing students: a pilot...

Advances in AI have introduced new opportunities in nursing education. Vignette-based item sets are valuable for evaluating clinical reasoning, but are time-consuming to develop. Generative AI offers a scalable approach to streamline item generation. This study aimed to develop and evaluate a vignette-based evaluation with generative artificial intelligence (VEGA), which automatically generates vignette-based item sets, defined as short clinical scenarios followed by structured assessment items.

A pilot study employed a mixed-methods design to evaluate the usability and user experiences of VEGA among nursing students. VEGA, a generative AI agent for vignette-based item set generation within the Collective AI on the Foundation AI platform, was developed based on the Analysis, Design, Development, Implementation, and Evaluation model and structured according to the NCSBN Clinical Judgment Measurement Model. Content validity was established through expert review and a preliminary survey prior to implementation. Quantitative data were collected through a post-usability survey administered to 12 undergraduate nursing students, and qualitative data were obtained through focus group interviews exploring their user experiences.

In the quantitative phase, VEGA demonstrated an overall usability score of 3.62 ± 1.04, with the highest domain score for information (4.25 ± 0.87). In the qualitative phase, focus group interviews identified four themes: individualized learning, enhancement of clinical reasoning, applicability in education and practice, and areas for improvement.

The AI-based item generation tool was perceived by nursing students as a potential educational resource for supporting engagement with clinical reasoning, with further improvements needed in technical stability, feedback depth, and multimedia integration. Given the small sample size and pilot study design, the findings should be interpreted as preliminary.