Developing and Validating a Coding Scheme for Clinical Reasoning in History Taking Using Generative AI-Based Virtual...

Area of Science:

• Medical Education
• Artificial Intelligence in Healthcare
• Clinical Reasoning Assessment

Background:

• Generative artificial intelligence (GenAI)-based virtual patients (VPs) are vital tools for practicing clinical history taking.
• A gap exists in effectively identifying and providing feedback on students' clinical reasoning during VP interactions.
• This limitation hinders the development of targeted instructional strategies.

Purpose of the Study:

• To develop and validate a coding scheme for assessing medical students' history-taking behaviors.
• To identify specific behaviors during interactions with GenAI-based VPs.
• To correlate these behaviors with academic performance metrics.

Main Methods:

• A coding scheme was inductively developed using systematic text condensation on 1030 dialogues from 210 second-year medical students across 4 cases.
• The scheme was validated for interrater reliability (κ≥0.85).
• Case 5 dialogues were analyzed to correlate history-taking behaviors with diagnostic accuracy, checklist scores, knowledge tests, and postencounter forms.

Main Results:

• A 12-behavior coding scheme across clinical reasoning, information gathering, and social interaction dimensions was established with high reliability.
• Clinical reasoning behaviors like summarizing and logical organization strongly correlated with performance metrics.
• Information gathering behaviors were linked to knowledge and thoroughness but not diagnostic accuracy.

Conclusions:

• A reliable, theory-informed coding scheme effectively identifies students' questioning behaviors and cognitive strategies during GenAI VP interactions.
• This scheme offers valuable insights into developing clinical reasoning in medical students.
• The approach enables scalable, real-time feedback for personalized learning and competency-based assessment in medical training.