Validity evidence for the communication learning assessment: Prior coursework as a known group

Effective physician-patient communication is a core component of high-quality medical care, yet it remains challenging to teach and assess systematically in medical education. The Communication Learning Assessment (CLA) is an artificial intelligence-assisted, video-based tool designed to evaluate and support students' development of key communication competencies. Establishing validity evidence for the CLA is essential to ensure that its scores reflect meaningful differences in communication competence rather than general verbal ability or related constructs.

This study examined the known-groups validity of the CLA by testing whether medical students with prior formal communication coursework demonstrated higher frequencies of patient-centered communication behaviors than those without such training.

Participants were 193 medical students (69.9% female) who completed the CLA as part of a larger experimental study. The CLA presented eight clinical vignettes requiring spoken responses to simulated patient scenarios. Participants' responses were scored by the ACTA artificial intelligence system, which identified the frequency of patient-centered communication behaviors. An independent samples t-test compared total behavior counts between students with and without prior communication coursework.

Students with prior communication coursework (n = 103, M = 14.02, SD = 3.10) demonstrated significantly more patient-centered communication behaviors than students without such training (n = 90, M = 12.84, SD = 3.10), t(191) = 2.81, p = .005, d = 0.34.

Students who had completed formal communication training displayed significantly stronger communication performance on the CLA, providing evidence of known-groups validity. These findings support the interpretation that CLA scores meaningfully capture differences in communication competence rather than general verbal skill. The CLA offers a promising, scalable approach for assessing and enhancing communication skills in medical education.