Robustness of a Large Language Model (LLM)-Based Virtual Patient for Japanese History-Taking Training Under Direct...

Large language model (LLM)-based virtual patients are increasingly used to scale history-taking practice in undergraduate and postgraduate medical education. For clinical simulation, reliability requires not only avoidance of harmful content but also role-consistent case fidelity, dialogue continuity, and adherence to constraints on what the simulated patient should disclose. We evaluated these dimensions as an operational robustness benchmark, not as evidence of deployment readiness. We systematically stress-tested a Japanese LLM-based virtual patient under six robustness conditions: clean inputs, noise, direct contamination, direct contamination with defense, indirect contamination, and indirect contamination with defense. Case fidelity was measured using the slot-level F1 score, a 0-to-1 measure combining precision and recall for expected case-history elements, excluding the opening greeting turn. Information coverage was measured using turn-target hit rate, defined as the fraction of prespecified target slots elicited at the intended interview turns. Proxy constraint-adherence outcomes included refusal, clarification, forbidden information leakage, contradiction, and role drift. Under clean inputs, case fidelity was high (0.947; 95% confidence interval (CI), 0.940-0.955), and turn-target hit rate was 0.946 (95% CI, 0.932-0.959). Noise preserved overall case fidelity (0.941; 95% CI, 0.934-0.949) but reduced turn-target hit rate (0.823; 95% CI, 0.806-0.841). Direct contamination caused severe degradation in fidelity (0.098; 95% CI, 0.052-0.144) and turn-target hit rate (0.077; 95% CI, 0.034-0.119). With preprocessing defense, performance returned to near-clean levels (fidelity, 0.945; 95% CI, 0.938-0.952; hit rate, 0.942; 95% CI, 0.930-0.955). Indirect contamination showed near-clean fidelity (0.946; 95% CI, 0.938-0.954) and hit rate (0.952; 95% CI, 0.940-0.965), with minimal additional benefit from defense. Refusal and clarification rates were 0 across all conditions, and role drift events were not observed. However, forbidden information leakage occurred at approximately 0.60 events per episode under the clean condition (0.600; 95% CI, 0.502-0.698), indicating incomplete constraint adherence despite high case fidelity. In this controlled sequential virtual patient benchmark, direct prompt contamination was the dominant failure mode in terms of case fidelity and information coverage, while noise primarily reduced target-slot acquisition. A simple preprocessing defense mitigated direct-contamination effects on fidelity and information coverage, but persistent forbidden-information leakage indicates that additional safeguards and external validation are required before claims of safe clinical or educational deployment can be made.