Related Experiment Video
Updated: Oct 26, 2026
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
Multilingual evaluation of a large language model-based primary care chatbot
Pei-Lun Chen1, Amogh Ananda Rao1, Sydney Pugh1
1AI-4-AI Lab, University of Pennsylvania, 3700 Hamilton Walk, Philadelphia, PA, 19104, United States.
Objective:
Large language model (LLM) chatbots show promise for pre-visit planning, but their English-centric development raises equity concerns for multilingual deployment. We evaluated the multilingual capabilities of the Primary Care Pre-visit Chatbot (PCP-Bot), a GPT-4o-based pre-visit chatbot, across English, Mandarin Chinese, Spanish, and Hindi.
Materials And Methods:
In this mixed-methods study, 31 bilingual participants (11 Mandarin, 10 Spanish, and 10 Hindi) role-played as patients across 5 synthetic primary care cases, each first in English, then in their second language. Participants completed validated instruments (Bot Usability Scale [BUS-11], NASA Task Load Index [NASA-TLX], Human-Computer Trust Scale [HCTS], and Technology Acceptance Model-Perceived Ease of Use [TAM-PEU]), an adapted summary quality rubric (fluency, adequacy, meaning, and severity), and provided open-ended feedback.
Results:
English interactions scored higher than non-English interactions on usability (difference = 0.15), conversation quality (0.62), and summary quality (0.25; all P < .001). Usability and conversation quality gaps varied by language (both P < .01), driven by Spanish and Hindi; Mandarin Chinese showed neither (P = .087). Trust, workload, and ease-of-use remained consistent across languages. Participants described PCP-Bot as natural and accurate overall but reported repetition, transcription errors, and more frequent breakdowns in non-English interactions.
Discussion:
Multilingual capability in pre-visit workflows appears feasible for LLMs under controlled conditions in the 4 high-resource languages evaluated, with high absolute performance in each. Disparities were language-dependent: conversational performance declined most for Spanish and Hindi, while summary quality showed a small, uniform English advantage, suggesting addressable language-specific weaknesses rather than a fixed limitation of English-developed systems.
Conclusion:
Large language models can support multilingual pre-visit interactions, but performance varies by language. Per-language validation and further testing are needed prior to clinical rollout.