When the Chatbot Leaks: Securing Patient-Facing Medical AI in the Age of Dual-Use Large Language Models

AbstractPatient-facing chatbots based on retrieval-augmented generation (RAG) are increasingly promoted to deliver curated, evidence-based health information. However, RAG is not a security boundary; a chatbot providing clinically accurate responses can still expose its instructions, knowledge base, and stored patient conversations through ordinary deployment errors. In an anonymized assessment of a publicly accessible patient-facing RAG chatbot, we used only a standard web browser, its built-in developer tools, and a commercial AI assistant to retrieve the system prompt, the full pipeline configuration, the knowledge base, and 1000 stored patient conversations without authentication or any specialized tool. The same dual-use capabilities that lower the barrier to building patient-facing AI now also lower the barrier to probing it. The health AI deployment process must extend beyond solely evaluating the model. It must evaluate the solution as a secure software system.