Development of a Guideline-Based Retrieval-Augmented Generation Chatbot Referencing Web-Based Cancer Clinical...

The widespread availability of medical information on the Internet has improved access to health-related knowledge for patients; however, it has also increased exposure to inaccurate medical information. Cancer care involves complex information, making it difficult for patients to identify accurate and evidence-based sources. Large language models enable natural language interaction but frequently generate hallucinations, limiting their application in medical information delivery. Retrieval-augmented generation (RAG) has the potential to mitigate these risks by grounding responses in external reference sources. This study describes the development and evaluation of a guideline-based RAG chatbot that uses publicly available, web-based cancer clinical practice guidelines. The system extracts URLs from the table-of-contents pages of a guideline, processes page text using morphological analysis and cosine similarity based on term frequency-inverse document frequency, and constructs reference information from highly relevant pages. A large language model uses these references to generate responses constrained to guideline content. The JLCS Guidebook for Lung Cancer Patients and Families was used as the reference guideline. Question sets included both in-scope cancer types covered by the guideline and out-of-scope cancer types to evaluate response control. Medical information was successfully extracted from the table-of-contents pages, with 98% of extracted URLs containing referenceable medical content. For in-scope questions, the chatbot generated responses by summarizing guideline content, and no hallucinations were identified during manual review under the defined test conditions. For out-of-scope questions, the chatbot consistently declined to answer and indicated that the available information was insufficient. The web structure of the guideline facilitated efficient scraping and organization of reference content at the URL level. This protocol provides practical guidance for constructing artificial intelligence systems that deliver medical information using web-based clinical practice guidelines.