Development of a cancer information chatbot model: Retrieval-augmented generation with data from the national center...

With the rapid advancement of digital health technologies, there is a growing need for reliable healthcare solutions. However, the vast amount of available cancer-related information and the challenges in identifying trustworthy sources highlight the requirement for systematic management.

This study aimed to develop a National Cancer Information Center-grounded RAG chatbot and to evaluate evidence traceability using automatic metrics.

We implemented a RAG-based chatbot using GPT-4o, FAISS vector search, and OpenAI embeddings, grounded in verified cancer-related data from the National Cancer Information Center. Two retrieval strategies were compared: (1) non-filtering retrieval based solely on vector similarity and (2) heuristic cancer-type filtering applied as a post-retrieval string-matching constraint. A total of 72 responses were evaluated using automatic evidence-traceability metrics, including retrieved evidence count, verified evidence count, cancer-matched evidence count, total answer sentence count, and evidence-aligned sentence count. Paired comparisons were conducted using the Wilcoxon signed-rank test with bootstrap confidence intervals and Holm correction.

The non-filtering strategy retrieved significantly more verified and cancer-matched evidence and produced more evidence-aligned answer sentences than heuristic filtering (all p<0.01). The total number of answer sentences did not differ significantly.

Heuristic cancer-type filtering degraded evidence grounding in a National Cancer Information Center-based RAG chatbot. Automatic traceability metrics provide a reproducible framework for evaluating and monitoring evidence-grounded performance.