Assessing AI-generated smoking cessation advice for patient education in primary care

Artificial intelligence (AI) has emerged as a promising tool to support smoking cessation in primary care, particularly for populations underserved by traditional interventions. However, the quality of AI-generated smoking cessation advice remains understudied, especially in low-resource settings and among vulnerable groups such as adolescents.

This study aims to evaluate AI-generated responses to smoking cessation questions for patient education in primary care, comparing different AI programs in terms of knowledge, readability, and quality.

Ten publicly accessible AI programs were prompted in Turkish with 24 standardized, open-ended smoking cessation questions framed as a patient consultation. Two family medicine specialists independently assessed each response's readability using Ateşman's Readability Index, reliability using the DISCERN instrument, and accuracy and motivational interviewing quality using a bespoke rubric and OARS (Open questions, Affirmations, Reflections, Summaries) framework. Inter-rater agreement was evaluated via intraclass correlation. Descriptive statistics were computed for readability scores, DISCERN ratings, and accuracy grades.

All AI programs provided at least partially correct answers to all questions. The average readability score was 54.90 (medium difficulty) according to Atesman's Index. The mean DISCERN score was 66 ± 5.2, indicating excellent quality. Three AI programs incorporated core motivational interviewing skills. The most accurately answered question concerned e-cigarettes' harm compared to traditional cigarettes, while medication advice was least evidence-based.

Free AI chatbots deliver reliably accurate and moderately readable smoking cessation advice, supporting their potential role as patient education adjuncts in primary care-particularly for individuals with at least a high school education. Further research should compare AI-assisted versus clinician-led interventions on smoking cessation outcomes.