Training the public health workforce for generative AI: validation of the Italian Chatbot Usability Questionnaire

Artificial intelligence (AI) is becoming increasingly relevant for public health practice. However, scalable training models and validated usability instruments for AI chatbots are still limited in non-English contexts. We aimed to (i) assess AI knowledge, attitudes, and practices (KAP) among a sample of Italian public health professionals, and (ii) translate, culturally adapt, and psychometrically validate an Italian version of the Chatbot Usability Questionnaire (CUQ-IT) by applying it to a workshop-developed vaccination counseling chatbot.

We conducted a three-phase study during the 57th Italian National Congress of Hygiene, Preventive Medicine and Public Health (Italy, October 2024): (i) use of an Italian translation and adaptation of the CUQ within a cross-sectional KAP survey; (ii) a 90-min theoretical-practical workshop with guided prompt engineering and collaborative development of a Custom GPT for vaccination counseling (VaxSense), followed by supervised interaction; (iii) post-workshop usability evaluation with CUQ-IT and psychometric testing. Construct adequacy was assessed using KMO and Bartlett's test. Internal consistency was estimated with Cronbach's α. Associations between KAP and usability ratings were explored.

Of 150 workshop attendees, 87 returned questionnaires (58%); 86 were eligible for KAP analyses, and 77 completed CUQ-IT for usability evaluation. Participants (mean age 35.9 ± 8.7; 55.8% females) reported moderate AI knowledge (mean 2.74 ± 1.05/5), positive attitudes (3.87 ± 0.73/5), and limited routine use (2.88 ± 1.11/5). CUQ-IT showed good sampling adequacy (KMO = 0.81) and significant item correlations (Bartlett's χ² (120) = 515.36, p-value <0.001). Internal consistency was high overall, with Cronbach's α = 0.881. The mean CUQ-IT usability score for VaxSense was 66.60 ± 15.29 (range 18.75-95.31). Participants with higher AI knowledge, more positive attitudes, and more frequent AI use tended to assign higher usability scores.

In our sample of Italian public health professionals, moderate baseline AI knowledge, positive attitudes, and limited routine use of AI emerged, while the workshop-developed vaccination chatbot achieved overall good usability. The significant gender differences observed in both self-rated AI knowledge and usability scores indicated that "AI readiness" may be uneven, even within trained professional groups, underscoring the need for inclusive capacity-building strategies. In the public health sector, training should be framed as a component of safe implementation: strengthening the ability to critically appraise outputs, recognize common failure modes, mitigate bias and equity risks, and apply privacy-by-design principles. CUQ-IT demonstrated robust psychometric performance and represents a standardized tool to benchmark chatbot usability in Italian settings. Future work should confirm its measurement structure in larger samples and assess whether targeted training may reduce capability gaps and support appropriate and sustained use of AI in public health practice.