When Do Physicians Trust AI? The Role of Clinical Context in Decision Support: SMART vignette study

Background Physicians' confidence in clinical AI tools and perceptions of their effectiveness can influence the adoption of these tools in practice, which in turn, could affect the quality of patient care. However, the factors shaping these perceptions remain poorly understood. Methods We surveyed U.S. family medicine and internal medicine physicians (n = 420) using a sequential multiple-assignment randomized trial (SMART) vignette design involving primary care scenarios with an AI-enabled clinical decision support system (AI-CDSS). We varied three vignette characteristics, each at two levels: clinical decision types (prescription or referral), amount of information provided about the AI-CDS tool (more or less), and the tool's recommendations (treat or do not treat). We used regression models to assess whether clinical decision type or amount of information about the AI-CDSS affected physicians' confidence and their perceptions of AI-CDSS's effectiveness, use of AI in clinical practice, and clinical decisions. We also examined associations between physicians' prior AI exposure and these perceptions. Results Physicians' prior exposure to AI was positively associated with their confidence in their decisions (β = 0.274, adjusted p = 0.042) and perceptions of effectiveness of the AI-CDSS (β = 0.350, adjusted p = 0.006). Neither clinical decision type nor amount of information significantly affected perceived confidence or effectiveness. Exploratory analyses revealed that physicians assigned to a prescription scenario reported lower trust in the AI-CDSS (β=-0.211, p = 0.027), fewer benefits of using it (β=-0.179, p = 0.042), and greater reliance on their own judgment (β = 0.154, p = 0.029) versus those assigned to a psychiatric referral scenario. Prior AI exposure was also associated with greater agreement that the AI result contributed to the final decision (β = 0.305, p = 0.010). Conclusions Our findings suggest that prior knowledge of and exposure to AI may enhance physicians' confidence in their clinical decisions and their perceptions of the effectiveness of AI-CDS tools, underscoring the importance of predeployment assessment of human factors in clinical AI implementation. Training in AI use and AI-assisted decision-making should be integrated into physicians' education, while factors influencing physicians' confidence merit continued examination.