A typology of physician input approaches to using AI chatbots for clinical decision-making

Recent studies have found that physicians with access to a large language model (LLM) chatbot during clinical reasoning tests may score no better to worse compared to the same chatbot performing alone with an input that included the entire clinical case. This study explores how physicians approach using LLM chatbots during clinical reasoning tasks and whether the amount of clinical case content included in the input affects performance. We conducted semi-structured interviews with U.S. physicians on experiences using an LLM chatbot and developed a typology based on input patterns. We then analyzed physician chat logs from two randomized controlled trials, coding each clinical case to an input approach type. Lastly, we used a linear mixed-effects model to compare the case scores of different input approach types. We identified four input approach types based on patterns of content amount: copy-paster (entire case), selective copy-paster (pieces of a case), summarizer (user-generated case summary), and searcher (short queries). Copy-pasting and searching were utilized most. No single type was associated with scoring higher on clinical cases. Other factors such as different prompting strategies, cognitive engagement, and interpretation of the outputs may have more impact and should be explored in future studies.