A typology of physician input approaches to using AI chatbots for clinical decision-making: a mixed methods study

Large language model (LLM) chatbots demonstrate high degrees of accuracy, yet recent studies found that physicians using these same chatbots may score no better to worse on clinical reasoning tests compared to the chatbot performing alone with researcher-curated prompts. It is unknown how physicians approach inputting information into chatbots.

This study aimed to identify how physicians interacted with LLM chatbots on clinical reasoning tasks to create a typology of input approaches, exploring whether input approach type was associated with improved clinical reasoning performance.

We carried out a mixed methods study in three steps. First, we conducted semi-structured interviews with U.S. physicians on experiences using an LLM chatbot and analyzed transcripts using the Framework Method to develop a typology based on input patterns. Next, we analyzed the chat logs of physicians who used a chatbot while solving clinical cases, categorizing each case to an input approach type. Lastly, we used a linear mixed-effects model to compare each input approach type with performance on the clinical cases.

We identified four input approach types based on patterns of "content amount": copy-paster (entire case), selective copy-paster (pieces of a case), summarizer (user-generated case summary), and searcher (short queries). Copy-pasting and searching were utilized most. No single type was associated with scoring higher on clinical cases.

This study adds to our understanding of how physicians approach using chatbots and identifies ways in which physicians intuitively interact with chatbots.

Purposeful training and support is needed to help physicians effectively use emerging AI technologies and realize their potential for supporting safe and effective medical decision-making in practice.