The Burden of Reviewing LLM-Generated Content

AbstractLarge language models (LLMs) in health care aim to reduce the cognitive and administrative burden for health care professionals. Yet the accuracy and completeness of LLM-generated material is still highly variable. Reviewing output for errors is critical to ensure documentation is accurate and complete. Unfortunately, humans tend to follow the path of least cognitive effort and will quickly become overly reliant on automation aids. Developers argue that LLM applications in health care are not automatic solutions as they require a human-in-the-loop approach, and that the risk of errors is adequately mitigated by relying on physician review. However, the perception of LLMs as outperforming humans may lead to overestimating the capabilities of the models. When humans begin to rely more heavily on LLM output, the automation argument is no longer valid. While the physician has the final determination in practice, this approach is grounded in two principles: the assumption that every physician will thoroughly review all output, every time, for every patient; and the reliance on disclaimers to cover liability and shift accountability from developers to physicians. This creates a new and tedious burden on physicians - proofreading content that was neither written nor dictated by the user is difficult to do well. Minimal research has been published to understand the net benefit on efficiency and cognitive burden when physicians' efforts are shifted from generating new content to reviewing content generated by LLMs. Furthermore, the use of LLMs increases physicians' accountability for something they did not write. The liability of missing a high-risk error in LLM output is a novel factor for consideration and not well understood. Further study on the use of LLMs to generate clinical documentation or clinical communication should consider the cognitive impact of proofreading and the increased accountability of physicians. We recommend integrating implementation science, user experience research, and collaborative input from policy makers, regulators, and professional medical associations to establish a shared accountability structure. This approach aims to protect patients while empowering physicians to perform their duties confidently, ultimately enhancing both patient safety and physician satisfaction.