Clinician Use of a General-Purpose Large Language Model in Hospital Medicine: A Mixed-Methods Pilot Study

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Clinical Workflow Optimization

Background:

• Real-world application of large language models (LLMs) in hospital medicine settings is not well-documented.
• The study addresses the need for understanding LLM integration into clinical practice.

Purpose of the Study:

• To evaluate the pilot implementation of a general-purpose LLM in a hospital medicine division.
• To assess user adoption, perceived benefits, and risks associated with LLM use in a clinical environment.

Main Methods:

• A two-month pilot study involving a general-purpose LLM application.
• Administration of two anonymous cross-sectional surveys at the beginning and end of the pilot.
• Analysis of user adoption rates, prompt frequency, and Net Promoter Score (NPS).

Main Results:

• High reported use of LLMs (over 88%) with moderate daily prompt frequency.
• Shift in LLM application from clinical reasoning/administrative tasks to documentation/summarization/research.
• Dominant perceived benefits included efficiency and time savings; key risks were hallucinations, source transparency, and privacy.

Conclusions:

• General-purpose LLMs can provide near-term value in supervised, lower-risk hospital medicine workflows.
• Successful clinical implementation requires structured governance, comprehensive training, and continuous evaluation.
• Addressing ethical and technical challenges is crucial for safe and effective LLM integration.