Implementation and User Evaluation of an On-Premise Large Language Model in a German University Hospital Setting:...

Large language models (LLMs) are increasingly used by employees at university hospitals for information retrieval or decision support. Self-hosted on-premise systems provide a secure environment and conform to data privacy and security regulations for handling sensitive personal data. Automation of standard procedures using an LLM application can substantially reduce time-consuming administrative tasks and facilitate the analysis of large datasets.

The objective of our study was to gather feedback from registered artificial intelligence (AI) users on the usability and common use cases of the on-premise LLM infrastructure we established at the University Medicine Magdeburg to optimize the models to the needs of our facility.

We developed an online questionnaire to which registered AI users were given access and were informed via email.

Of 322 registered AI users, 98 (30.4%) participated in the user survey. After filtering incomplete responses, results from 91 (28.3%) participants remained for further analysis. Speed and quality received overall high approval rates. Most of the users (n=57, 62.6%) used the platform at least once per week, and 44% (n=40) of the users reported saving at least 30 minutes of work per week by using our AI platform. A diverse set of use cases was observed, varying by profession; for example, health care and research professionals used the AI platform more frequently for creation and analysis tasks than administrative staff.

Our data indicate that the implementation of a self-hosted on-premise LLM was associated with positive perceptions among a diverse group of professionals working at a university hospital, saving time and meeting their individual needs.