A Secure User Interface for Preclinical Evaluation of AI in Patient Portal Message Management: Tutorial

Area of Science:

• Health Informatics
• Artificial Intelligence in Healthcare
• Clinical Workflow Optimization

Background:

• Increasing use of AI in patient portal message management necessitates preclinical evaluation.
• Direct AI testing in Electronic Health Record (EHR) systems presents safety, workflow, and data governance risks.
• Need for a controlled environment to test AI for patient portal messages prior to clinical integration.

Purpose of the Study:

• To report the technical feasibility of a secure user interface (UI) sandbox for AI experimentation in patient portal messaging.
• To enable clinical and technical teams to evaluate AI performance and risks before integration into live EHR systems.
• To assess the safety, workflow, and data governance implications of using AI for managing patient portal messages.

Main Methods:

• Developed a secure, web-based UI sandbox using Python 3 with a modular backend, operating within institutional firewalls.
• Implemented a deidentification pipeline to remove or replace personal health identifiers, with validated precision (82.1%).
• Tested large language model (LLM)-enabled tasks (authorship identification, categorization, criticality flagging, response drafting) using various prompting strategies on a deidentified EHR message corpus.

Main Results:

• The sandbox successfully executed end-to-end workflows for ingesting messages, performing AI analyses (single and batch), and presenting outputs.
• Deidentification pipeline achieved 95.1% sensitivity and 82.1% precision in masking personal health information.
• Prompting strategies were evaluated for interpretability in AI tasks, and response drafting generated editable clinician starting points; token-based cost readout provided transparent operating estimates.

Conclusions:

• A secure, UI-based sandbox provides a practical and safe method for health systems to evaluate AI for patient portal messaging before clinical integration.
• This framework supports exploratory testing, prompt iteration, and comparative analyses while preserving data governance boundaries.
• The sandbox approach generates evidence on AI feasibility, risks, and clinical fit in a controlled setting, mitigating risks associated with direct EHR testing.