An Acceptance Criteria Framework for Determining the Implementation Fit of Custom Large Language Models in Public...

Area of Science:

• Artificial Intelligence in Public Health
• Clinical Informatics
• Health Technology Assessment

Background:

• Large language models (LLMs) are increasingly used in health chatbots, shifting deployment risks from rule-based to LLM-enabled designs.
• LLM outputs are unpredictable and difficult to validate using traditional methods, posing challenges for public health interventions.
• Existing frameworks lack operational benchmarks for LLM deployment decisions in healthcare.

Purpose of the Study:

• To propose an Acceptance Criteria Framework (ACF) for determining the implementation readiness of LLM-enabled health tools.
• To operationalize deployment and implementation decisions by establishing prespecified minimum performance standards.
• To guide public health teams in assessing LLM performance variability and ethical considerations.

Main Methods:

• The ACF utilizes project-relevant and off-topic prompts for structured expert review.
• Prespecified thresholds are used to evaluate LLM performance and behavior under anticipated use.
• A documented decision record is produced, allowing for iterative reruns after model revisions.

Main Results:

• The ACF provides a systematic approach to assess LLM implementation fit, ensuring minimum performance standards are met.
• Case application in a tobacco cessation intervention demonstrated the framework's utility in guiding deployment decisions.
• The framework addresses practical and ethical challenges, including performance variability and inequitable language performance.

Conclusions:

• The proposed Acceptance Criteria Framework (ACF) offers a practical solution for evaluating LLM readiness in public health.
• Implementing the ACF can enhance the safety, effectiveness, and equity of LLM-based health interventions.
• The framework supports informed decision-making for the deployment of advanced AI tools in clinical and population health.