Proposed Context-of-Use Evaluation Framework for Medication Management Tasks Completed by Generative Artificial...

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Pharmacology

Background:

• Standardized evaluation methods for agentic artificial intelligence (AI) in medication management are currently lacking.
• Medication errors endorsed or missed by AI pose significant risks, necessitating robust performance evaluation constructs.
• The development of such constructs is essential for ensuring patient safety in AI-assisted healthcare.

Purpose of the Study:

• To develop a standardized grading framework for evaluating the performance of AI systems in medication management tasks.
• To create a comprehensive tool that assesses AI performance across various medication domains.
• To establish a reliable method for measuring AI's accuracy and safety in clinical decision support.

Main Methods:

• A mixed-methods approach combining literature review of comprehensive medication management (CMM) standards and best practices.
• Panel discussions with experts to refine evaluation domains and criteria.
• Iterative application of the framework to a set of clinical cases to assess its utility and reliability.

Main Results:

• A 5-panelist team developed the CMM Evaluation Framework with four dimensions: safety, factuality, completeness, and preference.
• The framework applies to three CMM skills: patient data collection, information analysis, and regimen design, with a 1-5 rating scale per dimension.
• High inter-rater reliability (Krippendorffs Alpha) was achieved, particularly in medication therapy problem and format categories for pneumonia cases.

Conclusions:

• The developed framework aligns with national CMM standards and expert clinical practice.
• It provides a structured approach to evaluate AI in medication management, allowing for practice variation while implementing safeguards against errors.
• Further research and pilot testing are recommended to validate and refine the framework for broader application.