A methodology for evaluation of knowledge-based systems in medicine

Area of Science:

• Medical Informatics
• Artificial Intelligence in Medicine
• Health Systems Engineering

Background:

• Knowledge-based systems (KBS) require thorough evaluation for safe and correct integration into clinical environments.
• User confidence in medical KBS is paramount for successful adoption and utilization.
• The medical domain presents unique challenges for system evaluation due to safety and correctness imperatives.

Purpose of the Study:

• To present a structured, iterative evaluation framework for medical knowledge-based systems.
• To detail a four-phased development evaluation cycle.
• To ensure the safety, correctness, and user acceptance of KBS in healthcare.

Main Methods:

• An iterative, four-phased development evaluation cycle was employed.
• Phase 1: Early prototype development and assessment.
• Phase 2: System validity testing.
• Phase 3: System functionality evaluation.
• Phase 4: Assessment of system impact on users and environment.

Main Results:

• The iterative cycle facilitates continuous improvement and validation of KBS.
• Each phase addresses specific aspects of system development and integration.
• The framework supports the development of reliable and trustworthy medical KBS.

Conclusions:

• A systematic, phased evaluation approach is crucial for medical knowledge-based systems.
• This iterative cycle enhances system safety, validity, functionality, and user confidence.
• Implementing this evaluation framework promotes the successful integration of KBS in healthcare settings.