Exploring subjective vs. objective issues in the validation of computer-based critiquing advice.

Area of Science:

• Medical Informatics
• Artificial Intelligence in Medicine
• Radiology

Background:

• Evaluating computer-based medical expert systems is crucial for development.
• Critiquing systems pose unique validation challenges due to their prose-based feedback.
• Subjective user interaction significantly influences system evaluation.

Purpose of the Study:

• To describe the DxCON expert system, designed for critiquing the radiologic workup of obstructive jaundice.
• To present the experimental validation of the DxCON system.
• To explore the impact of subjective factors on the validation of explanatory medical expert systems.

Main Methods:

• DxCON, an expert system, generates prose critiques for obstructive jaundice radiologic workups.
• Validation involved three independent judges interacting with the system's advice.
• Judges were allowed flexible, unstructured reactions to the system's output.

Main Results:

• The study details the design and implementation of DxCON's validation process.
• It highlights the challenges in evaluating expert systems with subjective, prose-based outputs.
• The validation process accommodated flexible, unstructured judge feedback.

Conclusions:

• Subjective issues significantly impact the design and execution of medical expert system validation.
• Flexible, unstructured evaluation methods are necessary for systems producing explanatory prose.
• This case study offers insights into validating complex medical AI systems.