Related Experiment Videos
Reference standards, judges, and comparison subjects: roles for experts in evaluating system performance.
1Department of Medical Informatics, Columbia University, New York, New York 10032, USA. hripcsak@columbia.edu
Journal of the American Medical Informatics Association : JAMIA
|December 26, 2001
Summary
Evaluating medical informatics systems is challenging due to a lack of reference standards. Clinical domain experts can help by performing tasks, judging system output, or serving as comparison subjects for reliable system performance assessment.
Area of Science:
- Medical Informatics
- Clinical Evaluation
- Health Systems Research
Background:
- Medical informatics systems aim for expert-level performance.
- Evaluating these systems is difficult due to the absence of clear reference standards.
- Assessing system sufficiency and reasonableness presents significant challenges.
Purpose of the Study:
- To address challenges in evaluating medical informatics system performance.
- To explore the utility of clinical domain experts in system evaluation.
- To delineate distinct roles for experts in study design and validation.
Main Methods:
- Clinical domain experts perform system tasks to generate reference standards.
- Experts directly evaluate the appropriateness of system-generated output.
- Experts serve as benchmarks for direct comparison with system performance.
Main Results:
- Defined three distinct roles for clinical domain experts in system evaluation.
- Highlighted implications of each role for study design, metrics, reliability, and validity.
- Emphasized the value of expert involvement in overcoming evaluation constraints.
Conclusions:
- Clinical domain experts are crucial for robust medical informatics system evaluation.
- Structured use of experts enhances the reliability and validity of performance assessments.
- Diagrams can clarify expert roles in complex evaluation methodologies.