Using experts feedback in clinical case resolution and arbitration as accuracy diagnosis methodology

Area of Science:

• Medical Informatics
• Artificial Intelligence in Medicine
• Clinical Decision Support Systems

Background:

• Evaluating the efficiency of medical diagnostic systems and clinical decision support systems is crucial for reliable healthcare.
• Existing methods may lack objectivity or comprehensive analysis of system performance.
• Expert feedback is a valuable but often complex component in system validation.

Purpose of the Study:

• To propose and validate a novel methodology for assessing medical diagnostic and clinical decision support systems.
• To compare expert feedback with system evaluations using objective performance metrics.
• To provide a reliable and fact-based process for system improvement and analysis.

Main Methods:

• A methodology comparing expert feedback on clinical cases with expert system evaluations.
• An arbitration process to ensure the correctness of results from both methods.
• Analysis using Precision, Recall, Accuracy, Specificity, and Matthews Correlation Coefficient (MCC) metrics (PRAS-M).

Main Results:

• The methodology allows for objective accuracy assessment of diagnostic systems based on real-world data.
• Analysis can be performed at the disease or knowledge base level, and on assessor efficiency.
• A real diagnostic system evaluation showed statistically significant improvements in recall (+27.54%) and MCC (+32.19%) compared to assessors.

Conclusions:

• The proposed methodology offers a reliable, objective, and fact-based approach to evaluating medical diagnostic systems.
• It facilitates the identification of key system characteristics and areas for enhancement.
• The methodology demonstrates practical applicability and yields statistically significant positive results.