A community-of-practice-based evaluation methodology for knowledge intensive computational methods and its...

Area of Science:

• Computational methods evaluation
• Health informatics
• Clinical decision support systems

Background:

• Knowledge-intensive computational methods require robust evaluation frameworks.
• Clinical decision support (CDS) methods, particularly those for multimorbidity, need in-depth analysis.
• Existing evaluation approaches may not fully capture the complexities of these methods.

Purpose of the Study:

• To develop a community-of-practice-based evaluation methodology for knowledge-intensive computational methods.
• To apply this methodology for a whitebox analysis of multimorbidity CDS (MGCDS) methods.
• To characterize functional features and inner workings of MGCDS methods.

Main Methods:

• Community-of-practice involvement in defining features and case studies.
• Developers solve case studies, detailing their computational methods.
• Qualitative analysis of solution reports to identify common themes and dimensions.
• Whitebox analysis focusing on functional features, processes, models, and data.

Main Results:

• Six research groups participated, submitting comprehensive solution reports.
• Four evaluation dimensions were identified: adverse interaction detection, management strategy representation, implementation paradigms, and human-in-the-loop support.
• Analysis revealed that while MGCDS methods offer diverse solutions, no single method is comprehensive.

Conclusions:

• The developed methodology provides a reusable benchmark for evaluating knowledge-intensive computational methods.
• The methodology was successfully applied to MGCDS methods, yielding insights into their capabilities.
• Future applications can extend to other computational methods and evaluation questions.