Computer-Interpretable Clinical Guidelines: A Review and Analysis of Evaluation Criteria for Authoring Methods

Area of Science:

• Health Informatics
• Clinical Guideline Development
• Software Engineering

Background:

• Numerous authoring tools and methods exist for creating computer-interpretable clinical guidelines (CIG).
• Evaluating these tools and methods is crucial for ensuring their effectiveness and usability.
• A comprehensive understanding of current evaluation practices is needed.

Purpose of the Study:

• To systematically review and synthesize the evaluation criteria used for computer-interpretable clinical guideline authoring tools and methods.
• To identify common themes and specific criteria within the literature.
• To pinpoint areas of CIG system evaluation that have been underexplored.

Main Methods:

• A literature review was conducted using controlled database searches.
• Snowballing techniques were employed to identify additional relevant studies.
• Evaluation criteria and methods from identified papers were manually extracted and categorized.
• A total of 32 relevant papers were analyzed.

Main Results:

• Sixty-eight distinct evaluation criteria were identified from the literature.
• These criteria were classified into ten overarching themes.
• The frequency of mention for each criterion indicated areas of focus and neglect in CIG system evaluations.
• Specific themes and criteria were found to be more or less frequently addressed.

Conclusions:

• The current evaluation of computer-interpretable clinical guideline authoring tools is characterized by a diverse set of criteria across ten themes.
• Significant variation exists in the emphasis placed on different evaluation aspects.
• Identifying neglected areas in evaluation is essential for future research and development in CIG authoring systems.