Making the AGREE tool more user-friendly: the feasibility of a user guide based on Boolean operators

Area of Science:

• Health Services Research
• Medical Informatics
• Clinical Guideline Development

Background:

• The Appraisal of Guidelines Research and Evaluation (AGREE) instrument is a widely used tool for assessing the quality of clinical practice guidelines.
• Existing AGREE item scoring can be ambiguous and subjective, potentially impacting the reliability of guideline quality assessments.
• Previous studies have explored modifications to the AGREE instrument to improve its performance.

Purpose of the Study:

• To reduce ambiguity and subjectivity in AGREE item scoring.
• To enhance the AGREE instrument's ability to differentiate between high- and low-quality guidelines.
• To develop and test a modified AGREE User Guide incorporating a Boolean-based dictionary and quality categorization criteria.

Main Methods:

• Conducted a literature review to identify existing AGREE instrument modifications.
• Developed a Boolean-based dictionary and seven criteria for guideline quality categorization within the AGREE User Guide.
• Tested the feasibility of the modified User Guide with three appraisers using three low back pain guidelines.
• Revised and re-tested the dictionary based on initial feasibility results.

Main Results:

• A literature review identified that 14% of studies modified AGREE items and 35% used consensus approaches to improve scoring consistency.
• Initial feasibility testing showed Pearson correlation coefficients ranging from 0.27 to 0.81.
• Revision and re-testing of the dictionary increased agreement levels (range 0.41 to 0.94).
• The revised dictionary decreased domain score variability but reduced reliability for inexperienced appraisers.

Conclusions:

• Appraisers found the Boolean-based AGREE User Guide easier to use, increasing their confidence in the tool.
• Good reliability was achieved in feasibility testing with the revised AGREE instrument.
• Further evaluation is needed to confirm the reliability and validity of the implemented changes, particularly for less experienced users.