Human computation as a new method for evidence-based knowledge transfer in Web-based guideline development groups:...

Area of Science:

• Health Sciences
• Medical Informatics
• Decision Science

Background:

• Existing clinical practice guideline development methods face methodological challenges and logistical demands.
• Novel consensus techniques are sought to enhance the rigor and efficiency of evidence-based guideline creation.
• Human computation systems, leveraging gamified knowledge aggregation, show promise for building knowledge databases.

Purpose of the Study:

• To evaluate the feasibility of a novel human computation (HC) consensus method against an informal face-to-face (IC) consensus method for guideline development.
• To compare the effectiveness of HC and IC methods in achieving group agreement and concordance with clinical evidence.

Main Methods:

• A randomized trial compared a web-based human computation (HC) method with an informal consensus (IC) method among master's students in nursing and obstetrics.
• Four lower back pain clinical scenarios (imaging, therapeutics, drugs, sick leave) were used to assess group (dis)agreement and evidence concordance.
• Statistical analyses included Cohen's d effect sizes, Wilcoxon signed rank tests, and Mann-Whitney U tests, with a Bonferroni adjusted alpha of .025.

Main Results:

• Human computation groups showed a trend toward greater improvement in evidence scores compared to informal consensus groups, though not statistically significant (e.g., d=0.56 for imaging, d=0.89 for drug use).
• No significant differences in the improvement of group agreement were found between HC and IC methods overall.
• HC groups demonstrated greater improvement in agreement for medical imaging (d=0.46) and drug use (d=0.31) scenarios; evidence arguments were rarely cited in informal discussions (6%).

Conclusions:

• The informal consensus (IC) method is suitable when evidence aligns with existing beliefs or is scarce.
• Human computation (HC) methods demonstrated superior performance in guideline development scenarios with evidence-based controversies.
• Human computation presents a viable methodology for guideline development, particularly when evidence challenges participants' existing beliefs, warranting further investigation in multidisciplinary settings.