Consensus conferences must include a systematic search and categorization of the evidence

Area of Science:

• Medical consensus building
• Evidence-based medicine
• Scientific literature review

Background:

• Consensus panels ideally integrate expert knowledge with literature evidence.
• Current consensus conferences often lack structured literature searches, relying on panelists' cited references.
• This study evaluates the effectiveness of panelists' reference lists in retrieving relevant literature.

Purpose of the Study:

• To assess the accuracy of literature retrieval by consensus panelists.
• To compare the literature cited by panelists with systematic database searches.
• To determine the sensitivity and specificity of panelists in identifying randomized controlled trials (RCTs).

Main Methods:

• Analysis of reference lists from nine surgeons participating in a consensus conference on common bile duct stones.
• Inclusion of all cited randomized controlled trials (RCTs).
• Comparison of panelists' cited RCTs against a comprehensive database search to calculate sensitivity and specificity.

Main Results:

• Panelists cited an average of only eight RCTs, with significant discrepancies between perceived and actual RCTs (23 out of 49 identified were actual RCTs).
• The sensitivity for correctly identifying an RCT was 0.21, and specificity was 0.80.
• RCTs with "randomized" in the title were more likely to be identified (RR 1.31).

Conclusions:

• Consensus panelists tend to rely on personal favorite papers rather than systematic literature searches, potentially biasing the evidence base.
• A biased evidence selection can compromise the foundation of consensus statements and clinical guidelines.
• Mandatory systematic literature searches are recommended for all consensus and guideline development processes to ensure comprehensive and unbiased evidence integration.