Addressing Limited Evidence in Guideline Development

Evidence-based guidelines are essential to optimize healthcare decision-making, yet in contexts such as rare diseases, emerging conditions, or specific subpopulations, direct evidence to estimate the magnitude of intervention effects may be minimal or absent. In these situations, guideline panels must formulate recommendations despite very low certainty evidence on benefits and harms. To address this dilemma, we explore three strategies that guideline panels may implement.

The first involves systematic recollection of expert clinical experiences, a structured, timely, and resource-efficient method to capture expert clinical experience. However, it is prone to the limitations of member recall and biases arising from clinicians' enthusiasm for particular interventions. The second involves leveraging patient registries, which may generate evidence comparable to well-conducted observational studies but are limited by data availability, quality (e.g., missing information), and representativeness. In particular, failure to capture key prognostic variables for adjusted analyses may result in very high risk of bias. The third approach involves conducting a multisite retrospective cohort study, which can generate direct evidence with merits of accurate and comprehensive data collection. It does, however, require extensive coordination, time, resources and may have missing data.

These approaches can help address evidence gaps by providing estimates of intervention effects closely aligned with the clinical question. Each method has, however, important methodological and practical limitations and is likely to generate only very low certainty evidence. Nevertheless, when transparently applied within established frameworks such as GRADE, these strategies can support the development of more informed and actionable recommendations in settings with limited evidence.

Why did we do this study? Healthcare guidelines are based on research showing the benefits and harms of different treatments. However, for rare diseases, emerging conditions, or specific groups of patients, there may be little or no relevant published research about how well treatments work or what harms they cause. Clinicians still need guidance, and guideline panels still need to make recommendations in these situations. We therefore explored three strategies that panels can use to help address these evidence gaps. What did we find? We explored three approaches. First, panels can systematically collect and summarize experts' clinical experiences. Second, they can use information from patient registries. Third, researchers can collect data from patient charts from several healthcare centres. Each approach can provide information that is closely related to the clinical question and may help estimate the benefits and harms of an intervention. What are the common challenges? Each approach has important limitations. Expert clinical experience may be affected by imperfect recall and clinicians' preferences for particular treatments. Registries may have missing or incomplete information, may not represent all relevant patients, and may lack important information needed to make appropriate comparisons between treatment groups. Multisite retrospective cohort studies can allow more accurate and comprehensive data collection but may require substantial time, resources, and coordination and can also have missing data. All three approaches are still likely to provide very low certainty evidence. What does this mean? These approaches can help guideline panels address important evidence gaps by providing information that is closely aligned with the clinical question. The resulting guidelines need to acknowledge the limitations of the approaches. Used in this way, these strategies can support more informed and actionable recommendations when evidence is limited.