Related Experiment Video
Updated: Dec 12, 2025
Author Spotlight: Evaluating the Adjuvant Efficacy and Safety of Angong Niuhuang Pill in Viral Encephalitis Treatment
Published on: April 19, 2024
Development of the Instrument to assess the Credibility of Effect Modification Analyses (ICEMAN) in randomized
Stefan Schandelmaier1, Matthias Briel2, Ravi Varadhan2
1Departments of Health Research Methods, Evidence, and Impact (Schandelmaier, Briel, Walsh, Thabane, Guyatt), Medicine (Walsh, Guyatt), Pediatrics (Thabane) and Anesthesia (Thabane), McMaster University, Hamilton, Ont.; Institute for Clinical Epidemiology and Biostatistics (Schandelmaier, Briel), Department of Clinical Research, Basel University, Basel, Switzerland; Division of Biostatistics and Bioinformatics (Varadhan), Sidney Kimmel Comprehensive Cancer Center, Johns Hopkins University, Baltimore, Md.; Department of Biostatistics (Schmid), Brown University School of Public Health, Brown University, Providence, RI; Indian institute of Public Health-Delhi (Devasenapathy), Public Health Foundation of India, New Delhi, India; VA Center for Clinical Management and Research (Hayward); Department of Internal Medicine (Hayward), University of Michigan School of Medicine; Department of Orthopaedic Surgery (Gagnier), University of Michigan; Department of Epidemiology (Gagnier), School of Public Health, University of Michigan, Ann Arbor, Mich.; Biostat (Borenstein), Englewood, NJ; Department of Social Dentistry (van der Heijden), Academic Center for Dentistry Amsterdam, University of Amsterdam and VU University Amsterdam, Amsterdam, Netherlands; Center for Evidence Synthesis in Health (Dahabreh) and Departments of Health Services, Policy, and Practice (Dahabreh) and Epidemiology (Dahabreh), School of Public Health, Brown University, Providence, RI; Chinese Evidence-Based Medicine Center (Sun), West China Hospital, Sichuan University, Chengdu, China; Institute of Medical Biometry and Statistics (Sauerbrei), Faculty of Medicine and Medical Center, University of Freiburg, Freiburg, Germany; Population Health Research Institute (Walsh), Hamilton Health Sciences/McMaster University, Hamilton, Ont.; Departments of Medicine (Ioannidis), Health Research and Policy (Ioannidis) and Biomedical Data Science (Ioannidis), and Statistics and Meta-Research Innovation Center at Stanford (METRICS) (Ioannidis), Stanford University, Stanford, Calif.; Biostatistics Unit (Thabane), St. Joseph's Healthcare, Hamilton, Ont. s.schandelmaier@gmail.com.
Background:
Most randomized controlled trials (RCTs) and meta-analyses of RCTs examine effect modification (also called a subgroup effect or interaction), in which the effect of an intervention varies by another variable (e.g., age or disease severity). Assessing the credibility of an apparent effect modification presents challenges; therefore, we developed the Instrument for assessing the Credibility of Effect Modification Analyses (ICEMAN).
Methods:
To develop ICEMAN, we established a detailed concept; identified candidate credibility considerations in a systematic survey of the literature; together with experts, performed a consensus study to identify key considerations and develop them into instrument items; and refined the instrument based on feedback from trial investigators, systematic review authors and journal editors, who applied drafts of ICEMAN to published claims of effect modification.
Results:
The final instrument consists of a set of preliminary considerations, core questions (5 for RCTs, 8 for meta-analyses) with 4 response options, 1 optional item for additional considerations and a rating of credibility on a visual analogue scale ranging from very low to high. An accompanying manual provides rationales, detailed instructions and examples from the literature. Seventeen potential users tested ICEMAN; their suggestions improved the user-friendliness of the instrument.
Interpretation:
The Instrument for assessing the Credibility of Effect Modification Analyses offers explicit guidance for investigators, systematic reviewers, journal editors and others considering making a claim of effect modification or interpreting a claim made by others.
Related Concept Videos
01:23Types of Biopharmaceutical Studies: Controlled and Non-Controlled Approaches
Non-controlled studies, commonly employed for initial exploration, lack a control group, rendering them susceptible to biases and external influences. In contrast,...
01:29Reliability and Validity
01:25Strategies for Assessing and Addressing Confounding
Confounding can be addressed at both the design phase of a study and through analytical methods after data...
01:17Cochran's Q Test
01:20Study Designs in Epidemiology
Observational studies are those where the researcher does not intervene but rather observes natural variations. They include cross-sectional, cohort, and...
01:09Odds Ratio