Related Experiment Video
Updated: Jun 26, 2026

A Naturalistic Setup for Presenting Real People and Live Actions in Experimental Psychology and Cognitive Neuroscience Studies
Published on: August 4, 2023
Synthesis of observational studies should consider credibility ceilings
Georgia Salanti1, John P A Ioannidis
1The Clinical Trials and Evidence-Based Medicine Unit, Department of Hygiene and Epidemiology, University of Ioannina, School of Medicine, Ioannina, Greece.
Objective:
Meta-analyses of observational studies often get spuriously precise results. We aimed to factor this skepticism in meta-analysis calculations.
Study Design And Setting:
We developed a simple sensitivity analysis starting from the assumption that any single observational study cannot give us more than a maximum certainty c% (called credibility ceiling) that an effect is in a particular direction and not in the other. Each study included in meta-analysis is adjusted for different credibility ceilings c and the consistency of the conclusion examined. We applied the method in three meta-analyses of observational studies with nominally statistically significant summary effects (mortality with teaching versus nonteaching health care; risk of non-Hodgkin's lymphoma with hair dyes; mortality with omega-3 fatty acids).
Results:
Between-study heterogeneity I(2) estimates dropped from 36%-72% without a ceiling effect to 0% with ceilings of 9%, 4%, and 4% in the three meta-analyses, respectively. Nominal statistical significance was lost with ceilings of 10%, 8%, and 11%, respectively. The likelihood ratios suggested that even with minimal ceiling effects, there was no strong support for the credibility of each of these three associations.
Conclusions:
Consideration of credibility ceilings allows conservative interpretation of observational evidence and can be applied routinely to meta-analyses of observational studies.
Related Concept Videos
Observational Studies
There are three types of observational studies – Prospective, retrospective, and cross-sectional.
Prospective Study
Prospective studies, also known as longitudinal or cohort studies, are carried out by collecting future data from groups sharing similar characteristics. One example of...
Naturalistic Observations
Bias
In statistics, a sampling bias is created when a sample is collected from a population, and some members of the population are not as likely to be chosen as others (remember, each member...
Data Collection by Observations
An astronomer viewing the motion and brightness of stars in the sky and recording the data is an example of observational data collection. A botanist recording...
Bias in Epidemiological Studies
Ethics in Research