Related Experiment Video
Updated: Dec 3, 2025
Inverse Probability of Treatment Weighting Propensity Score using the Military Health System Data Repository and National Death Index
Published on: January 8, 2020
When are randomized trials unnecessary? A signal detection theory approach to approving new treatments based on
Benjamin Djulbegovic1,2,3, Marianne Razavi1,2,3, Iztok Hozo4
1Department of Supportive Care Medicine, City of Hope National Medical Centre, Duarte, California, USA.
Rationale, Aims And Objectives:
New therapies are increasingly approved by regulatory agencies such as the Food and Drug Administration (FDA) and the European Medicines Agency (EMA) based on testing in non-randomized clinical trials. These treatments have typically displayed "dramatic effects" (ie, effects that are considered large enough to obviate the combined effects of biases and random errors that may affect the study results). The agencies, however, have not identified how large these effects should be to avoid the need for further testing in randomized controlled trials (RCTs). We investigated the effect size that would circumvent the need for further RCTs testing by the regulatory agencies. We hypothesized that the approval of therapeutic interventions by regulators is based on heuristic decision making whose accuracy can be best characterized by the application of signal detection theory (SDT).
Methods:
We merged the EMA and FDA database of approvals based on non-RCT comparisons. We excluded duplicate entries between the two databases. We included a total of 134 approvals of drugs and devices based on non-RCTs. We integrated Weber-Fechner law of psychophysics and recognition heuristics within SDT to provide descriptive explanations of the decisions made by the FDA and EMA to approve new treatments based on non-randomized studies without requiring further testing in RCTs.
Results:
Our findings suggest that when the difference between novel treatments and the historical control is at least one logarithm (base 10) of magnitude, the veracity of testing in non-RCTs seems to be established.
Conclusion:
Drug developers and practitioners alike can use the change in one logarithm of effect size as a benchmark to decide if further testing in RCTs should be pursued, or as a guide to interpreting the results reported in non-randomized studies. However, further research would be useful to better characterize the threshold of effect size above which testing in RCTs is not needed.
Insights
Regulatory agencies can approve new therapies based on non-randomized trials if the effect size is at least one log magnitude. This benchmark helps determine if further randomized controlled trials are necessary for novel treatments.
Area of Science:
- Medical research methodology
- Regulatory science
- Clinical trial design
Background:
- Regulatory agencies like the FDA and EMA approve new therapies based on non-randomized trials showing "dramatic effects."
- The threshold for "dramatic effects" that obviate the need for further randomized controlled trials (RCTs) has not been clearly defined.
- This study investigates the effect size required to bypass further RCTs, hypothesizing heuristic decision-making informed by signal detection theory (SDT).
Purpose of the Study:
- To determine the minimum effect size for approving therapies based on non-randomized studies.
- To apply signal detection theory (SDT) to understand regulatory decision-making for new treatments.
- To establish a benchmark for deciding when further randomized controlled trials (RCTs) are unnecessary.
Main Methods:
- Merged EMA and FDA databases of drug and device approvals based on non-randomized comparisons (n=134).
- Excluded duplicate entries between databases.
- Integrated Weber-Fechner law and recognition heuristics within SDT to model regulatory approval decisions.
Main Results:
- An effect size difference of at least one logarithm (base 10) between novel treatments and historical controls appears to establish the veracity of non-RCT testing.
- This finding suggests a quantifiable threshold for regulatory approval based on non-randomized studies.
Conclusions:
- A one-logarithm change in effect size can serve as a benchmark for drug developers and practitioners.
- This benchmark aids in deciding whether to pursue further RCTs or interpret results from non-randomized studies.
- Further research is recommended to refine the effect size threshold above which RCTs may not be needed.
More Related Videos
04:53A Clinical Trial Assessing the Safety, Efficacy, and Delivery of Olive-Oil-Based Three-Chamber Bags for Parenteral Nutrition
Published on: September 20, 2019
06:16Signal Acquisition, Score Interpretation, and Economics of a Non-Invasive Point-of-Care Test for Coronary Artery Disease
Published on: August 9, 2024
Related Concept Videos
01:20Bioequivalence Experimental Study Designs: Completely Randomized and Randomized Block Designs
01:11Blinding
01:13Randomized Experiments
Simple randomization
Simple...
01:16Clinical Trials
There are four phases in a clinical trial. A phase one...
01:11Clinical Trials: Overview
01:23Types of Biopharmaceutical Studies: Controlled and Non-Controlled Approaches
Non-controlled studies, commonly employed for initial exploration, lack a control group, rendering them susceptible to biases and external influences. In contrast,...