When are randomized trials unnecessary? A signal detection theory
approach to approving new treatments based on non-randomized studies
Abstract
Rationale, aims and objectives New therapies are increasingly approved
by regulatory agencies such as the Food and Drug Administration (FDA)
and the European Medicines Agency (EMA) based on testing in
non-randomized clinical trials. These treatments have typically
displayed “dramatic effects” (i.e., effects that are considered large
enough to obviate the combined effects of bias and random error). The
agencies, however, have not identified how large these effects should be
to avoid the need for further testing in randomized controlled trials
(RCTs). We investigated the effect size that would circumvent the need
for further RCTs testing by the regulatory agencies. We hypothesized
that the approval of therapeutic interventions by regulators is based on
heuristic decision-making whose accuracy can be best characterized by
the application of signal detection theory (SDT). Methods We merged the
EMA and FDA database of approvals based on non-RCT comparisons. We
excluded duplicate entries between the two databases. We included a
total of 134 approvals of drugs and devices based on non-RCTs. We
integrated Weber-Fechner law of psychophysics and recognition heuristics
within SDT to provide descriptive explanations of the decisions made by
the FDA and EMA to approve new treatments based on non-randomized
studies without requiring further testing in RCTs. Results Our findings
suggest that when the difference between novel treatments and the
historical control is at least one logarithm (base 10) of magnitude, the
veracity of testing in non-RCTs seems to be established. Conclusion Drug
developers and practitioners alike can use the change in one logarithm
of effect size as a benchmark to decide if further testing in RCTs
should be pursued, or as a guide to interpreting the results reported in
non-randomized studies. However, further research would be useful to
better characterize the threshold of effect size above which testing in
RCTs is not needed.