Rationale, aims, and objectives The current strategy of searching for an effective drug to treat COVID-19 relies mainly on repurposing existing therapies developed to target other diseases. There are currently more than four thousand active studies assessing the efficacy of existing drugs as therapies for COVID-19. The number of ongoing trials and the urgent need for a treatment poses the risk that false-positive results will be incorrectly interpreted as evidence for treatments’ efficacy and a ground for drug approval. Our purpose is to assess the risk of false-positive outcomes by analyzing the mechanistic evidence for the efficacy of exemplary candidates for repurposing, estimate false discovery rate, and discuss solutions to the problem of excessive hypothesis testing. Methods We estimate the expected number of false-positive results and probability of at least one false-positive result under the assumption that all tested compounds have no effect on the course of the disease. Later, we relax this assumption and analyze the sensitivity of the expected number of true-positive results to changes in the prior probability (π) that tested compounds are effective. Finally, we calculate False Positive Report Probability and expected numbers of false-positive and true-positive results for different thresholds of statistical significance, power of studies, and ratios of effective to non-effective compounds. We also review mechanistic evidence for the efficacy of two exemplary repurposing candidates (hydroxychloroquine and ACE2 inhibitors) and assess its quality to choose the plausible values of the prior probability (π) that tested compounds are effective against COVID-19. Results Our analysis shows that, due to the excessive number of statistical tests in the field of drug repurposing for COVID-19 and low prior probability (π) of the efficacy of tested compounds, positive results are far more likely to result from type-I error than reflect the effects of pharmaceutical interventions.

Our purpose is to assess epidemiological agent-based models– or ABMs - of the SARS-CoV-2 pandemic methodologically. The rapid spread of the outbreak requires fast-paced decision-making regarding mitigation measures. However, the evidence for the efficacy of non-pharmaceutical interventions such as imposed social distancing and school or workplace closures is scarce: few observational studies use quasi-experimental research designs, and conducting randomized controlled trials seems infeasible. Additionally, evidence from the previous coronavirus outbreaks of SARS and MERS lacks external validity, given the significant differences in contagiousness of those pathogens relative to SARS-CoV-2. To address the pressing policy questions that have emerged as a result of COVID-19, epidemiologists have produced numerous models that range from simple compartmental models to highly advanced agent-based models. These models have been criticized for involving simplifications and lacking empirical support for their assumptions. In order to address these voices and methodologically appraise epidemiological ABMs, we consider AceMod (the model of the COVID-19 epidemic in Australia) as an example of the modeling practice. Our case study shows that, although epidemiological ABMs involve simplifications of various sorts, the key characteristics of social interactions and the spread of SARS-CoV-2 are represented sufficiently accurately. This is the case because these modelers treat empirical results as inputs for constructing modeling assumptions and rules that the agents follow; and they use calibration to assert the adequacy to benchmark variables. Given this, we claim that the best epidemiological ABMs are models of actual mechanisms and deliver both mechanistic and difference-making evidence. Furthermore, the efficacy claims are not only internally valid but also adequately describe the effects of interventions in the targets of the models. We also discuss the limitations of ABMs and put forward policy recommendations.