Trustworthiness assessment as an inclusion criterion for systematic
reviews -- what is the impact on results?
Abstract
BACKGROUND There is increasing concern that a significant proportion of
randomised controlled trials (RCTs) included in Cochrane reviews may not
be trustworthy. Applying a trustworthiness screening tool (TST) has
already had a clinically important effect on several reviews published
by the Cochrane Pregnancy and Childbirth Group. OBJECTIVES We wanted to
assess the impact of removing untrustworthy RCTs from already- published
Cochrane reviews on a defined clinical area (ante- and post-natal
nutritional interventions). METHODS We applied the tool to 18 Cochrane
reviews (375 RCTs). The tool had four domains: i) is the research
governance trustworthy; ii) are the baseline characteristics
trustworthy; iii) is the study feasible; iv) are the results
plausible?). When additional information was needed, authors were
contacted using a standard template. At least two attempts were made to
contact the authors. At the end of the evaluation process each study was
classified as: i) included (YES to all domains); ii) excluded (retracted
study); or iii) awaiting classification (any NO to the TST questions).
RESULTS 95/375 studies (25%) were removed, affecting 14/18 (78%)
reviews. 13/18 reviews (72%) showed a difference in the Summary of
Findings tables (direction and size of effects and/or GRADE ratings).
6/18 Cochrane reviews (33%) were judged to require updating because of
important differences in either in their conclusions, implication for
practice, and/or implication for research. CONCLUSIONS Formal assessment
of trustworthiness and inclusion only of studies that satisfy
prespecified criteria for trustworthiness affect conclusions in a
relatively large number of Cochrane reviews, with potentially important
clinical implications for practice and research. The lack of consensus
regarding the best tool(s) for assessing trustworthiness cannot be an
excuse for ignoring this issue in future Cochrane reviews.