Speaker: Shaun Treweek
This is a talk at the International Clinical Trials Methodology Conference held in Harrogate, UK 2nd – 6th October 2022 (https://ictmc.org).
The talk is based on this article: https://trialsjournal.biomedcentral.com/articles/10.1186/s13063-022-06415-5
Abstract
Shaun Treweek (presenter), University of Aberdeen, UK
Stefania Pirosca, University of Aberdeen, UK
Frances Shiely, University College Cork, Ireland
Mike Clarke, Queen’s University Belfast, UK
Introduction
At the 2015 REWARD/EQUATOR conference on research waste, the late Doug Altman revealed that his only regret about his 1994 BMJ paper ‘The scandal of poor medical research’ was that he used the word ‘poor’ rather than ‘bad’. More than 20 years later, how much research is still bad? And what would improve things?
Methods
We focused on randomised trials and considered quality, size and cost. We randomly selected up to two quantitative intervention reviews published by each clinical Cochrane Review Group between May 2020 and April 2021. Data including risk of bias, number of participants, intervention type and country were extracted for all trials included in selected reviews. High risk of bias trials were classed as bad, low risk of bias trials were classed as good, and uncertain risk of bias trials were classed as just that, uncertain. A range for the cost of high risk of bias trials was estimated using three published estimates of trial cost per participant.
Results
We identified 96 eligible reviews authored by 546 reviewers from 49 of the 53 clinical Cochrane Review Groups. These reviews included 1,659 trials done in 84 countries. Of the 1,640 trials for which risk of bias information was available in the review, 1,013 (62%) were high risk of bias (bad), 494 (30%) were unclear (uncertain) and 133 (8%) were low risk of bias (good). Bad trials were spread across all clinical areas. Well over 220,000 participants (or 56% of all participants in the 1,640 trials) were in bad trials. The low estimate of the cost of bad trials was £726 million; our high estimate was over £8 billion.
Discussion
Most randomised trials are bad and most trial participants will be in a bad trial. We make five recommendations: trials should be neither funded (1) or given ethical approval (2) unless they have a statistician and methodologist on the team; trialists should use a risk of bias tool at design (3); more statisticians and methodologists should be trained and supported (4); and more funding should be put into applied methodology research and infrastructure (5).
The research community has tolerated bad trials for decades. This has to stop: we need to put rigour and methodology where it belongs– at the centre of our science.
