Risk of bias assessment of sequence generation: a study of 100 systematic reviews of trials

Background Systematic reviews of randomised trials guide policy and healthcare decisions. Yet, we observed that some reviews judge randomised trials as high or unclear risk of bias (ROB) for sequence generation, potentially introducing bias. However, to date, the extent of this issue has not been well examined. We evaluated the consistency in the ROB assessment for sequence generation of randomised trials in Cochrane and non-Cochrane reviews, and explored the reviewers’ judgement of the quality of evidence for the related outcomes. Methods Cochrane intervention reviews (01/01/2017–31/03/2017) were retrieved from the Cochrane Database of Systematic Reviews. We also searched for systematic reviews in ten general medical journals with highest impact factors (01/01/2016–31/03/2017). We examined the proportion of reviews that rated the sequence generation domain as high, low or unclear risk of selection bias. For reviews that had rated any randomised trials as high or unclear risk of bias, we examined the proportion that had assessed the quality of evidence. Results Overall, 100 systematic reviews were included in our analysis. We evaluated 64 Cochrane reviews which comprised of 984 randomised trials; 0.8% ( n = 8) and 52.2% ( n = 514) were rated as high and unclear ROB for sequence generation respectively. We further evaluated 36 non-Cochrane reviews which comprised of 1376 trials; 5.8% ( n = 80) and 39.6% ( n = 545) were rated as high and unclear ROB respectively. Ninety percent ( n = 10) of non-Cochrane reviews which rated randomised trials as high ROB for sequence generation did not report an underlying reason. All Cochrane reviews assessed the quality of evidence (GRADE). For the non-Cochrane reviews, only just over half had assessed the quality of evidence. Conclusion Systematic reviews of interventions frequently rate randomised trials as high or unclear ROB for sequence generation. In general, Cochrane reviews were more transparent than non-Cochrane reviews in ROB and quality of evidence assessment. The scientific community should more strongly promote consistent ROB assessment for sequence generation to minimise selection bias and support transparent quality of evidence assessment. Consistency ensures that appropriate conclusions are drawn from the data.