Can we rely on non-randomised studies? Findings from a meta-epidemiological review

Abstract Background Increasingly, health technology assessment (HTA) agencies must decide whether new medicines should be used routinely in the absence of randomised controlled trial (RCT) data, relying solely on non-randomised studies (NRS), which are at high risk of bias due to confounding. Against the background of increased availability and improved methods to analyse non-randomised data (e.g., propensity score methods and instrumental variables), it is important for decision-makers to have guidance on the analysis and interpretation of NRS to inform health economic evaluation. We therefore aimed to systematically and empirically assess the performance of NRS using different analytical methods as compared to RCTs and develop recommendations on the basis of our findings. Methods We conducted a large-scale meta-epidemiological review to obtain estimates of the discrepancy in treatment effects in matched RCTs and NRS of pharmacologic interventions from published meta-analyses indexed in MEDLINE and the Cochrane Database of Systematic Reviews. We also consulted with HTA bodies, regulators and academics from five European countries to learn from their experience with using non-randomised evidence. Results We compiled the largest dataset of clinical topics with matching RCTs and NRS using various analytical methods to date, covering >100 unique clinical questions. Incorporating information on direction of effect and effect size from >700 unique studies, the dataset can be used to evaluate discrepancies in treatment effects between study designs across a wide range of therapeutic areas. Conclusions An empirically based understanding of the risk of bias in NRS is required in order to promote the adequate use of non-randomised evidence as input for health economic decision-making.