Tool to assess risk of bias due to missing evidence in network meta-analysis (ROB-MEN): elaboration and examples

Selective outcome reporting and publication bias threaten the validity of systematic reviews and meta-analysis and ultimately can affect clinical decision-making. A rigorous methodology to evaluate the impact of this bias on the meta-analysis results of a network of interventions is still lacking. We present a tool to assess the Risk Of Bias due to Missing Evidence in Network meta-analysis (ROB-MEN) by expanding the methods previously developed for pairwise meta-analysis (ROB-ME, http://www.riskofbias.info). ROB-MEN first evaluates the risk of bias due to missing evidence for each pairwise comparison separately. This step considers possible bias due to the presence of studies with unavailable results ('known unknowns') and the potential for unpublished studies ('unknown unknowns'). The second step combines the overall judgements about the risk of bias due to missing evidence in pairwise comparisons with the percentage contribution of direct comparisons on the NMA estimates, the presence or absence of small-study effects, as evaluated by network meta-regression, and any bias from unobserved comparisons. Then, a level of "low risk", "some concerns" or "high risk" for the bias due to missing evidence is assigned to each NMA estimate, which is our tool's final output. We describe the methodology of ROB-MEN step-by-step using an illustrative example from a published NMA of non-diagnostic modalities for the detection of coronary artery disease in patients with low risk acute coronary syndrome. We also report a full application of the tool on a larger and more complex published network of 18 drugs from head-to-head studies for the acute treatment of adults with major depressive disorder. The ROB-MEN tool is the first tool for evaluating the risk of bias due to missing evidence in NMA and it is applicable to networks of all sizes and geometry.