The risk of bias in systematic reviews (ROBIS) tool showed fair reliability and good construct validity.

There is a movement from generic quality checklists towards a more domain based approach in critical appraisal tools. This study aims to report on a first experience with the newly developed risk of bias in systematic reviews (ROBIS) tool and compare it with A Measurement Tool to Assess Systematic Reviews (AMSTAR) that is the most common used tool to assess methodological quality of systematic reviews while assessing validity, reliability and applicability.Validation study with four reviewers based on 16 systematic reviews in the field of occupational health.IRR of all four raters was highest for domain 2 (Fleiss' kappa κ=0.56) and lowest for domain 4 (κ=0.04). For ROBIS median IRR was κ= 0.52 (range 0.13-0.88) for the experienced pair of raters compared to κ= 0.32 (range 0.12-0.76) for the less experienced pair of raters. The percentage of "yes" scores of each review of ROBIS ratings was strongly correlated with the AMSTAR ratings (rs= 0.76; p= 0.01).ROBIS has fair reliability and good construct validity to assess the risk of bias in systematic reviews. More validation studies are needed to investigate reliability and applicability, in particular.