Effect of rater instruction and standard area diagram structure on the accuracy of estimates of severity in Calonectria leaf blight

The objectives of this study were to evaluate the effect of rater instruction and two standard area diagrams (SAD), that varied on scale structure, on the accuracy and reliability of visual estimates of calonectria leaf blight (CLB) severity. The minimum and the maximum severities of the newly developed SAD were 0.8% and 90%, respectively, with eight intermediate severity values for the linear SAD (10, 20, 30, 40, 50, 60, 70, 80%) and amended-linear SAD (3, 5, 10, 15, 30, 45, 60, 75%). For validation, one group (21 raters) used the amended-linear SAD and another group (22 raters) used the linear SAD. For each group, three sequential assessments were made: 1) without any aid; 2) following specific instructions; and 3) using the respective SAD. Two raters with the highest overall accuracy and two with significant gains in accuracy after training were chosen, from each SAD group, for further evaluating the accuracy of unaided estimates after SAD use. The linear and amended-linear SAD similarly improved the overall accuracy of the estimates (CCC from 0.93 to 0.96). The overestimation of severity was reduced at the lower end of the scale (< 25% severity) when using the amended-linear SAD. A trend of increasing reliability was observed as raters progressed through the rounds of assessments, but there was no statistical difference. The less accurate raters during unaided estimates from both SAD groups improved their accuracy after the instruction and maintained the same level of accuracy in the two rounds after SAD use. Instruction was critical for inherently less accurate raters. We recommend the amended-linear SAD for assessments of CLB severity, which minimizes the overestimation at the lower severity levels.