Reliability of laparoscopic skills assessment on video: 8-year results of the endoscopic surgical skill qualification system in Japan.

Background and Purpose: The Japanese Urological Association and Japanese Society of Endourology established a urologic laparoscopic skills qualification system called the Endoscopic Surgical Skill Qualification (ESSQ) System in Urological Laparoscopy in 2004. The reliability of video assessments by referees was evaluated. Materials and Methods: Videos of nephrectomies or adrenalectomies performed by the applicants were assessed by two referees selected among a pool of 42 referees. From 2004 to 2011, 1308 urologists applied and 60.2% were qualified after video assessments. The results of skills assessments on 1220 videos that had fixed points by two referees were analyzed statistically. Results: The average number of videos that each referee assessed was 58.1, with a range of 16 to 87. The accordance rate of the results of the video assessment, pass or fail, by the two referees was 68.9%. The scores of the video assessment by each referee averaged 62.7 +/- 2.4 (standard deviation) (full score was set at 75 points and >= 60 points was needed to pass). There was a statistically significant difference in the average video assessment score among the referees (P<0.001), and five referees showed significantly higher or lower average scores than the other referees. The percentage qualification of the final decision made by the Referee Committee on the videos originally assessed by each referee showed no significant differences among the 42 referees. The accordance rate of the results from the video assessment by each referee with the final decision by the committee showed a statistically significant positive correlation with the number of videos assessed by each referee (r=0.404, P=0.0080). Conclusions: The ESSQ system showed moderate reliability for the video assessments by the referees. It was concluded that the video assessments by the referees were fair for all applicants, because the final qualification rates showed no significant differences among the referees.