282. validity and reliability of crowd-sourced and expert video assessment in minimally invasive esophagectomy using global rating scales

Abstract Minimally Invasive Esophagectomy (MIE) is a complex and technically demanding procedure and implementation is associated with a substantial learning curve. Suboptimal surgical performance in complex minimally invasive procedures has convincingly been related to less favorable patient outcomes. Structured video-based assessment by other surgeons can objectively evaluate surgical performance, but requires a surgeon’s scarce time. This study investigated whether crowdsourcing, collective surgical performance evaluation by anonymous and untrained laypeople, could be a valid alternative for MIE. Two video assessments types were conducted via the C-SATS platform with eight MIE operative videos. First, videos were cut into eight procedural phase-based video clips of maximum ten minutes. These 64 video clips were assessed with the validated Global Operative Assessment of Laparoscopic Skills (GOALS)‘s four domains (depth perception, bimanual dexterity, efficiency and tissue handling) using a 1-5 Likert scale. Each video clip was assessed by a pair of two MIE experts and > 30 crowd workers, recruited via Amazon Mechanical Turk. Second, the same expert pairs assessed the corresponding complete video with the procedure-specific MIE competency assessment tool (MIE-CAT). Experts assessed 128 clips and sixteen complete videos in 3.5 months. Crowd workers assessed 1984 clips in < two days. Expert scores showed a wider range (expert GOALS score range 7.50-19.50 versus 12.84-15.63 for crowds). When comparing experts and crowd workers scores, a poor correlation was found on all domains and total GOALS scores (ICC ≤ 0.42). When correlating GOALS scores with MIE-CAT scores a strong, positive correlation between experts’ scores was found (r¬s = 0.76, p = 0.03) while between crowd’s GOALS and expert’s MIE-CAT scores a small correlation was found (r¬s = 0.23, p = 0.58) (Figure 1). Despite rapid results from crowd workers, a poor agreement between crowd and experts’ GOALS scores was observed. MIE might be technically too difficult to assess for laypeople. Furthermore, GOALS might not be extensive enough to assess MIE performance in detail, while the procedure-specific MIE-CAT might be more comprehensive. However, the correlation between the experts’ MIE-CAT and GOALS scores indicates video clip assessments (instead of full length video assessments) could be helpful to shorten assessment time.