What'S Most Broken? Design And Evaluation Of A Tool To Guide Improvement Of An Intelligent Tutor

Intelligent Tutoring Systems (ITS) have great potential to change the educational landscape by bringing scientifically tested one-to-one tutoring to remote and under-served areas. However, effective ITSs are too complex to perfect. Instead, a practical guiding principle for ITS development and improvement is to fix what's most broken. This paper presents SPOT (Statistical Probe of Tutoring), a tool that mines data logged by an ITS to identify 'hot spots' most detrimental to its efficiency and effectiveness in terms of its software reliability, usability, task difficulty, student engagement, and other criteria. SPOT uses heuristics and machine learning to discover, characterize, and prioritize such hot spots in order to focus ITS refinement on what matters most. We applied SPOT to data logged by RoboTutor, an ITS that teaches children basic reading, writing and arithmetic. A panel-of-experts experimental evaluation found SPOT's selected video clips of RoboTutor's hot spots as significantly more informative than video clips selected randomly.