Automation Accuracy Is Good, but High Controllability May Be Better

When automating tasks using some form of artificial intelli- gence, some inaccuracy in the result is virtually unavoidable. In many cases, the user must decide whether to try the auto- mated method again, or fix it themselves using the available user interface. We argue this decision is influenced by both perceived automation accuracy and degree of task “control- lability” (how easily and to what extent an automated result can be manually modified). This relationship between accu- racy and controllability is investigated in a 750-participant crowdsourced experiment using a controlled, gamified task. With high controllability, self-reported satisfaction remained constant even under very low accuracy conditions, and over- all, a strong preference was observed for using manual con- trol rather than automation, despite much slower perfor- mance and regardless of very poor controllability.