Evaluating Improvement in Situation Awareness and Decision-Making Through Automation : (Poster)

Automated systems such as information extraction (IE) pipelines are designed to facilitate situation awareness by providing human decision makers with relevant information, but beyond the validity of the pipeline itself, designing the output of the pipeline for optimal human understanding should be a goal. This paper presents results comparing comprehension of text documents with and without markup from a (simulated) IE pipeline in a simulated intelligence task. While previous work suggests that markup hurts both objective and subjective measures of performance and preference, this paper uses handgenerated markup designed to be maximally accurate and task relevant, finding more favorable results. These results, however, still point toward the limitations of markup and the importance of the task it is intended to facilitate.