Tutorial Proposal: Opening the Black Box of Interaction in Visualization

Interaction (or human-computer interaction/HCI) is a key ingredient of modern visualization and visual analysis systems. It allows the user to manage the data and to explore its different aspects, as well as to shape its visual representation and to observe it from different perspectives – ultimately to pursue the user’s analytical goal. Yet, so far interaction is perceived from either of three different angles: through interaction activities (e.g., visualization tasks or input events), through interaction architectures (e.g., model-viewcontroller), or through interaction metaphors (e.g., direct manipulation). This 1/2-day tutorial for beginning audiences aims to bring these three perspectives on interaction together by first detailing the state-of-the-art for each of them and then putting them in the context of each other. This will be accompanied by a discussion of real world examples – i.e., actual interactive visualization techniques and systems – to highlight the benefits and challenges of such a comprehensive view on interaction. After completing this tutorial, participants can expect to have gained an extensive overview of interaction in visualization that will allow them to consider all three of the perspectives when designing and evaluating interactions. 1 TUTORIAL MOTIVATION While on the one hand it seems hard to overestimate interaction [10, 62], on the other hand a clear understanding of its inner functions remain a grand challenge in visualization [41, 56]. As a result of this standing challenge, there exist many interaction taxonomies, however there is no single agreed-upon reference model for the interaction process – a lack that is dealt with by the majority of publications on this subject in one of three ways: 1. Functionally through interaction activities that do not relate to an encompassing model of interaction, but are considered operations in their own right – e.g., data level tasks (annotate, filter, etc.) [14, 44] or hardware level events (pinch, click, etc.) [46, 53]. 2. Procedurally through interaction architectures that relate to a system’s internal processing model as a proxy for describing interaction mechanisms – e.g., interaction pipelining approaches [29], multi-threaded event handling [38], or the model-view-controller (MVC) pattern [30]. 3. Figuratively through interaction metaphors that relate to a user’s mental model as a proxy for describing interaction mechanisms – e.g., direct manipulation [49] or instrumental interaction [9]. All three of these perspectives are valid and valuable: An interaction designer would probably take on the figurative, user-centric view (the H in HCI), whereas a software engineer may rather think in terms of the procedural, system-centric view (the C in HCI), and a data analyst is most likely to perceive her task as a purely functional activity on the data that is invoked via input events (the I in HCI). Yet, in order to understand and describe interaction comprehensively, one needs to consider and to incorporate all three of these perspectives. This tutorial will enable its participants to do so. ∗e-mail: hjschulz@informatik.uni-rostock.de †e-mail: tatiana.von landesberger@gris.informatik.tu-darmstadt.de ‡e-mail: do@minik.us 2 TUTORIAL CONTENTS To achieve this aim, we discuss the state-of-the-art in all three of these aspects, while at the same time taking a fresh look at each of them by step-wise putting them into the context of each other. This is done in a bottom-up manner starting with a first part about the interaction activities as the functional building blocks that constitute interaction on the different levels of detail (hardware level, data level). These interaction activities are put together in a second part on interaction architecture that describes their interplay in forming the flow of interaction. Finally, a third part on interaction metaphors and guidelines discusses which of the activities to use in concert with which architectures and processes, so that the resulting interaction with the visualization becomes fluent and coherent. These three parts of the tutorial are outlined in the following. 2.1 PART I: Interaction Activities Interaction activities are usually thought of as being hierarchically organized. Depending on the different levels of granularity and the extent of this hierarchy, it can stretch from high-level cognitive activities to low-level perceptions [47], or from tasks to events [24]. Most research has been conducted on tasks (annotate, filter, etc.) [14, 44] that are related to the visualized data they shall be performed on [50, 65], and on events (pinch, click, etc.) [46, 53] that relate to an input device they are performed with [17, 20]. As a result, many task taxonomies for specific data types exist – for example, for graphs [32], for time-varying data [2], and for timevarying graphs [1]. Concrete examples discussed in this part will include, but not be limited to the interactive visualizations mentioned in [64, Sec.4], as well as those shown in [60]. The hierarchical relation between the data-level tasks and the hardware-level events is apparent, as a task is triggered by or composed of a number of input events. Yet there is not only a hierarchical relation of these levels, but also a procedural one: the user (sender) having a data-level task in mind has to encode this task into a series of hardware-level events (signals) that are sent through a number of input devices (channels) to the computer (receiver). So by thinking about interaction in these terms, its correspondence to a communication process becomes apparent. We will use this analogy between interaction and communication to discuss the challenges of interaction design that stem from it: How to encode the huge number of possible tasks onto the limited set of possible input events? (complex UI + minimal interaction vs. minimal UI + complex interaction [31]) How to deal with noise in this communications process? (magnification lenses to enlarge a target area [39], snap to grid mechanisms [12], etc.) Why is it so hard to define standard interaction vocabularies? [22] And which endeavors have been made in this direction? [11] 2.2 PART II: Interaction Architecture Interaction architecture and process models describe how the interaction is processed by the system. This differs from a workflow that concatenates tasks into an interaction sequence, as it models the flow of interaction from input/action to output/reaction through the system, including any effects it has on the system. These models are either abstract, on the level of interaction logic or on the technical level of the software implementation that realizes the interaction logic. For example, the simple activity of selection alone can behave according to thousands of different selection logics [63]. On the level of interaction logic, approaches exist to model the interaction process using various notations, such as the Unified Modeling Language (UML), in particular Sequence Diagrams [36] or the User Action Notation (UAN) [26] in combination with Task Transition Diagrams (TTD) [52]. Numerous others are surveyed in [19]. As it has recently been shown by the multilevel interaction model [40], it is even possible to include the hierarchical nature of interaction activities (cf. Part I) in such process models, effectively partitioning it into several interconnected models: the goal model, the behavior model, and the operation model. For the domain of visualization, recent work aimed to integrate interaction into the wide-spread pipeline model for visualization [29]. On the level of software implementation, the processing of interaction is commonly captured by design patterns, such as the modelview-controller pattern [30] and/or the data-context-interaction paradigm [18]. To not only provide the prospect to interact, but also the actual ability to do so, architectures such as multi-threaded event handling [38] aim to ensure a system’s reactiveness. While the software design patterns have been readily embraced by practitioners, their relation to and the benefits of the more formal models on the level of interaction logic often remain unclear [51]. Hence, we will discuss these relations and their benefits – including the need for an underlying model in order to capture interaction history and analytical provenance [37], as well as for dealing with issues of multitasking and interruption [35]. 2.3 PART III: Interaction Metaphors and Guidelines Interaction metaphors propose an understanding of interaction through a meaningful and non-technical model that can serve users and developers of interaction [28]. Using such a metaphor as a pointer towards which combination of actions and interaction logic will produce a coherent set of interactions, shaping the interaction space [48] in a way that “makes sense” to the user. Common examples include direct manipulation [49] that is performed immediately on the objects to be changed, and instrumental interaction [9] that is performed through an intermediate, such as a tool or an instrument. Lesser known metaphors are direct combination [27] and surface interaction [58]. Yet, it is neither always clear how to map all desired activities onto metaphors, nor how to map all functionality they suggest/demand onto the given interaction process architecture. Model-based approaches, such as [3], have been developed to aid in this mapping and to uncover mapping errors. As intuitive as metaphors are, as restricting they can be. Hence, often a number of interaction design guidelines are given instead to constrain the design of interaction in a way that also produces coherence and ease of use. Most common is the notion of “fluid” interaction [21] that minimizes interruptions (as discussed at the end of Part II) and aims to facilitate a distraction-free flow-like experience for the user [23]. These guidelines can also be thought of as a higher-level concept than concrete metapho