Touch and gesture: mediating content display, inscriptions, and gestures across multiple devices

Recent advances in computer design and technology have broadened the range of devices enabled for inscription and touch-based interaction and increased their adoption in collaborative work settings. Since most of the past research has focused on optimal use of individual devices, we now need to expand our understanding of how these devices are used in concert, particularly in collaborative work settings where touch and gesture facilitate communication and may interfere with the touch-based input. We conducted in situ observations of team meetings that involve the use of a tabletop computer, tablet personal computers (tablet PCs) with handwriting support, and a vertical display. The study shows how inscriptions and gestures naturally emerge around the content displayed on the devices and how important it is to maintain their spatial congruence. Furthermore, it reveals that the combination of the tablet PCs and the tabletop computer encourages the use of gestures and touch across devices as part of sense-making. While discussing the content, the users apply sequential and synchronous gestures to bind content and inscriptions across devices. Our observations of binding gestures extend the gesture taxonomies from previous studies and expand the notion of multi-touch beyond individual devices. We stipulate that designing support for touch and gestures across devices requires a holistic approach. Only through coordinated design of touch, inscription, and gesture input and consideration of broader usage scenarios, we can ensure minimal interference with naturally emerging touch and gestures and provide effective mechanism for disambiguating general user behavior from device input actions.