Afordance-Based and User-Defined Gestures for Spatial Tangible Interaction

Although mid-air hand gestures have been widely adopted by VR/AR products (e.g., Quest 2 and HoloLens), some drawbacks remain due to their lack of tangibility and tactile feedback. Opportunistic Tangible User Interfaces could address these shortcomings by repurposing existing objects in one's physical environment. However, there has yet to be a systematic investigation of the gestures that would be desirable when using opportunistic objects or how such gestures would be impacted by such objects. In this work, we conducted an elicitation study to investigate the desirability of object and gesture combinations across a variety of interactions. The results contribute (1) an opportunistic tangible UI gesture set for spatial interfaces, and (2) an Affordance-Based Object Selector Scheme that identifies ideal objects for tangible input given a desired input gesture, based on that object's physical affordances. Arising from these findings is the vision of the Adaptive Tangible User Interface, which supports the on-the-fly composition of tangible interfaces based on the affordances found in the physical environment and a user's input task.