Hand as Sensor: Virtualizing Interactions with Everyday Object Affordances using Wrist-worn Hand Pose Sensing

We are a group of HCI researchers from a capacitive technology company, and we have been collaborating on hardware to enable sensing the posture and anatomy of hands for use in mixed reality. Through this process, we have been exploring the motion and activity of hands; after discovering that our sensor is sensitive not only to dynamic hand pose but also to tendon load (such as is caused by holding or pressing on an object), we began exploring the idea of using passive held or touched objects as support for gestural input. Specifically, we are interested less in the passive haptics of existing objects, and more in their interactive affordances. This paper presents Ubiquitous Controls: a work-in-progress input technique measuring interactions between users’ hands and objects’ affordances powered by a wristband, which senses muscle pose and motion using capacitive-based mechanomyography (MMG). We use this technology to train machine learning models that map continuous object interactions—like “twisting a bottle cap” or “cutting with scissors”—to controls like dials or sliders. Across 10 participants, our models discriminated between 6 object interactions and identified the correct range state with mean accuracies of 85.5–98.5 %. Furthermore, 8 users interactively controlled UI widgets using objects and valued their haptic feedback and enhanced state reproducibility over freehand gestures. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s). EPO4VR’21, May 8–13, 2021, Yokohama, Japan © 2021 Copyright held by the owner/author(s). We also discuss some opportunities and discovered pitfalls of leveraging interactive affordances of everyday objects.