Snap-To-It: A User-Inspired Platform For Opportunistic Device Interactions

The ability to quickly interact with any nearby appliance from a mobile device would allow people to perform a wide range of one-time tasks (e.g., printing a document in an unfamiliar office location). However, users currently lack this capability, and must instead manually configure their devices for each appliance they want to use. To address this problem, we created Snap-To-It, a system that allows users to opportunistically interact with any appliance simply by taking a picture of it. Snap-To-It shares the image of the appliance a user wants to interact with over a local area network. Appliances then analyze this image (along with the user's location and device orientation) to see if they are being "selected," and deliver the corresponding control interface to the user's mobile device. Snap-To-It's design was informed by two technology probes that explored how users would like to select and interact with appliances using their mobile phone. These studies highlighted the need to be able to select hardware and software via a camera, and identified several novel use cases not supported by existing systems (e.g., interacting with disconnected objects, transferring settings between appliances). In this paper, we show how Snap-To-It's design is informed by our probes and how developers can utilize our system. We then show that Snap-To-It can identify appliances with over 95.3% accuracy, and demonstrate through a two-month deployment that our approach is robust to gradual changes to the environment.