Authoring sensor-based interactions by demonstration with direct manipulation and pattern recognition

CHI, pp. 145-154, 2007.

Cited by: 184|Bibtex|Views36|Links
EI
Keywords:
appropriate hardwaresensor input datainteraction designapplication logicpattern recognitionMore(9+)
Weibo:
This paper introduced techniques for authoring sensorbased interactions through programming by demonstration, where the crucial generalization step is user-editable through direct manipulation of thresholding and pattern matching

Abstract:

Sensors are becoming increasingly important in interaction design. Authoring a sensor-based interaction comprises three steps: choosing and connecting the appropriate hardware, creating application logic, and specifying the relationship between sensor values and application logic. Recent research has successfully addressed the first two i...More

Code:

Data:

0
Introduction
  • Sensing technologies are becoming pervasive, and sensor hardware is increasingly diverse and economical.
  • Existing visual tools (e.g., LabView [2]) were created with the intent of helping engineers and scientists to perform signal analysis; as such, they do not directly support interaction design.
  • This leaves users with a sizeable gulf of execution, the gap between their goals and the actions needed to attain those goals with the system [21].
  • One of the contributions of direct manipulation and WYSIWYG design tools for graphical interfaces is that they enable this “thinking through doing”—this paper’s research aims to provide a beneficial experience for sensor-based interaction
Highlights
  • Sensing technologies are becoming pervasive, and sensor hardware is increasingly diverse and economical
  • Our experience of first learning and later teaching physical interaction design, and our experience of deploying d.tools [19] in the classroom, has shown that specifying the relationship between sensor input and application logic remains problematic for designers for three reasons
  • We chose graphical instead of physical output since our study focused on authoring responses to sensor input, not on actuation
  • We introduced the set of available sensors, which comprised buttons, switches, capacitive touch sensors, light sensors, infrared distance rangers, resistive position sensors, forcesensitive resistors (FSRs), load cells, bend sensors, 2D joysticks, and 3D accelerometers
  • This paper introduced techniques for authoring sensorbased interactions through programming by demonstration, where the crucial generalization step is user-editable through direct manipulation of thresholding and pattern matching
Results
  • In the post-test survey, participants ranked Exemplar highly for decreasing the time required to build prototypes compared to their prior practice; for facilitating rapid modification; for enabling them to experiment more; and for helping them understand user experience.
  • Implementation composed less than a third of their design time
  • This rapid iteration enabled participants to explore up to four different control schemes for a game.
  • The authors see this as a success of enabling epistemic activity: participants spent their time on design thinking rather than implementation tinkering.
  • The absence of complaints by experts in the post-test surveys provides some support for the first hypothesis
Conclusion
  • This paper introduced techniques for authoring sensorbased interactions through programming by demonstration, where the crucial generalization step is user-editable through direct manipulation of thresholding and pattern matching.
  • Exemplar has been released as open source under the BSD license
Summary
  • Introduction:

    Sensing technologies are becoming pervasive, and sensor hardware is increasingly diverse and economical.
  • Existing visual tools (e.g., LabView [2]) were created with the intent of helping engineers and scientists to perform signal analysis; as such, they do not directly support interaction design.
  • This leaves users with a sizeable gulf of execution, the gap between their goals and the actions needed to attain those goals with the system [21].
  • One of the contributions of direct manipulation and WYSIWYG design tools for graphical interfaces is that they enable this “thinking through doing”—this paper’s research aims to provide a beneficial experience for sensor-based interaction
  • Results:

    In the post-test survey, participants ranked Exemplar highly for decreasing the time required to build prototypes compared to their prior practice; for facilitating rapid modification; for enabling them to experiment more; and for helping them understand user experience.
  • Implementation composed less than a third of their design time
  • This rapid iteration enabled participants to explore up to four different control schemes for a game.
  • The authors see this as a success of enabling epistemic activity: participants spent their time on design thinking rather than implementation tinkering.
  • The absence of complaints by experts in the post-test surveys provides some support for the first hypothesis
  • Conclusion:

    This paper introduced techniques for authoring sensorbased interactions through programming by demonstration, where the crucial generalization step is user-editable through direct manipulation of thresholding and pattern matching.
  • Exemplar has been released as open source under the BSD license
Related work
  • The research embodied in Exemplar was directly motivated by our experience with d.tools [19], which introduced a value entry by demonstration technique: in a visualization of a single sensor's signal, a user could copy the latest value into the threshold property of a state transition with a key press. Exemplar extends the d.tools work by introducing:

    • Direct manipulation techniques to control generalization from examples.

    • Pattern matching for complex signals with graphical editing of matching criteria.

    • Graphical feedback of matching criteria for sensor histories.

    • A raised ceiling by working with multiple sensors, multiple events, and extensible filters.

    • Evaluation through a lab study and CDN analysis. Exemplar also relates to three larger areas of work: research into programming by demonstration for ubiquitous computing, tools for musical controller design, and signal processing and analysis software. We discuss each in turn. Ubicomp PBD

    The closest predecessor to Exemplar in approach and scope is a CAPella [14]. This system focused on authoring binary context recognizers by demonstration (e.g., is there a meeting going on in the conference room?), through combining data streams from discrete sensors, a vision algorithm, and microphone input. Exemplar shares inspiration with a CAPella, but it offers important architectural contributions beyond this work. First, a CAPella was not a real-time interactive authoring tool: the authors of a CAPella reported the targeted iteration cycle to be on the order of days, not minutes as with Exemplar. Also, a CAPella did not provide strong support for continuous data. More importantly, a CAPella did not offer designers control over how the generalization step of the PBD algorithm was performed beyond marking regions. We believe that this limitation was partially responsible for the low recognition rates reported (between 50% and 78.6% for binary decisions).
Funding
  • We thank Wendy Ju for illustrating Figures 5 and 7, Intel for donating PCs, and MediaX/DNP for funding
Reference
  • 1 Arduino Physical Computing Platform, 2006. http://www.arduino.cc
    Findings
  • 2 LabView, 2006. National Instruments. http://www.ni.com/labview
    Findings
  • 3 SWT: The Standard Widget Toolkit, 2006. The Eclipse Foundation. http://www.eclipse.org/swt
    Findings
  • 4 Virtual Studio Technology (VST), 2006. Steinberg. http://steinbergcanada.com/technology/vst.htm
    Findings
  • 5 Andreae, P., Justified Generalization: Acquiring Procedures from Examples, Unpublished PhD Thesis, MIT, Dept. of Electrical Engineering and Computer Science, 1985.
    Google ScholarFindings
  • 6 Avrahami, D. and S. E. Hudson. Forming interactivity: A tool for rapid prototyping of physical interactive products. In Proceedings of DIS: ACM Conference on Designing Interactive Systems. pp. 141–46, 2002.
    Google ScholarLocate open access versionFindings
  • 7 Ballagas, R., M. Ringel, M. Stone, and J. Borchers. iStuff: A physical user interface toolkit for ubiquitous computing environments. In Proceedings of CHI: ACM Conference on Human Factors in Computing Systems. pp. 537–44, 2003.
    Google ScholarLocate open access versionFindings
  • 8 Barragan, H., Wiring: Prototyping Physical Interaction Design, Interaction Design Institute Ivrea, Italy, 2004.
    Google ScholarFindings
  • 9 Bellotti, V., M. Back, W. K. Edwards, R. E. Grinter, A. Henderson, and C. Lopes. Making sense of sensing systems: five questions for designers and researchers. In Proceedings of CHI: ACM Conference on Human Factors in Computing Systems. pp. 415–422, 2002.
    Google ScholarLocate open access versionFindings
  • 10 Bevilacqua, F., R. Müller, and N. Schnell, MnM: a Max/MSP mapping toolbox. In Proceedings of NIME: Conference on New Interfaces for Musical Expression. pp. 85–88, 2005.
    Google ScholarLocate open access versionFindings
  • 11 Blackwell, A. and T. Green, A Cognitive Dimensions Questionnaire, 2000. http://www.cl.cam.ac.uk/~afb21/ CognitiveDimensions/CDquestionnaire.pdf
    Findings
  • 12 Card, S. K., J. D. Mackinlay, and G. G. Roebertson. A morphological analysis of the design space of input devices. ACM Trans. Inf. Systems. 9(2): ACM Press. pp. 99–122, 1991.
    Google ScholarLocate open access versionFindings
  • 13 Cypher, A., ed. Watch What I Do - Programming by Demonstration. MIT Press: Cambridge, MA. 652 pp., 1993.
    Google ScholarFindings
  • 14 Dey, A. K., R. Hamid, C. Beckmann, I. Li, and D. Hsu. a CAPpella: Programming by demonstration of context-aware applications. In Proceedings of CHI: ACM Conference on Human Factors in Computing Systems. pp 33–40, 2004.
    Google ScholarLocate open access versionFindings
  • 15 Fails, J. and D. Olsen. A design tool for camera-based interaction. In Proceedings of CHI: ACM Conference on Human Factors in Computing Systems. pp 449–456, 2003.
    Google ScholarLocate open access versionFindings
  • 16 Green, T. R. G. Cognitive dimensions of notations. People and Computers V. pp. 443-60, 1989.
    Google ScholarFindings
  • 17 Green, T. R. G. and M. Petre. Usability Analysis of Visual Programming Environments: A ‘Cognitive Dimensions’ Framework. Journal of Visual Languages and Computing 7(2). pp. 131–74, 1996.
    Google ScholarLocate open access versionFindings
  • 18 Greenberg, S. and C. Fitchett. Phidgets: Easy development of physical interfaces through physical widgets. In Proceedings of UIST: ACM Symposium on User Interface Software and Technology. pp. 209–18, 2001.
    Google ScholarLocate open access versionFindings
  • 19 Hartmann, B., S. R. Klemmer, M. Bernstein, L. Abdulla, B. Burr, A. Robinson-Mosher and J. Gee. Reflective physical prototyping through integrated design, test, and analysis. In Proceedings of UIST 2006: ACM Symposium on User Interface Software and Technology. pp. 299–308, 2006.
    Google ScholarLocate open access versionFindings
  • 20 Hudson, S. and J. Mankoff. Rapid Construction of Functioning Physical Interfaces from Cardboard, Thumbtacks, Tin Foil and Masking Tape. In Proceedings of UIST: ACM Symposium on User Interface Software and Technology. pp 289–298, 2006.
    Google ScholarLocate open access versionFindings
  • 21 Hutchins, E. L., J. D. Hollan, and D. A. Norman. Direct Manipulation Interfaces. Human-Computer Interaction 1(4). pp. 311–38, 1985.
    Google ScholarLocate open access versionFindings
  • 22 Klemmer, S. R., B. Hartmann, and L. Takayama. How Bodies Matter: Five Themes for Interaction Design. In Proceedings of DIS: ACM Conference on Designing Interactive Systems. pp. 140–149, 2006.
    Google ScholarLocate open access versionFindings
  • 23 Klemmer, S. R., J. Li, J. Lin, and J. A. Landay. Papier-Mâché: Toolkit Support for Tangible Input. In Proceedings of CHI: ACM Conference on Human Factors in Computing Systems. pp. 399–406, 2004.
    Google ScholarLocate open access versionFindings
  • 24 Klemmer, S. R., A. K. Sinha, J. Chen, J. A. Landay, N. Aboobaker, and A. Wang. SUEDE: A Wizard of Oz Prototyping Tool for Speech User Interfaces. In Proceedings of UIST: ACM Symposium on User Interface Software and Technology. pp. 1–10, 2000.
    Google ScholarLocate open access versionFindings
  • 25 Li, Y. and J. A. Landay. Informal prototyping of continuous graphical interactions by demonstration. In Proceedings of UIST: ACM Symposium on User Interface Software and Technology. pp. 221–30, 2005.
    Google ScholarLocate open access versionFindings
  • 26 Lieberman, H., ed. Your Wish is my Command. Morgan Kaufmann. 416 pp., 2001.
    Google ScholarFindings
  • 27 Lieberman, H., F. Paternò, and V. Wulf, ed. End-User Development. Springer: Dordrecht, Netherlands. 492 pp., 2006.
    Google ScholarFindings
  • 28 Merrill, D. J. and J. A. Paradiso, Personalization, Expressivity, and Learnability of an Implicit Mapping Strategy for Physical Interfaces. In Extended Abstracts of CHI: ACM Conference on Human Factors in Computing Systems. 2005.
    Google ScholarLocate open access versionFindings
  • 29 Munch, S., J. Kreuziger, M. Kaiser, and R. Dillmann. Robot programming by demonstration – using machine learning and user interaction methods for the development of easy and comfortable robot programming systems. In Proceedings of International Symposium on Industrial Robots. 1994.
    Google ScholarLocate open access versionFindings
  • 30 Nardi, B. A., A Small Matter of Programming: Perspectives on End User Computing. Cambridge, MA: MIT Press. 1993.
    Google ScholarFindings
  • 31 Norman, D. A., Things that make us smart: Defending human attributes in the age of the machine: Addison Wesley Publishing Company. 290 pp., 1993.
    Google ScholarFindings
  • 32 Puckette, M., Pd, 2006. http://crca.ucsd.edu/~msp/software.html
    Findings
  • 33 Rogers, Y. and H. Muller. A framework for designing sensorbased interactions to promote exploration and reflection in play. International Journal of Human-Computer Studies 64(1). pp. 1–14, 2006.
    Google ScholarLocate open access versionFindings
  • 34 Sakoe, H. and S. Chiba. Dynamic programming algorithm optimization for spoken word recognition. IEEE Transactions on Acoustics, Speech, and Signal Processing 26(1): IEEE. pp. 43–49, 1978.
    Google ScholarLocate open access versionFindings
  • 35 Selker, T., A bike helmet built for road hazards, 2006. http://news.com.com/2300-1008_3-6111157-2.html
    Findings
  • 36 Shneiderman, B., Overview + Detail, in Readings in Information Visualization, S.K. Card, J.D. Mackinlay, and B. Shneiderman, ed. Morgan Kaufmann, 1996.
    Google ScholarFindings
  • 37 Steiner, H.-C., [hid] toolkit: a unified framework for instrument design. In Proceedings of NIME: Conference on New Interfaces for Musical Expression. pp. 140-143, 2005.
    Google ScholarLocate open access versionFindings
  • 38 Tufte, E. R., Envisioning Information. Cheshire, CT: Graphics Press LLC. 126 pp. 1990.
    Google ScholarFindings
Your rating :
0

 

Best Paper
Best Paper of CHI, 2007
Tags
Comments