Enhancing Cross-Device Interaction Scripting with Interactive Illustrations

CHI, pp. 5482-5493, 2016.

Cited by: 18|Bibtex|Views42|Links
EI
Keywords:
storyboardsinteractive illustrationform factorentire programmultiple deviceMore(8+)
Weibo:
FrameWire extracts interaction logic from a paper prototype walkthrough and the Designers’ Outpost captures site maps on a smart whiteboard. In contrast to these approaches, we extend the expressiveness of storyboards by introducing a new format based on the unique aspects of cro...

Abstract:

Cross-device interactions involve input and output on multiple computing devices. Implementing and reasoning about interactions that cover multiple devices with a diversity of form factors and capabilities can be complex. To assist developers in programming cross-device interactions, we created DemoScript, a technique that automatically a...More

Code:

Data:

0
Introduction
  • Wearable and mobile devices have introduced new ways for interaction with their unique capabilities and form factors.
  • Contact the Owner/Author.
  • Copyright is held by the owner/author(s).
  • Http://dx.doi.org/10.1145/2858036.2858382 manipulation [29, 32, 37], watch-phone interactions [6, 11, 16], multiple cube-size device gaming [34], and interactive augmented space via smart eyewear [10, 27]
  • CHI'16, May 07-12, 2016, San Jose, CA, USA ACM 978-1-4503-3362-7/16/05. http://dx.doi.org/10.1145/2858036.2858382 manipulation [29, 32, 37], watch-phone interactions [6, 11, 16], multiple cube-size device gaming [34], and interactive augmented space via smart eyewear [10, 27]
Highlights
  • Wearable and mobile devices have introduced new ways for interaction with their unique capabilities and form factors
  • FrameWire extracts interaction logic from a paper prototype walkthrough [24] and the Designers’ Outpost captures site maps on a smart whiteboard [20]. In contrast to these approaches, we extend the expressiveness of storyboards by introducing a new format based on the unique aspects of cross-device interaction
  • If the edited script is executable, the server partially executes Weave code by traversing the abstract syntax tree in order to store information, including device selections and properties. This data is sent in JSON back to the front-end UI, which highlights the code in the Editor, visualizes the storyboard using D3 [5], and handles developer interaction via jQuery2 and Bootstrap3
  • In the original script of task 3, auditory feedback was played before the event is triggered, and the script incorrectly specified a wrong device selection to initiate the application. These errors were visually distinguishable in the storyboard, and all the 4 participants were able to fix the script with DemoScript, whereas 2 of the
  • We evaluated DemoScript with 8 professional programmers and found that it outperformed the baseline condition in many ways to simplify programming tasks for complex interaction behaviors
Methods
  • Participants and Setups The authors recruited

    8 professional programmers (2 females), aged between 21 and 45 years (Mean=26) from an IT company.
  • Participants and Setups The authors recruited.
  • 8 professional programmers (2 females), aged between 21 and 45 years (Mean=26) from an IT company.
  • Participants were required to have moderate JavaScript programming knowledge and were selected randomly from volunteers via an internal study invitation.
  • 5 out of 8 participants had limited mobile programming experience; only 1 had programmed wearable devices.
  • None of them had used the Weave framework.
  • The study was conducted in a laboratory environment.
  • Each participant was compensated with a $40 gift card for their participation in a 90-minute session
Results
  • In the original script of task 3, auditory feedback was played before the event is triggered, and the script incorrectly specified a wrong device selection to initiate the application
  • These errors were visually distinguishable in the storyboard, and all the 4 participants were able to fix the script with DemoScript, whereas 2 of the other 4 participants failed to correct all the errors using the baseline system
Conclusion
  • The authors presented DemoScript, a technique that automatically analyzes and visualizes a cross-device interaction program while it is being written.
  • The authors introduced crossdevice storyboards, a novel visualization for cross-device development.
  • It is closely coupled with scripting by offering step-by-step execution of a selected portion or the entire program.
  • The authors evaluated DemoScript with 8 professional programmers and found that it outperformed the baseline condition in many ways to simplify programming tasks for complex interaction behaviors
Summary
  • Introduction:

    Wearable and mobile devices have introduced new ways for interaction with their unique capabilities and form factors.
  • Contact the Owner/Author.
  • Copyright is held by the owner/author(s).
  • Http://dx.doi.org/10.1145/2858036.2858382 manipulation [29, 32, 37], watch-phone interactions [6, 11, 16], multiple cube-size device gaming [34], and interactive augmented space via smart eyewear [10, 27]
  • CHI'16, May 07-12, 2016, San Jose, CA, USA ACM 978-1-4503-3362-7/16/05. http://dx.doi.org/10.1145/2858036.2858382 manipulation [29, 32, 37], watch-phone interactions [6, 11, 16], multiple cube-size device gaming [34], and interactive augmented space via smart eyewear [10, 27]
  • Methods:

    Participants and Setups The authors recruited

    8 professional programmers (2 females), aged between 21 and 45 years (Mean=26) from an IT company.
  • Participants and Setups The authors recruited.
  • 8 professional programmers (2 females), aged between 21 and 45 years (Mean=26) from an IT company.
  • Participants were required to have moderate JavaScript programming knowledge and were selected randomly from volunteers via an internal study invitation.
  • 5 out of 8 participants had limited mobile programming experience; only 1 had programmed wearable devices.
  • None of them had used the Weave framework.
  • The study was conducted in a laboratory environment.
  • Each participant was compensated with a $40 gift card for their participation in a 90-minute session
  • Results:

    In the original script of task 3, auditory feedback was played before the event is triggered, and the script incorrectly specified a wrong device selection to initiate the application
  • These errors were visually distinguishable in the storyboard, and all the 4 participants were able to fix the script with DemoScript, whereas 2 of the other 4 participants failed to correct all the errors using the baseline system
  • Conclusion:

    The authors presented DemoScript, a technique that automatically analyzes and visualizes a cross-device interaction program while it is being written.
  • The authors introduced crossdevice storyboards, a novel visualization for cross-device development.
  • It is closely coupled with scripting by offering step-by-step execution of a selected portion or the entire program.
  • The authors evaluated DemoScript with 8 professional programmers and found that it outperformed the baseline condition in many ways to simplify programming tasks for complex interaction behaviors
Related work
  • Our work touches on three research topics: enhanced code editors, cross-device IDEs, storyboards in interaction design, and iterating with examples. We here discuss how our work is related to each of these areas.

    Enhanced Code Editors Tools that visualize different aspects of source code and allow direct manipulation of abstractions can successfully lower the complexity of programming [28]. Prior work has attempted to integrate many software visualization techniques into code editors [8]. For example, Codelets shows interactive examples and documentation inside an IDE [31]. Stacksplorer visualizes the structure of call graphs [18]. Beyond enhancing IDEs for general programming tasks, research has also focused specifically on techniques for supporting the development of interactive applications. Most IDEs nowadays ship with a GUI builder to enable developers to graphically layout user interfaces. Xcode further allows developers to create UI sequences for Apple Watch and interactively monitor variables in a program [2, 3]. Juxtapose allows programmers to explore interface alternatives [14] using a linked editing technique [35], and DejaVu helps programmers understand computer vision-based applications that extract interaction logic from camera input [19].
Funding
  • Evaluated DemoScript with 8 professional programmers and found that DemoScript significantly improved development efficiency by helping developers interpret and manage cross-device interaction; it encourages testing to think through the script in a development process
  • Presents DemoScript, a novel technique that assists developers in programming cross-device interaction by interactively illustrating the behavior of the program
  • Focuses on preserving the flexibility and high complexity ceiling of authoring interactions in a general purpose programming language
  • Extends the expressiveness of storyboards by introducing a new format based on the unique aspects of cross-device interaction
Reference
  • Azza Abouzied, Joseph Hellerstein, and Avi Silberschatz. 2012. DataPlay: interactive tweaking and example-driven correction of graphical database queries. In Proc. UIST '12, 207-218.
    Google ScholarLocate open access versionFindings
  • Apple Inc. Swift. 2015. https://developer.apple.com/swift/
    Findings
  • Apple Inc. WatchKit. 2015. https://developer.apple.com/watchkit/
    Findings
  • Ira D. Baxter, Andrew Yahin, Leonardo Moura, Marcelo Sant'Anna, and Lorraine Bier. 1998. Clone Detection Using Abstract Syntax Trees. In Proc. ICSM '98, 368-377.
    Google ScholarLocate open access versionFindings
  • Michael Bostock, Vadim Ogievetsky, and Jeffrey Heer. 2011. D3 Data-Driven Documents. IEEE Transactions on Visualization and Computer Graphics 17, 12, 23012309.
    Google ScholarLocate open access versionFindings
  • Xiang 'Anthony' Chen, Tovi Grossman, Daniel J. Wigdor, and George Fitzmaurice. 2014. Duet: exploring joint interactions on a smart phone and a smart watch. In Proc. CHI '14, 159-168.
    Google ScholarLocate open access versionFindings
  • Pei-Yu (Peggy) Chi and Yang Li. 2015. Weave: Scripting Cross-Device Wearable Interaction. In Proc. CHI '15, 3923-3932.
    Google ScholarLocate open access versionFindings
  • Stephan Diehl. 2007. Software Visualization: Visualizing the Structure, Behaviour, and Evolution of Software. Springer-Verlag New York, Inc., Secaucus, NJ, USA.
    Google ScholarFindings
  • James Fogarty, Desney Tan, Ashish Kapoor, and Simon Winder. 2008. CueFlik: interactive concept learning in image search. In Proc. CHI '08, 29-38.
    Google ScholarLocate open access versionFindings
  • Jens Grubert, Matthias Heinisch, Aaron Quigley, and Dieter Schmalstieg. 2015. MultiFi: Multi Fidelity Interaction with Displays On and Around the Body. In Proc. CHI '15, 3933-3942.
    Google ScholarLocate open access versionFindings
  • Google Inc. Android Wear. 2015. https://www.android.com/wear/
    Findings
  • Google Inc. Chrome Apps. 2015. https://developer.chrome.com/apps/
    Findings
  • Björn Hartmann, Scott R. Klemmer, Michael Bernstein, Leith Abdulla, Brandon Burr, Avi Robinson-Mosher, and Jennifer Gee. 2006. Reflective physical prototyping through integrated design, test, and analysis. In Proc. UIST '06, 299-308.
    Google ScholarLocate open access versionFindings
  • Björn Hartmann, Loren Yu, Abel Allison, Yeonsoo Yang, and Scott R. Klemmer. 2008. Design as exploration: creating interface alternatives through parallel authoring and runtime tuning. In Proc. UIST '08, 91-100.
    Google ScholarLocate open access versionFindings
  • Ariya Hidayat. Esprima: ECMAScript parsing infrastructure for multipurpose analysis. 2015. http://esprima.org/
    Findings
  • Steven Houben and Nicolai Marquardt. 2015. WatchConnect: A Toolkit for Prototyping SmartwatchCentric Cross-Device Applications. In Proc. CHI '15, 1247-1256.
    Google ScholarLocate open access versionFindings
  • Melody Y. Ivory and Marti A Hearst. 2001. The state of the art in automating usability evaluation of user interfaces. ACM Comput. Surv. 33, 4, 470-516.
    Google ScholarLocate open access versionFindings
  • Thorsten Karrer, Jan-Peter Krämer, Jonathan Diehl, Björn Hartmann, and Jan Borchers. 2011. Stacksplorer: call graph navigation helps increasing code maintenance efficiency. In Proc. UIST '11, 217-224.
    Google ScholarLocate open access versionFindings
  • Jun Kato, Sean McDirmid, and Xiang Cao. 2012. DejaVu: integrated support for developing interactive camera-based programs. In Proc. UIST '12, 189-196.
    Google ScholarLocate open access versionFindings
  • Scott R. Klemmer, Mark W. Newman, Ryan Farrell, Mark Bilezikjian, and James A. Landay. 2001. The designers' outpost: a tangible interface for collaborative web site. In Proc. UIST '01, 1-10.
    Google ScholarLocate open access versionFindings
  • James A. Landay and Brad A. Myers. 1996. Sketching storyboards to illustrate interface behaviors. In Proc. CHI '09, 193-194.
    Google ScholarLocate open access versionFindings
  • Brian Lee, Savil Srivastava, Ranjitha Kumar, Ronen Brafman, and Scott R. Klemmer. 2010. Designing with interactive example galleries. In Proc. CHI '10, 22572266.
    Google ScholarLocate open access versionFindings
  • Yang Li and James A. Landay. 2008. Activity-based prototyping of ubicomp applications for long-lived, everyday human activities. In Proc. CHI '08, 13031312.
    Google ScholarLocate open access versionFindings
  • Yang Li, Xiang Cao, Katherine Everitt, Morgan Dixon, and James A. Landay. 2010. FrameWire: a tool for automatically extracting interaction logic from paper prototyping tests. In Proc. CHI '10, 503-512.
    Google ScholarLocate open access versionFindings
  • Henry Lieberman and Christopher Fry. 1995. Bridging the gulf between code and behavior in programming. In Proc. CHI '95, 480-486.
    Google ScholarLocate open access versionFindings
  • James Lin and James A. Landay. 2008. Employing patterns and layers for early-stage design and prototyping of cross-device user interfaces. In Proc. CHI '08, 1313-1322.
    Google ScholarLocate open access versionFindings
  • Microsoft Inc. HoloLens. 2015. https://www.microsoft.com/microsoft-hololens/
    Findings
  • Brad Myers, Scott E. Hudson, and Randy Pausch. 2000. Past, present, and future of user interface software tools. ACM Trans. Comput.-Hum. Interact. 7, 1, 3-28.
    Google ScholarLocate open access versionFindings
  • Michael Nebeling, Theano Mintsi, Maria Husmann, and Moira Norrie. 2014. Interactive development of cross-device user interfaces. In Proc. CHI '14, 27932802.
    Google ScholarLocate open access versionFindings
  • Donald A. Norman and Stephen W. Draper. 1986. User Centered System Design; New Perspectives on Human-
    Google ScholarFindings
  • Stephen Oney and Joel Brandt. 2012. Codelets: linking interactive documentation and example code in the editor. In Proc. CHI '12, 2697-2706.
    Google ScholarLocate open access versionFindings
  • Roman Rädle, Hans-Christian Jetter, Mario Schreiner, Zhihao Lu, Harald Reiterer, and Yvonne Rogers. 2015. Spatially-aware or Spatially-agnostic?: Elicitation and Evaluation of User-Defined Cross-Device Interactions. In Proc. CHI '15, 3913-3922.
    Google ScholarLocate open access versionFindings
  • Daniel Ritchie, Ankita Arvind Kejriwal, and Scott R. Klemmer. 2011. d.tour: style-based exploration of design example galleries. In Proc. UIST '11, 165-174.
    Google ScholarLocate open access versionFindings
  • Sifteo Inc. Sifteo Cubes. 2015. https://www.sifteo.com/
    Findings
  • Michael Toomim, Andrew Begel, and Susan L. Graham. 2004. Managing Duplicated Code with Linked Editing. In Proc. VLHCC '04: the 2004 IEEE Symposium on Visual Languages - Human Centric Computing, 173-180.
    Google ScholarLocate open access versionFindings
  • Khai N. Truong, Gillian R. Hayes, and Gregory D. Abowd. 2006. Storyboarding: an empirical determination of best practices and effective guidelines. In Proc. DIS '06, 12-21.
    Google ScholarLocate open access versionFindings
  • Jishuo Yang and Daniel Wigdor. 2014. Panelrama: enabling easy specification of cross-device web applications. In Proc. CHI '14, 2783-2792.
    Google ScholarLocate open access versionFindings
  • Mark Weiser. 1981. Program slicing. In Proc. ICSE '81: the 5th International Conference on Software engineering, 439-449.
    Google ScholarLocate open access versionFindings
Your rating :
0

 

Best Paper
Best Paper of CHI, 2016
Tags
Comments