HCI Bibliography Home | HCI Conferences | UIST Archive | Detailed Records | RefWorks | EndNote | Hide Abstracts
UIST Tables of Contents: 0708091011-111-212-112-213-113-214-114-215-115-2

Proceedings of the 2014 ACM Symposium on User Interface Software and Technology

Fullname:UIST'14: Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology
Editors:Hrvoje Benko; Mira Dontcheva; Daniel Wigdor
Location:Honolulu, Hawaii
Dates:2014-Oct-05 to 2014-Oct-08
Volume:1
Publisher:ACM
Standard No:ISBN: 978-1-4503-3069-5; ACM DL: Table of Contents; hcibib: UIST14-1
Papers:76
Pages:699
Links:Conference Website | Open Proceedings
  1. UIST 2014-10-05 Volume 1
    1. Opening keynote address
    2. Interacting with 3D data
    3. Working with crowds
    4. Building and using webpages
    5. Novel hardware I
    6. Modeling and prediction
    7. Developer tools I
    8. Fabrication
    9. Augmented reality I
    10. Novel hardware II
    11. Creative tools
    12. Collaboration
    13. Touch & gesture
    14. Video
    15. Input techniques
    16. Augmented reality II
    17. Developer tools II
    18. Closing keynote address

UIST 2014-10-05 Volume 1

Opening keynote address

Designing the user in user interfaces BIBAFull-Text 1
  Mark Bolas
In the good old days, the human was here, the computer there, and a good living was to be made by designing ways to interface between the two. Now we find ourselves unthinkingly pinching to zoom in on a picture in a paper magazine. User interfaces are changing instinctual human behavior and instinctual human behavior is changing user interfaces. We point or look left in the "virtual" world just as we point or look left in the physical.
   It is clear that nothing is clear anymore: the need for "interface" vanishes when the boundaries between the physical and the virtual disappear. We are at a watershed moment when to experience being human means to experience being machine. When there is not a user interface -- it is just what you do. When instinct supplants mice and menus and the interface insinuates itself into the human psyche.
   We are redefining and creating what it means to be human in this new physical/virtual integrated reality -- we are not just designing user interfaces, we are designing users.

Interacting with 3D data

A series of tubes: adding interactivity to 3D prints using internal pipes BIBAFull-Text 3-12
  Valkyrie Savage; Ryan Schmidt; Tovi Grossman; George Fitzmaurice; Björn Hartmann
3D printers offer extraordinary flexibility for prototyping the shape and mechanical function of objects. We investigate how 3D models can be modified to facilitate the creation of interactive objects that offer dynamic input and output. We introduce a general technique for supporting the rapid prototyping of interactivity by removing interior material from 3D models to form internal pipes. We describe this new design space of pipes for interaction design, where variables include openings, path constraints, topologies, and inserted media. We then present PipeDream, a tool for routing such pipes through the interior of 3D models, integrated within a 3D modeling program. We use two distinct routing algorithms. The first has users define pipes' terminals, and uses path routing and physics-based simulation to minimize pipe bending energy, allowing easy insertion of media post-print. The second allows users to supply a desired internal shape to which we fit a pipe route: for this we describe a graph-routing algorithm. We present several prototypes created using our tool to show its flexibility and potential.
FlatFitFab: interactive modeling with planar sections BIBAFull-Text 13-22
  James McCrae; Nobuyuki Umetani; Karan Singh
We present a comprehensive system to author planar section structures, common in art and engineering. A study on how planar section assemblies are imagined and drawn guide our design principles: planar sections are best drawn in-situ, with little foreshortening, orthogonal to intersecting planar sections, exhibiting regularities between planes and contours. We capture these principles with a novel drawing workflow where a single fluid user stroke specifies a 3D plane and its contour in relation to existing planar sections. Regularity is supported by defining a vocabulary of procedural operations for intersecting planar sections. We exploit planar structure properties to provide real-time visual feedback on physically simulated stresses, and geometric verification that the structure is stable, connected and can be assembled. This feedback is validated by real-world fabrication and testing. As evaluation, we report on over 50 subjects who all used our system with minimal instruction to create unique models.
Paper3D: bringing casual 3D modeling to a multi-touch interface BIBAFull-Text 23-32
  Patrick Paczkowski; Julie Dorsey; Holly Rushmeier; Min H. Kim
A 3D modeling system that provides all-inclusive functionality is generally too demanding for a casual 3D modeler to learn. In recent years, there has been a shift towards developing more approachable systems, with easy-to-learn, intuitive interfaces. However, most modeling systems still employ mouse and keyboard interfaces, despite the ubiquity of tablet devices, and the benefits of multi-touch interfaces applied to 3D modeling. In this paper, we introduce an alternative 3D modeling paradigm for creating developable surfaces, inspired by traditional papercrafting, and implemented as a system designed from the start for a multi-touch tablet. We demonstrate the process of assembling complex 3D scenes from a collection of simpler models, in turn shaped through operations applied to sheets of virtual paper. The modeling and assembling operations mimic familiar, real-world operations performed on paper, allowing users to quickly learn our system with very little guidance. We outline key design decisions made throughout the development process, based on feedback obtained through collaboration with target users. Finally, we include a range of models created in our system.
Cheaper by the dozen: group annotation of 3D data BIBAFull-Text 33-42
  Aleksey Boyko; Thomas Funkhouser
This paper proposes a group annotation approach to interactive semantic labeling of data and demonstrates the idea in a system for labeling objects in 3D LiDAR scans of a city. In this approach, the system selects a group of objects, predicts a semantic label for it, and highlights it in an interactive display. In response, the user either confirms the predicted label, provides a different label, or indicates that no single label can be assigned to all objects in the group. This sequence of interactions repeats until a label has been confirmed for every object in the data set. The main advantage of this approach is that it provides faster interactive labeling rates than alternative approaches, especially in cases where all labels must be explicitly confirmed by a person. The main challenge is to provide an algorithm that selects groups with many objects all of the same label type arranged in patterns that are quick to recognize, which requires models for predicting object labels and for estimating times for people to recognize objects in groups. We address these challenges by defining an objective function that models the estimated time required to process all unlabeled objects and approximation algorithms to minimize it. Results of user studies suggest that group annotation can be used to label objects in LiDAR scans of cities significantly faster than one-by-one annotation with active learning.

Working with crowds

Microtask programming: building software with a crowd BIBAFull-Text 43-54
  Thomas D. LaToza; W. Ben Towne; Christian M. Adriano; André van der Hoek
Microtask crowdsourcing organizes complex work into workflows, decomposing large tasks into small, relatively independent microtasks. Applied to software development, this model might increase participation in open source software development by lowering the barriers to contribution and dramatically decrease time to market by increasing the parallelism in development work. To explore this idea, we have developed an approach to decomposing programming work into microtasks. Work is coordinated through tracking changes to a graph of artifacts, generating appropriate microtasks and propagating change notifications to artifacts with dependencies. We have implemented our approach in CrowdCode, a cloud IDE for crowd development. To evaluate the feasibility of microtask programming, we performed a small study and found that a small crowd of 12 workers was able to successfully write 480 lines of code and 61 unit tests in 14.25 person-hours of time.
Situated crowdsourcing using a market model BIBAFull-Text 55-64
  Simo Hosio; Jorge Goncalves; Vili Lehdonvirta; Denzil Ferreira; Vassilis Kostakos
Research is increasingly highlighting the potential for situated crowdsourcing to overcome some crucial limitations of online crowdsourcing. However, it remains unclear whether a situated crowdsourcing market can be sustained, and whether worker supply responds to price-setting in such a market. Our work is the first to systematically investigate workers' behaviour and response to economic incentives in a situated crowdsourcing market. We show that the market-based model is a sustainable approach to recruiting workers and obtaining situated crowdsourcing contributions. We also show that the price mechanism is a very effective tool for adjusting the supply of labour in a situated crowdsourcing market. Our work advances the body of work investigating situated crowdsourcing.
Crowd-powered parameter analysis for visual design exploration BIBAFull-Text 65-74
  Yuki Koyama; Daisuke Sakamoto; Takeo Igarashi
Parameter tweaking is one of the fundamental tasks in the editing of visual digital contents, such as correcting photo color or executing blendshape facial expression control. A problem with parameter tweaking is that it often requires much time and effort to explore a high-dimensional parameter space. We present a new technique to analyze such high-dimensional parameter space to obtain a distribution of human preference. Our method uses crowdsourcing to gather pairwise comparisons between various parameter sets. As a result of analysis, the user obtains a goodness function that computes the goodness value of a given parameter set. This goodness function enables two interfaces for exploration: Smart Suggestion, which provides suggestions of preferable parameter sets, and VisOpt Slider, which interactively visualizes the distribution of goodness values on sliders and gently optimizes slider values while the user is editing. We created four applications with different design parameter spaces. As a result, the system could facilitate the user's design exploration.
Expert crowdsourcing with flash teams BIBAFull-Text 75-85
  Daniela Retelny; Sébastien Robaszkiewicz; Alexandra To; Walter S. Lasecki; Jay Patel; Negar Rahmati; Tulsee Doshi; Melissa Valentine; Michael S. Bernstein
We introduce flash teams, a framework for dynamically assembling and managing paid experts from the crowd. Flash teams advance a vision of expert crowd work that accomplishes complex, interdependent goals such as engineering and design. These teams consist of sequences of linked modular tasks and handoffs that can be computationally managed. Interactive systems reason about and manipulate these teams' structures: for example, flash teams can be recombined to form larger organizations and authored automatically in response to a user's request. Flash teams can also hire more people elastically in reaction to task needs, and pipeline intermediate output to accelerate completion times. To enable flash teams, we present Foundry, an end-user authoring platform and runtime manager. Foundry allows users to author modular tasks, then manages teams through handoffs of intermediate work. We demonstrate that Foundry and flash teams enable crowdsourcing of a broad class of goals including design prototyping, course development, and film animation, in half the work time of traditional self-managed teams.

Building and using webpages

Creating interactive web data applications with spreadsheets BIBAFull-Text 87-96
  Kerry Shih-Ping Chang; Brad A. Myers
While more and more data are available through web services, it remains difficult for end-users to create web applications that make use of these data without having to write complex code. We present Gneiss, a live programming environment that extends the spreadsheet metaphor to support creating interactive web applications that dynamically use local or web data from multiple sources. Gneiss closely integrates a spreadsheet editor with a web interface builder to let users demonstrate bindings between properties of web GUI elements and cells in the spreadsheet while working with real web service data. The spreadsheet editor provides two-way connections to web services, to both visualize and retrieve different data based on the user input in the web interface. Gneiss achieves rich interactivity without the need for event-based programming by extending the 'pull model' of formulas that is familiar to the spreadsheet users. We use a series of examples to demonstrate Gneiss's ability to create a variety of interactive web data applications.
Spreadsheet driven web applications BIBAFull-Text 97-106
  Edward Benson; Amy X. Zhang; David R. Karger
Creating and publishing read-write-compute web applications requires programming skills beyond what most end users possess. But many end users know how to make spreadsheets that act as simple information management applications, some even with computation. We present a system for creating basic web applications using such spreadsheets in place of a server and using HTML to describe the client UI. Authors connect the two by placing spreadsheet references inside HTML attributes. Data computation is provided by spreadsheet formulas. The result is a reactive read-write-compute web page without a single line of Javascript code. Nearly all of the fifteen HTML novices we studied were able to connect HTML to spreadsheets using our method with minimal instruction. We draw conclusions from their experience and discuss future extensions to this programming model.
AttachMate: highlight extraction from email attachments BIBAFull-Text 107-116
  Joshua Hailpern; Sitaram Asur; Kyle Rector
While email is a major conduit for information sharing in enterprise, there has been little work on exploring the files sent along with these messages -- attachments. These accompanying documents can be large (multiple megabytes), lengthy (multiple pages), and not optimized for the smaller screen sizes, limited reading time, and expensive bandwidth of mobile users. Thus, attachments can increase data storage costs (for both end users and email servers), drain users' time when irrelevant, cause important information to be missed when ignored, and pose a serious access issue for mobile users. To address these problems we created AttachMate, a novel email attachment summarization system. AttachMate can summarize the content of email attachments and automatically insert the summary into the text of the email. AttachMate also stores all files in the cloud, reducing file storage costs and bandwidth consumption. In this paper, the primary contribution is the AttachMate client/server architecture. To ground, support and validate the AttachMate system we present two upfront studies (813 participants) to understand the state and limitations of attachments, a novel algorithm to extract representative concept sentences (tested through two validation studies), and a user study of AttachMate within an enterprise.
Making the web easier to see with opportunistic accessibility improvement BIBAFull-Text 117-122
  Jeffrey P. Bigham
Many people would find the Web easier to use if content was a little bigger, even those who already find the Web possible to use now. This paper introduces the idea of opportunistic accessibility improvement in which improvements intended to make a web page easier to access, such as magnification, are automatically applied to the extent that they can be without causing negative side effects. We explore this idea with oppaccess.js, an easily-deployed system for magnifying web pages that iteratively increases magnification until it notices negative side effects, such as horizontal scrolling or overlapping text. We validate this approach by magnifying existing web pages 1.6x on average without introducing negative side effects. We believe this concept applies generally across a wide range of accessibility improvements designed to help people with diverse abilities.
Content-aware kinetic scrolling for supporting web page navigation BIBAFull-Text 123-127
  Juho Kim; Amy X. Zhang; Jihee Kim; Robert C. Miller; Krzysztof Z. Gajos
Long documents are abundant on the web today, and are accessed in increasing numbers from touchscreen devices such as mobile phones and tablets. Navigating long documents with small screens can be challenging both physically and cognitively because they compel the user to scroll a great deal and to mentally filter for important content. To support navigation of long documents on touchscreen devices, we introduce content-aware kinetic scrolling, a novel scrolling technique that dynamically applies pseudo-haptic feedback in the form of friction around points of high interest within the page. This allows users to quickly find interesting content while exploring without further cluttering the limited visual space. To model degrees of interest (DOI) for a variety of existing web pages, we introduce social wear, a method for capturing DOI based on social signals that indicate collective user interest. Our preliminary evaluation shows that users pay attention to items with kinetic scrolling feedback during search, recognition, and skimming tasks.

Novel hardware I

FlexSense: a transparent self-sensing deformable surface BIBAFull-Text 129-138
  Christian Rendl; David Kim; Sean Fanello; Patrick Parzer; Christoph Rhemann; Jonathan Taylor; Martin Zirkl; Gregor Scheipl; Thomas Rothländer; Michael Haller; Shahram Izadi
We present FlexSense, a new thin-film, transparent sensing surface based on printed piezoelectric sensors, which can reconstruct complex deformations without the need for any external sensing, such as cameras. FlexSense provides a fully self-contained setup which improves mobility and is not affected from occlusions. Using only a sparse set of sensors, printed on the periphery of the surface substrate, we devise two new algorithms to fully reconstruct the complex deformations of the sheet, using only these sparse sensor measurements. An evaluation shows that both proposed algorithms are capable of reconstructing complex deformations accurately. We demonstrate how FlexSense can be used for a variety of 2.5D interactions, including as a transparent cover for tablets where bending can be performed alongside touch to enable magic lens style effects, layered input, and mode switching, as well as the ability to use our device as a high degree-of-freedom input controller for gaming and beyond.
Hover Pad: interacting with autonomous and self-actuated displays in space BIBAFull-Text 139-147
  Julian Seifert; Sebastian Boring; Christian Winkler; Florian Schaub; Fabian Schwab; Steffen Herrdum; Fabian Maier; Daniel Mayer; Enrico Rukzio
Handheld displays enable flexible spatial exploration of information spaces -- users can physically navigate through three-dimensional space to access information at specific locations. Having users constantly hold the display, however, has several limitations: (1) inaccuracies due to natural hand tremors; (2) fatigue over time; and (3) limited exploration within arm's reach. We investigate autonomous, self-actuated displays that can freely move and hold their position and orientation in space without users having to hold them at all times. We illustrate various stages of such a display's autonomy ranging from manual to fully autonomous, which -- depending on the tasks -- facilitate the interaction. Further, we discuss possible motion control mechanisms for these displays and present several interaction techniques enabled by such displays. Our Hover Pad toolkit enables exploring five degrees of freedom of self-actuated and autonomous displays and the developed control and interaction techniques. We illustrate the utility of our toolkit with five prototype applications, such as a volumetric medical data explorer.
Graffiti fur: turning your carpet into a computer display BIBAFull-Text 149-156
  Yuta Sugiura; Koki Toda; Takayuki Hoshi; Youichi Kamiyama; Takeo Igarashi; Masahiko Inami
We devised a display technology that utilizes the phenomenon whereby the shading properties of fur change as the fibers are raised or flattened. One can erase drawings by first flattening the fibers by sweeping the surface by hand in the fiber's growth direction, and then draw lines by raising the fibers by moving the finger in the opposite direction. These material properties can be found in various items such as carpets in our living environments. We have developed three different devices to draw patterns on a "fur display" utilizing this phenomenon: a roller device, a pen device and pressure projection device. Our technology can turn ordinary objects in our environment into rewritable displays without requiring or creating any non-reversible modifications to them. In addition, it can be used to present large-scale image without glare, and the images it creates require no running costs to maintain.
LightRing: always-available 2D input on any surface BIBAFull-Text 157-160
  Wolf Kienzle; Ken Hinckley
We present LightRing, a wearable sensor in a ring form factor that senses the 2d location of a fingertip on any surface, independent of orientation or material. The device consists of an infrared proximity sensor for measuring finger flexion and a 1-axis gyroscope for measuring finger rotation. Notably, LightRing tracks subtle fingertip movements from the finger base without requiring instrumentation of other body parts or the environment. This keeps the normal hand function intact and allows for a socially acceptable appearance. We evaluate LightRing in a 2d pointing experiment in two scenarios: on a desk while sitting down, and on the leg while standing. Our results indicate that the device has potential to enable a variety of rich mobile input scenarios.
WristFlex: low-power gesture input with wrist-worn pressure sensors BIBAFull-Text 161-166
  Artem Dementyev; Joseph A. Paradiso
In this paper we present WristFlex, an always-available on-body gestural interface. Using an array of force sensitive resistors (FSRs) worn around the wrist, the interface can distinguish subtle finger pinch gestures with high accuracy (>80%) and speed. The system is trained to classify gestures from subtle tendon movements on the wrist. We demonstrate that WristFlex is a complete system that works wirelessly in real-time. The system is simple and light-weight in terms of power consumption and computational overhead. WristFlex's sensor power consumption is 60.7 uW, allowing the prototype to potentially last more then a week on a small lithium polymer battery. Also, WristFlex is small and non-obtrusive, and can be integrated into a wristwatch or a bracelet. We perform user studies to evaluate the accuracy, speed, and repeatability. We demonstrate that the number of gestures can be extended with orientation data from an accelerometer. We conclude by showing example applications.

Modeling and prediction

CommandSpace: modeling the relationships between tasks, descriptions and features BIBAFull-Text 167-176
  Eytan Adar; Mira Dontcheva; Gierad Laput
Users often describe what they want to accomplish with an application in a language that is very different from the application's domain language. To address this gap between system and human language, we propose modeling an application's domain language by mining a large corpus of Web documents about the application using deep learning techniques. A high dimensional vector space representation can model the relationships between user tasks, system commands, and natural language descriptions and supports mapping operations, such as identifying likely system commands given natural language queries and identifying user tasks given a trace of user operations. We demonstrate the feasibility of this approach with a system, CommandSpace, for the popular photo editing application Adobe Photoshop. We build and evaluate several applications enabled by our model showing the power and flexibility of this approach.
Predictive translation memory: a mixed-initiative system for human language translation BIBAFull-Text 177-187
  Spence Green; Jason Chuang; Jeffrey Heer; Christopher D. Manning
The standard approach to computer-aided language translation is post-editing: a machine generates a single translation that a human translator corrects. Recent studies have shown this simple technique to be surprisingly effective, yet it underutilizes the complementary strengths of precision-oriented humans and recall-oriented machines. We present Predictive Translation Memory, an interactive, mixed-initiative system for human language translation. Translators build translations incrementally by considering machine suggestions that update according to the user's current partial translation. In a large-scale study, we find that professional translators are slightly slower in the interactive mode yet produce slightly higher quality translations despite significant prior experience with the baseline post-editing condition. Our analysis identifies significant predictors of time and quality, and also characterizes interactive aid usage. Subjects entered over 99% of characters via interactive aids, a significantly higher fraction than that shown in previous work.
Tohme: detecting curb ramps in google street view using crowdsourcing, computer vision, and machine learning BIBAFull-Text 189-204
  Kotaro Hara; Jin Sun; Robert Moore; David Jacobs; Jon Froehlich
Building on recent prior work that combines Google Street View (GSV) and crowdsourcing to remotely collect information on physical world accessibility, we present the first 'smart' system, Tohme, that combines machine learning, computer vision (CV), and custom crowd interfaces to find curb ramps remotely in GSV scenes. Tohme consists of two workflows, a human labeling pipeline and a CV pipeline with human verification, which are scheduled dynamically based on predicted performance. Using 1,086 GSV scenes (street intersections) from four North American cities and data from 403 crowd workers, we show that Tohme performs similarly in detecting curb ramps compared to a manual labeling approach alone (F-measure: 84% vs. 86% baseline) but at a 13% reduction in time cost. Our work contributes the first CV-based curb ramp detection system, a custom machine-learning based workflow controller, a validation of GSV as a viable curb ramp data source, and a detailed examination of why curb ramp detection is a hard problem along with steps forward.
Zero-latency tapping: using hover information to predict touch locations and eliminate touchdown latency BIBAFull-Text 205-214
  Haijun Xia; Ricardo Jota; Benjamin McCanny; Zhe Yu; Clifton Forlines; Karan Singh; Daniel Wigdor
A method of reducing the perceived latency of touch input by employing a model to predict touch events before the finger reaches the touch surface is proposed. A corpus of 3D finger movement data was collected, and used to develop a model capable of three granularities at different phases of movement: initial direction, final touch location, time of touchdown. The model is validated for target distances >= 25.5cm, and demonstrated to have a mean accuracy of 1.05cm 128ms before the user touches the screen. Preference study of different levels of latency reveals a strong preference for unperceived latency touchdown feedback. A form of 'soft' feedback, as well as other uses for this prediction to improve performance, is proposed.
Detecting tapping motion on the side of mobile devices by probabilistically combining hand postures BIBAFull-Text 215-219
  William McGrath; Yang Li
We contribute a novel method for detecting finger taps on the different sides of a smartphone, using the built-in motion sensors of the device. In particular, we discuss new features and algorithms that infer side taps by probabilistically combining estimates of tap location and the hand pose -- the hand holding the device. Based on a dataset collected from 9 participants, our method achieved 97.3% precision and 98.4% recall on tap event detection against ambient motion. For detecting single-tap locations, our method outperformed an approach that uses inferred hand postures deterministically by 3% and an approach that does not use hand posture inference by 17%. For inferring the location of two consecutive side taps from the same direction, our method outperformed the two baseline approaches by 6% and 17% respectively. We discuss our insights into designing the detection algorithm and the implication on side tap-based interaction behaviors.

Developer tools I

Prefab layers and prefab annotations: extensible pixel-based interpretation of graphical interfaces BIBAFull-Text 221-230
  Morgan Dixon; Alexander Nied; James Fogarty
Pixel-based methods have the potential to fundamentally change how we build graphical interfaces, but remain difficult to implement. We introduce a new toolkit for pixel based enhancements, focused on two areas of support. Prefab Layers helps developers write interpretation logic that can be composed, reused, and shared to manage the multi-faceted nature of pixel-based interpretation. Prefab Annotations supports robustly annotating interface elements with metadata needed to enable runtime enhancements. Together, these help developers overcome subtle but critical dependencies between code and data. We validate our toolkit with (1) demonstrative applications and (2) a lab study that compares how developers build an enhancement using our toolkit versus state of the art methods. Our toolkit addresses core challenges faced by developers when building pixel based enhancements, potentially opening up pixel based systems to broader adoption.
Programming by manipulation for layout BIBAFull-Text 231-241
  Thibaud Hottelier; Ras Bodik; Kimiko Ryokai
We present Programming by Manipulation, a new programming methodology for specifying the layout of data visualizations, targeted at non-programmers. We address the two central sources of bugs that arise when programming with constraints: ambiguities and conflicts (inconsistencies). We rule out conflicts by design and exploit ambiguity to explore possible layout designs. Our users design layouts by highlighting undesirable aspects of a current design, effectively breaking spurious constraints and introducing ambiguity by giving some elements freedom to move or resize. Subsequently, the tool indicates how the ambiguity can be removed, by computing how the free elements can be fixed with available constraints. To support this workflow, our tool computes the ambiguity and summarizes it visually. We evaluate our work with two user-studies demonstrating that both non-programmers and programmers can effectively use our prototype. Our results suggest that our tool is 5-times more productive than direct programming with constraints.
Global beautification of layouts with interactive ambiguity resolution BIBAFull-Text 243-252
  Pengfei Xu; Hongbo Fu; Takeo Igarashi; Chiew-Lan Tai
Automatic global beautification methods have been proposed for sketch-based interfaces, but they can lead to undesired results due to ambiguity in the user's input. To facilitate ambiguity resolution in layout beautification, we present a novel user interface for visualizing and editing inferred relationships. First, our interface provides a preview of the beautified layout with inferred constraints, without directly modifying the input layout. In this way, the user can easily keep refining beautification results by interactively repositioning and/or resizing elements in the input layout. Second, we present a gestural interface for editing automatically inferred constraints by directly interacting with the visualized constraints via simple gestures. Our efficient implementation of the beautification system provides the user instant feedback. Our user studies validate that our tool is capable of creating, editing and refining layouts of graphic elements and is significantly faster than the standard snap-dragging and command-based alignment tools.
Deconstructing and restyling D3 visualizations BIBAFull-Text 253-262
  Jonathan Harper; Maneesh Agrawala
The D3 JavaScript library has become a ubiquitous tool for developing visualizations on the Web. Yet, once a D3 visualization is published online its visual style is difficult to change. We present a pair of tools for deconstructing and restyling existing D3 visualizations. Our deconstruction tool analyzes a D3 visualization to extract the data, the marks and the mappings between them. Our restyling tool lets users modify the visual attributes of the marks as well as the mappings from the data to these attributes. Together our tools allow users to easily modify D3 visualizations without examining the underlying code and we show how they can be used to deconstruct and restyle a variety of D3 visualizations.
InterState: a language and environment for expressing interface behavior BIBAFull-Text 263-272
  Stephen Oney; Brad Myers; Joel Brandt
InterState is a new programming language and environment that addresses the challenges of writing and reusing user interface code. InterState represents interactive behaviors clearly and concisely using a combination of novel forms of state machines and constraints. It also introduces new language features that allow programmers to easily modularize and reuse behaviors. InterState uses a new visual notation that allows programmers to better understand and navigate their code. InterState also includes a live editor that immediately updates the running application in response to changes in the editor and vice versa to help programmers understand the state of their program. Finally, InterState can interface with code and widgets written in other languages, for example to create a user interface in InterState that communicates with a database. We evaluated the understandability of InterState's programming primitives in a comparative laboratory study. We found that participants were twice as fast at understanding and modifying GUI components when they were implemented with InterState than when they were implemented in a conventional textual event-callback style. We evaluated InterState's scalability with a series of benchmarks and example applications and found that it can scale to implement complex behaviors involving thousands of objects and constraints.

Fabrication

WirePrint: 3D printed previews for fast prototyping BIBAFull-Text 273-280
  Stefanie Mueller; Sangha Im; Serafima Gurevich; Alexander Teibrich; Lisa Pfisterer; François Guimbretière; Patrick Baudisch
Even though considered a rapid prototyping tool, 3D printing is so slow that a reasonably sized object requires printing overnight. This slows designers down to a single iteration per day. In this paper, we propose to instead print low-fidelity wireframe previews in the early stages of the design process. Wireframe previews are 3D prints in which surfaces have been replaced with a wireframe mesh. Since wireframe previews are to scale and represent the overall shape of the 3D object, they allow users to quickly verify key aspects of their 3D design, such as the ergonomic fit. To maximize the speed-up, we instruct 3D printers to extrude filament not layer-by-layer, but directly in 3D-space, allowing them to create the edges of the wireframe model directly one stroke at a time. This allows us to achieve speed-ups of up to a factor of 10 compared to traditional layer-based printing. We demonstrate how to achieve wireframe previews on standard FDM 3D printers, such as the PrintrBot or the Kossel mini. Users only need to install the WirePrint software, making our approach applicable to many 3D printers already in use today. Finally, wireframe previews use only a fraction of material required for a regular print, making it even more affordable to iterate.
PrintScreen: fabricating highly customizable thin-film touch-displays BIBAFull-Text 281-290
  Simon Olberding; Michael Wessely; Jürgen Steimle
PrintScreen is an enabling technology for digital fabrication of customized flexible displays using thin-film electroluminescence (TFEL). It enables inexpensive and rapid fabrication of highly customized displays in low volume, in a simple lab environment, print shop or even at home. We show how to print ultra-thin (120 µm) segmented and passive matrix displays in greyscale or multi-color on a variety of deformable and rigid substrate materials, including PET film, office paper, leather, metal, stone, and wood. The displays can have custom, unconventional 2D shapes and can be bent, rolled and folded to create 3D shapes. We contribute a systematic overview of graphical display primitives for customized displays and show how to integrate them with static print and printed electronics. Furthermore, we contribute a sensing framework, which leverages the display itself for touch sensing. To demonstrate the wide applicability of PrintScreen, we present application examples from ubiquitous, mobile and wearable computing.
ShrinkyCircuits: sketching, shrinking, and formgiving for electronic circuits BIBAFull-Text 291-299
  Joanne Lo; Eric Paulos
In this paper we describe the development of ShrinkyCircuits, a novel electronic prototyping technique that captures the flexibility of sketching and leverages properties of a common everyday plastic polymer to enable low-cost, miniature, planar, and curved, multi-layer circuit designs in minutes. ShrinkyCircuits take advantage of inexpensive prestressed polymer film that shrinks to its original size when exposed to heat. This enables improved electrical characteristics though sintering of the conductive electrical layer, partial self-assembly of the circuit and components, and mechanically robust custom shapes Including curves and non-planar form factors. We demonstrate the range and adaptability of ShrinkyCircuits designs from simple hand drawn circuits with through-hole components to complex multilayer, printed circuit boards (PCB), with curved and irregular shaped electronic layouts and surface mount components. Our approach enables users to create extremely customized circuit boards with dense circuit layouts while avoiding messy chemical etching, expensive board milling machines, or time consuming delays in using outside PCB production houses.
Teegi: tangible EEG interface BIBAFull-Text 301-308
  Jérémy Frey; Renaud Gervais; Stéphanie Fleck; Fabien Lotte; Martin Hachet
We introduce Teegi, a Tangible ElectroEncephaloGraphy (EEG) Interface that enables novice users to get to know more about something as complex as brain signals, in an easy, engaging and informative way. To this end, we have designed a new system based on a unique combination of spatial augmented reality, tangible interaction and real-time neurotechnologies. With Teegi, a user can visualize and analyze his or her own brain activity in real-time, on a tangible character that can be easily manipulated, and with which it is possible to interact. An exploration study has shown that interacting with Teegi seems to be easy, motivating, reliable and informative. Overall, this suggests that Teegi is a promising and relevant training and mediation tool for the general public.

Augmented reality I

Individual variation in susceptibility to cybersickness BIBAFull-Text 309-317
  Lisa Rebenitsch; Charles Owen
We examined background characteristics of virtual reality participants in order to determine correlations to cybersickness. As 3D media and new VR display technologies from companies such as Occulus and Sony become more popular, the incidence of cybersickness is likely to increase. Understanding the impact of individual backgrounds on susceptibility can help shed light on which individuals are more likely to be impacted. Past history of motion sickness and video game play have the best predictive power of cybersickness of the factors studied. A model to estimate the likelihood of cybersickness using background characteristics is posed.
In-air gestures around unmodified mobile devices BIBAFull-Text 319-329
  Jie Song; Gábor Sörös; Fabrizio Pece; Sean Ryan Fanello; Shahram Izadi; Cem Keskin; Otmar Hilliges
We present a novel machine learning based algorithm extending the interaction space around mobile devices. The technique uses only the RGB camera now commonplace on off-the-shelf mobile devices. Our algorithm robustly recognizes a wide range of in-air gestures, supporting user variation, and varying lighting conditions. We demonstrate that our algorithm runs in real-time on unmodified mobile devices, including resource-constrained smartphones and smartwatches. Our goal is not to replace the touchscreen as primary input device, but rather to augment and enrich the existing interaction vocabulary using gestures. While touch input works well for many scenarios, we demonstrate numerous interaction tasks such as mode switches, application and task management, menu selection and certain types of navigation, where such input can be either complemented or better served by in-air gestures. This removes screen real-estate issues on small touchscreens, and allows input to be expanded to the 3D space around the device. We present results for recognition accuracy (93% test and 98% train), impact of memory footprint and other model parameters. Finally, we report results from preliminary user evaluations, discuss advantages and limitations and conclude with directions for future work.
ParaFrustum: visualization techniques for guiding a user to a constrained set of viewing positions and orientations BIBAFull-Text 331-340
  Mengu Sukan; Carmine Elvezio; Ohan Oda; Steven Feiner; Barbara Tversky
Many tasks in real or virtual environments require users to view a target object or location from one of a set of strategic viewpoints to see it in context, avoid occlusions, or view it at an appropriate angle or distance. We introduce ParaFrustum, a geometric construct that represents this set of strategic viewpoints and viewing directions. ParaFrustum is inspired by the look-from and look-at points of a computer graphics camera specification, which precisely delineate a location for the camera and a direction in which it looks. We generalize this approach by defining a ParaFrustum in terms of a look-from volume and a look-at volume, which establish constraints on a range of acceptable locations for the user's eyes and a range of acceptable angles in which the user's head can be oriented. Providing tolerance in the allowable viewing positions and directions avoids burdening the user with the need to assume a tightly constrained 6DoF pose when it is not required by the task. We describe two visualization techniques for virtual or augmented reality that guide a user to assume one of the poses defined by a ParaFrustum, and present the results of a user study measuring the performance of these techniques. The study shows that the constraints of a tightly constrained ParaFrustum (e.g., approximating a conventional camera frustum) require significantly more time to satisfy than those of a loosely constrained one. The study also reveals interesting differences in participant trajectories in response to the two techniques.
Through the combining glass BIBAFull-Text 341-350
  Diego Martinez Plasencia; Florent Berthaut; Abhijit Karnik; Sriram Subramanian
Reflective optical combiners like beam splitters and two way mirrors are used in AR to overlap digital contents on the users' hands or bodies. Augmentations are usually unidirectional, either reflecting virtual contents on the user's body (Situated Augmented Reality) or augmenting user's reflections with digital contents (AR mirrors). But many other novel possibilities remain unexplored. For example, users' hands, reflected inside a museum AR cabinet, can allow visitors to interact with the artifacts exhibited. Projecting on the user's hands as their reflection cuts through the objects can be used to reveal objects' internals. Augmentations from both sides are blended by the combiner, so they are consistently seen by any number of users, independently of their location or, even, the side of the combiner through which they are looking. This paper explores the potential of optical combiners to merge the space in front and behind them. We present this design space, identify novel augmentations/interaction opportunities and explore the design space using three prototypes.
Loupe: a handheld near-eye display BIBAFull-Text 351-354
  Kent Lyons; Seung Wook Kim; Shigeyuki Seko; David Nguyen; Audrey Desjardins; Mélodie Vidal; David Dobbelstein; Jeremy Rubin
Loupe is a novel interactive device with a near-eye virtual display similar to head-up display glasses that retains a handheld form factor. We present our hardware implementation and discuss our user interface that leverages Loupe's unique combination of properties. In particular, we present our input capabilities, spatial metaphor, opportunities for using the round aspect of Loupe, and our use of focal depth. We demonstrate how those capabilities come together in an example application designed to allow quick access to information feeds.

Novel hardware II

High rate, low-latency multi-touch sensing with simultaneous orthogonal multiplexing BIBAFull-Text 355-364
  Darren Leigh; Clifton Forlines; Ricardo Jota; Steven Sanders; Daniel Wigdor
We present "Fast Multi-Touch" (FMT), an extremely high frame rate and low-latency multi-touch sensor based on a novel projected capacitive architecture that employs simultaneous orthogonal signals. The sensor has a frame rate of 4000 Hz and a touch-to-data output latency of only 40 microseconds, providing unprecedented responsiveness. FMT is demonstrated with a high-speed DLP projector yielding a touch-to-light latency of 110 microseconds.
GaussStones: shielded magnetic tangibles for multi-token interactions on portable displays BIBAFull-Text 365-372
  Rong-Hao Liang; Han-Chih Kuo; Liwei Chan; De-Nian Yang; Bing-Yu Chen
This work presents GaussStones, a system of shielded magnetic tangibles design for supporting multi-token interactions on portable displays. Unlike prior works in sensing magnetic tangibles on portable displays, the proposed tangible design applies magnetic shielding by using an inexpensive galvanized steel case, which eliminates interference between magnetic tangibles. An analog Hall-sensor grid can recognize the identity of each shielded magnetic unit since each unit generates a magnetic field with a specific intensity distribution and/or polarization. Combining multiple units as a knob further allows for resolving additional identities and their orientations. Enabling these features improves support for applications involving multiple tokens. Thus, using prevalent portable displays provides generic platforms for tangible interaction design.
Tag system with low-powered tag and depth sensing camera BIBAFull-Text 373-382
  Hiroyuki Manabe; Wataru Yamada; Hiroshi Inamura
A tag system is proposed that offers a practical approach to ubiquitous computing. It provides small and low-power tags that are easy to distribute; does not need a special device to read the tags (in the future), thus enabling their use anytime, anywhere; and has a wide reading range in angle and distance that extends the design space of tag-based applications. The tag consists of a kind of liquid crystal (LC) and a retroreflector, and it sends its ID by switching the LC. A depth sensing camera that emits infrared (IR) is used as the tag reader; we assume that it will be part of the user's everyday devices, such as a smartphone. Experiments were conducted to confirm its potential, and a regular IR camera was also tested for comparison. The results show that the tag system has a wide readable range in terms of both distance (up to 8m) and viewing angle offset. Several applications were also developed to explore the design space. Finally, limitations of the current setup and possible improvements are discussed.
Trampoline: a double-sided elastic touch device for creating reliefs BIBAFull-Text 383-388
  Jaehyun Han; Jiseong Gu; Geehyuk Lee
Although reliefs are frequently used to add patterns to product surfaces, there is a lack of interaction techniques to model reliefs on the surface of virtual objects. We adopted the repoussé and chasing artwork techniques in an alternative interaction technique to model relief on virtual surfaces. To support this interaction technique, we developed the double-sided touchpad Trampoline that can detect the position and force of a finger touch on both sides. Additionally, Trampoline provides users with elastic feedback, as its surface consists of a stretchable fabric. We implemented a relief application with this device and the developed interaction technique. An informal user study showed that the proposed system can be a promising solution to create reliefs.
Skin buttons: cheap, small, low-powered and clickable fixed-icon laser projectors BIBAFull-Text 389-394
  Gierad Laput; Robert Xiao; Xiang 'Anthony' Chen; Scott E. Hudson; Chris Harrison
Smartwatches are a promising new interactive platform, but their small size makes even basic actions cumbersome. Hence, there is a great need for approaches that expand the interactive envelope around smartwatches, allowing human input to escape the small physical confines of the device. We propose using tiny projectors integrated into the smartwatch to render icons on the user's skin. These icons can be made touch sensitive, significantly expanding the interactive region without increasing device size. Through a series of experiments, we show that these 'skin buttons' can have high touch accuracy and recognizability, while being low cost and power-efficient.

Creative tools

Kitty: sketching dynamic and interactive illustrations BIBAFull-Text 395-405
  Rubaiat Habib Kazi; Fanny Chevalier; Tovi Grossman; George Fitzmaurice
We present Kitty, a sketch-based tool for authoring dynamic and interactive illustrations. Artists can sketch animated drawings and textures to convey the living phenomena, and specify the functional relationship between its entities to characterize the dynamic behavior of systems and environments. An underlying graph model, customizable through sketching, captures the functional relationships between the visual, spatial, temporal or quantitative parameters of its entities. As the viewer interacts with the resulting dynamic interactive illustration, the parameters of the drawing change accordingly, depicting the dynamics and chain of causal effects within a scene. The generality of this framework makes our tool applicable for a variety of purposes, including technical illustrations, scientific explanation, infographics, medical illustrations, children's e-books, cartoon strips and beyond. A user study demonstrates the ease of usage, variety of applications, artistic expressiveness and creative possibilities of our tool.
PortraitSketch: face sketching assistance for novices BIBAFull-Text 407-417
  Jun Xie; Aaron Hertzmann; Wilmot Li; Holger Winnemöller
We present PortraitSketch, an interactive drawing system that helps novices create pleasing, recognizable face sketches without requiring prior artistic training. As the user traces over a source portrait photograph, PortraitSketch automatically adjusts the geometry and stroke parameters (thickness, opacity, etc.) to improve the aesthetic quality of the sketch. We present algorithms for adjusting both outlines and shading strokes based on important features of the underlying source image. In contrast to automatic stylization systems, PortraitSketch is designed to encourage a sense of ownership and accomplishment in the user. To this end, all adjustments are performed in real-time, and the user ends up directly drawing all strokes on the canvas. The findings from our user study suggest that users prefer drawing with some automatic assistance, thereby producing better drawings, and that assistance does not decrease the perceived level of involvement in the creative process.
Painting with Bob: assisted creativity for novices BIBAFull-Text 419-428
  Luca Benedetti; Holger Winnemöller; Massimiliano Corsini; Roberto Scopigno
Current digital painting tools are primarily targeted at professionals and are often overwhelmingly complex for use by novices. At the same time, simpler tools may not invoke the user creatively, or are limited to plain styles that lack visual sophistication. There are many people who are not art professionals, yet would like to partake in digital creative expression. Challenges and rewards for novices differ greatly from those for professionals. In this paper, we leverage existing works in Creativity and Creativity Support Tools (CST) to formulate design goals specifically for digital art creation tools for novices. We implemented these goals within a digital painting system, called Painting with Bob. We evaluate the efficacy of the design and our prototype with a user study, and we find that users are highly satisfied with the user experience, as well as the paintings created with our system.
InterTwine: creating interapplication information scent to support coordinated use of software BIBAFull-Text 429-438
  Adam Fourney; Ben Lafreniere; Parmit Chilana; Michael Terry
Users often make continued and sustained use of online resources to complement use of a desktop application. For example, users may reference online tutorials to recall how to perform a particular task. While often used in a coordinated fashion, the browser and desktop application provide separate, independent mechanisms for helping users find and re-find task-relevant information. In this paper, we describe InterTwine, a system that links information in the web browser with relevant elements in the desktop application to create interapplication information scent. This explicit link produces a shared interapplication history to assist in re-finding information in both applications. As an example, InterTwine marks all menu items in the desktop application that are currently mentioned in the front-most web page. This paper introduces the notion of interapplication information scent, demonstrates the concept in InterTwine, and describes results from a formative study suggesting the utility of the concept.
Generating emotionally relevant musical scores for audio stories BIBAFull-Text 439-448
  Steve Rubin; Maneesh Agrawala
Highly-produced audio stories often include musical scores that reflect the emotions of the speech. Yet, creating effective musical scores requires deep expertise in sound production and is time-consuming even for experts. We present a system and algorithm for re-sequencing music tracks to generate emotionally relevant music scores for audio stories. The user provides a speech track and music tracks and our system gathers emotion labels on the speech through hand-labeling, crowdsourcing, and automatic methods. We develop a constraint-based dynamic programming algorithm that uses these emotion labels to generate emotionally relevant musical scores. We demonstrate the effectiveness of our algorithm by generating 20 musical scores for audio stories and showing that crowd workers rank their overall quality significantly higher than stories without music.

Collaboration

World-stabilized annotations and virtual scene navigation for remote collaboration BIBAFull-Text 449-459
  Steffen Gauglitz; Benjamin Nuernberger; Matthew Turk; Tobias Höllerer
We present a system that supports an augmented shared visual space for live mobile remote collaboration on physical tasks. The remote user can explore the scene independently of the local user's current camera position and can communicate via spatial annotations that are immediately visible to the local user in augmented reality. Our system operates on off-the-shelf hardware and uses real-time visual tracking and modeling, thus not requiring any preparation or instrumentation of the environment. It creates a synergy between video conferencing and remote scene exploration under a unique coherent interface. To evaluate the collaboration with our system, we conducted an extensive outdoor user study with 60 participants comparing our system with two baseline interfaces. Our results indicate an overwhelming user preference (80%) for our system, a high level of usability, as well as performance benefits compared with one of the two baselines.
Physical telepresence: shape capture and display for embodied, computer-mediated remote collaboration BIBAFull-Text 461-470
  Daniel Leithinger; Sean Follmer; Alex Olwal; Hiroshi Ishii
We propose a new approach to Physical Telepresence, based on shared workspaces with the ability to capture and remotely render the shapes of people and objects. In this paper, we describe the concept of shape transmission, and propose interaction techniques to manipulate remote physical objects and physical renderings of shared digital content. We investigate how the representation of user's body parts can be altered to amplify their capabilities for teleoperation. We also describe the details of building and testing prototype Physical Telepresence workspaces based on shape displays. A preliminary evaluation shows how users are able to manipulate remote objects, and we report on our observations of several different manipulation techniques that highlight the expressive nature of our system.
3D-board: a whole-body remote collaborative whiteboard BIBAFull-Text 471-479
  Jakob Zillner; Christoph Rhemann; Shahram Izadi; Michael Haller
This paper presents 3D-Board, a digital whiteboard capable of capturing life-sized virtual embodiments of geographically distributed users. When using large-scale screens for remote collaboration, awareness for the distributed users' gestures and actions is of particular importance. Our work adds to the literature on remote collaborative workspaces, it facilitates intuitive remote collaboration on large scale interactive whiteboards by preserving awareness of the full-body pose and gestures of the remote collaborator. By blending the front-facing 3D embodiment of a remote collaborator with the shared workspace, an illusion is created as if the observer was looking through the transparent whiteboard into the remote user's room. The system was tested and verified in a usability assessment, showing that 3D-Board significantly improves the effectiveness of remote collaboration on a large interactive surface.
RichReview: blending ink, speech, and gesture to support collaborative document review BIBAFull-Text 481-490
  Dongwook Yoon; Nicholas Chen; François Guimbretière; Abigail Sellen
This paper introduces a novel document annotation system that aims to enable the kinds of rich communication that usually only occur in face-to-face meetings. Our system, RichReview, lets users create annotations on top of digital documents using three main modalities: freeform inking, voice for narration, and deictic gestures in support of voice. RichReview uses novel visual representations and time-synchronization between modalities to simplify annotation access and navigation. Moreover, RichReview's versatile support for multi-modal annotations enables users to mix and interweave different modalities in threaded conversations. A formative evaluation demonstrates early promise for the system finding support for voice, pointing, and the combination of both to be especially valuable. In addition, initial findings point to the ways in which both content and social context affect modality choice.
Sensory triptych: here, near, out there BIBAFull-Text 491-496
  Eric Paulos; Chris Myers; Rundong Tian; Paxton Paulos
Sensory Triptych is a set of exploratory, interactive sensors designed for children that invite "new ways of seeing" our world from the perspective of the here (the earth, air, and water around us), near (things just out of sight), and out there (orbiting satellites and space junk) using familiar and novel interfaces, affordances, and narratives. We present a series of novel physical design prototypes that reframe sensing technologies for children that foster an early adoption of technology usage for exploring, understanding, communicating, sharing, and changing our world. Finally, we discuss how such designs expand the potential opportunities and landscapes for our future interactive systems and experiences within the UIST community.

Touch & gesture

Going to the dogs: towards an interactive touchscreen interface for working dogs BIBAFull-Text 497-507
  Clint Zeagler; Scott Gilliland; Larry Freil; Thad Starner; Melody Jackson
Computer-mediated interaction for working dogs is an important new domain for interaction research. In domestic settings, touchscreens could provide a way for dogs to communicate critical information to humans. In this paper we explore how a dog might interact with a touchscreen interface. We observe dogs' touchscreen interactions and record difficulties against what is expected of humans' touchscreen interactions. We also solve hardware issues through screen adaptations and projection styles to make a touchscreen usable for a canine's nose touch interactions. We also compare our canine touch data to humans' touch data on the same system. Our goal is to understand the affordances needed to make touchscreen interfaces usable for canines and help the future design of touchscreen interfaces for assistive dogs in the home.
Gaze-touch: combining gaze with multi-touch for interaction on the same surface BIBAFull-Text 509-518
  Ken Pfeuffer; Jason Alexander; Ming Ki Chong; Hans Gellersen
Gaze has the potential to complement multi-touch for interaction on the same surface. We present gaze-touch, a technique that combines the two modalities based on the principle of 'gaze selects, touch manipulates'. Gaze is used to select a target, and coupled with multi-touch gestures that the user can perform anywhere on the surface. Gaze-touch enables users to manipulate any target from the same touch position, for whole-surface reachability and rapid context switching. Conversely, gaze-touch enables manipulation of the same target from any touch position on the surface, for example to avoid occlusion. Gaze-touch is designed to complement direct-touch as the default interaction on multi-touch surfaces. We provide a design space analysis of the properties of gaze-touch versus direct-touch, and present four applications that explore how gaze-touch can be used alongside direct-touch. The applications demonstrate use cases for interchangeable, complementary and alternative use of the two modes of interaction, and introduce novel techniques arising from the combination of gaze-touch and conventional multi-touch.
Air+touch: interweaving touch & in-air gestures BIBAFull-Text 519-525
  Xiang 'Anthony' Chen; Julia Schwarz; Chris Harrison; Jennifer Mankoff; Scott E. Hudson
We present Air+Touch, a new class of interactions that interweave touch events with in-air gestures, offering a unified input modality with expressiveness greater than each input modality alone. We demonstrate how air and touch are highly complementary: touch is used to designate targets and segment in-air gestures, while in-air gestures add expressivity to touch events. For example, a user can draw a circle in the air and tap to trigger a context menu, do a finger 'high jump' between two touches to select a region of text, or drag and in-air 'pigtail' to copy text to the clipboard. Through an observational study, we devised a basic taxonomy of Air+Touch interactions, based on whether the in-air component occurs before, between or after touches. To illustrate the potential of our approach, we built four applications that showcase seven exemplar Air+Touch interactions we created.
SideSwipe: detecting in-air gestures around mobile devices using actual GSM signal BIBAFull-Text 527-534
  Chen Zhao; Ke-Yu Chen; Md Tanvir Islam Aumi; Shwetak Patel; Matthew S. Reynolds
Current smartphone inputs are limited to physical buttons, touchscreens, cameras or built-in sensors. These approaches either require a dedicated surface or line-of-sight for interaction. We introduce SideSwipe, a novel system that enables in-air gestures both above and around a mobile device. Our system leverages the actual GSM signal to detect hand gestures around the device. We developed an algorithm to convert the discrete and bursty GSM pulses to a continuous wave that can be used for gesture recognition. Specifically, when a user waves their hand near the phone, the hand movement disturbs the signal propagation between the phone's transmitter and added receiving antennas. Our system captures this variation and uses it for gesture recognition. To evaluate our system, we conduct a study with 10 participants and present robust gesture recognition with an average accuracy of 87.2% across 14 hand gestures.
Vibkinesis: notification by direct tap and 'dying message' using vibronic movement controllable smartphones BIBAFull-Text 535-540
  Shota Yamanaka; Homei Miyashita
We propose Vibkinesis, a smartphone that can control its angle and directions of movement and rotation. By separately controlling the vibration motors attached to it, the smartphone can move on a table in the direction it chooses. Vibkinesis can inform a user of a message received when the user is away from the smartphone by changing its orientation, e.g., the smartphone has rotated 90° to the left before the user returns to the smartphone. With this capability, Vibkinesis can notify the user of a message even if the battery is discharged. We also extend the sensing area of Vibkinesis by using an omni-directional lens so that the smartphone tracks the surrounding objects. This allows Vibkinesis to tap the user's hand. These novel interactions expand the mobile device's movement area, notification channels, and notification time span.

Video

Video lens: rapid playback and exploration of large video collections and associated metadata BIBAFull-Text 541-550
  Justin Matejka; Tovi Grossman; George Fitzmaurice
We present Video Lens, a framework which allows users to visualize and interactively explore large collections of videos and associated metadata. The primary goal of the framework is to let users quickly find relevant sections within the videos and play them back in rapid succession. The individual UI elements are linked and highly interactive, supporting a faceted search paradigm and encouraging exploration of the data set. We demonstrate the capabilities and specific scenarios of Video Lens within the domain of professional baseball videos. A user study with 12 participants indicates that Video Lens efficiently supports a diverse range of powerful yet desirable video query tasks, while a series of interviews with professionals in the field demonstrates the framework's benefits and future potential.
Glance: rapidly coding behavioral video with the crowd BIBAFull-Text 551-562
  Walter S. Lasecki; Mitchell Gordon; Danai Koutra; Malte F. Jung; Steven P. Dow; Jeffrey P. Bigham
Behavioral researchers spend considerable amount of time coding video data to systematically extract meaning from subtle human actions and emotions. In this paper, we present Glance, a tool that allows researchers to rapidly query, sample, and analyze large video datasets for behavioral events that are hard to detect automatically. Glance takes advantage of the parallelism available in paid online crowds to interpret natural language queries and then aggregates responses in a summary view of the video data. Glance provides analysts with rapid responses when initially exploring a dataset, and reliable codings when refining an analysis. Our experiments show that Glance can code nearly 50 minutes of video in 5 minutes by recruiting over 60 workers simultaneously, and can get initial feedback to analysts in under 10 seconds for most clips. We present and compare new methods for accurately aggregating the input of multiple workers marking the spans of events in video data, and for measuring the quality of their coding in real-time before a baseline is established by measuring the variance between workers. Glance's rapid responses to natural language queries, feedback regarding question ambiguity and anomalies in the data, and ability to build on prior context in followup queries allow users to have a conversation-like interaction with their data -- opening up new possibilities for naturally exploring video data.
Data-driven interaction techniques for improving navigation of educational videos BIBAFull-Text 563-572
  Juho Kim; Philip J. Guo; Carrie J. Cai; Shang-Wen (Daniel) Li; Krzysztof Z. Gajos; Robert C. Miller
With an unprecedented scale of learners watching educational videos on online platforms such as MOOCs and YouTube, there is an opportunity to incorporate data generated from their interactions into the design of novel video interaction techniques. Interaction data has the potential to help not only instructors to improve their videos, but also to enrich the learning experience of educational video watchers. This paper explores the design space of data-driven interaction techniques for educational video navigation. We introduce a set of techniques that augment existing video interface widgets, including: a 2D video timeline with an embedded visualization of collective navigation traces; dynamic and non-linear timeline scrubbing; data-enhanced transcript search and keyword summary; automatic display of relevant still frames next to the video; and a visual summary representing points with high learner activity. To evaluate the feasibility of the techniques, we ran a laboratory user study with simulated learning tasks. Participants rated watching lecture videos with interaction data to be efficient and useful in completing the tasks. However, no significant differences were found in task performance, suggesting that interaction data may not always align with moment-by-moment information needs during the tasks.
Video digests: a browsable, skimmable format for informational lecture videos BIBAFull-Text 573-582
  Amy Pavel; Colorado Reed; Björn Hartmann; Maneesh Agrawala
Increasingly, authors are publishing long informational talks, lectures, and distance-learning videos online. However, it is difficult to browse and skim the content of such videos using current timeline-based video players. Video digests are a new format for informational videos that afford browsing and skimming by segmenting videos into a chapter/section structure and providing short text summaries and thumbnails for each section. Viewers can navigate by reading the summaries and clicking on sections to access the corresponding point in the video. We present a set of tools to help authors create such digests using transcript-based interactions. With our tools, authors can manually create a video digest from scratch, or they can automatically generate a digest by applying a combination of algorithmic and crowdsourcing techniques and then manually refine it as needed. Feedback from first-time users suggests that our transcript-based authoring tools and automated techniques greatly facilitate video digest creation. In an evaluative crowdsourced study we find that given a short viewing time, video digests support browsing and skimming better than timeline-based or transcript-based video players.

Input techniques

Brain-based target expansion BIBAFull-Text 583-593
  Daniel Afergan; Tomoki Shibata; Samuel W. Hincks; Evan M. Peck; Beste F. Yuksel; Remco Chang; Robert J. K. Jacob
The bubble cursor is a promising cursor expansion technique, improving a user's movement time and accuracy in pointing tasks. We introduce a brain-based target expansion system, which improves the efficacy of bubble cursor by increasing the expansion of high importance targets at the optimal time based on brain measurements correlated to a particular type of multitasking. We demonstrate through controlled experiments that brain-based target expansion can deliver a graded and continuous level of assistance to a user according to their cognitive state, thereby improving task and speed-accuracy metrics, even without explicit visual changes to the system. Such an adaptation is ideal for use in complex systems to steer users toward higher priority goals during times of increased demand.
Pinch-to-zoom-plus: an enhanced pinch-to-zoom that reduces clutching and panning BIBAFull-Text 595-604
  Jeff Avery; Mark Choi; Daniel Vogel; Edward Lank
Despite its popularity, the classic pinch-to-zoom gesture used in modern multi-touch interfaces has drawbacks: specifically, the need to support an extended range of scales and the need to keep content within the view window on the display can result in the need to clutch and pan. In two formative studies of unimanual and bimanual pinch-to-zoom, we found patterns: zooming actions follows a predictable ballistic velocity curve, and users tend to pan the point-of-interest towards the center of the screen. We apply these results to design an enhanced zooming technique called Pinch-to-Zoom-Plus (PZP) that reduces clutching and panning operations compared to standard pinch-to-zoom behaviour.
Sensing techniques for tablet+stylus interaction BIBAFull-Text 605-614
  Ken Hinckley; Michel Pahud; Hrvoje Benko; Pourang Irani; François Guimbretière; Marcel Gavriliu; Xiang 'Anthony' Chen; Fabrice Matulic; William Buxton; Andrew Wilson
We explore grip and motion sensing to afford new techniques that leverage how users naturally manipulate tablet and stylus devices during pen + touch interaction. We can detect whether the user holds the pen in a writing grip or tucked between his fingers. We can distinguish bare-handed inputs, such as drag and pinch gestures produced by the nonpreferred hand, from touch gestures produced by the hand holding the pen, which necessarily impart a detectable motion signal to the stylus. We can sense which hand grips the tablet, and determine the screen's relative orientation to the pen. By selectively combining these signals and using them to complement one another, we can tailor interaction to the context, such as by ignoring unintentional touch inputs while writing, or supporting contextually-appropriate tools such as a magnifier for detailed stroke work that appears when the user pinches with the pen tucked between his fingers. These and other techniques can be used to impart new, previously unanticipated subtleties to pen + touch interaction on tablets.
Swipeboard: a text entry technique for ultra-small interfaces that supports novice to expert transitions BIBAFull-Text 615-620
  Xiang 'Anthony' Chen; Tovi Grossman; George Fitzmaurice
Ultra-small smart devices, such as smart watches, have become increasingly popular in recent years. Most of these devices rely on touch as the primary input modality, which makes tasks such as text entry increasingly difficult as the devices continue to shrink. In the sole pursuit of entry speed, the ultimate solution is a shorthand technique (e.g., Morse code) that sequences tokens of input (e.g., key, tap, swipe) into unique representations of each character. However, learning such techniques is hard, as it often resorts to rote memory. Our technique, Swipeboard, leverages our spatial memory of a QWERTY keyboard to learn, and eventually master a shorthand, eyes-free text entry method designed for ultra-small interfaces. Characters are entered with two swipes; the first swipe specifies the region where the character is located, and the second swipe specifies the character within that region. Our study showed that with less than two hours' training, Tested on a reduced word set, Swipeboard users achieved 19.58 words per minute (WPM), 15% faster than an existing baseline technique.
Improvements to keyboard optimization with integer programming BIBAFull-Text 621-626
  Andreas Karrenbauer; Antti Oulasvirta
Keyboard optimization is concerned with the design of keyboards for different terminals, languages, user groups, and tasks. Previous work in HCI has used random search based methods, such as simulated annealing. These "black box" approaches are convenient, because good solutions are found quickly and no assumption must be made about the objective function. This paper contributes by developing integer programming (IP) as a complementary approach. To this end, we present IP formulations for the letter assignment problem and solve them by branch-and-bound. Although computationally expensive, we show that IP offers two strong benefits. First, its structured non-random search approach improves the outcomes. Second, it guarantees bounds, which increases the designer's confidence over the quality of results. We report improvements to three keyboard optimization cases.

Augmented reality II

Designer's augmented reality toolkit, ten years later: implications for new media authoring tools BIBAFull-Text 627-636
  Maribeth Gandy; Blair MacIntyre
The Designer's Augmented Reality Toolkit (DART) was an augmented (AR) and mixed reality (MR) authoring tool targeted at new media designers. It was released in 2003 and was heavily used by a diverse population of creators for the next several years [28]. Ten years later, we approached a group of users to collect reflections on their use of DART, the artifacts they produced, their subsequent AR/MR authoring, their thoughts on the challenges of AR/MR authoring in general, and the state of modern tools. In this paper we present the findings from in-depth interviews with these DART developers and other AR experts. Their reflections provide insights on how to successfully engage non-technologists with new media and the challenges they face during authoring, the unique requirements of new media authoring, and how modern tools are still not meeting the needs of this type of author, highlighting where additional research is needed.
RoomAlive: magical experiences enabled by scalable, adaptive projector-camera units BIBAFull-Text 637-644
  Brett Jones; Rajinder Sodhi; Michael Murdock; Ravish Mehra; Hrvoje Benko; Andrew Wilson; Eyal Ofek; Blair MacIntyre; Nikunj Raghuvanshi; Lior Shapira
RoomAlive is a proof-of-concept prototype that transforms any room into an immersive, augmented entertainment experience. Our system enables new interactive projection mapping experiences that dynamically adapts content to any room. Users can touch, shoot, stomp, dodge and steer projected content that seamlessly co-exists with their existing physical environment. The basic building blocks of RoomAlive are projector-depth camera units, which can be combined through a scalable, distributed framework. The projector-depth camera units are individually auto-calibrating, self-localizing, and create a unified model of the room with no user intervention. We investigate the design space of gaming experiences that are possible with RoomAlive and explore methods for dynamically mapping content based on room layout and user position. Finally we showcase four experience prototypes that demonstrate the novel interactive experiences that are possible with RoomAlive and discuss the design challenges of adapting any game to any room.
Dyadic projected spatial augmented reality BIBAFull-Text 645-655
  Hrvoje Benko; Andrew D. Wilson; Federico Zannier
Mano-a-Mano is a unique spatial augmented reality system that combines dynamic projection mapping, multiple perspective views and device-less interaction to support face to face, or dyadic, interaction with 3D virtual objects. Its main advantage over more traditional AR approaches, such as handheld devices with composited graphics or see-through head worn displays, is that users are able to interact with 3D virtual objects and each other without cumbersome devices that obstruct face to face interaction. We detail our prototype system and a number of interactive experiences. We present an initial user experiment that shows that participants are able to deduce the size and distance of a virtual projected object. A second experiment shows that participants are able to infer which of a number of targets the other user indicates by pointing.
Tracs: transparency-control for see-through displays BIBAFull-Text 657-661
  David Lindlbauer; Toru Aoki; Robert Walter; Yuji Uema; Anita Höchtl; Michael Haller; Masahiko Inami; Jörg Müller
We present Tracs, a dual-sided see-through display system with controllable transparency. Traditional displays are a constant visual and communication barrier, hindering fast and efficient collaboration of spatially close or facing co-workers. Transparent displays could potentially remove these barriers, but introduce new issues of personal privacy, screen content privacy and visual interference. We therefore propose a solution with controllable transparency to overcome these problems. Tracs consists of two see-through displays, with a transparency-control layer, a backlight layer and a polarization adjustment layer in-between. The transparency-control layer is built as a grid of individually addressable transparency-controlled patches, allowing users to control the transparency overall or just locally. Additionally, the locally switchable backlight layer improves the contrast of LCD screen content. Tracs allows users to switch between personal and collaborative work fast and easily and gives them full control of transparent regions on their display.
HaptoMime: mid-air haptic interaction with a floating virtual screen BIBAFull-Text 663-667
  Yasuaki Monnai; Keisuke Hasegawa; Masahiro Fujiwara; Kazuma Yoshino; Seki Inoue; Hiroyuki Shinoda
We present HaptoMime, a mid-air interaction system that allows users to touch a floating virtual screen with hands-free tactile feedback. Floating images formed by tailored light beams are inherently lacking in tactile feedback. Here we propose a method to superpose hands-free tactile feedback on such a floating image using ultrasound. By tracking a fingertip with an electronically steerable ultrasonic beam, the fingertip encounters a mechanical force consistent with the floating image. We demonstrate and characterize the proposed transmission scheme and discuss promising applications with an emphasis that it helps us 'pantomime' in mid-air.

Developer tools II

Declarative interaction design for data visualization BIBAFull-Text 669-678
  Arvind Satyanarayan; Kanit Wongsuphasawat; Jeffrey Heer
Declarative visualization grammars can accelerate development, facilitate retargeting across platforms, and allow language-level optimizations. However, existing declarative visualization languages are primarily concerned with visual encoding, and rely on imperative event handlers for interactive behaviors. In response, we introduce a model of declarative interaction design for data visualizations. Adopting methods from reactive programming, we model low-level events as composable data streams from which we form higher-level semantic signals. Signals feed predicates and scale inversions, which allow us to generalize interactive selections at the level of item geometry (pixels) into interactive queries over the data domain. Production rules then use these queries to manipulate the visualization's appearance. To facilitate reuse and sharing, these constructs can be encapsulated as named interactors: standalone, purely declarative specifications of interaction techniques. We assess our model's feasibility and expressivity by instantiating it with extensions to the Vega visualization grammar. Through a diverse range of examples, we demonstrate coverage over an established taxonomy of visualization interaction techniques.
Laevo: a temporal desktop interface for integrated knowledge work BIBAFull-Text 679-688
  Steven Jeuris; Steven Houben; Jakob Bardram
Prior studies show that knowledge work is characterized by highly interlinked practices, including task, file and window management. However, existing personal information management tools primarily focus on a limited subset of knowledge work, forcing users to perform additional manual configuration work to integrate the different tools they use. In order to understand tool usage, we review literature on how users' activities are created and evolve over time as part of knowledge worker practices. From this we derive the activity life cycle, a conceptual framework describing the different states and transitions of an activity. The life cycle is used to inform the design of Laevo, a temporal activity-centric desktop interface for personal knowledge work. Laevo allows users to structure work within dedicated workspaces, managed on a timeline. Through a centralized notification system which doubles as a to-do list, incoming interruptions can be handled. Our field study indicates how highlighting the temporal nature of activities results in lightweight scalable activity management, while making users more aware about their ongoing and planned work.
Reflection: enabling event prediction as an on-device service for mobile interaction BIBAFull-Text 689-698
  Yang Li
By knowing which upcoming action a user might perform, a mobile application can optimize its user interface for accomplishing the task. However, it is technically challenging for developers to implement event prediction in their own application. We created Reflection, an on-device service that answers queries from a mobile application regarding which actions the user is likely to perform at a given time. Any application can register itself and communicate with Reflection via a simple API. Reflection continuously learns a prediction model for each application based on its evolving event history. It employs a novel method for prediction by 1) combining multiple well-designed predictors with an online learning method, and 2) capturing event patterns not only within but also across registered applications -- only possible as an infrastructure solution. We evaluated Reflection with two sets of large-scale, in situ mobile event logs, which showed our infrastructure approach is feasible.

Closing keynote address

Humane representation of thought: a trail map for the 21st century BIBAFull-Text 699
  Bret Victor
New representations of thought -- written language, mathematical notation, information graphics, etc -- have been responsible for some of the most significant leaps in the progress of civilization, by expanding humanity's collectively-thinkable territory. But at debilitating cost. These representations, having been invented for static media such as paper, tap into a small subset of human capabilities and neglect the rest. Knowledge work means sitting at a desk, interpreting and manipulating symbols. The human body is reduced to an eye staring at tiny rectangles and fingers on a pen or keyboard. Like any severely unbalanced way of living, this is crippling to mind and body. But less obviously, and more importantly, it is enormously wasteful of the vast human potential. Human beings naturally have many powerful modes of thinking and understanding. Most are incompatible with static media. In a culture that has contorted itself around the limitations of marks on paper, these modes are undeveloped, unrecognized, or scorned.
   We are now seeing the start of a dynamic medium. To a large extent, people today are using this medium merely to emulate and extend static representations from the era of paper, and to further constrain the ways in which the human body can interact with external representations of thought.
   But the dynamic medium offers the opportunity to deliberately invent a humane and empowering form of knowledge work. We can design dynamic representations which draw on the entire range of human capabilities -- all senses, all forms of movement, all forms of understanding -- instead of straining a few and atrophying the rest.
   This talk suggests how each of the human activities in which thought is externalized (conversing, presenting, reading, writing, etc) can be redesigned around such representations.