HCI Bibliography Home | HCI Conferences | UIST Archive | Detailed Records | RefWorks | EndNote | Hide Abstracts
UIST Tables of Contents: 99000102030405060708091011-111-212-112-213-113-214-114-215-1

Proceedings of the 2009 ACM Symposium on User Interface Software and Technology

Fullname:Proceedings of the 2009 ACM Symposium on User Interface and Software Technology
Editors:Andrew Wilson; François Guimbretière
Location:Victoria, British Columbia, Canada
Dates:2009-Oct-04 to 2009-Oct-07
Publisher:ACM
Standard No:ISBN 1-60558-745-1, 978-1-60558-745-5; ACM DL: Table of Contents hcibib: UIST09
Papers:35
Pages:262
Links:Conference Home Page
  1. Keynote address
  2. We have contact!
  3. Hold me, squeeze me
  4. The third dimension
  5. New solutions for old problems
  6. Mobile magic
  7. Waiter, can you please bring me a fork?
  8. Meat-space
  9. Keynote address
  10. A.I./U.I.
  11. I walk the line
  12. The tangled web we weave

Keynote address

Interactive viscosity BIBAKFull-Text 1-2
  Stephen L. Macknik; Susana Martinez-Conde
Your conscious experience is not a function of the world, it is a function of the neural networks of your brain. Therefore, the biology of the brain and consciousness is as fundamental to understanding the universe, as we know it, as the high-energy physics of subatomic particles. This is especially true for the study of sensory and cognitive illusions, since they represent effects that clearly stand out as not representing the real world. That is, since illusions don't match reality we can know that by studying illusions we are studying exactly what the brain is actually doing, and not just what we think the brain should be doing. Your brain does a staggering amount of pragmatic self-dealing guesswork and outright confabulation in order to construct the highly imperfect mental simulation of reality known as "consciousness." This is not to say that objective reality isn't "out there" in a very real sense -- but no one lives there. No one's ever even been there for a visit. Ironically, the fact that consciousness feels like a solid, robust, fact-rich transcript of reality is just one of the countless illusions your brain creates for itself.
   Illusions are not errors of the brain. Far from it. Illusions arise from processes that are critical to our survival. Our brains have developed illusory processes so that we may experience the world in a ready-to-consume manner. Remove the machinery of illusion, and you unwind the entire tapestry of human awareness. Illusions are those perceptual experiences that do not match the physical reality. They are therefore exquisite tools with which to analyze the neural correlates of human perception and consciousness. Neuroscientists have long known that they can only be sure of where they stand, in terms of correlating neural responses to awareness, when they correlate the awareness of an illusion to the brain's response, specifically because of the illusions' mismatch with reality. The study of illusions is therefore of critical importance to the understanding of the basic mechanisms of sensory perception and conscious awareness.
   If you've ever seen a good magician perform, you know how thrilling it is to watch the impossible happening before your eyes. The laws of physics, probability, psychology and common sense -- the four trusty compass points in your mental map of reality -- are suddenly turned into liabilities. Objects and people appear, vanish, levitate, transpose, transform, and with all your smarts you can't imagine how it's being done. Magicians are the premier artists of attention and awareness, and they manipulate our cognition like clay on a potter's wheel.
   And the mechanisms underlying magic perception have implications for our daily lives. The magical arts work because humans have hardwired processes of attention and awareness that are hackable. By understanding how magicians hack our brains, we can better understand how we work.
Keywords: illusions, magic, the mind

We have contact!

Ripples: utilizing per-contact visualizations to improve user interaction with touch displays BIBAKFull-Text 3-12
  Daniel Wigdor; Sarah Williams; Michael Cronin; Robert Levy; Katie White; Maxim Mazeev; Hrvoje Benko
We present Ripples, a system which enables visualizations around each contact point on a touch display and, through these visualizations, provides feedback to the user about successes and errors of their touch interactions. Our visualization system is engineered to be overlaid on top of existing applications without requiring the applications to be modified in any way, and functions independently of the application's responses to user input. Ripples reduces the fundamental problem of ambiguity of feedback when an action results in an unexpected behaviour. This ambiguity can be caused by a wide variety of sources. We describe the ambiguity problem, and identify those sources. We then define a set of visual states and transitions needed to resolve this ambiguity, of use to anyone designing touch applications or systems. We then present the Ripples implementation of visualizations for those states, and the results of a user study demonstrating user preference for the system, and demonstrating its utility in reducing errors.
Keywords: fat fingers, feedback ambiguity, multi-touch, no touch left behind, precision input, tabletop, touch
Contact area interaction with sliding widgets BIBAKFull-Text 13-22
  Tomer Moscovich
We show how to design touchscreen widgets that respond to a finger's contact area. In standard touchscreen systems a finger often appears to touch several screen objects, but the system responds as though only a single pixel is touched. In contact area interaction all objects under the finger respond to the touch. Users activate control widgets by sliding a movable element, as though flipping a switch. These Sliding Widgets resolve selection ambiguity and provide designers with a rich vocabulary of self-disclosing interaction mechanism. We showcase the design of several types of Sliding Widgets, and report study results showing that the simplest of these widgets, the Sliding Button, performs on-par with medium-sized pushbuttons and offers greater accuracy for small-sized buttons.
Keywords: interactive surfaces, touch input, touchscreen
Detecting and leveraging finger orientation for interaction with direct-touch surfaces BIBAKFull-Text 23-32
  Feng Wang; Xiang Cao; Xiangshi Ren; Pourang Irani
Current interactions on direct-touch interactive surfaces are often modeled based on properties of the input channel that are common in traditional graphical user interfaces (GUI) such as x-y coordinate information. Leveraging additional information available on the surfaces could potentially result in richer and novel interactions. In this paper we specifically explore the role of finger orientation. This property is typically ignored in touch-based interactions partly because of the ambiguity in determining it solely from the contact shape. We present a simple algorithm that unambiguously detects the directed finger orientation vector in real-time from contact information only, by considering the dynamics of the finger landing process. Results of an experimental evaluation show that our algorithm is stable and accurate. We then demonstrate how finger orientation can be leveraged to enable novel interactions and to infer higher-level information such as hand occlusion or user position. We present a set of orientation-aware interaction techniques and widgets for direct-touch surfaces.
Keywords: direct-touch surface, finger orientation, orientation aware interface

Hold me, squeeze me

Mouse 2.0: multi-touch meets the mouse BIBAKFull-Text 33-42
  Nicolas Villar; Shahram Izadi; Dan Rosenfeld; Hrvoje Benko; John Helmes; Jonathan Westhues; Steve Hodges; Eyal Ofek; Alex Butler; Xiang Cao; Billy Chen
In this paper we present novel input devices that combine the standard capabilities of a computer mouse with multi-touch sensing. Our goal is to enrich traditional pointer-based desktop interactions with touch and gestures. To chart the design space, we present five different multi-touch mouse implementations. Each explores a different touch sensing strategy, which leads to differing form-factors and hence interactive possibilities. In addition to the detailed description of hardware and software implementations of our prototypes, we discuss the relative strengths, limitations and affordances of these novel input devices as informed by the results of a preliminary user study.
Keywords: desktop computing, input devices, mouse, multi-touch, novel hardware, surface computing
PhotoelasticTouch: transparent rubbery tangible interface using an LCD and photoelasticity BIBAKFull-Text 43-50
  Toshiki Sato; Haruko Mamiya; Hideki Koike; Kentaro Fukuchi
PhotoelasticTouch is a novel tabletop system designed to intuitively facilitate touch-based interaction via real objects made from transparent elastic material. The system utilizes vision-based recognition techniques and the photoelastic properties of the transparent rubber to recognize deformed regions of the elastic material. Our system works with elastic materials over a wide variety of shapes and does not require any explicit visual markers. Compared to traditional interactive surfaces, our 2.5 dimensional interface system enables direct touch interaction and soft tactile feedback. In this paper we present our force sensing technique using photoelasticity and describe the implementation of our prototype system. We also present three practical applications of PhotoelasticTouch, a force-sensitive touch panel, a tangible face application, and a paint application.
Keywords: direct manipulation, interactive surfaces, multi-touch, photoelasticity, tabletop interfaces
A reconfigurable ferromagnetic input device BIBAKFull-Text 51-54
  Jonathan Hook; Stuart Taylor; Alex Butler; Nicolas Villar; Shahram Izadi
We present a novel hardware device based on ferromagnetic sensing, capable of detecting the presence, position and deformation of any ferrous object placed on or near its surface. These objects can include ball bearings, magnets, iron filings, and soft malleable bladders filled with ferrofluid. Our technology can be used to build reconfigurable input devices -- where the physical form of the input device can be assembled using combinations of such ferrous objects. This allows users to rapidly construct new forms of input device, such as a trackball-style device based on a single large ball bearing, tangible mixers based on a collection of sliders and buttons with ferrous components, and multi-touch malleable surfaces using a ferrofluid bladder. We discuss the implementation of our technology, its strengths and limitations, and potential application scenarios.
Keywords: ferromagnetic sensing, malleable surface, multi-touch, reconfigurable input device, tangibles
A practical pressure sensitive computer keyboard BIBAKFull-Text 55-58
  Paul H. Dietz; Benjamin Eidelson; Jonathan Westhues; Steven Bathiche
A pressure sensitive computer keyboard is presented that independently senses the force level on every depressed key. The design leverages existing membrane technologies and is suitable for low-cost, high-volume manufacturing. A number of representative applications are discussed.
Keywords: force pressure sensitive keyboard

The third dimension

EverybodyLovesSketch: 3D sketching for a broader audience BIBAKFull-Text 59-68
  Seok-Hyung Bae; Ravin Balakrishnan; Karan Singh
We present EverybodyLovesSketch, a gesture-based 3D curve sketching system for rapid ideation and visualization of 3D forms, aimed at a broad audience. We first analyze traditional perspective drawing in professional practice. We then design a system built upon the paradigm of ILoveSketch, a 3D curve drawing system for design professionals. The new system incorporates many interaction aspects of perspective drawing with judicious automation to enable novices with no perspective training to proficiently create 3D curve sketches. EverybodyLovesSketch supports a number of novel interactions: tick-based sketch plane selection, single view definition of arbitrary extrusion vectors, multiple extruded surface sketching, copy-and-project of 3D curves, freeform surface sketching, and an interactive perspective grid. Finally, we present a study involving 49 high school students (with no formal artistic training) who each learned and used the system over 11 days, which provides detailed insights into the popularity, power and usability of the various techniques, and shows our system to be easily learnt and effectively used, with broad appeal.
Keywords: 3D sketching, axis widget, gestural interface, learnability, perspective sketching, sketch surface
A screen-space formulation for 2D and 3D direct manipulation BIBAKFull-Text 69-78
  Jason L. Reisman; Philip L. Davidson; Jefferson Y. Han
Rotate-Scale-Translate (RST) interactions have become the de facto standard when interacting with two-dimensional (2D) contexts in single-touch and multi-touch environments. Because the use of RST has thus far focused almost entirely on 2D, there are not yet standard techniques for extending these principles into three dimensions. In this paper we describe a screen-space method which fully captures the semantics of the traditional 2D RST multi-touch interaction, but also allows us to extend these same principles into three-dimensional (3D) interaction. Just like RST allows users to directly manipulate 2D contexts with two or more points, our method allows the user to directly manipulate 3D objects with three or more points. We show some novel interactions, which take perspective into account and are thus not available in orthographic environments. Furthermore, we identify key ambiguities and unexpected behaviors that arise when performing direct manipulation in 3D and offer solutions to mitigate the difficulties each presents. Finally, we show how to extend our method to meet application-specific control objectives, as well as show our method working in some example environments.
Keywords: constraints, direct manipulation, multi-touch, optimization, pressure

New solutions for old problems

Activity analysis enabling real-time video communication on mobile phones for deaf users BIBAKFull-Text 79-88
  Neva Cherniavsky; Jaehong Chon; Jacob O. Wobbrock; Richard E. Ladner; Eve A. Riskin
We describe our system called MobileASL for real-time video communication on the current U.S. mobile phone network. The goal of MobileASL is to enable Deaf people to communicate with Sign Language over mobile phones by compressing and transmitting sign language video in real-time on an off-the-shelf mobile phone, which has a weak processor, uses limited bandwidth, and has little battery capacity. We develop several H.264-compliant algorithms to save system resources while maintaining ASL intelligibility by focusing on the important segments of the video. We employ a dynamic skin-based region-of-interest (ROI) that encodes the skin at higher quality at the expense of the rest of the video. We also automatically recognize periods of signing versus not signing and raise and lower the frame rate accordingly, a technique we call variable frame rate (VFR).
   We show that our variable frame rate technique results in a 47% gain in battery life on the phone, corresponding to an extra 68 minutes of talk time. We also evaluate our system in a user study. Participants fluent in ASL engage in unconstrained conversations over mobile phones in a laboratory setting. We find that the ROI increases intelligibility and decreases guessing. VFR increases the need for signs to be repeated and the number of conversational breakdowns, but does not affect the users' perception of adopting the technology. These results show that our sign language sensitive algorithms can save considerable resources without sacrificing intelligibility.
Keywords: computer vision, mobileasl, region-of-interest, sign language, variable frame rate, video compression
User guided audio selection from complex sound mixtures BIBAKFull-Text 89-92
  Paris Smaragdis
In this paper we present a novel interface for selecting sounds in audio mixtures. Traditional interfaces in audio editors provide a graphical representation of sounds which is either a waveform, or some variation of a time/frequency transform. Although with these representations a user might be able to visually identify elements of sounds in a mixture, they do not facilitate object-specific editing (e.g. selecting only the voice of a singer in a song). This interface uses audio guidance from a user in order to select a target sound within a mixture. The user is asked to vocalize (or otherwise sonically represent) the desired target sound, and an automatic process identifies and isolates the elements of the mixture that best relate to the user's input. This way of pointing to specific parts of an audio stream allows a user to perform audio selections which would have been infeasible otherwise.
Keywords: audio interfaces
TapSongs: tapping rhythm-based passwords on a single binary sensor BIBAKFull-Text 93-96
  Jacob Otto Wobbrock
TapSongs are presented, which enable user authentication on a single "binary" sensor (e.g., button) by matching the rhythm of tap down/up events to a jingle timing model created by the user. We describe our matching algorithm, which employs absolute match criteria and learns from successful logins. We also present a study of 10 subjects showing that after they created their own TapSong models from 12 examples (< 2 minutes), their subsequent login attempts were 83.2% successful. Furthermore, aural and visual eavesdropping of the experimenter's logins resulted in only 10.7% successful impostor logins by subjects. Even when subjects heard the target jingles played by a synthesized piano, they were only 19.4% successful logging in as imposters. These results are attributable to subtle but reliable individual differences in people's tapping, which are supported by prior findings in music psychology.
Keywords: binary sensors, jingles, mobile devices, password entry, rhythm, songs, tapping, temporal strings, user authentication
Collabio: a game for annotating people within social networks BIBAKFull-Text 97-100
  Michael Bernstein; Desney Tan; Greg Smith; Mary Czerwinski; Eric Horvitz
We present Collabio, a social tagging game within an online social network that encourages friends to tag one another. Collabio's approach of incentivizing members of the social network to generate information about each other produces personalizing information about its users. We report usage log analysis, survey data, and a rating exercise demonstrating that Collabio tags are accurate and augment information that could have been scraped online.
Keywords: human computation, social computing, social tagging

Mobile magic

Disappearing mobile devices BIBAKFull-Text 101-110
  Tao Ni; Patrick Baudisch
In this paper, we extrapolate the evolution of mobile devices in one specific direction, namely miniaturization. While we maintain the concept of a device that people are aware of and interact with intentionally, we envision that this concept can become small enough to allow invisible integration into arbitrary surfaces or human skin, and thus truly ubiquitous use. This outcome assumed, we investigate what technology would be most likely to provide the basis for these devices, what abilities such devices can be expected to have, and whether or not devices that size can still allow for meaningful interaction. We survey candidate technologies, drill down on gesture-based interaction, and demonstrate how it can be adapted to the desired form factors. While the resulting devices offer only the bare minimum in feedback and only the most basic interactions, we demonstrate that simple applications remain possible. We complete our exploration with two studies in which we investigate the affordance of these devices more concretely, namely marking and text entry using a gesture alphabet.
Keywords: gesture, input device, interaction technique, miniaturization, mobile device, sensor, ubicomp, wearable
SemFeel: a user interface with semantic tactile feedback for mobile touch-screen devices BIBAKFull-Text 111-120
  Koji Yatani; Khai Nhut Truong
One of the challenges with using mobile touch-screen devices is that they do not provide tactile feedback to the user. Thus, the user is required to look at the screen to interact with these devices. In this paper, we present SemFeel, a tactile feedback system which informs the user about the presence of an object where she touches on the screen and can offer additional semantic information about that item. Through multiple vibration motors that we attached to the backside of a mobile touch-screen device, SemFeel can generate different patterns of vibration, such as ones that flow from right to left or from top to bottom, to help the user interact with a mobile device. Through two user studies, we show that users can distinguish ten different patterns, including linear patterns and a circular pattern, at approximately 90% accuracy, and that SemFeel supports accurate eyes-free interactions.
Keywords: mobile device, multiple vibration motors, tactile feedback, touch screen
Abracadabra: wireless, high-precision, and unpowered finger input for very small mobile devices BIBAKFull-Text 121-124
  Chris Harrison; Scott E. Hudson
We present Abracadabra, a magnetically driven input technique that offers users wireless, unpowered, high fidelity finger input for mobile devices with very small screens. By extending the input area to many times the size of the device's screen, our approach is able to offer a high C-D gain, enabling fine motor control. Additionally, screen occlusion can be reduced by moving interaction off of the display and into unused space around the device. We discuss several example applications as a proof of concept. Finally, results from our user study indicate radial targets as small as 16 degrees can achieve greater than 92% selection accuracy, outperforming comparable radial, touch-based finger input.
Keywords: cursor, finger input, gesture, interaction techniques, magnetic, mobile devices, pointing, small screens
Virtual shelves: interactions with orientation aware devices BIBAKFull-Text 125-128
  Frank Chun Yat Li; David Dearman; Khai N. Truong
Triggering shortcuts or actions on a mobile device often requires a long sequence of key presses. Because the functions of buttons are highly dependent on the current application's context, users are required to look at the display during interaction, even in many mobile situations when eyes-free interactions may be preferable. We present Virtual Shelves, a technique to trigger programmable shortcuts that leverages the user's spatial awareness and kinesthetic memory. With Virtual Shelves, the user triggers shortcuts by orienting a spatially-aware mobile device within the circular hemisphere in front of her. This space is segmented into definable and selectable regions along the phi and theta planes. We show that users can accurately point to 7 regions on the theta and 4 regions on the phi plane using only their kinesthetic memory. Building upon these results, we then evaluate a proof-of-concept prototype of the Virtual Shelves using a Nokia N93. The results show that Virtual Shelves is faster than the N93's native interface for common mobile phone tasks.
Keywords: kinesthetic memory, mobile computing, spatial memory, spatially aware devices

Waiter, can you please bring me a fork?

Bonfire: a nomadic system for hybrid laptop-tabletop interaction BIBAKFull-Text 129-138
  Shaun K. Kane; Daniel Avrahami; Jacob O. Wobbrock; Beverly Harrison; Adam D. Rea; Matthai Philipose; Anthony LaMarca
We present Bonfire, a self-contained mobile computing system that uses two laptop-mounted laser micro-projectors to project an interactive display space to either side of a laptop keyboard. Coupled with each micro-projector is a camera to enable hand gesture tracking, object recognition, and information transfer within the projected space. Thus, Bonfire is neither a pure laptop system nor a pure tabletop system, but an integration of the two into one new nomadic computing platform. This integration (1) enables observing the periphery and responding appropriately, e.g., to the casual placement of objects within its field of view, (2) enables integration between physical and digital objects via computer vision, (3) provides a horizontal surface in tandem with the usual vertical laptop display, allowing direct pointing and gestures, and (4) enlarges the input/output space to enrich existing applications. We describe Bonfire's architecture, and offer scenarios that highlight Bonfire's advantages. We also include lessons learned and insights for further development and use.
Keywords: ambient interaction, computer vision, extended display, gestures, laptop, micro-projector, object recognition, peripheral display, surface, tabletop, tangible bits
Interactions in the air: adding further depth to interactive tabletops BIBAKFull-Text 139-148
  Otmar Hilliges; Shahram Izadi; Andrew D. Wilson; Steve Hodges; Armando Garcia-Mendoza; Andreas Butz
Although interactive surfaces have many unique and compelling qualities, the interactions they support are by their very nature bound to the display surface. In this paper we present a technique for users to seamlessly switch between interacting on the tabletop surface to above it. Our aim is to leverage the space above the surface in combination with the regular tabletop display to allow more intuitive manipulation of digital content in three-dimensions. Our goal is to design a technique that closely resembles the ways we manipulate physical objects in the real-world; conceptually, allowing virtual objects to be 'picked up' off the tabletop surface in order to manipulate their three dimensional position or orientation. We chart the evolution of this technique, implemented on two rear projection-vision tabletops. Both use special projection screen materials to allow sensing at significant depths beyond the display. Existing and new computer vision techniques are used to sense hand gestures and postures above the tabletop, which can be used alongside more familiar multi-touch interactions. Interacting above the surface in this way opens up many interesting challenges. In particular it breaks the direct interaction metaphor that most tabletops afford. We present a novel shadow-based technique to help alleviate this issue. We discuss the strengths and limitations of our technique based on our own observations and initial user feedback, and provide various insights from comparing, and contrasting, our tabletop implementations.
Keywords: 3D, 3D graphics, computer vision, depth-sensing cameras, holoscreen, interactive surfaces, surfaces, switchable diffusers, tabletop
Augmenting interactive tables with mice & keyboards BIBAKFull-Text 149-152
  Björn Hartmann; Meredith Ringel Morris; Hrvoje Benko; Andrew D. Wilson
This note examines the role traditional input devices can play in surface computing. Mice and keyboards can enhance tabletop technologies since they support high fidelity input, facilitate interaction with distant objects, and serve as a proxy for user identity and position. Interactive tabletops, in turn, can enhance the functionality of traditional input devices: they provide spatial sensing, augment devices with co-located visual content, and support connections among a plurality of devices. We introduce eight interaction techniques for a table with mice and keyboards, and we discuss the design space of such interactions.
Keywords: interactive tabletops, surface computing
ARC-Pad: absolute+relative cursor positioning for large displays with a mobile touchscreen BIBAKFull-Text 153-156
  David C. McCallum; Pourang Irani
We introduce ARC-Pad (Absolute+Relative Cursor pad), a novel technique for interacting with large displays using a mobile phone's touchscreen. In ARC-Pad we combine ab-solute and relative cursor positioning. Tapping with ARC-Pad causes the cursor to jump to the corresponding location on the screen, providing rapid movement across large distances. For fine position control, users can also clutch using relative mode. Unlike prior hybrid cursor positioning techniques, ARC-Pad does not require an explicit switch between relative and absolute modes. We compared ARC-Pad with the relative positioning commonly found on touchpads. Users were given a target acquisition task on a large display, and results showed that they were faster with ARC-Pad, without sacrificing accuracy. Users welcomed the benefits associated with ARC-Pad.
Keywords: absolute position, clutching, cursor, touchpad

Meat-space

Using fNIRS brain sensing in realistic HCI settings: experiments and guidelines BIBAKFull-Text 157-166
  Erin Treacy Solovey; Audrey Girouard; Krysta Chauncey; Leanne M. Hirshfield; Angelo Sassaroli; Feng Zheng; Sergio Fantini; Robert J. K. Jacob
Because functional near-infrared spectroscopy (fNIRS) eases many of the restrictions of other brain sensors, it has potential to open up new possibilities for HCI research. From our experience using fNIRS technology for HCI, we identify several considerations and provide guidelines for using fNIRS in realistic HCI laboratory settings. We empirically examine whether typical human behavior (e.g. head and facial movement) or computer interaction (e.g. keyboard and mouse usage) interfere with brain measurement using fNIRS. Based on the results of our study, we establish which physical behaviors inherent in computer usage interfere with accurate fNIRS sensing of cognitive state information, which can be corrected in data analysis, and which are acceptable. With these findings, we hope to facilitate further adoption of fNIRS brain sensing technology in HCI research.
Keywords: BCI, brain-computer interface, fNIRS, functional near-infrared spectroscopy, human cognition
Enabling always-available input with muscle-computer interfaces BIBAKFull-Text 167-176
  T. Scott Saponas; Desney S. Tan; Dan Morris; Ravin Balakrishnan; Jim Turner; James A. Landay
Previous work has demonstrated the viability of applying offline analysis to interpret forearm electromyography (EMG) and classify finger gestures on a physical surface. We extend those results to bring us closer to using muscle-computer interfaces for always-available input in real-world applications. We leverage existing taxonomies of natural human grips to develop a gesture set covering interaction in free space even when hands are busy with other objects. We present a system that classifies these gestures in real-time and we introduce a bi-manual paradigm that enables use in interactive systems. We report experimental results demonstrating four-finger classification accuracies averaging 79% for pinching, 85% while holding a travel mug, and 88% when carrying a weighted bag. We further show generalizability across different arm postures and explore the tradeoffs of providing real-time visual feedback.
Keywords: electromyography (EMG), input, interaction, muscle-computer interface
Optically sensing tongue gestures for computer input BIBAKFull-Text 177-180
  T. Scott Saponas; Daniel Kelly; Babak A. Parviz; Desney S. Tan
Many patients with paralyzing injuries or medical conditions retain the use of their cranial nerves, which control the eyes, jaw, and tongue. While researchers have explored eye-tracking and speech technologies for these patients, we believe there is potential for directly sensing explicit tongue movement for controlling computers. In this paper, we describe a novel approach of using infrared optical sensors embedded within a dental retainer to sense tongue gestures. We describe an experiment showing our system effectively discriminating between four simple gestures with over 90% accuracy. In this experiment, users were also able to play the popular game Tetris with their tongues. Finally, we present lessons learned and opportunities for future work.
Keywords: gestures, infrared, tongue-computer interface

Keynote address

The rise of the expert amateur: DIY culture and citizen science BIBAKFull-Text 181-182
  Eric Paulos
We are at an important technological inflection point. Most of our computing systems have been designed and built by professionally trained experts (i.e. us -- computer scientists, engineers, and designers) for use in specific domains and to solve explicit problems. Artifacts often called "user manuals" traditionally prescribed the appropriate usage of these tools and implied an acceptable etiquette for interaction and experience. A fringe group of individuals usually labeled "hackers" or "nerds" have challenged this producer-consumer model of technology by hacking novel hardware and software features to "improve" our research and products while a similar creative group of technicians called "artists" have re-directed the techniques, tools, and tenets of accepted technological usage away from their typical manifestations in practicality and product. Over time the technological artifacts of these fringe groups and the support for their rhetoric have gained them a foothold into computing culture and eroded the established power discontinuities within the practice of computing research. We now expect our computing tools to be driven by an architecture of open participation and democracy that encourages users to add value to their tools and applications as they use them. Similarly, the bar for enabling the design of novel, personal computing systems and "hardware remixes" has fallen to the point where many non-experts and novices are readily embracing and creating fascinating and ingenious computing artifacts outside of our official and traditionally sanctioned academic research communities.
   But how have we as "expert" practitioners been influencing this discussion? By constructing a practice around the design and development of technology for task based and problem solving applications, we have unintentionally established such work as the status quo for the human computing experience. We have failed in our duty to open up alternate forums for technology to express itself and touch our lives beyond productivity and efficiency. Blinded by our quest for "smart technologies" we have forgotten to contemplate the design of technologies to inspire us to be smarter, more curious, and more inquisitive. We owe it to ourselves to rethink the impact we desire to have on this historic moment in computing culture. We must choose to participate in and perhaps lead a dialogue that heralds an expansive new acceptable practice of designing to enable participation by experts and non-experts alike. We are in the milieu of the rise of the "expert amateur".
   We must change our mantra: "not just usability but usefulness and relevancy to our world, its citizens, and our environment".
   We must design for the world and what matters.
   This means discussing our computing research alongside new keywords such as the economy, the environment, activism, poverty, healthcare, famine, homelessness, literacy, religion, and politics.
   This talk will explore the design territory and potential opportunities for all of us to collaborate and benefit as a society from this cultural movement.
Keywords: DIY culture, citizen science

A.I./U.I.

Sikuli: using GUI screenshots for search and automation BIBAKFull-Text 183-192
  Tom Yeh; Tsung-Hsiang Chang; Robert C. Miller
We present Sikuli, a visual approach to search and automation of graphical user interfaces using screenshots. Sikuli allows users to take a screenshot of a GUI element (such as a toolbar button, icon, or dialog box) and query a help system using the screenshot instead of the element's name. Sikuli also provides a visual scripting API for automating GUI interactions, using screenshot patterns to direct mouse and keyboard events. We report a web-based user study showing that searching by screenshot is easy to learn and faster to specify than keywords. We also demonstrate several automation tasks suitable for visual scripting, such as map navigation and bus tracking, and show how visual scripting can improve interactive help systems previously proposed in the literature.
Keywords: automation, image search, online help
CommunityCommands: command recommendations for software applications BIBAKFull-Text 193-202
  Justin Matejka; Wei Li; Tovi Grossman; George Fitzmaurice
We explore the use of modern recommender system technology to address the problem of learning software applications. Before describing our new command recommender system, we first define relevant design considerations. We then discuss a 3 month user study we conducted with professional users to evaluate our algorithms which generated customized recommendations for each user. Analysis shows that our item-based collaborative filtering algorithm generates 2.1 times as many good suggestions as existing techniques. In addition we present a prototype user interface to ambiently present command recommendations to users, which has received promising initial user feedback.
Keywords: recommender
Mining web interactions to automatically create mash-ups BIBAKFull-Text 203-212
  Jeffrey P. Bigham; Ryan S. Kaminsky; Jeffrey Nichols
The deep web contains an order of magnitude more information than the surface web, but that information is hidden behind the web forms of a large number of web sites. Metasearch engines can help users explore this information by aggregating results from multiple resources, but previously these could only be created and maintained by programmers. In this paper, we explore the automatic creation of metasearch mash-ups by mining the web interactions of multiple web users to find relations between query forms on different web sites. We also present an implemented system called TX2 that uses those connections to search multiple deep web resources simultaneously and integrate the results in context in a single results page. TX2 illustrates the promise of constructing mash-ups automatically and the potential of mining web interactions to explore deep web resources.
Keywords: deep web, mash-ups, meta-search, programming-by-example, web forms

I walk the line

Relaxed selection techniques for querying time-series graphs BIBAKFull-Text 213-222
  Christian Holz; Steven Feiner
Time-series graphs are often used to visualize phenomena that change over time. Common tasks include comparing values at different points in time and searching for specified patterns, either exact or approximate. However, tools that support time-series graphs typically separate query specification from the actual search process, allowing users to adapt the level of similarity only after specifying the pattern. We introduce relaxed selection techniques, in which users implicitly define a level of similarity that can vary across the search pattern, while creating a search query with a single-gesture interaction. Users sketch over part of the graph, establishing the level of similarity through either spatial deviations from the graph, or the speed at which they sketch (temporal deviations). In a user study, participants were significantly faster when using our temporally relaxed selection technique than when using traditional techniques. In addition, they achieved significantly higher precision and recall with our spatially relaxed selection technique compared to traditional techniques.
Keywords: similarity queries, time-series data
Integrated videos and maps for driving directions BIBAKFull-Text 223-232
  Billy Chen; Boris Neubert; Eyal Ofek; Oliver Deussen; Michael F. Cohen
While onboard navigation systems are gaining in importance, maps are still the medium of choice for laying out a route to a destination and for way finding. However, even with a map, one is almost always more comfortable navigating a route the second time due to the visual memory of the route. To make the first time navigating a route feel more familiar, we present a system that integrates a map with a video automatically constructed from panoramic imagery captured at close intervals along the route. The routing information is used to create a variable speed video depicting the route. During playback of the video, the frame and field of view are dynamically modulated to highlight salient features along the route and connect them back to the map. A user interface is demonstrated to allow exploration of the combined map, video, and textual driving directions. We discuss the construction of the hybrid map and video interface. Finally, we report the results of a study that provides evidence of the effectiveness of such a system for route following.
Keywords: driving directions, map, routes, video
Perceptual interpretation of ink annotations on line charts BIBAKFull-Text 233-236
  Nicholas Kong; Maneesh Agrawala
Asynchronous collaborators often use freeform ink annotations to point to visually salient perceptual features of line charts such as peaks or humps, valleys, rising slopes and declining slopes. We present a set of techniques for interpreting such annotations to algorithmically identify the corresponding perceptual parts. Our approach is to first apply a parts-based segmentation algorithm that identifies the visually salient perceptual parts in the chart. Our system then analyzes the freeform annotations to infer the corresponding peaks, valleys or sloping segments. Once the system has identified the perceptual parts it can highlight them to draw further attention and reduce ambiguity of interpretation in asynchronous collaborative discussions.
Keywords: deixis, freeform ink annotation, line charts, pointing, selection, visual perception, visualization

The tangled web we weave

Changing how people view changes on the web BIBAKFull-Text 237-246
  Jaime Teevan; Susan T. Dumais; Daniel J. Liebling; Richard L. Hughes
The Web is a dynamic information environment. Web content changes regularly and people revisit Web pages frequently. But the tools used to access the Web, including browsers and search engines, do little to explicitly support these dynamics. In this paper we present DiffIE, a browser plug-in that makes content change explicit in a simple and lightweight manner. DiffIE caches the pages a person visits and highlights how those pages have changed when the person returns to them. We describe how we built a stable, reliable, and usable system, including how we created compact, privacy-preserving page representations to support fast difference detection. Via a longitudinal user study, we explore how DiffIE changed the way people dealt with changing content. We find that much of its benefit came not from exposing expected change, but rather from drawing attention to unexpected change and helping people build a richer understanding of the Web content they frequent.
Keywords: change, dynamic information, re-finding, revisitation, web browsing, web dynamics
Overview based example selection in end user interactive concept learning BIBAKFull-Text 247-256
  Saleema Amershi; James Fogarty; Ashish Kapoor; Desney Tan
Interaction with large unstructured datasets is difficult because existing approaches, such as keyword search, are not always suited to describing concepts corresponding to the distinctions people want to make within datasets. One possible solution is to allow end users to train machine learning systems to identify desired concepts, a strategy known as interactive concept learning. A fundamental challenge is to design systems that preserve end user flexibility and control while also guiding them to provide examples that allow the machine learning system to effectively learn the desired concept. This paper presents our design and evaluation of four new overview based approaches to guiding example selection. We situate our explorations within CueFlik, a system examining end user interactive concept learning in Web image search. Our evaluation shows our approaches not only guide end users to select better training examples than the best performing previous design for this application, but also reduce the impact of not knowing when to stop training the system. We discuss challenges for end user interactive concept learning systems and identify opportunities for future research on the effective design of such systems.
Keywords: end-user interactive concept learning
The web page as a WYSIWYG end-user customizable database-backed information management application BIBAKFull-Text 257-260
  David R. Karger; Scott Ostler; Ryan Lee
Dido is an application (and application development environment) in a web page. It is a single web page containing rich structured data, an AJAXy interactive visualizer/editor for that data, and a "metaeditor" for WYSIWYG editing of the visualizer/editor. Historically, users have been limited to the data schemas, visualizations, and interactions offered by a small number of heavyweight applications. In contrast, Dido encourages and enables the end user to edit (not code) in his or her web browser a distinct ephemeral interaction "wrapper" for each data collection that is specifically suited to its intended use. Dido's active document metaphor has been explored before but we show how, given today's web infrastructure, it can be deployed in a small self-contained HTML document without touching a web client or server.
Keywords: databases, end user programming, wysiwyg