HCI Bibliography Home | HCI Conferences | FAC Archive | Detailed Records | RefWorks | EndNote | Hide Abstracts
FAC Tables of Contents: 070911131415

FAC 2007: 3rd International Conference on Foundations of Augmented Cognition

Fullname:FAC 2007: 3rd International Conference on Augmented Cognition
Note:Volume 16 of HCI International 2007
Editors:Dylan Schmorrow; Leah Reeves
Location:Beijing, China
Dates:2007-Jul-22 to 2007-Jul-27
Series:Lecture Notes in Computer Science 4565
Standard No:ISBN: 978-3-540-73215-0 (print), 978-3-540-73216-7 (online); hcibib: FAC07
Links:Online Proceedings | Publisher Book Page
  1. Part I: Augmented Cognition Methods and Techniques
  2. Part II: Applications of Augmented Cognition

Part I: Augmented Cognition Methods and Techniques

Development of Gauges for the QinetiQ Cognition Monitor BIBAKFull-Text 3-12
  Andy Belyavin; Chris Ryder; Blair Dickson
This paper describes the development of a new version of the calibration procedure for the QinetiQ Cognition Monitor so that it can be implemented to support the development of a cognitive cockpit at NAVAIR. A new signal cleaning procedure for processing the electro-encephalogram (EEG) automatically is outlined and the results from tests in the UK and US are summarized. It is concluded that estimates of the content of the EEG signal at high frequencies are important to gauges measuring verbal and spatial workload. The combination of results from UK and US tests suggests that the cleaning procedure is effective, although increased robustness of the verbal gauge is desirable.
Keywords: EEG; signal cleaning; calibration; cognitive workload
Quantitative EEG Changes Under Continuous Wakefulness and with Fatigue Countermeasures: Implications for Sustaining Aviator Performance BIBAKFull-Text 13-22
  Carlos Cardillo; Michael Russo; Patricia LeDuc; William Torch
Sleep management, naps, and pharmacological countermeasures may be combined to assist operators requiring around the clock tasks. We used QEEG methodologies to elucidate the CNS effects of stimulants (caffeine, modafinil, and dextroamphetamine) combined with sleep deprivation. Thirty-two UH-60 pilots were tested during 87 hours of continuous wakefulness using frequency analysis to quantify eight EEG channels for up to 20 frequency bands. Data were analyzed using brain mapping techniques and repeated measure analysis of variance. After 50 hours awake, all groups showed the sleep deprivation effects: increases in slow-waves and decreases in alpha activity. Caffeine and modafinil groups appeared to have the greatest degree of effect, producing delays on the electrophysiological deterioration for up to 46 hours into the sleep deprivation cycle. Additional analysis of these data could systematically correlate cognitive tasks and QEEG data for each pharmacologic intervention.
Keywords: QEEG; CNS; Brain Mapping; Artifact; Epoch
Exploring Calibration Techniques for Functional Near-Infrared Imaging (fNIR) Controlled Brain-Computer Interfaces BIBAKFull-Text 23-29
  Peter Wubbels; Erin M. Nishimura; Evan D. Rapoport; Benjamin A. Darling; Dennis Proffitt; Traci H. Downs; J. Hunter Downs
Functional near-infrared sensing (fNIR) enables real-time, noninvasive monitoring of cognitive activity by measuring the brain's hemodynamic and metabolic responses. We have demonstrated the ability for non-vocal and non-physical communications through detecting directed changes in cognitive tasks. Building upon past research, this paper reports methods that allow the calibration of the fNIR oxygenation signal to better be used in more complex communicative and selection tasks. This work is then discussed in the context of a faster, continuous fNIR brain-computer interface framework.
Keywords: Functional Near-Infrared Imagining; Brain-Computer Interface; fNIR; BCI
A Sensor Positioning System for Functional Near-Infrared Neuroimaging BIBAKFull-Text 30-37
  Ping He; Betty Yang; Sarah Hubbard; Justin Estepp; Glenn F. Wilson
In cognitive studies using functional near-infrared (fNIR) techniques, the optical sensors are placed over the scalp of the subject. In order to document the actual sensor location, a system is needed that can measure the 3D position of an arbitrary point on the scalp with a high precision and repeatability and express sensor location in reference to the international 10-20 system for convenience. In addition, in cognitive studies using functional magnetic resonance imaging (fMRI), the source location is commonly expressed using Talairach system. In order to correlate the results from the fNIR study with that of the fMRI study, one needs to project the source location in Talairach coordinates onto a site on the scalp for the placement of the fNIR sensors. This paper reports a sensor positioning system that is designed to achieve the above goals. Some initial experimental data using this system are presented.
Keywords: 10-20 system; brain mapping; fNIR; neuroimaging; Talairach
Ad-Hoc Wireless Body Area Network for Augmented Cognition Sensors BIBAKFull-Text 38-46
  Curtis S. Ikehara; Edoardo Biagioni; Martha E. Crosby
There is a "spaghetti" of wires when physiological sensors are used for augmented cognition tying a user down to a fixed location. Besides being visually unappealing, there are practical issues created by the "spaghetti" that have a negative impact on the adoption of sensor based augmented cognition technologies. A wireless sensor network can support sensors commonly used in augmented cognition. This paper describes the benefits and issues of implementing an ideal wireless network of physiological sensors using Bluetooth and other related types of networking approaches.
Keywords: Ad hoc network; wireless; biosensor; augmented cognition
Integrating Innovative Neuro-educational Technologies (I-Net) into K-12 Science Classrooms BIBAKFull-Text 47-56
  Ronald H. Stevens; Trysha Galloway; Chris Berka
With the U.S. facing a decline in science, math and engineering skills, there is a need for educators in these fields to team with engineers and cognitive scientists to pioneer novel approaches to science education. There is a strong need for the incorporation problem solving and emerging neuroscience technologies into mainstream classrooms, and for students and teachers to experience what it means at a very personal level, to engage in and struggle with solving difficult science problems. An innovating and engaging way of doing this is by making the problem solving process visible through the use of real-time electroencephalography cognitive metrics. There are educational, task, and measurement challenges that must be addressed to accomplish this goal. In this paper we detail some of these challenges, and possible solutions, to develop a framework for a new set of Interactive Neuro-Educational Technologies (I-Net).
Keywords: EEG; Problem solving; Skill Acquisition; Cognitive Workload
The Impact of Direct Data Entry by Sensory Devices on EMR Systems BIBAKFull-Text 57-64
  David Pager; Dennis J. Streveler; Luz Marina Quiroga
This paper takes an interdisciplinary look at how the electronic record is likely to evolve in the future. From what new sources will data be drawn? Will such data be directly recorded from sensory devices and from one's personal "memex"? Will these data enable a new set of outputs to more fully interconnect patients with their health care system. The paper considers the combined impact of a host of emerging technologies:
  • 1. the impact of the networking phenomenon
  • 2. the impact of adding robust patient access to the EMR
  • 3. the impact of the growing emergence of PHRs
  • 4. the impact of emerging technologies on the usability of the EMR
  • 5. the impact of direct sensory input devices
  • 6. the impact of bioinformatics and genomics
  • 7. the impact of the personal memex
    Keywords: Electronic Medical Record; Sensory Devices; Human-Computer Interface Issues; Biomedical Technology
  • Event-Related Brain Dynamics in Continuous Sustained-Attention Tasks BIBAKFull-Text 65-74
      Ruey-Song Huang; Tzyy-Ping Jung; Scott Makeig
    Event-related brain dynamics of electroencephalographic (EEG) activity in a continuous compensatory tracking task (CTT) and in a continuous driving simulation were analyzed by independent component analysis (ICA) and time-frequency techniques. We showed that changes in the level of subject performance are accompanied by distinct changes in EEG spectrum of a class of bilateral posterior independent EEG components. During periods of high-error (drowsy) performance, tonic alpha band EEG power was significantly elevated, compared to that during periods of low-error (alert) performance. In addition, characteristic transient (phasic) alpha and other band increases and decreases followed critical task events, depending on current performance level. These performance-related and event-related spectral changes were consistently observed across subjects and sessions, and were remarkably similar across the two continuous sustained-attention tasks.
    Keywords: EEG; ICA; brain dynamics; driving; drowsiness
    Information Filtering, Expertise and Cognitive Load BIBAKFull-Text 75-83
      David N. Chin
    Information filtering can be used to reduce cognitive load. However the expertise level of the user will greatly affect the effectiveness of information filtering. Any attempt to use neurophysiological measures of cognitive load for information filtering should take these effects into account in the design of the information filtering system. Combining information filtering, neurophysiological measurements of cognitive load and user modeling of expertise can improve performance. An integrated architecture for combining these techniques is described along with its application to routing information within a crisis management team.
    Keywords: information filtering; cognitive load; expertise; stereotypes; neurophysiological measures
    Using Eye Blinks as a Tool for Augmented Cognition BIBAKFull-Text 84-93
      Ric Heishman; Zoran Duric
    The human face comprises a complex system integrated from tissue, bone and electricity. Biometrics associated with this region provide useful information for a wide range of research disciplines. For those interested in augmented cognition, the metrics and behaviors inherent to eye blinks are particularly valuable in the interpretation and understanding of an individual's affective and cognitive states. Our work involves a novel integration of computer vision techniques for observing and interpreting the biometric information flow inherent in human eye blinks, and using these behavioral patterns to gain insight into the cognitive engagement and fatigue levels of individual subjects. Of particular interest are behavioral ambiguities -- both across multiple subjects and in individual subjects across various scenarios -- that present problems to both the observation and interpretation processes. Our work is pertinent to system development efforts across a wide range of applications, including driver fatigue, medical patient monitoring and critical system operator vigilance.
    Keywords: Eye; blink; cognition; affective computing; motion
    Assessing Information Presentation Preferences with Eye Movements BIBAKFull-Text 94-102
      Laurel King; Martha E. Crosby
    This study investigates the relationship between participants' self-reported high verbal or high visual information preferences and their performance and eye movements during analytical reasoning problems. Twelve participants, six male and six female, were selected as being more visual than verbal or more verbal than visual in approach, based on the results of a questionnaire administered to 140 college students. Selected participants were tested for individual differences in spatial ability and working memory capacity. They completed a repeated measures experiment while their eye movements were tracked to examine any correlation with their stated preference for verbal or visual information presentation. Performance on analytical reasoning problems with and without an optional diagram is compared between groups and within-subjects. Due to the small number of participants, between-group differences, although indicated, were mostly statistically insignificant. Within-subject analysis is still being completed, but trends in diagram usage are examined.
    Keywords: information presentation; eye tracking; analytical reasoning; problem representation
    Inclusive Design for Brain Body Interfaces BIBAKFull-Text 103-112
      Paul Gnanayutham; Jennifer George
    In comparison to all types of injury, those to the brain are among the most likely to result in death or permanent disability. A certain percentage of these brain-injured people cannot communicate, recreate, or control their environment due to severe motor impairment. This group of individuals with severe head injury has received little from assistive technology. Brain computer interfaces have opened up a spectrum of assistive technologies, which are particularly appropriate for people with traumatic brain-injury, especially those who suffer from "locked-in" syndrome. Previous research in this area developed brain body interfaces so that this group of brain-injured people can communicate, recreate and launch applications communicate using computers despite the severity of their brain injury, except for visually impaired and comatose participants. This paper reports on an exploratory investigation carried out with visually impaired using facial muscles or electromyography (EMG) to communicate using brain body interfaces.
    Keywords: Brain-Body Interface; Inclusive design; Neuro-rehabilitation; Assistive Technology and visual impairment; EEG; EMG and EOG
    A Human Computer Interface Using SSVEP-Based BCI Technology BIBAKFull-Text 113-119
      Chuan Jia; Honglai Xu; Bo Hong; Xiaorong Gao; Zhiguang Zhang; Shangkai Gao
    To address the issue of system simplicity and subject applicability, a brain controlled HCI system derived from steady state visual evoked potential (SSVEP) based brain computer interface (BCI) is proposed in this paper. Aiming at an external input device for personal computer, key issues of hardware and software design for better performance and user-friendly interface are introduced systematically. With proper parameter customization for each individual, an average information transfer rate of 46bits/min was achieved in the operation of dialing a phone number. With encouraging online performance and advantages of system simplicity, the proposed HCI using SSVEP-based BCI technology is promising for a substitute of standard computer input device for both health and disabled computer users.
    Keywords: brain-computer interface (BCI); steady state visual evoked potential (SSVEP); input device
    Enhanced P300-Based Cursor Movement Control BIBAKFull-Text 120-126
      Zhongwei Ma; Xiaorong Gao; Shangkai Gao
    In order to build a high-performance brain-computer interface (BCI) for cursor movement control, a P300-based BCI system using a five-select oddball paradigm was designed and implemented. We found that high intensity visual stimuli (HIVS) can improve the performance of BCI. 9 subjects participated in the test of the proposed BCI system. Each subject completed 40 epochs with HIVS and low intensity visual stimuli (LIVS) respectively. The preprocessed data were classified by support vector machines (SVM). The averaged waveforms both from HIVS and LIVS proved that this new paradigm can elicit evident P300 potentials. Furthermore, the results indicated the information transfer rate (ITR) of HIVS could reach 5.4 bit/min, which was higher than 4.6 bit/min of LIVS.
    Keywords: brain-computer interface (BCI); P300; support vector machine (SVM); information transfer rate (ITR); stimulus intensity; electrophysiological (EEG)
    Low Power Technology for Wearable Cognition Systems BIBAKFull-Text 127-136
      David C. Yates; Alexander J. Casson; Esther Rodríguez-Villegas
    This paper analyses a key tradeoff behind miniature devices intended to monitor cognition-related parameters. These devices are supposed to be worn by people that would otherwise be carrying on a normal life and this factor imposes important constraints in the design. They have to be wireless, wearable, discrete, low maintenance and reliable. In order to reduce power intelligence will be built into the sensors aiming to reduce the data transmission to only that information that it is strictly necessary. This intelligence will be in the form of an algorithm which will be required to be implemented in electronic circuits as part of the system. The complexity of the algorithm affects the complexity of the electronics and hence the power consumption. This, in turn affects the size of the battery and the overall size of the device. For the sensor to be low maintenance the device must operate for extended periods from the battery, adding more constraints to the power consumption of the electronic circuits. The battery must be kept small so that the overall size of the device is small and lightweight enough to be worn on the body and the more discrete the device the higher consumer compliance. A tradeoff has to be met between the algorithm complexity, the power consumption of the electronics required to realize the latter, the power consumption required to transmit data and the battery size and lifetime.
    Keywords: Ambulatory EEG; low-power; wearable; wireless; cognition
    Novel Hybrid Bioelectrodes for Ambulatory Zero-Prep EEG Measurements Using Multi-channel Wireless EEG System BIBAKFull-Text 137-146
      Robert Matthews; Neil J. McDonald; Harini Anumula; Jamison Woodward; Peter J. Turner; Martin A. Steindorf; Kaichun Chang; Joseph M. Pendleton
    This paper describes a wireless multi-channel system for zero-prep electroencephalogram (EEG) measurements in operational settings. The EEG sensors are based upon a novel hybrid (capacitive/resistive) bioelectrode technology that requires no modification to the skin's outer layer. High impedance techniques developed for QUASAR's capacitive electrocardiogram (ECG) sensors minimize the sensor's susceptibility to common-mode (CM) interference, and permit EEG measurements with electrode-subject impedances as large as 107 Ω. Results for a side-by-side comparison between the hybrid sensors and conventional wet electrodes for EEG measurements are presented. A high level of correlation between the two electrode technologies (>99 subjects seated) was observed. The electronics package for the EEG system is based upon a miniature, ultra-low power microprocessor-controlled data acquisition system and a miniaturized wireless transceiver that can operate in excess of 72 hours from two AAA batteries.
    Keywords: EEG; biosensors; high impedance; wireless
    Measuring Cognitive Task Load on a Naval Ship: Implications of a Real World Environment BIBAFull-Text 147-156
      Marc Grootjen; Mark A. Neerincx; Jochum C. M. van Weert; Khiet P. Truong
    Application of more and more automation in process control shifts the operator's task from manual to supervisory control. Increasing system autonomy, complexity and information fluctuations make it extremely difficult to develop static support concepts that cover all critical situations after implementing the system. Therefore, support systems in dynamic domains should be dynamic as the domain itself. This paper elaborates on the state information needed from the operator to generate effective mitigation strategies. We describe implications of a real world experiment onboard three frigates of the Royal Netherlands Navy. Although new techniques allow us to measure, combine and gain insight in physiological, subjective and task information, many practical issues need to be solved.
    Measuring Spatial Factors in Comparative Judgments About Large Numerosities BIBAFull-Text 157-165
      Catherine Sophian
    Numerical information is crucial to successful performance on many tasks. Accordingly, as a basis for developing augmented cognition applications, it is important to understand how people apprehend numerical information and whether there are systematic limitations on their ability to do so accurately. This paper reports research on the role of non-numerical spatial information and of numerical representations in adults' judgments about large-numerosity spatial arrays. Arrays that contained more open space tended to be perceived as less numerous than ones with less open space. Further, the accuracy with which viewers estimated the arrays' numerosities bore little relation to their success in identifying the more numerous array in each pair. Numerical judgments thus are heavily influenced by spatial information that is not necessarily a reliable cue to numerosity. While some information about absolute numerosity is extracted in making numerical comparisons, it appears to be very imprecise.
    Augmented Metacognition Addressing Dynamic Allocation of Tasks Requiring Visual Attention BIBAKFull-Text 166-175
      Tibor Bosse; Willem A. van Doesburg; Peter-Paul van Maanen; Jan Treur
    This paper discusses the use of cognitive models as augmented metacognition on task allocation for tasks requiring visual attention. In the domain of naval warfare, the complex and dynamic nature of the environment makes that one has to deal with a large number of tasks in parallel. Therefore, humans are often supported by software agents that take over part of these tasks. However, a problem is how to determine an appropriate allocation of tasks. Due to the rapidly changing environment, such a work division cannot be fixed beforehand: dynamic task allocation at runtime is needed. Unfortunately, in alarming situations the human does not have the time for this coordination. Therefore system-triggered dynamic task allocation is desirable. The paper discusses the possibilities of such a system for tasks requiring visual attention.
    Keywords: Visual attention; cognitive modeling; augmented metacognition
    Highly Configurable Software Architecture Framework for Acquisition and Visualization of Biometric Data BIBAKFull-Text 176-185
      Jan Stelovsky
    The research in augmented cognition and its practical applications rely heavily on the acquisition and evaluation of biometrics data. We propose software architecture that offers unified approach to the integration of emerging hardware and evaluation technologies. In this paper we focus on the software layers that combine the data events and offer visual representations of the results. In particular, we show that the common evaluation of the collected data as well as the commonly used graphical depictions of the results can be achieved using a fully modular and extendible software architecture.
    Keywords: software architecture; visualization; biometrics; eye-tracking
    Simulation Fidelity Design Informed by Physiologically-Based Measurement Tools BIBAKFull-Text 186-194
      Jack Maxwell Vice; Corinna E. Lathan; Anna D. Lockerd; James M. Hitt
    Virtual environments (VE's) and simulations are being employed for training applications in a wide variety of disciplines, both military and civilian. The common assumption is that the more realistic the VE, the better the transfer of training to real world tasks. However, some aspects of task content and fidelity may result in stronger transfer of training than even the most high fidelity simulations. A physiologically-based system capable of dynamically detecting changes in operator behavior and physiology throughout a VE experience and comparing those changes to operator behavior and physiology in real-world tasks, could potentially determine which aspects of VE fidelity will have the highest impact on transfer of training. Thus, development of training assessment and guidance tools that utilize operator behavior and physiology to determine VE effectiveness and transfer of training are needed.
    Keywords: virtual reality; simulation; transfer of training; physiology; behavior; training effectiveness
    Reverse Engineering the Visual System Via Genetic Programs BIBAKFull-Text 195-200
      Diglio A. Simoni
    We propose a datamining based method for automated reverse engineering of search strategies during active visual search tasks. The method uses a genetic program (GP) that evolves populations of fuzzy decision trees and selects an optimal one. Previous psychophysical observations of subjects engaged in a simple search task result in a database of stimulus conditions and concomitant measures of eye gaze information and associated psychophysical metrics that globally describe the subjects search strategies. Fuzzy rules about the likely design properties of the components of the visual system involved in selecting fixation location during search are defined based on these metrics. A fitness function that incorporates both the fuzzy rules and the information in the database is used to conduct GP based datamining. The information extracted through the GP process is the internal design specification of the visual system vis-à-vis active visual search.
    Keywords: active visual search; eye tracking; psychophysics; fuzzy logic; datamining; knowledge discovery; genetic programming; reverse engineering
    EEG-Based Estimation of Mental Fatigue: Convergent Evidence for a Three-State Model BIBAKFull-Text 201-211
      Leonard J. Trejo; Kevin Knuth; Raquel Prado; Roman Rosipal; Karla Kubitz; Rebekah Kochavi; Bryan Matthews; Yuzheng Zhang
    Two new computational models show that the EEG distinguishes three distinct mental states ranging from alert to fatigue. State 1 indicates heightened alertness and is frequently present during the first few minutes of time on task. State 2 indicates normal alertness, often following and lasting longer than State 1. State 3 indicates fatigue, usually following State 2, but sometimes alternating with State 1 and State 2. Thirty-channel EEGs were recorded from 16 subjects who performed up to 180 min of nonstop computer-based mental arithmetic. Alert or fatigued states were independently confirmed with measures of subjects' performance and pre- or post-task mood. We found convergent evidence for a three-state model of fatigue using Bayesian analysis of two different types of EEG features, both computed for single 13-s EEG epochs: 1) kernel partial least squares scores representing composite multichannel power spectra; 2) amplitude and frequency parameters of multiple single-channel autoregressive models.
    Keywords: EEG; mental fatigue; alertness; computational models; situation awareness; performance monitoring; augmented cognition
    Augmenting Task-Centered Design with Operator State Assessment Technologies BIBAFull-Text 212-219
      Karl F. Van Orden; Erik Viirre; David A. Kobus
    Task-Centered Design (TCD) of human-system interfaces focuses on supporting the user throughout all phases of tasks, from initiation to completion. TCD typically requires software that monitors aspects of system information to trigger tasks, develop user-friendly information sets, propose task solutions and actions, and confirm actions as directed and approved by the operator. The operator monitors tasks awaiting completion on a Task Manager display. We demonstrate that moment-to-moment operator workload monitoring is greatly facilitated by TCD. Workload estimates were obtained every 2-min over the course of a 35-min test session during an air defense command and control scenario. Workload was readily modeled by the task loading, and the density of track icons on the display. A second study related the unitary workload estimates to NASA TLX workload subscales. Unpublished data from our laboratory indicated that eye activity measures (e.g., blink frequency and duration, pupil diameter, fixation frequency and dwell time) did not improve the estimation of workload. These findings indicate that at least for well-executed TCD systems, eye tracking technologies may be best employed to monitor for fatigue and incongruities between the focus of attention and task requirements. Recent findings using EEG hold promise for the identification of specific brain signatures of confusion, orientation, and loss of situational awareness. Thus the critical element of human directed systems is good initial design. Understanding of the task will lead to system automation that can balance the workload of the operator, who is functioning in a normal state. However, physiological monitoring will be most useful if operators veer beyond their normal conditions and are confused, overloaded, disoriented or have other impairments to their abilities. By detecting the operator's loss of function early, inappropriate operator inputs can potentially be avoided.
    Augmented Cognition and Cognitive State Assessment Technology -- Near-Term, Mid-Term, and Long-Term Research Objectives BIBAKFull-Text 220-228
      Leah Reeves; Dylan Schmorrow; Kay M. Stanney
    The 1st Augmented Cognition International (ACI) conference was held in July 2005 in conjunction with the HCI International conference in Las Vegas, Nevada. A full day working group session was held during this inaugural ACI conference to facilitate the development of an Augmented Cognition R&D agenda for the near- (1-2 years), medium- (within 5 years) and long-term (> 5 years). Working group attendees included scientists, developers, and practitioners from government, academia, and industry who were invited to participate based on their numerous years of experience and expertise in the Augmented Cognition and related fields. This article highlights key results of the workshop discussions that were focused on Cognitive State Assessment (CSA) R&D objectives, particularly with regard to the design and implementation of CSA tools and techniques.
    Keywords: Augmented Cognition; human factors; cognitive state assessment; sensors; design; neuroergonomics; neurotechnologies; neurophysiological

    Part II: Applications of Augmented Cognition

    Augmented Cognition, Universal Access and Social Intelligence in the Information Society BIBAFull-Text 231-240
      Ray Adams; Satinder P. Gill
    The two concepts of universal access and augmented cognition have both contributed significantly to providing the intended users of modern information and communication technology with the necessary resources to achieve enhanced interaction and performance. The two concepts share a number of important features including; the improvement of user performance, the use of concepts from cognitive psychology, a consideration of user modelling, a user sensitive approach, support for customisation, personalisation, adaptation and adaptive systems. They differentially emphasise; short term and long term demands, ambient intelligence, ubiquitous computing, people with disabilities, the Information Society and social skills. Since the present research programme (CIRCUA) is focussed upon the design and evaluation of universally accessible systems within a vocational context, the concepts of universal access and augmented are both very relevant, though both need to draw more upon the concept of social intelligence if they to tackle key issues of the Information Society.
    Intent Driven Interfaces to Ubiquitous Computers BIBAKFull-Text 241-250
      Neil G. Scott; Martha E. Crosby
    An intent driven interface allows a person to control a computer by stating an intended outcome rather than entering the sequence of tasks required to achieve the same outcome. Techniques that were originally developed as part of a universal access accelerator for individuals with disabilities are now being applied as convenience and productivity tools for accessing any computer based device, appliance or system. An intelligent universal serial bus (USB) Hub, called the iTASK Module, determines user intent independently of the input source or the system that is accessed. iTASK Modules can be interconnected to support multi-user collaboration and sharing of system resources without requiring any hardware or software changes to the accessed system.
    Keywords: IDEAL; iTASK Module; NIP; intent; ubiquitous; USB
    Foundations for Creating a Distributed Adaptive User Interface BIBAKFull-Text 251-257
      Don Kemper; Larry D. Davis; Cali M. Fidopiastis; Denise M. Nicholson
    Distributed simulation allows multiple users to develop and improve interactions without having to be collocated. To enhance such interaction, we present the foundation for a distributed, multi-modal, adaptive user interface. First, the interface concept is placed within the context of a closed-loop human system. Next, the present prototype implementation is described. Then, the concept of modifying interface elements based upon a combination of actual, physically simulated, and virtual devices is discussed. Finally, we discuss the possibility for self-adaptation, design challenges, and directions for future development.
    Keywords: Multi-Modal Adaptive User Interface; Closed-Loop Human Systems
    EMMA: An Adaptive Display for Virtual Therapy BIBAKFull-Text 258-265
      Mariano Alcañiz Raya; Cristina Botella; Beatriz Rey; Rosa María Baños; José Antonio Lozano; Nuria Lasso de la Vega; Diana Castilla; Javier Montesa; Antonio Hospitaler
    Environments used up to now for therapeutic applications are invariable ones. Their contents can not be changed neither by the therapist nor by the patient. However, this is a technical issue that can be solved with current technology. In this paper, we describe a virtual environment that has been developed taking into account this factor. The main technical feature of the environment is that its aspect can be modified controlled by the therapist that conducts the clinical sessions depending on the emotions that the patient is feeling at each moment, and depending on the purpose of the clinical session. The environment has been applied for the treatment of post traumatic stress disorder, pathological bereavement, and adjustment disorder in adult population. In the paper we present some data showing its utility for the treatment of a phobia in a 9-year-old child.
    Keywords: Virtual Reality; Adaptive display; Virtual therapy
    Closed-Loop Adaptive Decision Support Based on Automated Trust Assessment BIBAFull-Text 266-275
      Peter-Paul van Maanen; Tomas Klos; Kees van Dongen
    This paper argues that it is important to study issues concerning trust and reliance when developing systems that are intended to augment cognition. Operators often under-rely on the help of a support system that provides advice or that performs certain cognitive tasks autonomously. The decision to rely on support seems to be largely determined by the notion of relative trust. However, this decision to rely on support is not always appropriate, especially when support systems are not perfectly reliable. Because the operator's reliability estimations are typically imperfectly aligned or calibrated with the support system's true capabilities, we propose that the aid makes an estimation of the extent of this calibration (under different circumstances) and intervenes accordingly. This system is intended to improve overall performance of the operator-support system as a whole. The possibilities in terms of application of these ideas are explored and an implementation of this concept in an abstract task environment has been used as a case study.
    A Closed-Loop Adaptive System for Command and Control BIBAFull-Text 276-285
      Tjerk de Greef; Henryk F. R. Arciszewski
    On Navy ships, technological developments enable crews to work more efficiently and effectively. However, in such complex, autonomous, and information-rich environments a competition for the users' attention is going on between different information items, possibly leading to a cognitive overload. This overload originates in the limitations of human attention and constitutes a well-known and well-studied bottleneck in human information processing. The concept of adaptive automation promises a solution to the overwhelmed operator by shifting the amount of work between the human and the system in time, while maintaining a high level of situation awareness. One of the most critical challenges in developing adaptive human-machine collaboration concerns the design of a trigger mechanism. This paper discusses and evaluates a number of possible triggers for the usage in closed-loop adaptive automation from the perspective of command and control.
    Attuning In-Car User Interfaces to the Momentary Cognitive Load BIBAKFull-Text 286-293
      Marieka Hoedemaeker; Mark A. Neerincx
    Cars, trucks and busses are more and more equipped with functions and services that drivers are supposed to operate and understand. The most important developments in this area are the Advanced Driver Assistance Systems (ADAS) and In Vehicle Information Systems (IVIS). In order to make sure that the driver understands and appreciates (comfort) these services and traffic safety is not at risk (distraction, workload), the HMI's (Human Machine Interfaces) of all these functions should be attuned to each other, to the driver, and to the context. For attuning the functions to each other, a HMI platform is needed on which these functions are integrated. For attuning the functions to the driver it is necessary to have knowledge about the momentary state of the driver and of the intentions of the driver at a certain moment. For attuning the functions to the context, it is required to sense the relevant environmental conditions or states. This paper shows that a recent cognitive task load model from process control domain can be applied for the design of adaptive in-car user interfaces. Furthermore, current developments of such interfaces are being discussed.
    Keywords: In-car services; workload; adaptive user interface; central management
    EEG-Based Drivers' Drowsiness Monitoring Using a Hierarchical Gaussian Mixture Model BIBAFull-Text 294-303
      Roman Rosipal; Björn Peters; Göran Kecklund; Torbjörn Åkerstedt; Georg Gruber; Michael Woertz; Peter Anderer; Georg Dorffner
    We developed an EEG-based probabilistic model, which effectively predicts drowsiness levels of thirty-two subjects involved in a moving base driving simulator experiment. A hierarchical Gaussian mixture model (hGMM) with two mixture components at the lower hierarchical level is used. Each mixture models data density distribution of one of the two drowsiness cornerstones/classes represented by 4-second long EEG segments with low and high drowsiness levels. We transfer spectral contents of each EEG segment into a compact form of autoregressive model coefficients. The Karolinska drowsiness scoring method is used to initially label data belonging to individual classes. We demonstrate good agreement between Karolinska drowsiness scores and the predicted drowsiness, when the hGMM is applied to continuously monitor drowsiness over the time-course of driving sessions. The computations associated with the approach are fast enough to build up a practical real-time drowsiness monitoring system.
    The Effect of Fatigue on Cognitive and Psychomotor Skills of Surgical Residents BIBAFull-Text 304-313
      Kanav Kahol; Mark Smith; Stephanie Mayes; Mary Deka; Vikram Deka; John Ferrara; Sethuraman Panchanathan
    Surgical residents are exposed to a significant amount of cognitive load during call. While various efforts have been made to quantify the effect of fatigue and sleep deprivation on the psychomotor skills of surgical residents, there is very little investigations into the effect of these factors on cognitive skills. However, this is an important issue in medical curriculum design, as much of the medical errors are procedural in nature and are not psychomotor. In this paper, we present a study that aimed to quantify the effect of fatigue on cognitive skills. We employed hand movement data for developing a proficiency measure of surgical skill. The difference in proficiencies measured through hand movement post call and pre call was determined. The simulation tasks were designed to challenge working memory, attention of the user. The results showed a significant difference in hand movement proficiencies as well as behavioral errors pre and post-call. EEG Data was also gathered during simulation tasks pre and post call through the B-Alert® Bluetooth EEG technology. The B-Alert® software was analyzed to reveal ratings of alertness/drowsiness, engagement, mental workload and distraction. The results showed statistically significant difference in EEG ratings in pre call and post call condition.
    Assessing the Real-Time Cognitive Capabilities of First Responders Using Emerging Technologies in Manikin Simulators BIBAKFull-Text 314-322
      Kathleen Kihmm Connolly; Lawrence Burgess
    Medical triage can be a highly stressful situation in which decisions and task performance may have life or death consequences. Individual responses in stressful situations may affect task performance. Increased injury or casualties may occur without proper training and competency of the first-responder. The emerging technologies of advanced manikin simulators have afforded anatomic, physiological, and pharmacologic realism, which can be dynamically programmed in real-time. This has increased the capabilities and realism of manikin simulations, thus allowing advanced learning techniques that were not previously possible. By employing physiological measures of the learner to determine areas of overwhelming task complexity, which may degrade performance, a method such that the training can be adjusted to the real-time cognitive needs/load of the learner (adaptive scaffolding) can be applied. This has the potential to enhance learning and human data processing in medical triage training.
    Keywords: Triage; manikin simulators; first-responder; physiological sensors; adaptive scaffolding
    Physiologic System Interfaces Using fNIR with Tactile Feedback for Improving Operator Effectiveness BIBAKFull-Text 323-328
      Erin M. Nishimura; Evan D. Rapoport; Benjamin A. Darling; Dennis Proffitt; Traci H. Downs; J. Hunter Downs
    This paper explores the validation of tactile mechanisms as an effective means of communications for integration into a physiologic system interface (PSI). Tactile communications can offer a channel that only minimally interferes with a primary or concurrent task. The PSI will use functional brain imaging techniques, specifically functional near-infrared imaging (fNIR), to determine cognitive workload in language and visual processing areas of the brain. The resulting closed-loop system will thus have the capability of providing the operator with necessary information by using the modality most available to the user, thus enabling effective multi-tasking and minimal task interference.
    Keywords: physiologic system interfaces; functional near-infrared (fNIR); tactile; tactile communications
    A Model for Visio-Haptic Attention for Efficient Resource Allocation in Multimodal Environments BIBAKFull-Text 329-336
      Priyamvada Tripathi; Kanav Kahol; Anusha Sridaran; Sethuraman Panchanathan
    Sequences of visual and haptic exploration were obtained on surfaces of different curvature from human subjects. We then extracted regions of interest (ROI) from the data as a function of number of times a subject fixated on a certain location on object and amount of time spent on such each location. Simple models like a plane, cone, cylinder, paraboloid, hyperboloid, ellipsoid, simple-saddle and a monkey-saddle were generated. Gaussian curvature representation of each point on all the surfaces was pre-computed. The surfaces have been previously tested for haptic and visual realism and distinctness by human subjects in a separate experiment. Both visual and haptic rendering were subsequently used for exploration by human subjects to study whether there is a similarity between the visual ROI and haptic ROIs. Additionally, we wanted to see if there is a correlation between curvature values and the ROIs thus obtained. A multiple regression model was further developed to see if this data can be used to predict the visual exploration path using haptic curvature saliency measures.
    Keywords: Vision; Haptics; Eye movements; Attention; Saliency; Regions of Interest
    Towards Attention-Guided Human-Computer Collaborative Reasoning for Spatial Configuration and Design BIBAKFull-Text 337-345
      Sven Bertel
    In this contribution, we investigate the interrelation between visual focus and higher-level cognitive processing during diagrammatic problem solving. It is argued that eye movement data can be employed for the detection and prediction of model selection in mental model-based reasoning contexts. The argument is substantiated by results from an explorative eye tracking study. Implications for the role of cognitive models in human-computer collaborative reasoning and potential application domains are discussed.
    Keywords: Visual focus; focus of attention; eye tracking; problem solving; human-computer collaborative reasoning; computational cognitive modeling
    Automated SAF Adaptation Tool (ASAT) BIBAKFull-Text 346-353
      Roy Stripling; Joseph T. Coyne; Anna Cole; Daniel Afergan; Raymond L. Barnes; Kelly A. Rossi; Leah Reeves; Dylan Schmorrow
    The purpose of this paper is to describe a new, user-friendly tool that will enable researchers and instructors to setup and run virtual environment scenarios that adapt to the VE user's real-time performance and cognitive status. This tool, the Automated SAF Adaptation Tool (ASAT), will work with existing performance and cognitive state assessment software, and with existing semi-automated forces (SAF) behavior engines. ASAT will collect processed performance and cognitive state data from the assessment software and trigger SAF behavior setting manipulations that were pre-selected by the SAF operator. A key feature of ASAT is the ability to setup and execute these real-time manipulations without the need to alter code in either the assessment software or the SAF engines.
    Keywords: semi-automated forces; SAF; real-time; adaptation; virtual environment; training; cognitive state
    Unobtrusive Multimodal Emotion Detection in Adaptive Interfaces: Speech and Facial Expressions BIBAKFull-Text 354-363
      Khiet P. Truong; David A. van Leeuwen; Mark A. Neerincx
    Two unobtrusive modalities for automatic emotion recognition are discussed: speech and facial expressions. First, an overview is given of emotion recognition studies based on a combination of speech and facial expressions. We will identify difficulties concerning data collection, data fusion, system evaluation and emotion annotation that one is most likely to encounter in emotion recognition research. Further, we identify some of the possible applications for emotion recognition such as health monitoring or e-learning systems. Finally, we will discuss the growing need for developing agreed standards in automatic emotion recognition research.
    Keywords: emotion detection; emotion recognition; classification; speech; facial expression; emotion database
    Embedding Hercule Poirot in Networks: Addressing Inefficiencies in Digital Forensic Investigations BIBAKFull-Text 364-372
      Barbara Endicott-Popovsky; Deborah A. Frincke
    Forensic investigations on networks are not scalable in terms of time and money [1]. Those investigations that do occur consume months of attention from the very experts who should be investing in more productive activities, like designing and improving network performance [1]. Given these circumstances, organizations often must select which cases to pursue, ignoring many that could be prosecuted, if time allowed. Recognizing the exponential growth in the number of crimes that employ computers and networks that become subject to digital evidence procedures, researchers and practitioners, alike, have called for embedding forensics -- essentially integrating the cognitive skills of a detective into the network [2, 3, 4]. The premise is that the level of effort required to document incidents can thus be reduced, significantly. This paper introduces what technical factors might reflect those detecting skills, leading to solutions that could offset the inefficiencies of current practice.
    Keywords: Network forensics; digital forensics; computer crime; augmented cognition
    The Future of Augmented Cognition Systems in Education and Training BIBAKFull-Text 373-379
      Erica D. Palmer; David A. Kobus
    As adaptive interfaces increase in sophistication and application, augmented cognition systems are becoming accessible to a wider variety of users in real-world settings. The potential for using closed-loop augmented cognition systems in education and training is immense, and will be instrumental in meeting growing demands for distance learning and remote training. Augmented cognition technologies can be applied in numerous ways to dynamically tailor instruction to the user's cognitive style and skill level. Examples of such applications are discussed, along with their implications for enhanced educational and training programs of the future.
    Keywords: Augmented cognition; education; training
    An Adaptive Instructional Architecture for Training and Education BIBAKFull-Text 380-384
      Denise M. Nicholson; Cali M. Fidopiastis; Larry D. Davis; Dylan Schmorrow; Kay M. Stanney
    Office of Naval Research (ONR) initiatives such as Human Performance Training and Education (HPT&E) as well as Virtual Technologies and Environments (VIRTE) have primarily focused on developing the strategies and technologies for creating multimodal reality or simulation based content. Resulting state-of-the-art training and education prototype simulators still rely heavily on instructors to interpret performance data, and adapt instruction via scenario generation, mitigations, feedback and after action review tools. Further research is required to fully close the loop and provide automated, adaptive instruction in these learning environments. To meet this goal, an ONR funded initiative focusing on the Training and Education arm of the HPT&E program will address the processes and components required to deliver these capabilities in the form of an Adaptive Instructional Architecture (AIA). An overview of the AIA as it applies to Marine Corps Warfighter training protocols is given as well as the theoretical foundations supporting it.
    Keywords: adaptive training systems; augmented cognition; simulation
    AFFectIX -- An Affective Component as Part of an E-Learning-System BIBAKFull-Text 385-393
      Karina Oertel; Robin Kaiser; Jörg Voskamp; Bodo Urban
    This paper presents a system component, so called AFFectIX, as an affix to an e-learning system which was that way enhanced with affective abilities. AFFectIX is based on an emotion recognition sensor system and aims to reply negative emotions during human-computer interaction and to provoke an optimum emotional level for the learning process. It was implemented as a first prototype and evaluated by an ad-hoc sample of ten participants. First findings indicate a slight tendency for more satisfaction and learning success.
    Keywords: Affective Computing; Usability; Emotion Recognition; E-Learning
    Performance Compared to Experience Level in a Virtual Reality Surgical Skills Trainer BIBAKFull-Text 394-399
      Christoph Aschwanden; Lawrence Burgess; Kevin Montgomery
    A virtual reality (VR) manual skills experiment was conducted comparing Human performance measures to experiences indicated on a questionnaire handed out. How much do past experiences influence human performance on a VR surgical skills simulator? Performance measures included; time, accuracy, efficiency of motion and errors. Past experiences are among video games and computer proficiency. Results showed little or no relations between experience level and performance. Significant results could only be established for computer gaming experience versus task completion time, F(1, 22) = 3.3, p = .083. Participants familiar with computer gaming were able to carry out tasks faster than their counterparts.
    Keywords: Skills Training; Experience; Performance; Surgery; Laparoscopy; Endoscopy; Gaming; Joystick; Virtual Reality; Simulation; Time; Accuracy; Efficiency; Motion; Errors; HCI; VRMSS; SPRING
    Exploring Neural Trajectories of Scientific Problem Solving Skill Acquisition BIBAKFull-Text 400-408
      Ronald H. Stevens; Trysha Galloway; Chris Berka
    We have modeled changes in electroencephalography (EEG) -- derived measures of cognitive workload, engagement, and distraction as individuals developed and refined their problem solving skills in science. Subjects performing a series of problem solving simulations showed decreases in the times needed to solve the problems; however, metrics of high cognitive workload and high engagement remained the same. When these indices were measured within the navigation, decision, and display events in the simulations, significant differences in workload and engagement were often observed. In addition, differences in these event categories were also often observed across a series of the tasks, and were variable across individuals. These preliminary studies suggest that the development of EEG-derived models of the dynamic changes in cognitive indices of workload, distraction and engagement may be an important tool for understanding the development of problem solving skills in secondary school students.
    Keywords: EEG; Problem solving; Skill Acquisition; Cognitive Workload
    Towards a Closed-Loop Training System: Using a Physiological-Based Diagnosis of the Trainee's State to Drive Feedback Delivery Choices BIBAKFull-Text 409-414
      Amy Bolton; Gwendolyn E. Campbell; Dylan Schmorrow
    Designers of a closed loop scenario based training systems must have specifications to drive the decisions of whether or not performance feedback is appropriate in response to student behavior, the most effective content of that feedback, and the optimal time and method of delivery. In this paper, we propose that physiological measures, when interpreted in conjunction with information about the learning objective, task environment and student performance, could provide the data necessary to inform effective, automated decision processes. In addition, we present an overview of both the relevant literature in this area and some ongoing work that is explicitly evaluating these hypotheses.
    Keywords: Simulation based training; physiological measures; feedback; training interventions
    Aiding Tomorrow's Augmented Cognition Researchers Through Modeling and Simulation Curricula BIBAKFull-Text 415-423
      Julie M. Drexler; Randall Shumaker; Denise M. Nicholson; Cali M. Fidopiastis
    Research in the newly emerged field of Augmented Cognition (AugCog) has demonstrated great potential to develop more intelligent computational systems capable of monitoring and adapting the systems to the changing cognitive state of human operators in order to minimize cognitive bottlenecks and improve task performance. As the AugCog field rapidly expands, an increasing number of researchers will be needed to conduct basic and applied research in this burgeoning field. However, due to its multidisciplinary nature and cutting-edge technological applications, most traditional academic disciplines cannot support the training needs of future AugCog researchers. Accordingly, an established Modeling and Simulation (M&S) graduate curriculum is described, which provides a broad basis of interdisciplinary knowledge and skills as well as depth of knowledge within a specific area of the M&S field. Support for use of the flexible M&S curriculum to provide the requisite multifaceted foundational training in Augmented Cognition principles is also presented.
    Keywords: Augmented Cognition; Modeling; Simulation; Cognitive Neuroscience; Adaptive Technology; Human-Computer Interaction; Curriculum Development
    Designing for Augmented Cognition -- Problem Solving for Complex Environments BIBAFull-Text 424-433
      Joseph Juhnke; Timothy Mills; Jennifer Hoppenrath
    The objective of this paper was to aggregate research done during several different Small Business Innovative Research (SBIR) grants as they apply to the design of complex environments and Augmented Cognition. This paper provides a high level exploration of the definition of situational awareness (SA), the action loop, an advanced mitigation framework, and a repeatable design methodology that was created to overcome several key mistakes made by UI designers. The discussion is illustrated using a recent user interface metaphor design project that maximizes information flow in a novel F-35 Joint Strike Fighter Cockpit. While testing is not complete on the resulting UI metaphor, initial observations indicate that the results of using these models and processes offer a significant improvement in performance and user acceptance appears to be high.
    Making the Giant Leap with Augmented Cognition Technologies: What Will Be the First "Killer App"? BIBAKFull-Text 434-438
      Chris Forsythe; Chris Berka; Robert Matthews; John Wagner
    This paper highlights key topic areas to be discussed the authors in a panel format during the Augmented Cognition thematic area paper session: "Augmented Cognition Lessons Learned and Future Directions for Enabling 'Anyone, Anytime, Anywhere' Applications". The term "killer app" has been part of the vernacular in the commercial computer software and electronic devices industry to refer to breakthrough technologies [2]. A "killer app" generally emerges with the development of related technologies that extends over some time and involves numerous variations on a basic concept. Hypotheses may be offered with respect to the conditions that will be needed to enable a similar situation with augmented cognition technologies. This paper and resulting panel session will address the numerous concepts that have emerged from the augmented cognition field to date and postulate how and when this field's first "killer app" may emerge (e.g., 5, 10, 15, or more years from now).
    Keywords: Augmented Cognition; human factors; ergonomics; design; computer science; neurotechnology; killer app
    Augmenting Cognition: Reviewing the Symbiotic Relation Between Man and Machine BIBAFull-Text 439-448
      Tjerk de Greef; Kees van Dongen; Marc Grootjen; Jasper Lindenberg
    One of the goals of augmented cognition is creation of adaptive human-machine collaboration that continually optimizes performance of the human-machine system. Augmented Cognition aims to compensate for temporal limitations in human information processing, for instance in the case of overload, cognitive lockup, and underload. Adaptive behavior, however, may also have undesirable side effects. The dynamics of adaptive support may be unpredictable and may lead to human factors problems such as mode errors, 'out-of-the-loop' problems, and trust related issues. One of the most critical challenges in developing adaptive human-machine collaboration concerns system mitigations. A combination of performance, effort and task information should be taken into account for mitigation strategies. This paper concludes with the presentation of an iterative cognitive engineering framework, which addresses the adaptation strategy of the human and machine in an appropriate manner carefully weighing the costs and benefits.