HCI Bibliography Home | HCI Conferences | FAC Archive | Detailed Records | RefWorks | EndNote | Hide Abstracts
FAC Tables of Contents: 070911131415

FAC 2009: 5th International Conference on Foundations of Augmented Cognition. Neuroergonomics and Operational Neuroscience

Fullname:FAC 2009: 5th International Conference on Augmented Cognition. Neuroergonomics and Operational Neuroscience
Note:Volume 16 of HCI International 2009
Editors:Dylan Schmorrow; Ivy V. Estabrooke; Marc Grootjen
Location:San Diego, California
Dates:2009-Jul-19 to 2009-Jul-24
Publisher:Springer-Verlag
Series:Lecture Notes in Computer Science 5638
Standard No:ISBN: 978-3-642-02811-3 (print), 978-3-642-02812-0 (online); hcibib: FAC09
Papers:94
Pages:845
Links:Online Proceedings | Publisher Book Page
  1. Understanding Human Cognition and Behavior in Complex Tasks and Environments
  2. Cognitive Modeling, Perception, Emotion and Interaction
  3. Cognitive Load and Performance
  4. Electroencephalography and Brain Activity Measurement
  5. Physiological Measuring
  6. Augmented Cognition in Training and Education
  7. Brain-Computer Interfaces
  8. Rehabilitation and Cognitive Aids

Understanding Human Cognition and Behavior in Complex Tasks and Environments

A Generic Personal Assistant Agent Model for Support in Demanding Tasks BIBAFull-Text 3-12
  Tibor Bosse; Rob Duell; Mark Hoogendoorn; Michel C. A. Klein; Rianne van Lambalgen; Andy van der Mee; Rogier Oorburg; Alexei Sharpanskykh; Jan Treur; Michael de Vos
Human task performance may vary depending on the characteristics of the human, the task and the environment over time. To ensure high effectiveness and efficiency of the execution of tasks, automated personal assistance may be provided to task performers. A personal assistant agent may constantly monitor the human's state and task execution, analyse the state of the human and task, and intervene when a problem is detected. This paper proposes a generic design for a Personal Assistant agent model which can be deployed in a variety of domains. Application of the Personal Assistant model is illustrated by a case study from the naval domain.
Adaptive Interfaces in Driving BIBAKFull-Text 13-19
  Rino F. T. Brouwer; Marieka Hoedemaeker; Mark A. Neerincx
The automotive domain is an excellent domain for investigating augmented cognition methods, and one of the domains that can provide the applications. We developed, applied and tested indirect (or derived) measures to estimate driver state risks, validated by direct state-sensing methods, with major European vehicle manufacturers, suppliers and research institutes in the project AIDE (Adaptive Integrated Driver-vehicle InterfacE). The project developed an interface with the driver that integrates different advanced driver assistant systems and in-vehicle information systems and adapted the interface to different driver or traffic conditions. This paper presents an overview of the AIDE project and will then focus on the adaptation aspect of AIDE. Information presented to the driver could be adapted on basis of environmental conditions (weather and traffic), and on basis of assessed workload, distraction, and physical condition of the driver. The adaptation of how information is presented to the driver or the timing of when information is presented to the driver is of importance. Adapting information, however, also results in systems that are less transparent to the driver.
Keywords: In-car services; workload; adaptive user interface; central management
Using Context to Identify Difficult Driving Situations in Unstructured Environments BIBAFull-Text 20-29
  Kevin R. Dixon; Justin D. Basilico; J. Chris Forsythe; Wilhelm E. Kincses
We present a context-based machine-learning approach for identifying difficult driving situations using sensor data that is readily available in commercial vehicles. The goal of this system is improve vehicle safety by alerting drivers to potentially dangerous situations. The context-based approach is a two-step learning process by first performing unsupervised learning to discover meaningful regularities, or "contexts," in the vehicle data and then performing supervised learning, mapping the current context to a measure of driving difficulty. To validate the benefit of this approach, we collected driving data from a set of experiments involving both on-road and off-road driving tasks in unstructured environments. We demonstrate that context recognition greatly improves the performance of identifying difficult driving situations and show that the driving-difficulty system achieves a human level of performance on cross-validation data.
Neurally-Driven Adaptive Decision Aids BIBAKFull-Text 30-34
  Alexandra Geyer; Jared Freeman; Denise M. Nicholson; Cali M. Fidopiastis; Phan Luu; Joseph Cohn
Warfighters are constantly challenged with increasingly complex mission environments, roles, and tasks, which require rapid and accurate decision making. Most current military and commercial decision aids leverage a single strategy: they retrieve and fuse information about well-defined objects and events for the user. Such aids effectively discourage users from considering contextual information and patterns that may help them recognize or think critically about hostile or innocent events. If a decision aiding system were to be truly effective, its adaptive strategies should be driven by more than manipulation of well-defined information presented to the user. In this paper, we propose several critical factors -- (1) Information state, (2) User cognitive state, and (3) Interaction state -- that will enable for discern what must be decided and by when; discriminate which cognitive state and process are in play; and assess interactions (queries, selections, etc.) with the information. Most importantly, these factors will allow for a decision aid to capitalize on -- the distinctly human ability to find meaning in swarm of objects and events being perceived.
Keywords: adoptive decision aids; intuition; cognitive state; warfighters
Understanding Brain, Cognition, and Behavior in Complex Dynamic Environments BIBAKFull-Text 35-41
  Scott E. Kerick; Kaleb McDowell
Many challenges remain for understanding how the human brain functions in complex dynamic environments. For example, how do we measure brain physiology of humans interacting in their natural environments where data acquisition systems are intrusive and environmental and biological artifacts severely confound brain source signals? How do we understand the full context within which the human brain is operating? How do we know which information is most meaningful to extract from the data? How can we best utilize that extracted information and what are the implications for human performance? The papers comprising this section address these questions from conceptual, technical, and applied perspectives. It is clearly seen that significant progress has been made since the inception of the Augmented Cognition program and that, to overcome these challenges, a continued multidisciplinary approach is required across basic and applied research from cognitive scientists, neuroscientists, computer scientists, and engineers.
Keywords: electroencephalography (EEG); natural environment; operational neuroscience; Augmented Cognition; cognitive engineering; human dimension
Designing a Control and Visualization System for Off-Highway Machinery According to the Adaptive Automation Paradigm BIBAKFull-Text 42-50
  Stefano Marzani; Francesco Tesauri; Luca Minin; Roberto Montanari; Caterina Calefato
This paper aims at describing the requirements of an off-highway human-machine system able to recognize potential risky situations and consequently prevent them. The developed methodology is based on two techniques derived from the field of human factors studies, namely the Hierarchical Task Analysis (HTA) and the Function Allocation (FA), which have been integrated and revised to suit the specific domain of off-highway machinery. The paradigms of adaptive automation and persuasive technology will be followed in the design process. After the off-highway domain analysis a system aiming at improving operator and machine safety is presented. The information system extends the human intelligence monitoring the stability of the machine.
Keywords: Adaptive Automation; Collision; Function Allocation; Human-Machine Interaction; Hierarchical Task Analysis; Off-Highway Vehicles; Overun; Rollover; Runover; Safety; Tractors
Context-Dependent Force-Feedback Steering Wheel to Enhance Drivers' On-Road Performances BIBAKFull-Text 51-57
  Luca Minin; Stefano Marzani; Francesco Tesauri; Roberto Montanari; Caterina Calefato
In this paper the topic of the augmented cognition applied to the driving task, and specifically to the steering maneuver, is discussed. We analyze how the presence of haptic feedback on the steering wheel could help drivers to perform a visually-guided task by providing relevant information like vehicle speed and trajectory. Starting from these considerations, a Context-Dependant Steering Wheel force feedback (CDSW) had been developed, able to provide to the driver the most suitable feeling of the vehicle dynamics according to the driven context. With a driving simulator the CSWD software had been tested twice and then compared with a traditional steering wheel.
Keywords: adaptive steering wheel; augmented cognition; driver performance; force feedback; haptic feedback; lane change
Where Is My Stuff? Augmenting Finding and Re-finding Information by Spatial Locations and Icon Luminance BIBAKFull-Text 58-67
  J. Michelle Moon; Wai-Tat Fu
We studied how spatial locations and luminance affect finding and re-finding information in a desktop environment. In an experiment conducted with computer icons, fixed locations led to more frequent accesses to icons while change of luminance led to worse recall of icon titles and locations. In an analysis of icon access transition, a sequential search pattern was identified in earlier sessions, which suggests that participants were minimizing efforts in external search and were not utilizing internal memory of titles and locations yet. In later sessions, icon accesses were more focused to information directly relevant to search tasks as participants started using titles and locations for re-finding icons. Results are consistent with the notion that information search behavior is adaptive to the cost-benefit structure of the interface, and search strategies are adaptive to different external representations of icons. Results also suggest that both external representations and human information processes are critical in determining the effectiveness of different GUI designs.
Keywords: Adaptive human behavior; re-finding information; spatial memory; interface design
Adaptive Work-Centered and Human-Aware Support Agents for Augmented Cognition in Tactical Environments BIBAFull-Text 68-77
  Martijn Neef; Peter-Paul van Maanen; Peter Petiet; Maartje Spoelstra
We introduce a support system concept that offers both work-centered and human-aware support for operators in tactical command and control environments. The support system augments the cognitive capabilities of the operator by offering instant, personalized task and work support. The operator obtains support by entering into a collaborative agreement with support agents. Such an agreement creates opportunities to establish adaptive capabilities and human-aware support features. We describe the concept in and propose an experimental design to evaluate its effectiveness in tactical environments.
Designing Cognition-Centric Smart Room Predicting Inhabitant Activities BIBAKFull-Text 78-87
  Andrey Ronzhin; Alexey Karpov; Irina S. Kipyatkova
Assignment of easy-to-use and well-timed services staying invisible for a user is one of important features of ambient intelligent. Multimodal user interface capable to perceive speech, movements, poses and gestures of participants in order to determinate their needs provides the natural and intuitively understandable way of interaction with the developed intelligent meeting room. Awareness of the room about spatial position of the participants, their current activities, roles in a current event, their preferences helps to predict more accurately the intentions and needs of participants. Technological framework, equipment and description of technologies applied to the intelligent meeting room are presented. Some scenarios and data structures used for a formalization of context and behavior information from practical human-human, human-machine and machine-machine interaction are discussed.
Keywords: ambient intelligence; cognitive-centric design; multimodal interfaces; context awareness; smart home; intelligent meeting room
Context-Aware Team Task Allocation to Support Mobile Police Surveillance BIBAKFull-Text 88-97
  Jan Willem Streefkerk; Myra P. van Esch-Bussemakers; Mark A. Neerincx
To optimally distribute tasks within police teams during mobile surveillance, a context-aware task allocation system is designed and evaluated with end-users. This system selects and notifies appropriate team members of current incidents, based on context information (officer availability, officer proximity to the incident and incident priority) and decision rules. Eight teams of three experienced police officers evaluated this system in a surveillance task through a virtual environment, comparing it to a non-adaptive system. Task performance, communication, workload and preferences were measured. Results show that team communication, decision making and response times improve using the adaptive system and that this system is preferred. We conclude that context-aware task allocation helps police teams to coordinate incidents efficiently.
Keywords: Context-aware computing; mobile computing; police surveillance; task allocation; notification
Operational Brain Dynamics: Data Fusion Technology for Neurophysiological, Behavioral, and Scenario Context Information in Operational Environments BIBAKFull-Text 98-104
  Don M. Tucker; Phan Luu
Classical laboratory studies of human performance have always required some form of data integration, such as the synchronization of stimulus display, behavioral accuracy, and reaction time. Studies of performance in operational environments have typically been limited in the precision of behavioral observations. As improved digital informatics have expanded the laboratory data acquisition from a few bytes to terabytes, there has been a similar expansion in both the opportunities and the challenges for data fusion.
Keywords: EEG; information systems; brain activity; neuroergonomics

Cognitive Modeling, Perception, Emotion and Interaction

Characterizing Cognitive Adaptability via Robust Automated Knowledge Capture BIBAFull-Text 107-113
  Robert G. Abbott; Chris Forsythe
Applications such as individually tailored training and behavior emulation call for cognitive models tailored to unique individuals on the basis of empirical data. While the study of individual differences has been a mainstay of psychology, a prevailing assumption in cognitive theory and related modeling has been that cognitive processes are largely invariant across individuals and across different conditions for an individual. Attention has focused on identifying a universally correct set of components and their interactions. At the same time, it is known that aptitudes for specific skills vary across individuals and different individuals will employ different strategies to perform the same task [3]. Moreover, individuals will perform tasks differently over time and under different conditions (e.g. Taylor et al, 2004). To reach their full potential, systems designed to augment cognitive performance must thus account for such between- and within-individual differences in cognitive processes. We propose that cognitive adaptability is a trait necessary to explain the inherently dynamic nature of cognitive processes as individuals adapt their available resources to ongoing circumstances. This does not imply a "blank slate;" humans are predisposed to process information in particular ways. Instead, we assert that given variation in the structure and functioning of the brain, there exists inherent flexibility that may be quantified and used to predict differences in cognitive performance between individuals and for a given individual over time. This paper presents an early report on research we are undertaking to discover the dynamics of cognitive adaptability, with emphasis on a task environment designed to evoke and quantify adaptation in controlled experiments.
Implications of User Anxiety in the Evaluation of Deception in Web Sites BIBAKFull-Text 114-119
  Brent Auernheimer; Marie Iding; Martha E. Crosby
In a pilot study, we investigated the effect of anxiety on users' susceptibility to deceptive information on Web pages. Specifically, we manipulated the perceived control and associated anxiety of participants with and without visual disabilities as they used an assistive technology, a screen reader. Preliminary findings indicated that anxious participants (i.e., without visual disabilities) using the unfamiliar assistive technology were more susceptible to deception and expressed more suspicion regarding the Web pages. We interpret these preliminary findings as consistent with the work of Whitson and Galinsky [1] and discuss implications for further research in Web site credibility determinations and users' susceptibility to deception.
Keywords: Deception; Web Sites; Anxiety; Control; Assistive Technology
Investigation of Sleepiness Induced by Insomnia Medication Treatment and Sleep Deprivation BIBAKFull-Text 120-127
  Ioanna Chouvarda; Emmanouil Michail; Athina Kokonozi; Luc Staner; Nathalie Domis; Nicos Maglaveras
The main objective of this work is the study of EEG signals in order to investigate sleepiness induced from drug administration for insomnia and sleep deprivation. Data used in this work were obtained from real experiments in FORENAP, France and in CERTH, Thessaloniki, Greece. The features under consideration are Power Spectrum in certain frequency areas, alpha slow-wave index (ASI) and Fractal Dimension (FD) for placebo and verum subjects. Studying these features in the above groups, we found that sleepiness due to hypnotic medication and due to sleep deprivation can cause different behaviour in brain activity at certain locations. These EEG characteristics could be used for the classification of the medication intake (verum or placebo) and its effect.
Keywords: insomnia; sleep deprivation; EEG signals
Activity Awareness and Social Sensemaking 2.0: Design of a Task Force Workspace BIBAKFull-Text 128-137
  Gregorio Convertino; Lichan Hong; Les Nelson; Peter Pirolli; Ed H. Chi
Task forces of expert knowledge workers would benefit from more advanced web tools supporting activity awareness and social sensemaking. This paper proposes the design of a task force workspace, which is under development. It introduces the problem through a scenario, specifies requirements, illustrates a modeling approach and the mockups of the functions in the proposed workspace. Design issues and future work are finally discussed.
Keywords: Awareness; Sensemaking; Task Force; Roles; User Modeling; CSCW Design; RSS or Atom Feeds
Use of Deception to Improve Client Honeypot Detection of Drive-by-Download Attacks BIBAKFull-Text 138-147
  Barbara Endicott-Popovsky; Julia Narvaez; Christian Seifert; Deborah A. Frincke; Lori Ross O'Neil; Chiraag Uday Aval
This paper presents the application of deception theory to improve the success of client honeypots at detecting malicious web page attacks from infected servers programmed by online criminals to launch drive-by-download attacks. The design of honeypots faces three main challenges: deception, how to design honeypots that seem real systems; counter-deception, techniques used to identify honeypots and hence defeating their deceiving nature; and counter counter-deception, how to design honeypots that deceive attackers. The authors propose the application of a deception model known as the deception planning loop to identify the current status on honeypot research, development and deployment. The analysis leads to a proposal to formulate a landscape of the honeypot research and planning of steps ahead.
Keywords: deception; counter-deception; honeypots; drive-by-downloads; cyber-attacks
Capturing and Building Expertise in Virtual Worlds BIBAKFull-Text 148-154
  Jared Freeman; Webb Stacy; Jean MacMillan; Georgiy Levchuk
Model-driven simulation can make the design and delivery of instruction more efficient and effective. We describe two computational models that support both the design and delivery of instruction. BEST (the Benchmarked Experiential System for Training) can guide experts through the space of domain problems during the knowledge engineering phase of instructional design; it can guide trainees through the space of training objectives during instruction. PRESTO (Pedagogically Relevant Engineering of Scenarios for Training Objectives) builds scenarios on the fly to elicit the knowledge of experts during instructional design, and to satisfy the instructional objectives of trainees.
Keywords: Adaptive instruction; knowledge engineering; Constraint Logic Programming; Markov Decision Process
Conformity out of Diversity: Dynamics of Information Needs and Social Influence of Tags in Exploratory Information Search BIBAKFull-Text 155-164
  Ruogu Kang; Thomas George Kannampallil; Jibo He; Wai-Tat Fu
We studied the dynamic effects of information needs and social influence of tags in an exploratory search task. Although initially differences in information needs led to diversity in tag choices, this diversity disappeared as participants collaboratively tagged the same set of resources. Our findings are in general consistent with the notion that people conform to the collective interpretation of contents in an information system. In addition, our results showed that conformity does not only arise out of imitation of behavior, but also from the same underlying semantic interpretation or knowledge structures of users as they engage in informal collaboration through the social tagging system. Implications for design of social information system are discussed.
Keywords: Exploratory search; tag choice; information needs diversity; semantic interpretation of tags
Trail Patterns in Social Tagging Systems: Role of Tags as Digital Pheromones BIBAKFull-Text 165-174
  Thomas George Kannampallil; Wai-Tat Fu
The popularity of social information systems has been driven by their ability to help users manage, organize and share online resources. Though the research exploring the use of tags is relatively new, two things are widely acknowledged in the research community: (a) tags act as a medium for social collaboration, navigation and browsing and (b) an overall stable equilibrium exists among tag patterns due to the social nature of the tagging process. But there is very little agreement on what causes these stable patterns. In this paper, we take an evolutionary perspective to understand the process of tagging to investigate whether tags act as "way finders" or digital pheromones in social tagging systems. We investigate the existence of tag trails based on a semantic similarity measure among existing tags. We found that over 50% of the resources we evaluated exhibited strong trail patterns. The implications of these patterns for the design and management of social tagging systems is discussed.
Keywords: Social Tagging Systems (STS); Stigmergy; Pheromones; Web 2.0
Real-Time Emotional State Estimator for Adaptive Virtual Reality Stimulation BIBAKFull-Text 175-184
  Davor Kukolja; Sinisa Popovic; Branimir Dropuljic; Marko Horvat; Kresimir Cosic
The paper presents design and evaluation of emotional state estimator based on artificial neural networks for physiology-driven adaptive virtual reality (VR) stimulation. Real-time emotional state estimation from physiological signals enables adapting the stimulations to the emotional response of each individual. Estimation is first evaluated on artificial subjects, which are convenient during software development and testing of physiology-driven adaptive VR stimulation. Artificial subjects are implemented in the form of parameterized skin conductance and heart rate generators that respond to emotional inputs. Emotional inputs are a temporal sequence of valence/arousal annotations, which quantitatively express emotion along unpleasant-pleasant and calm-aroused axes. Preliminary evaluation of emotional state estimation is also performed with a limited set of humans. Human physiological signals are acquired during simultaneous presentation of static pictures and sounds from valence/arousal-annotated International Affective Picture System and International Affective Digitized Sounds databases.
Keywords: Real-Time Emotional State Estimator; Adaptive Virtual Reality Stimulation; Artificial Neural Network; Stimuli Generation; Physiological Measurements
User's Motion for Shape Perception Using CyARM BIBAKFull-Text 185-191
  Ryo Mizuno; Kiyohide Ito; Tetsuo Ono; Junichi Akita; Takanori Komatsu; Makoto Okamoto
We have developed a new sensing device, named by "CyARM". CyARM is one of Active Perception device, based on activeness of human perception. In order to clarify information for shape perception using CyARM, an experiment had conducted. Results of this experiment show that shapes with sharp edges are identified easily, but shapes with smooth edges are identified with greater difficulty. In other words, the subjects perceived changes in distance from the sensor to the object as object edges. Furthermore, from the results of multi-dimensional scaling, it is suggested that object shapes perceived by the subjects were classified according to the sharpness of the edge and by the ratio of height and width. In addition, motion analyses were conducted. The result shows, it is suggested that user tend to swing arm to front arm mainly in motion of lateral direction.
Keywords: Active Perception Device; CyARM; Shape Perception
Human Control Modeling Based on Multimodal Sensory Feedback Information BIBAKFull-Text 192-201
  Edwardo Murakami; Toshihiro Matsui
In order to simulate the human control behavior during a manipulation task in a remote controlled or in a X-by-wire systems, first it is necessary to measure and analyze the human control characteristics. The aim of this research is to measure the operator reaction time and analyze the human visual and force sensory feedback integration related to a manipulation task. Using the developed master-slave type experimental device it was possible to identify and build a human operator control model related to different sensory feedback. The human model related to visual feedback solely and visual/force feedback was identified using the techniques of system identification methods.
Keywords: Human-Machine Interface; System Identification; Reaction Time; Sensory Feedback Information
Potential and Challenges of Body Area Networks for Affective Human Computer Interaction BIBAKFull-Text 202-211
  Julien Penders; Bernard Grundlehner; Ruud J. M. Vullers; Bert Gyselinckx
The Human++ program aims at achieving highly miniaturized, wireless, intelligent and autonomous body sensor nodes to assist our health, comfort and lifestyle. In this paper the concept of body area network is applied to wireless monitoring of emotions, thus opening a new, affective, dimension in human computer interaction. A prototype body area network targeting the monitoring of physiological responses from the autonomous system is introduced, and tested for the classification of discrete emotions. Using data fusion and regression analysis, we show that the wireless physiological data can be mapped in real-time to an estimation of an individual's arousal level. Results in a controlled environment are presented, and specific challenges that need to be overcome for a widespread use of the technology are discussed. Finally, we show how advances in micro-power generation devices may lead to fully autonomous systems in the future.
Keywords: Ambulatory; Body area networks; Emotion monitoring; Ultra-low-power; Wireless
Experimental Assessment of Accuracy of Automated Knowledge Capture BIBAKFull-Text 212-216
  Susan M. Stevens; Chris Forsythe; Robert G. Abbott; Charles J. Gieseler
The U.S. armed services are widely adopting simulation-based training, largely to reduce costs associated with live training. However simulation-based training still requires a high instructor-to-student ratio which is expensive. Intelligent tutoring systems target this need, but they are often associated with high costs for knowledge engineering and implementation. To reduce these costs, we are investigating the use of machine learning to produce models of expert behavior for automated student assessment. A key concern about the expert modeling approach is whether it can provide accurate assessments on complex tasks of real-world interest. This study evaluates of the accuracy of model-based assessments on a complex task. We trained employees at Sandia National Laboratories on a Navy simulator and then compared their simulation performance to the performance of experts using both automated and manual assessment. Results show that automated assessments were comparable to the manual assessments on three metrics.
Keywords: Automated assessment; Naval training systems; simulation-based training; intelligent tutoring systems

Cognitive Load and Performance

Eye Movement as Indicators of Mental Workload to Trigger Adaptive Automation BIBAKFull-Text 219-228
  Tjerk de Greef; Harmen Lafeber; Herre van Oostendorp; Jasper Lindenberg
This research describes an approach to objective assessment of mental workload, by analyzing differences in pupil diameter and several aspects of eye movement (fixation time, saccade distance, and saccade speed) under different levels of mental workload. In an experiment, these aspects were measured by an eye-tracking device to examine whether these are indeed indicators for mental workload. Pupil diameter and fixation time both show a general significant increase if the mental workload increases while saccade distance and saccade speed do not show any significant differences. This assessment of mental workload could be a trigger for aiding the operator of an information system, in order to meet operational requirements.
Keywords: mental workload; adaptive automation; eye movement; pupil diameter; saccade; fixation time
Impact of Automation and Task Load on Unmanned System Operator's Eye Movement Patterns BIBAKFull-Text 229-238
  Cali M. Fidopiastis; Julie M. Drexler; Daniel Barber; Keryl Cosenzo; Michael J. Barnes; Jessie Y. C. Chen; Denise M. Nicholson
Eye tracking under naturalistic viewing conditions may provide a means to assess operator workload in an unobtrusive manner. Specifically, we explore the use of a nearest neighbor index of workload calculated using eye fixation patterns obtained from operators navigating an unmanned ground vehicle under different task loads and levels of automation. Results showed that fixation patterns map to the operator's experimental condition suggesting that systematic eye movements may characterize each task. Further, different methods of calculating the workload index are highly correlated, r(46) = .94, p = .01. While the eye movement workload index matches operator reports of workload based on the NASA TLX, the metric fails on some instances. Interestingly, these departure points may relate to the operator's perceived attentional control score. We discuss these results in relation to automation triggers for unmanned systems.
Keywords: Adaptive Automation; Unmanned Ground Systems; Eye Tracking; Workload
Combining Electroencephalograph and Functional Near Infrared Spectroscopy to Explore Users' Mental Workload BIBAKFull-Text 239-247
  Leanne M. Hirshfield; Krysta Chauncey; Rebecca Gulotta; Audrey Girouard; Erin Treacy Solovey; Robert J. K. Jacob; Angelo Sassaroli; Sergio Fantini
We discuss the physiological metrics that can be measured with electroencephalography (EEG) and functional near infrared spectroscopy (fNIRs). We address the functional and practical limitations of each device, and technical issues to be mindful of when combining the devices. We also present machine learning methods that can be used on concurrent recordings of EEG and fNIRs data. We discuss an experiment that combines fNIRs and EEG to measure a range of user states that are of interest in HCI. While our fNIRS machine learning results showed promise for the measurement of workload states in HCI, our EEG results indicate that more research must be done in order to combine these two devices in practice.
Keywords: fNIRs; EEG; near infrared spectroscopy; workload
Detecting Intentional Errors Using the Pressures Applied to a Computer Mouse BIBAKFull-Text 248-253
  Curtis S. Ikehara; Martha E. Crosby
Intentional errors are considered a form of deceit. In this pilot study, the pressures applied to a computer mouse will be analyzed to determine if it is possible to detect intentional errors. Twenty participants ranging in age from 18 to 21 years performed a task involving intentionally making errors when instructed. A comparison will be made between the pressures applied to a computer mouse when answering the questions with the intention of being correct and with the intention of making an error. The data will need to be normalized for each individual to obtain accurate results. The analysis of the pressures may indicate that there are detectable variations within some individuals. Due to the preliminary nature of this study further research will be required.
Keywords: Intentional errors; deceit; pressure sensitive computer mouse
Visual Navigation Patterns and Cognitive Load BIBAKFull-Text 254-259
  Laurel A. King
Eye tracking technology is a prospective tool for augmenting cognition in real-time in response to screen navigation and other eye movements that can be monitored. This paper examines eye movements associated with differences in problem complexity. The experiment utilized constraint satisfaction problems of differing difficulty measured by the number of steps necessary to complete and the relative time required to solve it. Participants were observed and tested through an eye-tracking experiment to see if correlations between visual navigation and problem complexity were present. Eye movement patterns, in particular pupil size, have been used to measure cognitive load in other contexts [6-9]. The results showed overall increases in fixations and pupil size that corresponded to increases in problem complexity.
Keywords: Cognitive load; eye tracking; analytical reasoning
Modeling the Cognitive Task Load and Performance of Naval Operators BIBAKFull-Text 260-269
  Mark A. Neerincx; Stefan Kennedie; Marc Grootjen; Franc Grootjen
Operators on naval ships have to act in dynamic, critical and high-demand task environments. For these environments, a cognitive task load (CTL) model has been proposed as foundation of three operator support functions: adaptive task allocation, cognitive aids and resource feedback. This paper presents the construction of such a model as a Bayesian network with probability relationships between CTL and performance. The network is trained and tested with two datasets: operator performance with an adaptive user interface in a lab-setting and operator performance on a high-tech sailing ship. The "Naïve Bayesian network" tuned out to be the best choice, providing performance estimations with 86% and 74% accuracy for respectively the lab and ship data. Overall, the resulting model nicely generalizes over the two datasets. It will be used to estimate operator performance under momentary CTL-conditions, and to set the thresholds of the load-mitigation strategies for the three support functions.
Keywords: mental load; emotion; Bayesian networks; cognitive engineering; Defense and Space operations
Impact on Performance and Process by a Social Annotation System: A Social Reading Experiment BIBAKFull-Text 270-278
  Les Nelson; Gregorio Convertino; Peter Pirolli; Lichan Hong; Ed H. Chi
Social annotation systems such as SparTag.us and del.icio.us have been designed to encourage individual reading and marking behaviors that, when shared, accumulate to build collective knowledge spaces. Prior work reported on the experimental design and performance effects observed in a controlled study of SparTag.us. Study participants working independently on a sensemaking task who had access to a set of expert annotations were compared against participants using SparTag.us without those annotations and participants using only office software for annotation support. A learning effect favored the participants exposed to expert annotations. In this paper, we analyze the behavioral data captured during the experiment and identify differences in the work process that can explain the performance effects reported previously.
Keywords: Convergent measures; social annotation systems; evaluation; social sensemaking
Proposing Strategies to Prevent the Human Error in Automated Industrial Environments BIBAKFull-Text 279-288
  José A. do N. Neto; Maria de Fátima Queiroz Vieira; Charles Santoni; Daniel Scherer
This paper presents a process to conceive strategies to prevent the human error when operating industrial systems. The process adopts a broader view to error prevention, going beyond the error analysis to consider the user profile, the task and context description. The error classification is done according to a task execution cognitive model. The conceived strategies focus on the human interface component of those systems since it is this work's premise that the human interface design has a strong impact on the human error rate.
Keywords: Human-Machine Interfaces; Human error; Cognitive model
Wearable Modular Device for Facilitation of Napping and Optimization of Post-nap Performance BIBAKFull-Text 289-298
  Djordje Popovic; Giby Raphael; Robin Johnson; Gene Davis; Chris Berka
Sleep deprivation-induced deficiencies in performance can be associated with high financial and human costs. Napping is an effective countermeasure, but the effects depend on previously accumulated sleep debt and timing, duration and sleep architecture of the naps. Long-term assessment of sleep architecture of nap/sleep episodes could yield an estimate of the accumulated sleep debt and help optimize the napping schedule. Moreover, sensory stimulation coupled with real-time assessment of sleep states could optimize sleep architecture and duration of each nap. With these goals in mind we designed a wearable device, dubbed Nap Cap, which integrates real-time EEG analysis with audio, visual and thermal stimulation. The prototype was evaluated on seven subjects (fully rested vs. sleep-deprived). While the prototype provided high quality EEG and comfort, sensory stimulation did not significantly influence sleep architecture. Evaluation of more paradigms of sensory stimulation on larger samples is warranted before final conclusions can be made.
Keywords: Nap; Sleep Deprivation; Performance Optimization; Wearable Devices
Converging Minds: Assessing Team Performance Using Psychophysiological Measures BIBAKFull-Text 299-303
  Aniket A. Vartak; Siddharth S. Somvanshi; Cali M. Fidopiastis; Denise M. Nicholson
Effective teams are an integral component to the success and the advancement of any organization. This issue emphasizes the need to develop valid measures for team performance especially in operational environments. The use of psychophysiological data has been proposed as a candidate for developing these team-level measures. In this paper, we review past research in the field and discuss two contrasting approaches to model human cognition used in the context of teams. We then propose a test-bed for evaluating these models for human-in-the loop adaptive systems using psychophysiological measures.
Keywords: Team Performance; Team Cognition; Psychophysiology; Social Cybernetics; Information Processing; Closed-Loop Human Systems
Measuring Cognitive Workload in Non-military Scenarios Criteria for Sensor Technologies BIBAKFull-Text 304-310
  Jörg Voskamp; Bodo Urban
Augmented Cognition manifesting in the DARPA project is becoming of more and more interest to non-military application areas. First areas it is going to be applied are in flight control and power plant control. Measuring cognitive workload in the context of Augmented Cognition is bound to the application of sensor technologies and frameworks which are going to be applied to users. It is necessary to make Augmented Cognition Application in non-military areas as comfortable to the user as possible as we do not want to disturb her but to support her in her tasks. In this paper we will define criteria to be considered when designing Augmented Cognition applications in non-military environments.
Keywords: Augmented Cognition; Application; Sensors systems; sensor criteria
Combined Effects of Sleep Deprivation, Narrow Space, Social Isolation and High Cognitive Workload on Cognitive Ability of Chinese Operators BIBAKFull-Text 311-316
  Yijing Zhang; Xueyong Liu; Zhizhong Li; Bin Wu; Fang Liu; Xiaolu Jing; Jun Wang; Haibo Qin; Su Wu
This study aims to investigate the effects of sleep deprivation on the cognitive abilities of Chinese subjects under a combined scenario of narrow space, social isolation, and high cognitive workload. Twelve subjects participated in the experiment of 72-hour sleep deprivation, and took 15 cognitive ability tests for three times at the first day of sleep deprivation (level 1), the second day of sleep deprivation (level 2) and the third day of sleep deprivation (level 3) respectively. The result data analyses show that: most of the cognitive abilities do not change significantly, but the value of special graphics search is increased significantly with the increasing of sleep deprivation time (p=0.01). In addition, when four cognitive ability tests were combined into one complex measure, the effect of sleep deprivation becomes more significant. The results mainly support the Hockey's compensatory control model. And they may be due to two other reasons: one is that the combined stressors will counteract each other; the other is that learning effect can improve operators' performance. The results also imply that sensitive and complex measures need to be developed and used to reflect the compound effects of sleep deprivation under such combined situation.
Keywords: Sleep deprivation; narrow space; social isolation; high cognitive workload; cognitive ability

Electroencephalography and Brain Activity Measurement

Quantifying the Feasibility of Compressive Sensing in Portable Electroencephalography Systems BIBAKFull-Text 319-328
  Amir M. Abdulghani; Alexander J. Casson; Esther Rodríguez-Villegas
The EEG for use in augmented cognition produces large amounts of compressible data from multiple electrodes mounted on the scalp. This huge amount of data needs to be processed, stored and transmitted and consumes large amounts of power. In turn this leads to physically large EEG units with limited lifetimes which limit the ease of use, and robustness and reliability of the recording. This work investigates the suitability of compressive sensing, a recent development in compression theory, for providing online data reduction to decrease the amount of system power required. System modeling which incorporates a review of state-of-the-art EEG suitable integrated circuits shows that compressive sensing offers no benefits when using an EEG system with only a few channels. It can, however, lead to significant power savings in situations where more than approximately 20 channels are required. This result shows that the further investigation and optimization of compressive sensing algorithms for EEG data is justified.
Keywords: Compressive Sensing; Electroencephalogram; Power efficient; Wireless Systems
Are You Really Looking? Finding the Answer through Fixation Patterns and EEG BIBAFull-Text 329-338
  Anne-Marie Brouwer; Maarten A. Hogervorst; Pawel Herman; Frank Kooi
Eye movement recordings do not tell us whether observers are 'really looking' or whether they are paying attention to something else than the visual environment. We want to determine whether an observer's main current occupation is visual or not by investigating fixation patterns and EEG. Subjects were presented with auditory and visual stimuli. In some conditions, they focused on the auditory information whereas in others they searched or judged the visual stimuli. Observers made more fixations that are less cluttered in the visual compared to the auditory tasks, and they were less variable in their average fixation location. Fixated features revealed which target the observers were looking for. Gaze was not attracted more by salient features when performing the auditory task. 8-12 Hz EEG oscillations recorded over the parieto-occipital regions were stronger during the auditory task than during visual search. Our results are directly relevant for monitoring surveillance workers.
"What Was He Thinking?": Using EEG Data to Facilitate the Interpretation of Performance Patterns BIBAKFull-Text 339-347
  Gwendolyn E. Campbell; Christine L. Belz; Phan Luu
Previous research has demonstrated that EEG data can be used to identify and remove unintentional responses from a data set (guesses and slips). This study sought to determine if removing this error variance has a significant impact on the interpretation of a trainee's performance. Participants were taught to recognize tank silhouettes. Multiple linear regression models were built for each participant based on three sets of their data: 1) all trials of their performance data, 2) only trials that were learned according to a state space analysis, and 3) their intentional data as identified by EEG. When compared to an expert model, each of the three models for every participant yielded a different diagnosis, indicating that filtering performance data with EEG data changes the interpretation of a participant's competence.
Keywords: electroencephalography; training; student modeling
Motion-Sickness Related Brain Areas and EEG Power Activates BIBAKFull-Text 348-354
  Yu-Chieh Chen; Jeng-Ren Duann; Chun-Ling Lin; Shang-Wen Chuang; Tzyy-Ping Jung; Chin-Teng Lin
This study investigates electroencephalographic (EEG) correlates of motion sickness in a virtual-reality based driving simulator. The driving simulator comprised an actual automobile mounted on a Stewart motion platform with six degrees of freedom, providing both visual and vestibular stimulations to induce motion-sickness in a manner that is close to that in daily life. EEG data were acquired at a sampling rate of 500 Hz using a 32-channel EEG system. The acquired EEG signals were analyzed using independent component analysis (ICA) and time-frequency analysis to assess EEG correlates of motion sickness. Subject's degree of motion-sickness was simultaneously and continuously reported using an onsite joystick, providing non-stop psychophysical references to the recorded EEG changes. Five Motion-sickness related brain processes with equivalent dipoles located in the left motor, the parietal, the right motor, the occipital and the occipital midline areas were consistently identified across all subjects. These components exhibited distinct spectral suppressions or augmentation in motion sickness. The results of this study could lead to a practical human-machine interface for noninvasive monitoring of motion sickness of drivers or passengers in real-world environments.
Keywords: EEG; ICA; motion-sickness; delta; theta; alpha; time-frequency
Building Dependable EEG Classifiers for the Real World -- It's Not Just about the Hardware BIBAKFull-Text 355-364
  Gene Davis; Djordje Popovic; Robin R. Johnson; Chris Berka; Mirko Mitrovic
One of the major deficiencies with the EEG-based classifiers used in today's laboratory settings is that they are often ill suited for the real world. In many cases the classifiers that were painstakingly developed in the controlled laboratory environment become unreliable with increased mobility of the user. In addition to increased mobility, many real world scenarios impose constraints on data collection that cannot be accommodated by the lab-created classifier. Addressing these issues throughout the development process of EEG-based classifiers by building hardware, software, and algorithms intended for use in the real world should result in more dependable classifiers. With this approach we were able to collect and classify data on a research vessel at sea, in the desert by night, on dismounted soldiers in the training field, and everywhere between.
Keywords: Electroencephalogram (EEG); Mobile EEG; Operational Neuroscience; Engagement; Workload; Drowsiness
Improved Team Performance Using EEG- and Context-Based Cognitive-State Classifications for a Vehicle Crew BIBAFull-Text 365-372
  Kevin R. Dixon; Konrad Hagemann; Justin Basilico; J. Chris Forsythe; Siegfried Rothe; Michael Schrauf; Wilhelm E. Kincses
We present an augmented cognition (AugCog) system that utilizes two sources to assess cognitive state as a basis for actions to improve operator performance. First, continuous EEG is measured and signal processing algorithms utilized to identify patterns of activity indicative of high cognitive demand. Second, data from the automobile is used to infer the ongoing driving context. Subjects participated as eleven 2-person crews consisting of a driver/ navigator and a commander/gunner. While driving a closed-loop test route, the driver received through headphones a series of communications and had to perform two secondary tasks. Certain segments of the route were designated as threat zones. The commander was alerted when entering a threat zone and their task was to detect targets mounted on the roadside and engage those targets To determine targeting success, a photo was taken with each activation of the trigger and these photos were assessed with respect to the position of the reticle relative to the target. In a secondary task, the commander was presented a series of communications through headphones. Our results show that it is possible to reliably discriminate different cognitive states on the basis of neuronal signals. Results also confirmed our hypothesis: improved performance at the crew level in the AugCog condition for a secondary communications tasks, as compared to a control condition, with no change in performance for the primary tasks.
Note: Best Paper Award
Detecting Frontal EEG Activities with Forehead Electrodes BIBAKFull-Text 373-379
  Jeng-Ren Duann; Po-Chuan Chen; Li-Wei Ko; Ruey-Song Huang; Tzyy-Ping Jung; Chin-Teng Lin
This study demonstrates the acquisitions of EEG signals from non-hairy forehead sites and tested the feasibility of using the forehead EEG in detecting drowsiness-related brain activities. A custom-made 15-channel forehead EEG-electrode patch and 28 scalp electrodes placed according to the International 10-20 system were used to simultaneously record EEG signals from the forehead and whole-head regions, respectively. A total of five subjects were instructed to perform a night-time long-haul driving task for an hour in a virtual-reality based driving simulator comprising a real car mounted on a 6 degree-of-freedom Steward motion platform and a immersive VR environment with 360 degree projection scenes. Separate independent component analyses were applied to the forehead and whole-head EEG data for each individual subject. For the whole-head independent component (IC) set, the frontal central midline (FCM) IC with an equivalent dipole source located in the anterior cingulate cortex was selected for further analysis. For the forehead IC set, the IC with its theta power changes highly correlated with subject's driving performance was selected. The EEG power changes of the selected forehead ICs were then used to predict driving performance based on a linear regression model. The results of this study showed that it is feasible to accurately estimate quantitatively the changing level of driving performance using the EEG features obtained from the forehead non-hairy channels, and the estimation accuracy was comparable to that using the EEG features of the whole-head recordings.
Keywords: Forehead EEG; Drowsiness; Driving performance; Independent component analysis (ICA)
The Effectiveness of Feedback Control in a HCI System Using Biological Features of Human Beings BIBAKFull-Text 380-389
  Mariko Fujikake Funada; Miki Shibukawa; Yoshihide Igarashi; Takashi Shimizu; Tadashi Funada; Satoki P. Ninomija
The purpose of this paper is to clarify the brain activities of human beings engaged in their tasks. Response time, correctness ratios, and Event Related Potentials (ERPs) are useful indexes of the brain activities of a subject at his task in the experiments. We analyze these indexes by a method called the Principle Component Analysis. Then we characterize the brain activities while he is engaged in the task. Finally we discuss the effectiveness of feedback control in a HCI system using these indexes.
Keywords: event related potentials; feedback control; principle component analysis; response time
Bayesian Reconstruction of Perceptual Experiences from Human Brain Activity BIBAKFull-Text 390-393
  Jack L. Gallant; Thomas Naselaris; Ryan J. Prenger; Kendrick N. Kay; Dustin Stansbury; Michael Oliver; An Vu; Shinji Nishimoto
A method for decoding the subjective contents of perceptual systems in the human brain would have broad practical utility for communication and as a brain-machine interface. Previous approaches to this problem in vision have used linear classifiers to solve specific problems, but these approaches were not general enough to solve complex problems such as reconstructing subjective perceptual states. We have developed a new approach to these problems based on quantitative encoding models that explicitly describe how visual stimuli are (nonlinearly) transformed into brain activity. We then invert these encoding models in order to decode activity evoked by novel images or movies, providing reconstructions with unprecedented fidelity. Here we briefly review these results and the potential uses of perceptual decoding devices.
Keywords: Bayesian; vision; brain-machine interface; brain-computer interface; brain reading
Tonic Changes in EEG Power Spectra during Simulated Driving BIBAKFull-Text 394-403
  Ruey-Song Huang; Tzyy-Ping Jung; Scott Makeig
Electroencephalographic (EEG) correlates of driving performance were studied using an event-related lane-departure paradigm. High-density EEG data were analyzed using independent component analysis (ICA) and Fourier analysis. Across subjects and sessions, when reaction time to lane-departure events increased, several clusters of independent component activities in the occipital, posterior parietal, and middle temporal cortex showed tonic power increases in the delta, theta, and alpha bands. The strongest of these tonic power increases occurred in the alpha band in the occipital and parietal regions. Other independent component clusters in the somatomotor and frontal regions showed less or no significant increase in all frequency bands as RT increased. This study demonstrates additional evidence of the close and specific links between cortical brain activities (via changes in EEG spectral power) and performance (reaction time) during sustained-attention tasks. These results may also provide insights into the development of human-computer interfaces for countermeasures for drowsy driving.
Keywords: EEG; ICA; driving; alertness; delta; theta; alpha; reaction time
P300 Based Single Trial Independent Component Analysis on EEG Signal BIBAFull-Text 404-410
  Kun Li; Ravi Sankar; Yael Arbel; Emanuel Donchin
A Brain Computer Interface (BCI) is a device that allows the user to communicate with the world without utilizing voluntary muscle activity (i.e., using only the electrical activity of the brain). It makes use of the well-studied observation that the brain reacts differently to different stimuli, as a function of the level of attention allotted to the stimulus stream and the specific processing triggered by the stimulus. In this article we present a single trial independent component analysis (ICA) method that is working with a BCI system proposed by Farwell and Donchin. It can dramatically reduce the signal processing time and improve the data communicating rate. This ICA method achieved 76.67% accuracy on single trial P300 response identification.
Directed Components Analysis: An Analytic Method for the Removal of Biophysical Artifacts from EEG Data BIBAKFull-Text 411-416
  Phan Luu; Robert Frank; Scott Kerick; Don M. Tucker
Artifacts generated by biophysical sources (such as muscles, eyes, and heart) often hamper the use of EEG for the study of brain functions in basic research and applied settings. These artifacts share frequency overlap with the EEG, making frequency filtering inappropriate for their removal. Spatial decomposition methods, such as principal and independent components analysis, have been employed for the removal of the artifacts from the EEG. However, these methods have limitations that prevent their use in operational environments that require real-time analysis. We have introduced a directed components analysis (DCA) that employs a spatial template to direct the selection of target artifacts. This method is computationally efficient, allowing it to be employed in real-world applications. In this paper, we evaluate the effect of spatial undersampling of the scalp potential field on the ability of DCA to remove blink artifacts.
Keywords: EEG; artifact; brain activity; neuroergonomic
Functional Near-Infrared Spectroscopy and Electroencephalography: A Multimodal Imaging Approach BIBAKFull-Text 417-426
  Anna C. Merzagora; Meltem Izzetoglu; Robi Polikar; Valerie Weisser; Banu Onaral; Maria T. Schultheis
Although neuroimaging has greatly expanded our knowledge about the brain-behavior relation, combining multiple neuroimaging modalities with complementing strengths can overcome some limitations encountered when using a single modality. Valuable candidates for a multimodal approach are functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG). fNIRS is an imaging technology that localizes hemodynamic changes within the cortex. However, hemodynamic activation is an intrinsically slow process. On the other hand, EEG has excellent time resolution by directly measuring the manifestation of the brain electrical activity at the scalp. Based on their complementary strengths, the integration of fNIRS and EEG may provide higher spatiotemporal resolution than either method alone. In this effort, we integrate fNIRS and EEG to evaluate the behavioral performance of six healthy adults in a working memory task. To this end, features extracted from fNIRS and EEG were used separately, as well as in combination, and their performances were compared against each other.
Keywords: multimodal neuroimaging; functional near-infrared spectroscopy; EEG; pattern classification; working memory; n-back; P300
Transcranial Doppler: A Tool for Augmented Cognition in Virtual Environments BIBAKFull-Text 427-436
  Beatriz Rey; Mariano Alcañiz Raya; Valery Naranjo; José Tembl; Vera Parkhutik
In this work, we propose the use of Transcranial Doppler Monitoring (TCD) as a tool to measure brain activity during the exposure to virtual environments (VE) that can be used in Augmented Cognition (AugCog) systems. The technique is non-invasive, and can be easily integrated with virtual reality (VR) settings. Its high temporal resolution allows the correlation of changes in brain activity to specific events in the VE. In this paper, the TCD technique is described, and results from two studies developed in our group combining TCD with VR are summarized. Possible applications of TCD in the AugCog field are finally discussed.
Keywords: Augmented Cognition; Virtual Reality; Transcranial Doppler; Neurophysiological Data; Cognitive State Assessment
Predicting Intended Movement Direction Using EEG from Human Posterior Parietal Cortex BIBAKFull-Text 437-446
  Yijun Wang; Scott Makeig
The posterior parietal cortex (PPC) plays an important role in motor planning and execution. Here, we investigated whether noninvasive electroencephalographic (EEG) signals recorded from the human PPC can be used to decode intended movement direction. To this end, we recorded whole-head EEG with a delayed saccade-or-reach task and found direction-related modulation of event-related potentials (ERPs) in the PPC. Using parietal EEG components extracted by independent component analysis (ICA), we obtained an average accuracy of 80.25% on four subjects in binary single-trial EEG classification (left versus right). These results show that in the PPC, neuronal activity associated with different movement directions can be distinguished using EEG recording and might, thus, be used to drive a noninvasive brain-machine interface (BMI).
Keywords: posterior parietal cortex (PPC); electroencephalography (EEG); independent component analysis (ICA); brain-machine interface (BMI)

Physiological Measuring

Enhancing Text-Based Analysis Using Neurophysiological Measures BIBAKFull-Text 449-458
  Adrienne Behneman; Natalie Kintz; Robin Johnson; Chris Berka; Kelly S. Hale; Sven Fuchs; Par Axelsson; Angela Baskin
Intelligence analysts are faced with the demanding task of identifying patterns in large volumes of complex, textual sources and predicting possible outcomes based on perceived patterns. To address this need, the Advanced Neurophysiology for Intelligence Text Analysis (ANITA) system is being developed to provide a real-time analysis system using EEG to monitor analysts' processing of textual data during evidence gathering. Both conscious and unconscious 'interest' are identified by the neurophysiological sensors based on the analyst's mental model, as related to specific sentences, indicating relevance to the analysis goal. By monitoring the evidence gathering process through neurophysiological sensors and implementation of real-time strategies, more accurate and efficient extraction of evidence may be achieved. This paper outlines an experiment that focused on identifying distinct changes in EEG signals that can be used to decipher sentences of relevance versus those of irrelevance to a given proposition.
Keywords: EEG; Reading; Relevancy; Alpha; Theta
Affective Computer-Generated Stimulus Exposure: Psychophysiological Support for Increased Elicitation of Negative Emotions in High and Low Fear Subjects BIBAKFull-Text 459-468
  Christopher G. Courtney; Michael E. Dawson; Anne M. Schell; Thomas D. Parsons
The present study examined physiological measures of affect when viewing images from the International Affective Picture System (IAPS), computer-generated still images, and computer-generated videos of feared and non-feared stimuli. Twenty low fear (LF) and twelve high fear (HF) individuals viewed static and moving images of spiders and snakes. In both LF and HF subjects, computer-generated video images elicited more intense affective responses than the IAPS images and the computer-generated stills. Computer-generated still images were as effective in eliciting fear responses as the IAPS. These results suggest that computer-generated images can be as or more effective as the IAPS in eliciting fear. Regardless of modality, HF subjects showed stronger physiological responses to their specifically feared stimulus (snake or spider) than to a non-feared stimulus.
Keywords: Psychophysiology; Fear; EMG; skin conductance; VR; startle
Applying Real Time Physiological Measures of Cognitive Load to Improve Training BIBAFull-Text 469-478
  Joseph T. Coyne; Carryl Baldwin; Anna Cole; Ciara Sibley; Daniel M. Roberts
This paper discusses how the fields of augmented cognition and neuroergonomics can be expanded into training. Several classification algorithms based upon EEG data and occular data are discussed in terms of their ability to classify operator state in real time. These indices have been shown to enhance operator performance within adaptive automation paradigms. Learning is different from performing a task that one is familiar with. According to cognitive load theory (CLT), learning is essentially the act of organizing information from working memory into long term memory. However, our working memory system has a bottleneck in this process, such that when training exceeds working memory capacity, learning is hindered. This paper discusses how CLT can be combined with multiple resource theory to create a model of adaptive training. This new paradigm hypothesizes that a system that can monitor working memory capacity in real time and adjust training difficulty can improve learning.
Considerations for Designing Response Quantification Procedures in Non-traditional Psychophysiological Applications BIBAFull-Text 479-487
  A. V. Iyer; L. D. Cosand; Christopher G. Courtney; Albert A. Rizzo; Thomas D. Parsons
Psychophysiological assessment in the context of virtual environments is a promising means for benchmarking the efficacy and ecological validity of virtual reality scenarios. When applied to human-computer interaction, psychophysiological and affective computing approaches may increase facility for development of the next generation of human-computer systems. Such systems have the potential to use psychophysiological signals for user-feedback and adaptive responding. As the composition of investigating teams becomes diverse in keeping with interdisciplinary trends, there is a need to review de-facto standards of psychophysiological response quantification and arrive at consensus protocols adequately addressing the concerns of basic researchers and application developers. The current paper offers a demonstration of the ways in which such consensus scoring protocols may be derived. Electromyographic eye-blink scoring from an immersion investigation is used as an illustrative case study.
Neurophysiological Measures of Brain Activity: Going from the Scalp to the Brain BIBAKFull-Text 488-494
  Phan Luu; Catherine Poulsen; Don M. Tucker
Behavior, such as reaction time and correctness of a response, is the most studied output of the mind in the fields of psychology and human factors. With the advent of modern neuroimaging technologies, opportunities exist for direct study of the mind's machinery: the brain. Moreover, there are opportunities for applying these technologies to solve a host of educational and engineering challenges, such as how to design better interfaces with computer systems or how to better educate and train students. The electroencephalogram (EEG) is a direct reflection of the functioning brain, and technologies that enable recording of the EEG have been in existence for more than 50 years. Within the past decade substantial progress has been made in EEG technology, permitting a direct view into the brain. We cover these advances in this paper, which include dense-sensor array technology and physics-based computational head models, and present several examples of how they have been applied.
Keywords: Dense-Array EEG; neuroergonomic
Parsimonious Identification of Physiological Indices for Monitoring Cognitive Fatigue BIBAKFull-Text 495-503
  Lance J. Myers; J. Hunter Downs
The objective of this study was to identify a parsimonious set of physiological measures that could be used to best predict cognitive fatigue levels. A 37 hour sleep deprivation study was conducted to induce reduced levels of alertness and cognitive impairment as measured by a psychomotor vigilance test. Non-invasive, wearable and ambulatory sensors were used to acquire cardio-respiratory and motion data during the sleep deprivation. Subsequently 23 potential predictors were derived from the raw sensor data. The least absolute shrinkage and selection operator, along with a cross validation strategy was used to create a sparse model and identify a minimum predictor subset that provided the best prediction accuracy. Final predictor selection was found to vary with task and context. Depending on context selected predictors indicated elevated levels of sympathetic nervous system activity, increased restlessness during engaging tasks and increased cardio-respiratory synchronization with increasing cognitive fatigue.
Keywords: cognitive fatigue; heart rate variability; feature selection; wearable sensors
In-Helmet Oxy-hemoglobin Change Detection Using Near-Infrared Sensing BIBAKFull-Text 504-513
  Erin M. Nishimura; Christopher A. Russell; J. Patrick Stautzenberger; Harvey Ku; J. Hunter Downs
Near-infrared (NIR) sensing in flight applications can provide critical objective indicators of crew state. By monitoring oxy-hemoglobin concentrations, a NIR sensor can detect changes in flight crew physiology in response to both cognitive demands and extreme conditions related to flight applications, including gravity-induced loss of consciousness (G-LOC) and hypoxia. A custom NIR sensor was created for in-helmet monitoring of oxy-hemoglobin in flight. This wearable, wireless sensor addresses requirements for flight applications and was applied to a case study that examines the raw optical signal and oxy-hemoglobin response to Valsalva maneuvers performed at 1g.
Keywords: Near-infrared sensing; functional brain imaging; oxy-hemoglobin; hemodynamics; physiologic monitoring
Assessment of Psychophysiological Differences of West Point Cadets and Civilian Controls Immersed within a Virtual Environment BIBAKFull-Text 514-523
  Thomas D. Parsons; Christopher G. Courtney; Louise Cosand; Arvind Iyer; Albert A. Rizzo; Kelvin S. Oie
An important question for ecologically valid virtual environments is whether cohort characteristics affect immersion. If a method for assessing a certain neurocognitive capacity (e.g. attentional processing) is adapted to a cohort other than the one that was used for the initial normative distribution, data obtained in the new cohort may not be reflective of the neurocognitive capacity in question. We assessed the psychophysiological impact of different levels of immersion upon persons from two cohorts: 1) civilian university students; and 2) West Point Cadets. Cadets were found to have diminished startle eyeblink amplitude compared with civilians, which may reflect that cadets experienced less negative affect during the scenario in general. Further, heart rate data revealed that Cadets had significantly lower heart rates than Civilians in the "low" but not "high" immersion condition. This suggests that "low" immersion conditions may not have the ecological validity necessary to evoke consistent affect across cohorts.
Keywords: virtual environment; psychophysiological assessment; immersion; ecological validity; neuropsychology
Characterizing the Psychophysiological Profile of Expert and Novice Marksmen BIBAKFull-Text 524-532
  Nicholas Pojman; Adrienne Behneman; Natalie Kintz; Robin Johnson; Gregory K. W. K. Chung; Sam O. Nagashima; Paul Espinosa; Chris Berka
Marksmanship training includes a combination of classroom instruction and field practice involving the instantiation of a well-defined set of sensory, motor, and cognitive skills. 10 expert marksmen and 30 novices participated in a study that measured marksman performance during simulated ballistics shooting of a M4 replica infrared rifle. Participants' physiology and performance were quantified while they completed a battery of neurocognitive tests. Experts demonstrated consistent and more accurate shot performance across all trials. Compared to novices, experts evidenced lower levels of sympathetic activation as measured by heart rate variability during the neurocognitive tasks. Factor analysis identified experts as having above normal visuospatial processing speeds and sustained attention, reflecting experts as having better performance during vigilance neurocognitive tasks. Identifying physiological metrics of experts during neurocognitive testing opens the door to individualized novice instruction to help to improve specific areas flagged as below normal during or prior to novice marksmanship instruction.
Keywords: Electroencephalogram (EEG); Electrocardiogram (EKG); Marksmanship; Expert; Heart Rate Variability; Neurocognitive testing; psychomotor skill acquisition
Assessing Cognitive State with Multiple Physiological Measures: A Modular Approach BIBAKFull-Text 533-542
  Lee W. Sciarini; Denise M. Nicholson
The purpose of this effort is to introduce a novel approach which can be used to determine how multiple minimally intrusive physiological sensors can be used together and validly applied to areas such as Augmented Cognition and Neuroergonomics. While researchers in these fields have established the utility of many physiological measures for informing when to adapt systems, the use of such measures together remains limited. Specifically, this effort will provide a contextual explanation of cognitive state, workload, and the measurement of both; provide a brief discussion on several relatively noninvasive physiological measures; explore what a modular cognitive state gauge should consist of; and finally, propose a framework based on the previous items that can be used to determine the interactions of the various measures in relation to the change of cognitive state.
Keywords: Augmented Cognition; Neuroergonomics; Physiological Measures
Neuro-NIRS: Analysis of Neural Activities Using NIRS BIBAKFull-Text 543-552
  Hiroshi Tamura; Miki Fuchigami; Akira Okada
Analysis method for quick component of NIRS is explained. In order to compare quantitatively the amount of quick component, absolute average operation is applied. The result Q is then formulated as the product of magnitude M and density D. By characteristic differences of M and D functional features of each channel are discussed. The method is applied to text entry task to mobile phone. Most of 34 channels under study D values diverged in rest state, but they converged at task state. 6 channels among 34, showed specific responses to specific task.
Keywords: neural activities; NIRS; quick components; text entry; mobile phone
Eye Movements and Pupil Size Reveal Deception in Computer Administered Questionnaires BIBAKFull-Text 553-562
  Andrea K. Webb; Douglas J. Hacker; Dahvyn Osher; Anne E. Cook; Dan J. Woltz; Sean Kristjansson; John C. Kircher
An oculomotor test is described that uses pupil diameter and eye movements during reading to detect deception. Forty participants read and responded to statements on a computerized questionnaire about their possible involvement in one of two mock crimes. Twenty guilty participants committed one of two mock crimes, and 20 innocent participants committed no crime. Guilty participants demonstrated speeded and accurate reading when they encountered statements about their crime and increases in pupil size. A discriminant function of oculomotor measures successfully discriminated between guilty and innocent participants and between the two groups of guilty participants. Results suggest that oculomotor tests may be of value for pre-employment and security screening applications.
Keywords: Oculomotor measures; pupil size; deception
Physiological-Based Assessment of the Resilience of Training to Stressful Conditions BIBAKFull-Text 563-571
  Mikhail Zotov; J. Forsythe; Vladimir Petrukovich; Inga Akhmedova
Russian applied psychophysiology has a wide experience of using the heart rate variability (HRV) measures for the assessment of operator workload. However, 'workload indexes' that have received a wide practical application, such as tension index (TI), are not sensitive to the moment-to-moment changes of operator physiological arousal level during the performance of cognitive tasks. In this connection, a new method of HRV analysis called CS-index is offered. This index permits to identify moment-to-moment changes of operator's functional state. The presented research shows that CS-index is sensitive to task load factors, such as task difficulty level and stressful conditions and allows to differentiate experienced and novice operators during their performance on a simulator. If the CS-index proves to be reliable enough, its combination with the Automated Expert Modeling for Automated Student Evaluation (AEMASE) approach can considerably raise the efficiency of operator training.
Keywords: heart rate variability; cognitive workload; training

Augmented Cognition in Training and Education

Tunnel Operator Training with a Conversational Agent-Assistant BIBAKFull-Text 575-584
  Eric Buiël; Jan Lubbers; Willem A. van Doesburg; Tijmen Muller
A tunnel operator monitors and regulates the flow of traffic inside a tunnel. Tunnel operators need to train in a simulator regularly in order to maintain proficiency in handling incident situations. During quiet working hours, the operator has enough time for training. But generally at that time no instructor or colleague operators are present to provide instruction, advises, and feedback. As a solution, we have designed an automated training system. The system employs a conversational agent which supports the operator's situation assessment tasks. The agent exhibits peer behavior which is unobtrusively directed by didactic strategies. In this paper we present the design, development and application of the agent.
Keywords: Agent-Based Modeling and Training; Cognitive Modeling; Constructive Learning; Intelligent Virtual Agent
Evaluating Training with Cognitive State Sensing Technology BIBAFull-Text 585-594
  Patrick L. Craven; Patrice D. Tremoulet; Joyce Barton; Steven J. Tourville; Yaela Dahan-Marks
Five different training techniques (classroom, video, game-based, computer-based, and simulator) were compared using neurophysiological measurements. The best performance was displayed by individuals in the classroom and video conditions. These participants also displayed the lowest levels of cognitive workload and the highest levels of engagement. The poorest performance on the training was exhibited by individuals in the computer-based and game conditions. These participants also displayed the highest levels of cognitive workload, the lowest levels of engagement, and computer-based had the highest levels of drowsiness. As expected, the testing phases of the training had the highest levels of workload. In general, engagement dropped and distraction increased during the training phase when the material was first presented to participants. However, participants who could keep engagement high during this period performed better. This suggests that mental state monitoring during training could help provide a mechanism for alleviating distraction and inattention and boost training efficacy.
Identifying the Nature of Knowledge Using the Pressures Applied to a Computer Mouse BIBAKFull-Text 595-600
  Martha E. Crosby; Curtis S. Ikehara; Wendy S. Ark
The nature of knowledge retention is not that a student either knows or doesn't. Using signal detection theory, the correct and incorrect responses a student provides can each be subdivided into two more levels of knowledge using the student's confidence of answer correctness. The proposed study will attempt to link confidence of answer correctness to the categorized pressures applied to a computer mouse allowing for the partitioning of responses. Twenty participants that were part of a pedagogical methods study will be retested using a computer-based multiple choice test and pressure sensitive computer mouse. Participants will also rate their confidence of answer correctness. It is hypothesized that the analyzed pressures applied to the computer mouse will indicate the confidence of answer correctness. Using the categorized pressures from the computer mouse allows for the real-time assessment of a student's knowledge to guide pedagogical follow-up.
Keywords: knowledge; pressure sensitive computer mouse; confidence of correctness
Realizing Adaptive Instruction (Ad-In): The Convergence of Learning, Instruction, and Assessment BIBAKFull-Text 601-610
  Edward Dieterle; John Murray
In this paper, we define adaptive instruction, or Ad-In, as applied to sophisticated skills development systems that target learning and assessment in a highly individualized and interactive manner. We argue that the successful design and use of such systems rely heavily upon the interrelationships among learning styles, instructional theories, and assessment methods, in the context of personalized learning. We outline and structure the links among these topics by drawing upon recent empirical studies of virtual environments and augmented realities. The paper also presents a candidate architecture for applying Ad-In concepts in an intelligent interactive environment for skills development.
Keywords: Adaptive instruction; augmented reality; immersion; intelligent tutoring assistant; multi-user virtual environment; neomillennial learning styles
Adaptive Learning via Social Cognitive Theory and Digital Cultural Ecosystems BIBAFull-Text 611-619
  Joseph Juhnke; Adam R. Kallish
This paper will look at the human predisposition to oral tradition and its effectiveness as a learning tool to convey mission-critical information. After exploring the effectiveness of the conveyance of information, the paper will examine current adaptive learning research and develop a system that will marry the strengths of oral tradition with those of an optimal adaptive learning environment. Emphasis will be made in the area of military service personnel stationed in contested cultures, the aiding of their arrival and once established their continual improvement processes. This paper will then illustrate a digital cultural ecosystem that leverages the strengths of current industry thinking in digital community development and social architecture combining the adaptive learning models discussed earlier to create a dynamic digital social ecology that could significantly improve the transition process by exposing service personnel to the collective learning of all of the personnel currently and previously deployed to a particular region. It will illustrate tools and techniques that can be used to filter the quality of the collective intelligence, the dynamic categorization of new narrative and the selective recommendation of content as an adaptive learning technique. This system will incorporate a virtual environment to test the quality of learning before the military personnel are deployed and a capture and debrief system that will enable the continual improvement of service personnel as they complete missions during their deployment.
The Interaction between Chinese University Students' Computer Use and Their Attitudes toward Computer in Learning and Innovation BIBAKFull-Text 620-629
  Ye Liu; Xiaolan Fu
A survey study investigated the dynamic interaction between Chinese university students' computer use and their attitudes toward computer in learning and innovation. The relationships among attitudes toward computer in learning (ACL), attitudes toward innovation (ATI), and self-perception on computer skill (SPCS), were also examined. Participants were 292 university students from three universities in Beijing, and all of them had used computer and Internet before. The results showed that: (a) Previous computer use could predict recent computer use with ACL, ATI, and SPCS as mediate variables; (b) males were more confident than females in SPCS, but there was no gender difference in either ACL or ATI; and (c) the participants' notion of innovation was significantly more positive than their innovative action.
Keywords: Computer use; Computer attitude; Attitudes toward innovation; Self-perception on computer skill; Gender difference
Peak Performance Trainer (PPT™): Interactive Neuro-educational Technology to Increase the Pace and Efficiency of Rifle Marksmanship Training BIBAKFull-Text 630-639
  Giby Raphael; Chris Berka; Djordje Popovic; Gregory K. W. K. Chung; Sam O. Nagashima; Adrienne Behneman; Gene Davis; Robin Johnson
Marksmanship training involves a combination of classroom instructional learning and field practice involving the instantiation of a well-defined set of sensory, motor and cognitive skills. Current training procedures rely heavily on conventional classroom instruction often with qualitative assessment based on observation (i.e. coaching). We have developed a novel device called the Peak Performance Trainer (PPT™) which can accelerate the progression from novice-to-expert based on automated inferences from neurophysiological measurements. Our previous work has revealed specific EEG correlates to stages of skill acquisition in simple learning and memory tasks. We have incorporated this knowledge as well as an array of other physiological metrics to develop a field-deployable training technology with continuous physiological monitoring in combination with simultaneous measures of performance, workload, engagement and distraction, accuracy, speed and efficiency. This paper outlines the features of the PPT and the preliminary results of its use in marksmanship training.
Keywords: EEG; Heart rate; Alpha; Theta; Haptics
The Quality of Training Effectiveness Assessment (QTEA) Tool Applied to the Naval Aviation Training Context BIBAKFull-Text 640-649
  Tom Schnell; Rich Cornwall; Melissa Walwanis; Jeff Grubb
Today, flight trainers use objective measures of task performance and additional estimated, subjective data to assess the cognitive workload and situation awareness of trainees. This data is very useful in training assessment but trainees can succeed at performing a task purely by accident (referred to as "miserable success"). Additionally the trainee can be in a less than optimal for learning cognitive state when the instructor operator applies brute force training tasks and methods with little regard to the learning curve which can result in the training being too easy or, more often, too difficult, thereby inducing negative learning. In order to provide the instructor with additional quantitative data on student performance, we have designed the Quality of Training Effectiveness Assessment (QTEA) concept. QTEA is conceived as a system that allows the trainer to assess a student in real-time using sensors that can quantify the cognitive and physiological workload.
Keywords: Neurocognitive measures; operator state characterization; flight training
Perceptually-Informed Virtual Environment (PerceiVE) Design Tool BIBAKFull-Text 650-657
  Anna Skinner; Jack Maxwell Vice; Corinna E. Lathan; Cali M. Fidopiastis; Chris Berka; Marc M. Sebrechts
Virtual environments (VE's) are becoming more and more prevalent as training tools for both military and civilian applications. The common assumption is that the more realistic the VE, the better the transfer of training to real world tasks. However, some aspects of task content and fidelity may result in stronger transfer of training than even the most high fidelity simulations. This research effort seeks to demonstrate the technical feasibility of a Perceptually-informed Virtual Environment (PerceiVE) Design Tool, capable of dynamically detecting changes in operator behavior and physiology throughout a VE experience and comparing those changes to operator behavior and physiology in real-world tasks. This approach could potentially determine which aspects of VE fidelity will have the highest impact on transfer of training. A preliminary study was conducted in which psychophysiological and performance data were compared for a visual search tasks with low and high fidelity conditions. While no significant performance effects were found across conditions, event-related potential (ERP) data revealed significant differences between the low and high fidelity stimulus conditions. These results suggest that psychophysiological measures may provide a more sensitive and objective measure for determining VE fidelity requirements.
Keywords: Psychophysiological Measures; Virtual Environments; Fidelity; Transfer of Training; Simulation Design
Can Neurophysiologic Synchronies Provide a Platform for Adapting Team Performance? BIBAKFull-Text 658-667
  Ronald H. Stevens; Trysha Galloway; Chris Berka; Marcia Sprang
We have explored using neurophysiologic patterns as an approach for developing a deeper understanding of how teams collaborate when solving time-critical, complex real-world problems. Fifteen students solved substance abuse management simulations individually, and then in teams of three while measures of mental workload (WL) and engagement (E) were generated by electroencephalography (EEG). High and low workload and engagement levels were identified at each epoch for each team member and vectors of these measures were clustered by self organizing artificial neural networks. The resulting patterns, termed neurophysiologic synchronies, differed for the five teams reflecting the teams' efficiency. When the neural synchronies were compared across the collaboration, segments were identified where different synchronies were preferentially expressed. This approach may provide an approach for monitoring the quality of team work during complex, real-world and possible one of a kind problem solving, and for adaptively modifying the teamwork flow when optimal synchronies are not frequent.
Keywords: Collaboration; EEG; Neurophysiologic synchrony
Seeing the World through an Expert's Eyes: Context-Aware Display as a Training Companion BIBAFull-Text 668-677
  Marc T. Tomlinson; Michael Howe; Bradley C. Love
Responsive Adaptive Display Anticipates Requests (RADAR) is a domain general system that learns to highlight an individual's preferred information displays, given the current context. Previous studies with human subjects in a video game environment demonstrate that RADAR is an effective cognitive aid. RADAR increases situation awareness and reduces cognitive load by anticipating and providing task relevant information. Additionally, because RADAR's fit to a user's behavior encapsulates the user's situation-driven information preferences, RADAR also excels as a descriptive and predictive assessment tool. Here, we focus RADAR as a training aid. We test the hypothesis that novices can benefit from training under a RADAR model derived from an expert's behavioral patterns. The results indicate that novices exposed to an expert's information preferences through RADAR rapidly learn to conform to the expert's preferences.
Translating Learning Theories into Physiological Hypotheses BIBAKFull-Text 678-686
  Jennifer J. Vogel-Walcutt; Denise M. Nicholson; Clint A. Bowers
The battlefield has become an increasingly more complicated setting in which to operate. Additional stressors, complexity, and novel situations have challenged not only those in the field, but consequently also those in training. More information must be imparted to the trainees, yet more time is not available. Thus, in this paper, we consider one way to optimize the delivery and acquisition of knowledge that can be meaningfully applied to the field setting. We hypothesize that for learning efficiency to be maximized, we need to keep learners in a constant state of engagement and absorption. As such, we consider neuro-physiological hypotheses that can help prescribe mitigation strategies to reduce the impact of sub-optimal learning.
Keywords: Learning efficiency; Augmented cognition; Adaptive training
Adapting Instruction BIBAFull-Text 687-695
  Wallace H. Wulfeck
It is often claimed that adapting instruction to an individual's progress, personal characteristics or preferences will somehow increase learning, and there have been literally thousands of studies over at least the past 100 years exploring this idea. This presentation will review various approaches to adapting instruction, such as changing the rate, difficulty, sequence or structure, instructional strategy or instructional media on the basis of learner progress, prior knowledge, aptitudes or preferences, under various forms of instructor, learner, program, or opponent control. This paper gives an organizing framework, describes some of the theoretical underpinnings for particular adaptations, and describes experimental and practical criteria for evaluating claims of efficacy and efficiency of instructional adaptations.

Brain-Computer Interfaces

Assessment of Cognitive Neural Correlates for a Functional Near Infrared-Based Brain Computer Interface System BIBAKFull-Text 699-708
  Hasan Ayaz; Patricia A. Shewokis; Scott C. Bunce; Maria T. Schultheis; Banu Onaral
Functional Near Infrared Spectroscopy (fNIR) is a promising brain imaging technology that relies on optical techniques to detect changes of hemodynamic responses within the prefrontal cortex in response to sensory, motor, or cognitive activation. fNIR is safe, non-invasive, affordable, and highly portable. The objective of this study is to determine if biomarkers of neural activity generated by intentional cognitive activity, as measured by fNIR, can be used to communicate directly from the brain to a computer. A bar-size-control task based on a closed-loop system was designed and tested with 5 healthy subjects across two days. Comparisons of the average task and rest period oxygenation changes are significantly different (p<0.01). The average task completion time (reaching +90%) decreases with practice: day1 (mean 52.3 sec) and day2 (mean 39.1 sec). These preliminary results suggest that a closed-loop fNIR-based BCI can allow for a human-computer interaction with a mind switch task.
Keywords: Brain Computer Interface; fNIR; Near Infrared Spectroscopy
Systems and Strategies for Accessing the Information Content of fNIRS Imaging in Support of Noninvasive BCI Applications BIBAKFull-Text 709-718
  Randall L. Barbour; Harry L. Graber; Yong Xu; Yaling Pei; Glenn R. Wylie; Gerald T. Voelbel; John DeLuca; Andrei V. Medvedev
An essential component for a practical noninvasive brain-computer interface (BCI) system is data recording technology that can access the information-processing activity of the brain with high fidelity and throughput. Functional near-infrared spectroscopic (fNIRS) imaging is a methodology that shows promise in meeting this need, having a demonstrated sensitivity to both the slow hemodynamic response that follows neuroactivation and to the lower amplitude fast optical response that is considered a direct correlate of neuroactivation. In this report we summarize the technology integration strategy we have developed that permits detection of both signal types with a single measuring platform, and present results that document the ability to detect these data types transcranially in response to two different visual paradigms. Also emphasized is the effectiveness of different data analysis approaches that serve to isolate signals of interest. The findings support the practical utility of NIRS-based imaging methods for development of BCI applications.
Keywords: Diffuse Optical Tomography; fNIRS imaging; fast signal; combinatorial Hb States; Neuroactivation; Visual Stimulus; NIRS Technology
Brain-Computer Interaction BIBAKFull-Text 719-723
  Peter Brunner; Gerwin Schalk
Detection and automated interpretation of attention-related or intention-related brain activity carries significant promise for many military and civilian applications. This interpretation of brain activity could provide information about a person's intended movements, imagined movements, or attentional focus, and thus could be valuable for optimizing or replacing traditional motor-based communication between a person and a computer or other output devices. We describe here the objective and preliminary results of our studies in this area.
Keywords: Brain-computer interface; BCI; Neural Engineering; Neural Prosthesis
P300 Based Brain Computer Interfaces: A Progress Report BIBAKFull-Text 724-731
  Emanuel Donchin; Yael Arbel
Brain-Computer Interfaces (BCI) are the only means of communication available to patients who are locked-in, that is for patients who are completely paralyzed yet are fully conscious. We focus on the status of the P300-BCI first described by Farwell and Donchin (1988). This system has now been tested with several dozen ALS patients and some have been using this approach for communication at a very extensive level. More recently, we have adapted this BCI (in collaboration with the laboratory of Dr. Rajiv Dubey) to the control of a robotic arm. In this presentation we will discuss the special problems of human computer interaction that occur within the context of such a BCI. The special needs of the users forced the development of variants of this system, each with advantages and disadvantages. The general principles that can be derived from the experience we have had with this BCI will be reviewed.
Keywords: Brain Computer Interface (BCI); P300; wheelchair-mounted robotic arm (WMRA)
Goal-Oriented Control with Brain-Computer Interface BIBAKFull-Text 732-740
  Günter Edlinger; Clemens Holzner; Christoph Groenegress; Christoph Guger; Mel Slater
A brain-computer interface (BCI) is a new communication channel between the human brain and a digital computer. Such systems have been designed to support disabled people for communication and environmental control. In more recent research also BCI control in combination with Virtual Environments (VE) gains more and more interest. Within this study we present experiments combining BCI systems and VE for navigation and control purposes just by thoughts. Results show that the new P300 based BCI system allows a very reliable control of the VR system. Of special importance is the possibility to select very rapidly the specific command out of many different choices. The study suggests that more than 80% of the population could use such a BCI within 5 minutes of training only. This eliminates the usage of decision trees as previously done with BCI systems.
Keywords: Brain-computer interface; virtual reality; P300 evoked potential
Wearable and Wireless Brain-Computer Interface and Its Applications BIBAKFull-Text 741-748
  Chin-Teng Lin; Li-Wei Ko; Che-Jui Chang; Yu-Te Wang; Chia-Hsin Chung; Fu-Shu Yang; Jeng-Ren Duann; Tzyy-Ping Jung; Jin-Chern Chiou
This study extends our previous work on mobile & wireless EEG acquisition to a truly wearable and wireless human-machine interface, NCTU Brain-Computer-Interface-headband (BCI-headband), featuring: (1) dry Micro-Electro-Mechanical System (MEMS) EEG electrodes with 400 ganged contacts for acquiring signals from non-hairy sites without use of gel or skin preparation; (2) a miniature data acquisition circuitry; (3) wireless telemetry; and (4) online signal processing on a commercially available cell phone or a lightweight, wearable digital signal processing module. The applicability of the NCTU BCI-headband to EEG monitoring in real-world environments was demonstrated in a sample study: cognitive-state monitoring and management of participants performing normal tasks.
Keywords: Dry electrodes; brain computer interface; mobile and wireless EEG
Mind Monitoring via Mobile Brain-Body Imaging BIBAKFull-Text 749-758
  Scott Makeig
Current brain-computer interface (BCI) research attempts to estimate intended operator body or cursor movements from his/her electroencephalographic (EEG) activity alone. More general methods of monitoring operator cognitive state, intentions, motivations, and reactions to events might be based on continuous monitoring of the operator's (EEG) as well as his of her body and eye movements and, to the extent possible, her or his multisensory input. Joint modeling of this data should attempt to identify individualized modes of brain/body activity and/or reactivity that appear in the operator's brain and/or behavior in distinct cognitive contexts, if successful producing, in effect, a new mobile brain/body imaging (MoBI) modality. Robust MoBI could allow development of new brain/body-system interface (BBI) designs performing multidimensional monitoring of an operator's changing cognitive state including their movement intentions and motivations and ('top-down') apprehension of sensory events.
Keywords: cognitive monitoring; electroencephalography (EEG); motion capture; independent component analysis (ICA); brain-computer interface (BCI); mobile brain/body imaging (MoBI); human-computer interface (HCI)
Utilizing Secondary Input from Passive Brain-Computer Interfaces for Enhancing Human-Machine Interaction BIBAFull-Text 759-771
  Thorsten O. Zander; Christian Kothe; S. Welke; Matthias Roetting
A Brain-Computer Interface (BCI) directly translates patterns of brain activity to input for controlling a machine. The introduction of methods from statistical machine learning [1] to the field of brain-computer interfacing (BCI) had a deep impact on classification accuracy. It also minimized the effort needed to build up the skill of controlling a BCI system [2]. This enabled other fields of research to adapt methods from BCI research for their own purposes [3, 4]. Team PhyPA, the research group for physiological parameters of the chair for Human-Machine Systems (HMS) of the Technical University of Berlin, focuses on enabling new communication channels for HMS. Especially the use of passive BCIs (pBCI) [3, 4], not dependent on any intended action of the user, show a high potential for enhancing the interaction in HMS [5]. Additionally, as actual classification rates are still below the threshold for efficient primary control [6, 7] in HMS, we focus on establishing a secondary, BCI-based communication channel. This kind of interaction does not necessarily disturb the primary mode of interaction, providing a low usage cost and hence an efficient way of enhancement. We have designed several applications following this approach. Here we are going to present briefly the results from two studies, which show the capabilities arising from the use of passive and secondary BCI interaction. First, we show that a pBCI can be utilized to gain valuable information about HMSs, which are hard to detect by exogeneous factors. By mimicking a typical BCI interaction, we have been able to identify and isolate a factor inducing non-stationarities with a deep impact on the feature dynamics. The retained information can be utilized for automatically triggered classifier adaptation. And second, we show that pBCIs are indeed capable to enhance common HMS interaction outside the laboratory. With this, we would like to feed back our experiences made with the use of BCIs for HMS. We hope to provide new and useful information about brain dynamics which might be helpful for ongoing research in augmented cognition.

Rehabilitation and Cognitive Aids

Augmented Cognition as Rehabilitation: Facilitating Neuroplasticity? BIBAKFull-Text 775-781
  Michael Feuerstein; Gina Luff; Mark Peugeot; Miki Moskowitz; Briana Todd
Different types of brain injury are associated with deficits in working memory, executive functioning, and information processing speed, which can impact performance at work. Augmented Cognition (AugCog), a technology developed to improve human performance in complex tasks, may have potential for optimizing cognitive functioning in the context of work for those with mild to moderate cognitive deficits. AugCog is a way to accommodate or augment function thus improving the performance of the operator. This approach may facilitate neuroplasticity that can occur following injury to the brain. The authors will provide the rationale, operational structure, and potential application to occupational rehabilitation.
Keywords: Augmented cognition; occupational rehabilitation; cognitive limitations; mild brain injury
Embodying Meaning in Bio-cognitive Aid Design BIBAFull-Text 782-791
  Daniel Garrison; Victoria Garrison
Through analysis involving components of theory, concept, and semantics, we synthesize a set of insights and relations existing through embodiment that have applicability to the design of cognitive aids. Of note, we explore the areas of embodied interaction, embodied user interfaces, and embodied cognition. Additionally, we demonstrate how meaning is created and distributed in and through the areas. We then apply findings to the development of an embodied user interface developed to cognitively aid in renovation projects.
CI Therapy: A Method for Harnessing Neuroplastic Changes to Improve Rehabilitation after Damage to the Brain BIBAFull-Text 792-799
  L. V. Gauthier; E. Taub
Constraint-Induced Movement (CI) therapy has been successfully implemented for treating motor deficit resulting from a variety of previously intractable neurological conditions such as traumatic brain injury. CI therapy's efficacy can be attributed to two interrelated mechanisms: overcoming "learned nonuse" and neuroplasticity. Voxel-based morphometry (VBM) analyses have demonstrated that CI therapy produces lasting structural changes to the human brain. Patients that received full CI therapy demonstrated profuse grey matter increases in sensory and motor areas and hippocampus, whereas those who received only intensive motor practice did not. The magnitude of the observed structural changes was correlated with the extent to which the patient regained use of the impaired arm for daily activities. These findings demonstrate that the two mechanisms believed to underlie improvement from CI therapy, overcoming "learned nonuse" and neuroplasticity, act synergistically. Therefore, a bidirectional approach to treating brain injury, one that targets both brain and behavior, is suggested.
Augmented Cognition Design Approaches for Treating Mild Traumatic Brain Injuries BIBAKFull-Text 800-809
  Kay M. Stanney; Kelly S. Hale; David Jones
Augmented cognition could serve as an innovative rehabilitation approach for mild traumatic brain injuries, where issues with cognition, behavior, and affective responses are monitored in real-time and mitigation strategies are triggered to resolve performance or behavior issues. Such mitigations could guide individuals in addressing the current situation (e.g., performance decrement, undesired behavior, negative affective response), as well as provide rehabilitation support to improve performance and behavior in subsequent situations. This paper focuses on mitigation strategies that are suitable for an augmented cognition rehabilitation setting, with the goal of supporting recovery from suboptimal performance and providing rehabilitation tools in real-time, operational context.
Keywords: Augmented cognition; mild traumatic brain injury; mitigation strategies
Brain Processes and Neurofeedback for Performance Enhancement of Precision Motor Behavior BIBAKFull-Text 810-817
  B. Hatfield; A. Haufler; José L. Contreras-Vidal
Based on a number of empirical investigations of cerebral cortical dynamics during precision aiming tasks (i.e. marksmanship) employing electroencephalography (EEG) refinement of cortical activity and attenuation of nonessential cortico-cortical communication with the motor planning regions of the brain results in superior performance. Employment of EEG neurofeedback during the aiming period of target shooting designed to reduce cortical activation resulted in improved performance in skilled marksmen. Such an effect implies that refinement of cortical activity is causally related to performance. Recently, we examined cerebral cortical dynamics during the stress of competitive target shooting and observed increased activation and cortico-cortical communication between non-motor and motor regions relative to a practice-alone condition. As predicted, this finding was associated with degradation of shooting performance. These findings imply that neurofeedback targeted to brain regions related to emotional responding may preserve the cortical dynamics associated with superior performance resulting in improved accuracy of precision aiming performance.
Keywords: electroencephalography (EEG); psychomotor performance; cognitive neuroscience; stress; kinematics
Long Term Repair of Learning Disability through Short-Term Reduction of CNS Inhibition BIBAKFull-Text 818-825
  H. Craig Heller; Damien Colas; Norman F. Ruby; Fabian Fernandez; Bayarasaikhan Chuluun; Martina Blank; Craig C. Garner
Learning disabilities are serious societal problems contributing to a loss of quality of life for affected individuals and their families. We hypothesized that the learning disability in Down Syndrome and perhaps in other neurodegenerative disorders is due to an imbalance between inhibitory and excitatory tone in the CNS. Specifically, we predicted that reduction of GABA related inhibition would improve learning. We used the TsDn65 mouse model of Down Syndrome and treated adult mice with daily doses of different GABA antagonists. Following treatments learning performance of these mice in several rodent learning tasks was indistinguishable from the performance of wild type mice, and the learning improvement lasted for months after the treatment ended. We are now exploring the mechanism of this durable neuroplastic effect and asking whether it would generalize to other learning disorders or optimize learning in wild type mice.
Keywords: GABA; picrotoxin; pentylenetetrazole; bilobilide; flumazinil; Down Syndrome; TsDn65 mice; novel object recognition
Development of Sensitive, Specific, and Deployable Methods for Detecting and Discriminating mTBI and PTSD BIBAKFull-Text 826-835
  Robin R. Johnson; Djordje Popovic; Deborah Perlick; Dennis Dyck; Chris Berka
This paper presents a theoretical framework for the development of non-invasive methods for detection and discrimination between mild traumatic brain injury (mTBI) and post-traumatic stress disorder (PTSD). Growing use of IEDs and increased pace of multiple deployment cycles in current conflicts has lead to significant increases in exposure to risks for these conditions. Co-morbidity of these conditions is common, diagnostically challenging, and controversial. Development of easy to use, deployable diagnostic tools would allow for accurate early identification and intervention. Early intervention increases the potential for positive outcomes for both the individual and their families. In addition, the appropriately designed system could be used epidemiologically to screen returning soldiers for these conditions that may otherwise not be appropriately assessed until much later, if at all. The framework presented here proposes that a wireless, portable EEG/EKG based device may be an appropriate platform upon which to develop such an assessment tool.
Keywords: Electroencephalogram (EEG); Electrocardiogram (EKG); Post-Traumatic Stress Disorder (PTSD); mild Traumatic Brain Injury (mTBI)
Physiologically Driven Rehabilitation Using Virtual Reality BIBAKFull-Text 836-845
  Angela M. Salva; Antonio J. Alban; Mark D. Wiederhold; Brenda K. Wiederhold; Lingjun Kong
Creating a platform that allows the fusion of real-time physiological measurements and virtual reality (VR) simulation will greatly improve present human-computer interaction, adaptive displays, military training, and anxiety therapy. The Virtual Reality Medical Center (VRMC) has developed a physiologically-driven rehabilitation platform that correctly assesses user anxiety levels based on multiple real time physiological measures, determines the optimal level of physiological arousal for each individual user, and automates the virtual simulation to the proper intensity for each user. Additionally, VRMC collaborates with UCF to develop novel, state-of-the-art sensors to be integrated within the platform that are capable of measuring electrocardiogram, (EEG), skin conductance, gait, and pupillometry. In Phase I VRMC developed a capability to monitor, fuse, and evaluate physiological measures (heart rate, skin conductance, skin temperature, and respiration) in real time to assess user anxiety levels. The physiological data collected will be used to assess user anxiety levels in real time as neutral, low, or high with 90% accuracy and to determine the optimal level of physiological arousal for each individual user.
Keywords: physiological measurement; stroke; traumatic brain injury; cerebrovascular accident; rehabilitation; cognitive rehabilitation; simulation; mixed reality