HCI Bibliography Home | HCI Conferences | FAC Archive | Detailed Records | RefWorks | EndNote | Hide Abstracts
FAC Tables of Contents: 070911131415

FAC 2011: 6th International Conference on Foundations of Augmented Cognition. Directing the Future of Adaptive Systems

Fullname:FAC 2011: 6th International Conference on Augmented Cognition. Directing the Future of Adaptive Systems
Note:Volume 20 of HCI International 2011
Editors:Dylan Schmorrow; Cali M. Fidopiastis
Location:Orlando, Florida
Dates:2011-Jul-09 to 2011-Jul-14
Publisher:Springer-Verlag
Series:Lecture Notes in Computer Science 6780
Standard No:ISBN: 978-3-642-21851-4 (print), 978-3-642-21852-1 (online); hcibib: FAC11
Papers:75
Pages:666
Links:Online Proceedings | Publisher Book Page | Conference Webpage
  1. Theories, Models and Technologies for Augmented Cognition
  2. Neuroscience and Brain Monitoring
  3. Augmented Cognition, Social Computing and Collaboration
  4. Augmented Cognition for Learning
  5. Augmented Cognition and Interaction
  6. Augmented Cognition in Complex Environments

Theories, Models and Technologies for Augmented Cognition

The Brain as Target Image Detector: The Role of Image Category and Presentation Time BIBAFull-Text 3-12
  Anne-Marie Brouwer; Jan B. F. van Erp; Bart Kappé; Anne E. Urai
The brain can be very proficient in classifying images that are hard for computer algorithms to deal with. Previous studies show that EEG can contribute to sorting shortly presented images in targets and non-targets. We examine how EEG and classification performance are affected by image presentation time and the kind of target: humans (a familiar category) or kangaroos (unfamiliar). Humans are much easier detected as indicated by behavioral data, EEG and classifier performance. Presentation of humans is reflected in the EEG even if observers were attending to kangaroos. In general, 50ms presentation time decreased markers of detection compared to 100ms.
Implementation of fNIRS for Monitoring Levels of Expertise and Mental Workload BIBAKFull-Text 13-22
  Scott C. Bunce; Kurtulus Izzetoglu; Hasan Ayaz; Patricia A. Shewokis; Meltem Izzetoglu; Kambiz Pourrezaei; Banu Onaral
An accurate measure of mental workload would help improve operational safety and efficacy in many environments that involve multitasking or sustained vigilance. The current study utilized functional near-infrared spectroscopy (fNIRs) to examine the relationship of the hemodynamic response in dorsolateral prefrontal cortex (DLPFC) as it related to mental workload, level of expertise, and task performance. DLPFC responses were monitored with fNIRs while 8 participants (4 with high practice, 4 novices) completed a quasirealistic computerized Warship Commander Task with various levels of difficulty. The results show that greater expertise was associated with relatively lower oxygenation (less neural activity) at low to moderate levels of taskload, but higher oxygenation and better performance at high levels of taskload. For novices, oxygenation was higher at moderate levels of taskload, but dropped precipitously at higher levels of taskload, along with performance, consistent with disengaging from the task. Results are interpreted within a "scaffolding-storage" framework.
Keywords: Optical Brain Imaging; functional near infrared spectroscopy; mental workload; expertise; practice; fNIR
Challenges and Solutions with Augmented Cognition Technologies: Precursor Issues to Successful Integration BIBAKFull-Text 23-29
  Joseph Cohn
Today's combat environment requires increasingly complex interactions between human operators and their systems. Whereas in the past, the roles of human and system were clearly delineated, with the integration of advanced technologies into the C4ISR toolkit, the distinct parsing of tasks has given way to paradigms in which the human operator's roles and responsibilities must dynamically change according to task and context. Yet, current methodologies for integrating the human into the system have not kept pace with this shift. An important consequence of this mismatch between human operator and system is that failures often lead to catastrophic and unrecoverable accidents (O'Connor & Cohn, 2010). In order to reintegrate the human element back into the system, new approaches for representing operator performance, in terms of their individual cognitive and behavioral capacities, limitations and changing needs are required.
Keywords: Neuroscience; Cognition; Autonomy; Human Systems; Information Processing; Adaptive; Cognitive Architecture
Augmenting Brain and Cognition by Aerobic Exercise BIBAKFull-Text 30-38
  Kirk I. Erickson
Cognitive function declines in late adulthood and this is preceded by atrophy of the prefrontal cortex, hippocampal formation, and parietal cortex. Despite significant loss of brain tissue in late adulthood, decline is not ubiquitous across all older adults. In fact, some adults age quite successfully with minimal decline. This suggests that brain deterioration might not be an inevitable consequence of aging. In fact, mounting evidence suggests that participation in regular aerobic exercise is effective at enhancing cognitive and brain health in late adulthood. In this paper we discuss the evidence that cardiorespiratory fitness and aerobic exercise augments cognition by increasing gray matter volume in prefrontal and hippocampal brain regions.
Keywords: Aging; brain; atrophy; exercise
Neurological Advances and Ethical/Legal Conundrums: Lessons from History BIBAKFull-Text 39-45
  Cheryl Erwin
The scientific advances in the neurosciences are exciting and promise to advance our understanding of the human mind. The ethical and legal issues raised by neuroscience are distinctive but they are not unique to the twenty-first century. The ethical issues raised by these technologies deserve attention even while the science is in development. History teaches us to reflect on our humanity using insights from many disciplines and many times.
Keywords: Neuroethics; neurolaw; neuroprivacy; neuropolicy; research ethics; regulation of emerging technologies
Individual Differences and the Science of Human Performance BIBAKFull-Text 46-54
  Michael Trumbo; Susan M. Stevens-Adams; Stacey M. L. Hendrickson; Robert G. Abbott; Michael Haass; Chris Forsythe
This study comprises the third year of the Robust Automated Knowledge Capture (RAKC) project. In the previous two years, preliminary research was conducted by collaborators at the University of Notre Dame and the University of Memphis. The focus of this preliminary research was to identify relationships between cognitive performance aptitudes (e.g., short-term memory capacity, mental rotation) and strategy selection for laboratory tasks, as well as tendencies to maintain or abandon these strategies. The current study extends initial research by assessing electrophysiological correlates with individual tendencies in strategy selection. This study identifies regularities within individual differences and uses this information to develop a model to predict and understand the relationship between these regularities and cognitive performance.
Keywords: Individual Differences; EEG; Memory Span; RAT; Attentional Beam; Mental Rotation; Ruff Attention Task; Raven's Matrices; Box Folding; Dual Task; Barton's; Binary; Stroop; N-back; Mismatch Negativity; P300; Oddball; Semantic Memory; Episodic Memory; Go/No-Go; Flanker; Line Drawing; MAT-B
Cognition: What Does It Have to Do with the Brain? BIBAKFull-Text 55-59
  Alexandra Geyer
The emergence of new non-invasive technologies for assessing the structure and the function of the human brain has provided us with means to investigate the neural substrates underlying cognitive processes with the goal of achieving a better understanding of cognition. This paper is focused on discussing the contributions that assessment of neural processes underlying cognition brings to our understanding of cognition as well as the impact of this understanding of cognition on military operations.
Keywords: cognition; neural substrates; intent; warfighters
The Evolution of Human Systems: A Brief Overview BIBAKFull-Text 60-66
  Jeff Grubb; Joseph Cohn
Recently, there has been a profound resurgence interest in expanding the effectiveness of human machine systems. The motivation for this interest stems not only from the growing realization that better designed systems -- tailored to augment their user's innate skills and capabilities -- will enable users to 'do more', but also from the fact that the world with which we interact is becoming increasingly reliant on machines. In the past, the human machine interface was bridged through engineering based principles, but, with our expanding understanding of how the human brain drives behavior it is now possible to consider, as never before, human machine design efforts that fully address human and machine needs at the same time.
Keywords: Neuroscience; Cognition; Automation; Human Systems; Cognitive Model
The Influence of Culture on Memory BIBAKFull-Text 67-76
  Angela H. Gutchess; Aliza J. Schwartz; Aysecan Boduroglu
The study of cognition across cultures offers a useful approach to both identifying bottlenecks in information processing and suggesting culture-specific strategies to alleviate these limitations. The recent emphasis on applying cognitive neuroscience methods to the study of culture further aids in specifying which processes differ cross-culturally. By localizing cultural differences to distinct neural regions, the comparison of cultural groups helps to identify candidate information processing mechanisms that can be made more efficient with augmented cognition and highlights the unique solutions that will be required for different groups of information processors.
Keywords: cognition; culture; memory; strategies; fMRI
Using Computational Modeling to Assess Use of Cognitive Strategies BIBAKFull-Text 77-86
  Michael J. Haass; Laura E. Matzen
Although there are many strategies and techniques that can improve memory, cognitive biases generally lead people to choose suboptimal memory strategies. In this study, participants were asked to memorize words while their brain activity was recorded using electroencephalography (EEG). The participants' memory performance and EEG data revealed that a self-testing (retrieval practice) strategy could improve memory. The majority of the participants did not use self-testing, but computational modeling revealed that a subset of the participants had brain activity that was consistent with this optimal strategy. We developed a model that characterized the brain activity associated with passive study and with explicit memory testing. We used that model to predict which participants adopted a self-testing strategy, and then evaluated the behavioral performance of those participants. This analysis revealed that, as predicted, the participants whose brain activity was consistent with a self-testing strategy had better memory performance at test.
Keywords: Memory; computational modeling; electroencephalography
Advances and Challenges in Signal Analysis for Single Trial P300-BCI BIBAFull-Text 87-94
  Kun Li; Vanitha Narayan Raju; Ravi Sankar; Yael Arbel; Emanuel Donchin
In this paper a brief introduction to some of the goals, recent developments, and open problems in BCI research are given. We mainly focus on presenting our research work in signal processing for single-trial P300-BCI and discuss our current plans for improving the BCI method.
Characterizing the Performance Limits of High Speed Image Triage Using Bayesian Search Theory BIBAKFull-Text 95-103
  Santosh Mathan; Kenneth E., II Hild; Yonghong Huang; Misha Pavel
The rapid serial visual presentation (RSVP) modality has been used in conjunction with neurophysiological and behavioral responses to identify targets within large volumes of imagery efficiently. The research reported here uses optimal search theory to characterize the limits of this approach. Search theory is used to inform the estimation of detection functions. These functions provide a principled basis for selecting presentation parameters that balance search efficiency and accuracy. Detection functions are also used to characterize individual differences in search performance and to assess the extent to which the RSVP presentation modality generalizes across a class of complex targets.
Keywords: EEG; Search Theory; Rapid Serial Visual Presentation; Visual Psychophysics; Detection Functions; Target Detection
Facial Recognition: An Enabling Technology for Augmented Cognition Applications BIBAKFull-Text 104-111
  Denise M. Nicholson; Christine Podilchuk; Kathleen Bartlett
Research in Augmented Cognition (AugCog) investigates computational methods, technologies, and non-invasive neurophysiological tools to adapt computational systems to the changing cognitive state of human operators to improve task performance. Closed-loop AugCog systems contain four components: 1) operational or simulated environment, 2) automated sensors to monitor and assess cognitive state via behavior and/or physiology, 3) adaptive interface, and 4) computational decision architecture that directs AugCog adaptations. Since cognitive state is influenced by environment, a critical challenge for AugCog systems is capture of situational awareness (SA) within the decision architecture. Previously, AugCog systems have been demonstrated within simulated environments that provide SA and ground truth data to drive intelligent decision architecture. In live operating environments, electronic C4 systems (i.e., communications), provide a limited model of operator "state," but emerging facial recognition/analysis technology can provide detection, identification, and tracking of humans in the environment to increase the accuracy of the AugCog system's SA.
Keywords: Augmented Cognition; facial recognition; situation awareness; biometrics; environmental monitoring
Analysis of Multiple Physiological Sensor Data BIBAKFull-Text 112-119
  Lauren Reinerman-Jones; Grant Taylor; Keryl Cosenzo; Stephanie J. Lackey
Physiological measures offer many benefits to psychological research including objective, non-intrusive assessment of affective and cognitive states. However, this utility is limited by analysis techniques available for testing data recorded by multiple physiological sensors. The present paper presents one set of data that was attained from a repeated measures design with a nominal independent variable for analysis. Specifically, the International Affective Picture System (IAPS; Lang, Bradley, & Cuthbert, 2008), a series of images known to convey seven different emotions, was presented to participants while measures of their neurological activity (Electroencephalogram; EEG), heart rate (Electrocardiogram; ECG), skin conductance (Galvanic Skin Respond; GSR), and pupillary response were taken. Subsequently, a discussion of statistics available for analyzing responses attained from the various sensors is presented. Such statistics include correlation, ANOVA, MANOVA, regression, and discriminant function analysis. The details on design limitations are addressed and recommendations are given for employing each statistical option.
Keywords: EEG; ECG; Eye Tracking; Statistical Analyses; Emotion
Exploring New Methodologies for the Analysis of Functional Magnetic Resonance Imaging (fMRI) Following Closed-Head Injuries BIBAKFull-Text 120-128
  Peter B. Walker; Ian N. Davidson
An increasing amount of research has focused on the use of newer and alternative data analytic approaches to multi-dimensional data sets. The primary aim of this paper is to introduce two data analytic approaches as they have been applied to image scans from functional Magnetic Resonance Imaging (fMRI). The first approach involves loading data from fMRI scans into multi-dimensional cubes and performing tensor decomposition. In addition, we introduce a second approach involving the use of network modeling that attempts to identify stable networks in fMRI scans across time. Discussion will be focused on the application of these approaches to the modeling and rehabilitation following closed-head injury.
Keywords: fMRI; Tensor Decomposition; Graph/Network Modeling

Neuroscience and Brain Monitoring

EEG Knows Best: Predicting Future Performance Problems for Targeted Training BIBAKFull-Text 131-136
  Gwendolyn E. Campbell; Christine L. Belz; Charles P. R. Scott; Phan Luu
Many uses for neurophysiological data in training have been proposed in the literature [6], [10], and [11]. However, to date it has not been demonstrated that the use of EEG yields performance diagnoses that are actually more accurate. The current study investigated the capability of EEG to accurately diagnose performance difficulties by examining the predictive ability of an accurate diagnosis on future performance. The data from this study suggests that using EEG to filter a trainee's performance data prior to analysis on a computer based tank identification task yields a more accurate diagnosis than analyzing the data with the traditional statistical methods.
Keywords: electroencephalography; training; neurophysiology
Computational Cultural Neuroscience: Implications for Augmented Cognition BIBAKFull-Text 137-142
  Joan Y. Chiao
From perceiving objects in space to recognizing emotions at a distance, culture affects how people think, feel, reason as well as the neurobiological mechanisms underlying these processes. Here I review recent evidence from cultural neuroscience, introduce the notion of computational cultural neuroscience -- the development of computational and formal models of how culture affects neurobiological mechanisms and vice versa -- and finally, discuss the implications of computational cultural neuroscience for research in augmented cognition.
Keywords: cultural neuroscience; computational cultural neuroscience; augmented cognition
Enhancing Team Performance Using Neurophysiologic Synchronies in a Virtual Training Environment BIBAKFull-Text 143-152
  Marianne Clark; Kimberly Cellucci; Chris Berka; Daniel J. Levendowski; Jonny Trejo; Amy Kruse; Ron Stevens
A study was conducted to investigate the use of neurophysiologic synchronies as a measurement of team cognition (1) in a military-style virtual environment simulation. Neurophysiologic synchronies (NS), defined as the second-by-second quantitative co-expression of the levels of cognitive measures by individual members of a team (8), were found to be useful in monitoring the quality of teamwork and to be a means to identify more optimal patterns of team interaction which can be used to provide feedback during training. In the current study, findings showed promise for further research in the collection of NS. A framework is also proposed to support the research and training of team cognition.
Keywords: Team performance; team cognition; shared mental models; collaboration; neurophysiologic synchronies; electroencephalography (EEG); virtual environments; mission rehearsal training; RealWorld
Theoretical Transpositions in Brain Function and the Underpinnings of Augmented Cognition BIBAFull-Text 153-158
  Cali M. Fidopiastis
Augmented Cognition (AugCog) explores behavior in real-time and in real world settings. This research avenue is a departure from standard experimental approaches such as those accepted in the fields of Cognitive Psychology and the Neurosciences. AugCog as a field of study, therefore, has the potential to up-end some of the tried-and-true laboratory based findings on such topics as learning and transfer of learning. Steeped in history from both the biological systems perspective and the cognition neuroscience vantage, the future of AugCog seems contingent on its success at merging these paradigms and concurrently producing analysis tools with which to keep peering into the brain as it functions in operational environments. In this paper, we review the theories that drive Augmented Cognition approaches and evaluate their capacity to keep the field moving forward.
Non-invasive Functional Brain Biomarkers for Cognitive-Motor Performance Assessment: Towards New Brain Monitoring Applications BIBAKFull-Text 159-168
  Rodolphe J. Gentili
Along with theoretical advances in neuroscience research, recent neurotechnological developments provide portable recording and processing systems that can be employed for real-time assessment in applied military environments. This article provides a brief overview of research related to non-invasive brain biomarkers derived from brain signals that can track brain dynamics during cognitive-motor performance. Potential applications of such brain biomarkers for military personnel such as neurofeedback for accelerated learning as well as brain monitoring for performance assessment and rehabilitation are discussed.
Keywords: Cognitive-motor performance; EEG/fNIRS biomarkers; alpha power; phase synchronization; brain monitoring; neurofeedback; rehabilitation
Estimating the Level of Motion Sickness Based on EEG Spectra BIBAKFull-Text 169-176
  Li-Wei Ko; Chun-Shu Wei; Tzyy-Ping Jung; Chin-Teng Lin
Motion sickness (MS) is a normal response to real, perceived, or even anticipated movement. People tend to get motion sickness on a moving boat, train, airplane, car, or amusement park rides. Although many motion sickness-related biomarkers have been identified, but how to estimate human's motion sickness level (MSL) is a big challenge in the operational environment. Traditionally, questionnaire and physical check are the common ways to passively evaluate subject's sickness level. Our past studies had investigated the EEG activities correlated with motion sickness in a virtual-reality based driving simulator. The driving simulator comprised an actual automobile mounted on a Stewart motion platform with six degrees of freedom, providing both visual and vestibular stimulations to induce motion-sickness in a manner that is close to that in daily life. EEG data were acquired at a sampling rate of 500 Hz using a 32-channel EEG system. The acquired EEG signals were analyzed using independent component analysis (ICA) and time-frequency analysis to assess EEG correlates of motion sickness. Subject's degree of motion-sickness was simultaneously and continuously reported using an onsite joystick, providing non-stop psychophysical references to the recorded EEG changes. We found that the parietal, motor, occipital brain regions exhibited significant EEG power changes in response to vestibular and visual stimuli. Based on these findings and experimental results, this study aims to develop an EEG-based system to estimate subject's motion sickness level upon the EEG power spectra from motion-sickness related brain areas. The MS evaluation system can be applied to early detection of the subject's motion sickness and prevent its uncomfortable syndromes in our daily life. Furthermore, the experiment results could also lead to a practical human-machine interface for noninvasive monitoring of motion sickness of drivers or passengers in real-world environments.
Keywords: EEG; ICA; motion-sickness; estimation; time-frequency; driving cognition
Combining fNIRS and EEG to Improve Motor Cortex Activity Classification during an Imagined Movement-Based Task BIBAFull-Text 177-185
  Darren J. Leamy; Rónán Collins; Tomás Ward
This work serves as an initial investigation into improvements to classification accuracy of an imagined movement-based Brain Computer Interface (BCI) by combining the feature spaces of two unique measurement modalities: functional near infrared spectroscopy (fNIRS) and electroencephalography (EEG). Our dual-modality system recorded concurrent and co-locational hemodynamic and electrical responses in the motor cortex during an imagined movement task, participated in by two subjects. Offline analysis and classification of fNIRS and EEG data was performed using leave-one-out cross-validation (LOOCV) and linear discriminant analysis (LDA). Classification of 2-dimensional fNIRS and EEG feature spaces was performed separately and then their feature spaces were combined for further classification. Results of our investigation indicate that by combining feature spaces, modest gains in classification accuracy of an imagined movement-based BCI can be achieved by employing a supplemental measurement modality. It is felt that this technique may be particularly useful in the design of BCI devices for the augmentation of rehabilitation therapy.
The Frustration Status and Noise Proof Feature During Perception of the Auditory Images BIBAKFull-Text 186-193
  Sergey Lytaev; Yuliaj Surovitskaj
Tests for modeling of the human status at recognition of target and non target stimulus with auditory evoked potentials (AEPs) registration; emotional neutral and significant information-psychological influences with EEG registration and analysis of fractal dynamics (AFD) were applied. From the moment of signal presentation the greatest difference of AEPs at target stimulation is marked in frontal areas of the left hemisphere through 16-18 ms. Emotionally-neutral and emotionally-significant psycho-informational influences provided the most conclusive AFD EEG data. Essentially, personal frustration is activated when the subject perceives situations to be threatening to his or her self-esteem and self-evaluation. Individuals with high levels of frustration are inclined to perceive a wide range of situations as threatening and therefore, will respond according to what they think the situation dictates.
Keywords: Auditory Evoked Potentials (AEPs); Brain Mapping; EEG; Information-Psychological Influences
Cultural Neuroscience and Individual Differences: Implications for Augmented Cognition BIBAKFull-Text 194-198
  Laura E. Matzen
Technologies that augment human cognition have the potential to enhance human performance in a wide variety of domains. However, there are a number of individual differences in brain activity that must be taken into account during the development, validation, and application of augmented cognition tools. A growing body of research in cultural neuroscience has shown that there are substantial differences in how people from different cultural backgrounds approach various cognitive tasks. In addition, there are many other types of individual differences and even changes in a single individual over time that have implications for augmented cognition research and development. The aim of this session is to highlight a few of those differences and to discuss how they might impact augmented cognition technologies.
Keywords: Cultural neuroscience; individual differences
Towards a Software Toolkit for Neurophysiological Data Collection and Analysis BIBAFull-Text 199-202
  James Niehaus; Peter Weyhrauch
Modern devices such as cell phones, handheld computers, and technical equipment enable professional users to communicate, understand, and act more efficiently and effectively. However, these new systems often increase cognitive workload, and may even introduce performance errors. System analysts can decrease these errors by identifying a users cognitive performance deficits and addressing them through training, improved performance support, and redesigned operational systems. To identify these deficits, neurocognitive measurements of indicators such as cognitive workload and attention can be approximated with high accuracy by using non-invasive sensors to measure brain activity and other physiological indicators. Thus, we are designing and demonstrating the feasibility of a toolkit for system analysts to use neurocognitive measurements to recommend additional training for individual users, performance support for all users of the system, and the redesign of system interfaces or components. This research addresses a clear need for an extensible, general-purpose, stand-alone neurocognitive assessment toolkit that can be incorporated into new and existing technology development with little to no integration effort.
From Sound to Meaning: Changes in EEG Source-Localized Brain Activity with Foreign-Language Training BIBAKFull-Text 203-211
  Catherine Poulsen; Phan Luu; Colin Davey; Don M. Tucker; Joey Nelson
Learning a foreign language is a complex human task, involving multiple processes and a dynamic network of brain activity. The present study used 256-channel dense-array electroencephalography (dEEG) and linear-inverse source analysis (sLORETA) to identify changes in brain activity during the early stages of language training. Twenty native English speakers attended two 50-minute sessions of computer-assisted, virtual-reality Dari language instruction. Training-specific changes in neural activity were observed in both articulatory-motor and semantic processing regions, including increases in left posterior inferior temporal gyrus and left lateral inferior frontal regions. Also observed was increasing left lateralization, and an increase in mediotemporal regions suggestive of memory reconsolidation. These findings illustrate the ability to track changes with training in recognized language-processing brain regions using source-localized EEG recorded while listening to continuous, naturalistic speech. Subsequent research will explore individual differences and the development of adaptive training based on neural indices.
Keywords: language learning; training; dense-array EEG; linear-inverse source analysis; electroencephalography; event-related potentials
Analyzing Neural Correlates of Attentional Changes during the Exposure to Virtual Environments: Application of Transcranial Doppler Monitoring BIBAKFull-Text 212-220
  Beatriz Rey; Vera Parkhutik; José Tembl; Mariano Alcañiz Raya
Transcranial Doppler monitoring (TCD) has been proposed as a tool to be used in Augmented Cognition (AugCog) systems to monitor brain activation during the performance of different cognitive tasks. In the present study, the main goal is to analyze variations in blood flow velocity (BFV) measured by TCD during the exposure to a virtual reality environment when there are changes in the focus of attention of the participants. Two abrupt events are forced during the navigation in a virtual environment in order to change their focus of attention to the real world. In one of them, the screen goes completely blue, and in the other one, a mesh appears in front of the virtual environment making it difficult to visualize. Results show that BFV values in both middle cerebral arteries remain similar when the first event occurs, but there is an increase during the second event. The origin of this increment may probably be found in the higher difficulty of having a mesh in front of the virtual environment, requiring more attention than before. These results show that changes in the stimuli can generate modifications in BFV that can be monitored by TCD, and can be useful for AugCog applications.
Keywords: Augmented Cognition; Virtual Reality; Transcranial Doppler; Neurophysiological Data; Cognitive State Assessment
Neuroergonomic Assessment of Simulator Fidelity in an Aviation Centric Live Virtual Constructive (LVC) Application BIBAKFull-Text 221-230
  Tom Schnell; Alex Postnikov; Nancy Hamel
This paper describes a recent human factors study that was performed on a flight simulator and in a fighter trainer jet aircraft to quantify the cognitive effects of simulator fidelity. There are many parameters that could be manipulated to affect physical fidelity in a simulator and we want to point out that in this study we make no claims of having covered a large portion of the possible fidelity design space. Rather, this study provides a comparison of trainee performance in a low to mid-level simulator with the performance obtained in a real fighter jet training aircraft using state-of-the-art operator state characterization equipment. As this study is ongoing, only partial data is shown in this paper.
Keywords: Neurocognitive measures; operator state characterization; flight training
Brain Activity of Young and Adult Hebrew Speakers during Lexical Decision Task: fNIR Application to Language BIBAKFull-Text 231-239
  Itamar Sela; Tzipi Horowitz-Kraus; Meltem Izzetoglu; Patricia A. Shewokis; Kurtulus Izzetoglu; Banu Onaral; Zvia Breznitz
The process of reading activates a large-scale neural network which includes different cortical brain regions. This network is thought to be age-dependent and changes throughout the process of reading acquisition. The frontal lobe is considered to be related to higher, executive, functions. We conducted a functional Near InfraRed Spectroscopy (fNIR) study in order to compare frontal lobe performance during a Lexical Decision Task (LDT) among two different age-groups: children and adults. Data indicated significant differences with age in LDT behavioral performance, and brain activity in the upper left frontal lobe. The young group exhibited slower reaction times and lower accuracy in addition to differences both in the level of blood oxygenation as well in the blood oxygenation timeline. The current study's results suggest 1) the involvement of the frontal lobe during the process of reading and that 2) frontal lobe activity is modified with the age of maturity.
Keywords: Neuroimaging; fNIR; Lexical Decision Task; Developmental language
Brain in the Loop: Assessing Learning Using fNIR in Cognitive and Motor Tasks BIBAKFull-Text 240-249
  Patricia A. Shewokis; Hasan Ayaz; Meltem Izzetoglu; Scott C. Bunce; Rodolphe J. Gentili; Itamar Sela; Kurtulus Izzetoglu; Banu Onaral
The skill acquisition process and learning assessments are dependent upon the quality and extent of practice of the tasks. Typically, learning is inferred from behavioral and cognitive results without taking into account the role of the brain in the learning loop. In this paper we discuss the neural mechanisms of learning and skill acquisition using fNIR with 3D spatial navigation tasks (e.g., MazeSuite), a center-out reaching movement task during which adaptation to new tool use was performed and mathematical problem solving tasks. Further, this research study compared and contrasted multiple analysis methods, which include general linear models of repeated measures during acquisition, retention and transfer phases of learning, learning curve analyses, the testing of fit of various learning models (i.e., power, exponential or other non-linear functions) and relationships between neural activation and behavioral measures.
Keywords: Practice; Learning; Optical Brain Imaging; Analysis Methods; Functional Near Infrared Spectroscopy; fNIR; Prefrontal Cortex
Neurocognitive Patterns: Using Brain, Behavior, and Context to Infer User Intent BIBAKFull-Text 250-256
  Webb Stacy
Neurocognitive Patterns is a system that will offer execution options to users as soon as they form an intention to act. It will accomplish this by combining neural signals, user behavior, and contextual knowledge to determine when a user has a goal, and what that goal is. Because it will leverage the user's neural signals and behavioral history, the options it will provide to the user will be available quickly. Because it will leverage real-time contextual and background knowledge, its estimates concerning the user's goal will be accurate. Our initial target domain is UAV operators, but we expect it will be of use to other military decision-makers in Command and Control settings. We also expect that Neurocognitive Patterns will be a useful tool in Cognitive Neuroscience in general for interpreting neural signals in the presence of salient contextual information.
Keywords: Neural Signals; Behavioral Measures; User Intent; Using Contextual Information
Behavioral and Brain Dynamics of Team Coordination Part I: Task Design BIBAKFull-Text 257-264
  Emmanuelle Tognoli; A. J. Kovacs; B. Suutari; Daniel Afergan; Joseph T. Coyne; G. Gibson; Roy Stripling; J. A. Scott Kelso
In this study, pairs of subjects performed a team-intensive task with the shared goal of clearing a virtual room from threats. Our goal was to identify signatures of efficient team work from a dynamic analysis of both subjects' brain signals and behavioral performance. An ecologically valid task of room clearing was designed and a novel analysis framework was developed to address the challenge of understanding complex, continuous social processes at both behavioral and brain levels. In the present paper, we detail the design of the task, and present validation techniques undertaken to acquire and analyze high-quality and accurately timed neurobehavioral information. A companion paper will discuss the neurobehavioral findings and their implications.
Keywords: Neuromarkers; EEG; neurobehavioral dynamics; social behavior; complexity
Using Neurophysiological Data to Inform Feedback Timing: A Pilot Study BIBAKFull-Text 265-274
  Jennifer J. Vogel-Walcutt; Julian Abich
In an effort to achieve a level of knowledge comparable to that which typically results from individual tutoring, innovative models of adaptive computer-based training are continually being tested and refined. Despite these efforts, adaptive computerized training programs still fall significantly short of the gold standard of one-on-one instruction. In response, this study used a previously developed model defining when to apply instructional feedback during learning in order to improve efficiency. Specifically, we compared the combination of performance and neuro-physiological indices to performance alone as indicators for when to adapt training. Contrary to our hypotheses, this study failed to demonstrate positive impact on knowledge acquisition, knowledge application, perceived cognitive load, or training efficiency. However, based on observational data, it is suspected that participants in neither group possessed enough available working memory capacity to attend to the supporting material. Consequently, this may account for the lack of differential findings.
Keywords: Feedback; EEG; physiological measures; simulation based training; adaptive intelligent systems

Augmented Cognition, Social Computing and Collaboration

Modelling User Behaviour and Interactions: Augmented Cognition on the Social Web BIBAFull-Text 277-287
  Ching-man Au Yeung; Tomoharu Iwata
Social sharing on the Web has become very popular in recent years. However, as the amount of information grows rapidly it becomes difficult for a user to discover relevant information. The principle of augmented cognition can be applied to help users on the Social Web. This can be done by modelling the behaviours and interactions of the users in a system in order to discover implicit relations among the users. We describe two related approaches to model user behaviours for different types of social sharing sites. We show that the methods can be used to help users identify social relations that are more important to them, as well as items that are more relevant to their interests.
Brain Signatures of Team Performance BIBAKFull-Text 288-297
  Silke Dodel; Joseph Cohn; Jochen Mersmann; Phan Luu; Chris Forsythe; Viktor Jirsa
We report results from a dual electroencephalography (EEG) study, in which two-member teams performed a simulated combat scenario. Our aim was to distinguish expert from novice teams by their brain dynamics. Our findings suggest that dimensionality increases in the joint brain dynamics of the team members is a signature of increased task demand, both objective, e.g. increased task difficulty, and subjective, e.g. lack of experience in performing the task. Furthermore in each team we identified a subspace of joint brain dynamics related to team coordination. Our approach identifies signatures specific to team coordination by introducing surrogate team data as a baseline for joint brain dynamics without team coordination. This revealed that team coordination affects the subspace itself in which the joint brain dynamics of the team members are evolving, but not its dimensionality. Our results confirm the possibility to identify signatures of team coordination from the team members' brain dynamics.
Keywords: team; coordination; manifold; dimension; brain; dynamics; subspace; EEG
Team Coordination Dynamics and the Interactive Approach: Emerging Evidence and Future Work BIBAKFull-Text 298-307
  Jamie C. Gorman
In the study of coordination and teamwork, the primacy of team interaction is emphasized in an interactive approach. The interactive approach lies in stark contrast to the traditional, shared cognition approach to understanding team cognition. An overview of team coordination dynamics, an interactive approach rooted in nonlinear dynamics, is provided. Results from a series of experiments on team coordination dynamics are summarized. Finally, future research directions, inspired by those results, are considered.
Keywords: Nonlinear dynamics; Teams; Team coordination; Teamwork
Performance-Based Metrics for Evaluating Submarine Command Team Decision-Making BIBAKFull-Text 308-317
  Eric Jones; Ronald Steed; Frederick Diedrich; Robert Armbruster; Cullen Jackson
Successful submarine operations -- those that accomplish the mission while maintaining security and safety -- depend on numerous factors, including the capabilities of various sensor systems, the reliability of algorithms, and the proficiency of the crew. Among the most critical elements is Command Team decision-making and the underlying processes that create a cohesive and effective team. As a team, submarine commanders must successfully contend with complexities associated with safety and security as they build an understanding of the operational environment in order to accomplish their mission. Hence, opportunities to enhance training to support Command Team decision-making are essential. This paper describes a framework used to develop performance measures to support formative assessment of the submarine Command Team. Results are reported here from a study at the Naval Submarine School concerning the validity and utility of the measures in relation to capturing essential aspects of performance.
Keywords: performance measures; formative assessment; decision-making; teamwork; submarine
Multi-Modal Measurement Approach to Team Cohesion BIBAKFull-Text 318-324
  Camilla C. Knott; Alexandra Geyer; Jason Sidman; Emily Wiese
Team performance is a function, in part, of team cohesion: a dynamic process that is reflected in the tendency of a group to remain united in the pursuit of its goals and objectives (Carron 1982). We propose that a multi-modal measurement approach that integrates data from a variety of sources is critical to forming a comprehensive understanding of the relationship between team cohesion and performance, and can afford measurement of the hard-to-assess social component of team cohesion. Moreover, the use of a multi-modal measurement technique can afford flexibility in measuring across a variety of environments and selecting the most relevant measurement tools to minimize the technical footprint required for the assessment of teams and individuals in an operational environment.
Keywords: Team cohesion; multi-modal measurement; team performance
Communications-Based Automated Assessment of Team Cognitive Performance BIBAFull-Text 325-334
  Kiran Lakkaraju; Susan M. Stevens-Adams; Robert G. Abbott; Chris Forsythe
In this paper we performed analysis of speech communications in order to determine if we can differentiate between expert and novice teams based on communication patterns. Two pairs of experts and novices performed numerous test sessions on the E-2 Enhanced Deployable Readiness Trainer (EDRT) which is a medium-fidelity simulator of the Naval Flight Officer (NFO) stations positioned at bank end of the E-2 Hawkeye. Results indicate that experts and novices can be differentiated based on communication patterns. First, experts and novices differ significantly with regard to the frequency of utterances, with both expert teams making many fewer radio calls than both novice teams. Next, the semantic content of utterances was considered. Using both manual and automated speech-to-text conversion, the resulting text documents were compared. For 7 of 8 subjects, the two most similar subjects (using cosine-similarity of term vectors) were in the same category of expertise (novice/expert). This means that the semantic content of utterances by experts was more similar to other experts, than novices, and vice versa. Finally, using machine learning techniques we constructed a classifier that, given as input the text of the speech of a subject, could identify whether the individual was an expert or novice with a very low error rate. By looking at the parameters of the machine learning algorithm we were also able to identify terms that are strongly associated with novices and experts.
Visual Analytics of Social Networks: Mining and Visualizing Co-authorship Networks BIBAKFull-Text 335-345
  Carson Kai-Sang Leung; Christopher L. Carmichael; Eu Wern Teh
Co-authorship networks are examples of social networks, in which researchers are linked by their joint publications. Like many other instances of social networks, co-authorship networks contain rich sets of valuable data. In this paper, we propose a visual analytic tool, called SocialVis, to analyze and visualize these networks. In particular, SocialVis first applies frequent pattern mining to discover implicit, previously unknown and potential useful social information such as teams of multiple frequently collaborating researchers, their composition, and their collaboration frequency. SocialVis then uses a visual representation to present the mined social information so as to help users get a better understanding of the networks.
Keywords: Human-computer interaction; data mining; frequent patterns; social network analysis and mining; social computing; social information; data visualization; information and knowledge visualization; visualizing social interaction; augmented cognition
The Crowdsourcing Design Space BIBAKFull-Text 346-355
  Yasuaki Sakamoto; Yuko Tanaka; Lixiu Yu; Jeffrey V. Nickerson
Crowdsourcing is a new kind of organizational structure, one that is conducive to large amounts of short parallel work: thousands of individuals may work for several minutes on tasks, their outputs aggregated into a useful product or service. The dimensions of this new organizational form are described. Areas for future research are identified, focusing on open-ended tasks and the coordination structures that might foster collective creativity.
Keywords: Crowdsourcing; distributed cognition; organizational design; peer production; collective creativity; human computation
Developing Systems for the Rapid Modeling of Team Neurodynamics BIBAKFull-Text 356-365
  Ronald H. Stevens; Trysha Galloway; Chris Berka; Peter Wang
Cognitive Neurophysiologic synchronies (NS) are a low level data stream derived from EEG measurements that can be collected and analyzed in near real time and in realistic settings. We are using NS to develop systems that can rapidly determine the functional status of a team with the goals of being able to assess the quality of a teams' performance / decisions, and to adaptively rearrange the team or task components to better optimize the team. EEG-derived measures of engagement from Submarine Piloting and Navigation team members were normalized and pattern classified by self-organizing artificial neural networks and hidden Markov models. The temporal expression of these patterns were mapped onto team events and related to the frequency of team members' speech. Standardized models were created using pooled data from multiple teams and were used to compare NS expression across teams, training sessions and levels of expertise. These models have also been incorporated into software systems that can provide for rapid (minutes) after training feedback to the team and provide a framework for future real-time monitoring.
Keywords: Collaboration; EEG; Neurophysiologic synchrony
Mapping Cognitive Attractors onto the Dynamic Landscapes of Teamwork BIBAKFull-Text 366-375
  Ronald H. Stevens; Jamie C. Gorman
The objective of this study was to apply ideas from complexity theory to derive new models of teamwork. The measures include EEG-derived measures of Engagement and Workload obtained from submarine piloting and navigation (SPAN) teams and communication streams from Uninhibited Air Vehicle Synthetic Task Environments (UAV-STE). We show that despite large differences in the data streams and modeling, similar changes are seen in the respective order parameters in response to task perturbations and the experience of the team. These changes may provide a pathway for future adaptive training systems as both order parameters could conceivably be modeled and reported in real time.
Keywords: Complexity; Teamwork; EEG; Neurophysiologic synchrony; Nonlinear dynamics
Behavioral and Brain Dynamics of Team Coordination Part II: Neurobehavioral Performance BIBAKFull-Text 376-382
  Emmanuelle Tognoli; A. J. Kovacs; B. Suutari; Daniel Afergan; Joseph T. Coyne; G. Gibson; Roy Stripling; J. A. Scott Kelso
In this study, pairs of subjects performed a team-intensive task with the shared goal of clearing a virtual room from threats. The neurobehavioral dynamics of both subjects was analyzed to identify signatures of efficient team work. An ecologically valid task of room clearing was designed and a novel analysis framework was developed to address the challenge of understanding complex, continuous social processes at both behavioral and brain levels. A companion paper detailed the design of the neurobehavioral task and its associated dynamical analysis framework. In this paper, we present candidate neuromarkers for efficient room clearing and discuss key theoretical issues relating to successful team coordination.
Keywords: Neuromarkers; EEG; neurobehavioral dynamics; social behavior; complexity
Feature Selection in Crowd Creativity BIBAKFull-Text 383-392
  Lixiu Yu; Yasuaki Sakamoto
Crowdsourcing is emerging as a wellspring of creative designs. This paper examines the mechanisms that support collective design. A sequential combination system is described: one crowd generates designs, and another crowd combines these designs. Previous experiments showed that the combined designs were judged more creative than the initial designs. The current work extends this previous research by examining the combination process of the designs more closely, looking at how features of the designs were selected and integrated into later designs. Participants preferred atypical features to typical ones for integration, and given a choice, selected practical but less atypical features over impractical but more atypical features. We conclude that crowds attend to both novelty and practicality of the features, and that the presence of atypical yet practical features contributes to the increased creativity of the combined designs.
Keywords: Crowdsourcing; collective creativity; combination; feature selection

Augmented Cognition for Learning

Augmented Cognition Methods for Evaluating Serious Game Based Insider Cyber Threat Detection Training BIBAKFull-Text 395-403
  Terence S. Andre; Cali M. Fidopiastis; Tiffany R. Ripley; Anna L. Oskorus; Ryan E. Meyer; Robert A. Snyder
DoD investments into cyber threat defense are ongoing; however, little attention is paid to training personnel to detect and prevent threats to cyber networks that come from internal sources. Supervisors need to know what behavioral signs to watch for that might indicate an employee intends to commit an insider crime. Monitoring employee workstations is proving an ineffective means of determining insider threats. Training is needed to provide examples of the numerous ways cyber threats are achieved. An interactive role-play game environment may provide an appropriate instructional delivery system to train supervisors. Such a training system should employ instructional support features, aids, and feedback to the trainer and the trainee. The training system should also provide adaptive learning pathways to facilitate accelerated learning where individual assessments show mastery of specific content. Creating such a system not only requires appropriate training materials, but also a means to assess the systems efficacy. Augmented cognition methods and techniques for evaluating the cognitive state of a learner provide a real-time, objective means of evaluating training delivery and content. In this paper we discuss our efforts to assess learner engagement using psychophysiological measures.
Keywords: accelerated learning; adaptive training; learner engagement; psychophysical measures
Ongoing Efforts towards Developing a Physiologically Driven Training System BIBAKFull-Text 404-412
  Joseph T. Coyne; Ciara Sibley; Carryl Baldwin
There have been a number of successes of real-time application of physiological measures in operational environments such as with the control of remotely piloted vehicles (RPV). More recently, similar techniques have been investigated within the context of improving learning. A major challenge of the learning environment is that an individual's ability to perform the task, and thus their workload experienced during the task, are constantly changing. Cognitive Load Theory provides insight into how workload interacts with learning. One aspect of this theory is that as information is learned it reduces working memory demands. This paper discusses results from an RPV training study investigating the effects of workload and learning on pupil diameter. Specifically, pupil diameter decreased overtime as the task difficulty was held constant, and increased as new information was presented. The results of these studies are discussed in terms of how they can be used in a physiologically driven adaptive training system.
Keywords: Augmented Cognition; Pupil Diameter; Training; Workload
A Hierarchical Adaptation Framework for Adaptive Training Systems BIBAKFull-Text 413-421
  Sven Fuchs; Angela Carpenter; Meredith Carroll; Kelly S. Hale
Real-time adaptation is challenging in both operational and training environments, as the system must be able to identify what, why, and when mitigation is needed, and how best to mitigate to optimize the human-system interaction. Training systems have additional complexities, as the sole goal is not to optimize performance as in operational environments, but to optimize training, which may involve more error allowance for learning opportunities. This paper outlines a proposed hierarchical adaptation framework for adaptive training systems, involving diagnoses of learning state, performance, and expertise. It will also discuss candidate approaches to obtaining the necessary measurements using physiological and neurophysiological processes, provide some guidance for designing strategies for optimal adaptation, and highlight current challenges and future research areas.
Keywords: Adaptive Training; Augmented Cognition; Training Systems
Developing and Automating a Prototype for Assessing Levels of Student Involvement BIBAKFull-Text 422-431
  Curtis S. Ikehara; Martha E. Crosby
The proposed project objective is to develop and automate a methodological technology for objectively measuring a student's affective states, cognitive states and levels of involvement during computer-mediated instruction. Passive devices will record gaze activity, facial expressions and body motions while students are doing computer mediate instruction. From these measurements, a sensor fusion classification algorithm will be developed to provide an automated assessment of affective states, cognitive states and levels of involvement of the student. This automated assessment system will be validated using student interviews and rater observations. The system will provide detailed categorized information never before available to researchers. For the instructor, a large class could be equipped and assessed in real-time so that an instructor can appropriately focus attention to improve the learning environment or for student evaluation during instruction and for self-evaluation of instructional strategies after instruction.
Keywords: Cognition; real-time passive sensors; computer mediated instruction; gaze; body motion; facial expression; affective; student involvement
Considering Cognitive Traits of University Students with Dyslexia in the Context of a Learning Management System BIBAKFull-Text 432-441
  Carolina Mejía; Alicia Díaz; Juan E. Jiménez; Ramón Fabregat
This paper studies the cognitive processes involved in reading among Spanish-speaking university students with dyslexia, and proposes to evaluate these processes to identify specific cognitive traits. On this basis, an automated battery for the assessment of cognitive processes was designed to be included in a learning management system (LMS). To integrate this battery into the LMS, a web service architecture that works independently of the LMS was designed. The assessment battery has been built based on a multimodal communication mechanism that delivers evaluation tasks using the visual, auditory, and speech communication channels of human-computer interaction.
Keywords: Dyslexia; cognitive traits; user model; university students; multimodal communication
Improving Students' Meta-cognitive Skills within Intelligent Educational Systems: A Review BIBAKFull-Text 442-451
  Alejandro Peña Ayala; Michiko Kayashima; Riichiro Mizoguchi; Rafael Dominguez de Leon
Metacognition aims at monitoring and regulating one's thinking devoted to problem-solving processes and learning habits among others cognitive tasks. Hence, individuals engaged in better acquisition of domain knowledge achieve higher scores when they are bewaring of how to exploit their metacognitive faculties. Thus, we present a review of some models and methods with the purpose to understand what metacognition is and know how stimulate metacognitive skills. In addition, we propose a Metacognition-Driven Learning paradigm as a reference to guide the design of Intelligent Educational Systems oriented to improve students' metacognitive skills.
Keywords: Metacognition; metacognitive skills; metacognitive models; Metacognition-Driven Learning; Intelligent Educational Systems
Interactive Neuro-Educational Technologies (I-NET): Development of a Novel Platform for Neurogaming BIBAKFull-Text 452-461
  Giby Raphael; Adrienne Behneman; Veasna Tan; Nicholas Pojman; Chris Berka
The advances in sophisticated, immersive and highly engaging video gaming technology have resulted in the introduction of "serious gaming" as platforms for training. A virtual environment that mimics reality as closely as possible is an effective instructional medium and also serves as a performance improvement/evaluation platform. However, the current methodologies suffer from several limitations: 1) conventional qualitative evaluation techniques that are removed from the trainee's actual experience in both time and context 2) open loop platforms fail to support adaptive training and scenarios or leverage repeatability to accelerate training 3) failure to adapt to individual's current psychophysiological state, limiting skill acquisition rates 4) multi-person tasks that lack tools for objective assessment and prediction of team cohesion or performance. As part of our initiative to invent a suite of Interactive Neuro-Educational Technologies (I-NET), we have developed a Neurogaming platform that will help resolve many of these limitations.
Keywords: EEG; Neuroergonomics; Neurosensing; Augmented Cognition
Learning in Virtual Worlds: A New Path for Supporting Cognitive Impaired Children BIBAKFull-Text 462-471
  Laura Anna Ripamonti; Dario Maggiorini
We have adopted the serious game perspective to design, develop, and test a prototypal application, in a virtual world, aimed at teaching children affected by Down Syndrome how to read a clock. The main idea has been to offer them a new and intriguing learning environment to reduce the sense of frustration they often are burdened with during educational activities. In particular, an approach based on serious gaming has been coupled with the Feuerstein's method, which is currently spreading as an effective support to teaching activities aimed at impaired kids. The prototype has been developed adopting a playcentric process and has been tested with a group of children who were unable to read the time.
Keywords: serious games; videogames; healthcare; virtual world; usability; augmented cognition; Down Syndrome

Augmented Cognition and Interaction

A Longitudinal Study of P300 Brain-Computer Interface and Progression of Amyotrophic Lateral Sclerosis BIBAKFull-Text 475-483
  Nathan A. Gates; Christopher K. Hauser; Eric W. Sellers
BCI can provide communication for people locked in by amyotrophic lateral sclerosis (ALS). Empirical examination of how disease progression affects brain-computer interface (BCI) performance has not been investigated. This pilot study uses a longitudinal design to investigate changes in P300-BCI use as ALS disability increases. We aimed to (a) examine the relationship between BCI accuracy and the ALS/Functional Rating Scale and (b) examine changes in the event-related potential (ERP) components across time. Eight subjects have been enrolled in the study. BCI accuracy was measured and ERP components were assessed by a principal component analysis (PCA). Two subjects have been followed for an average of nine-months, and BCI accuracy is 99.6%. While many research obstacles remain, these preliminary data help elucidate the relationship between BCI performance and disease progression.
Keywords: Amyotrophic lateral sclerosis; electroencephalogram; brain-computer interface; P300 event-related potential; assistive communication
Discovering Context: Classifying Tweets through a Semantic Transform Based on Wikipedia BIBAKFull-Text 484-492
  Yegin Genc; Yasuaki Sakamoto; Jeffrey V. Nickerson
By mapping messages into a large context, we can compute the distances between them, and then classify them. We test this conjecture on Twitter messages: Messages are mapped onto their most similar Wikipedia pages, and the distances between pages are used as a proxy for the distances between messages. This technique yields more accurate classification of a set of Twitter messages than alternative techniques using string edit distance and latent semantic analysis.
Keywords: Text classification; Wikipedia; semantics; context; cognition; latent semantic analysis
Toward a Wearable, Neurally-Enhanced Augmented Reality System BIBAKFull-Text 493-499
  David H. Goldberg; R. Jacob Vogelstein; Diego A. Socolinsky; Lawrence B. Wolff
Augmented reality systems hold great promise, but as they become more complex they can become more challenging to use. Incorporating neural interfaces into augmented reality systems can dramatically increase usability and utility. We explore these issues in the context of Equinox Corporation's Night REAPER™ system-an augmented reality system for dismounted warfighters. We describe the current Night REAPER system and then survey some of the potential enhancements and unique design challenges associated with the addition of a neural interface. Signals, sensors, and decoding techniques for the system's brain-machine interface are discussed.
Keywords: augmented reality; brain-machine interface; wearable systems
Interface Design Challenge for Brain-Computer Interaction BIBAKFull-Text 500-506
  Jeremy Hill; Peter Brunner; Theresa M. Vaughan
Great things can be achieved even with very low bandwidth. Stephen Hawking has been able to break new ground in theoretical physics just by twitching his hand and cheek. Jean-Dominique Bauby was able to write a best-selling memoir by blinking one eyelid. By reading and decoding "brain-waves", the field of brain-computer interfacing (BCI) is poised to open up the possibility of such expression, even for people who can no longer move a single muscle. A BCI still requires an HCI front-end to be of practical use, but many currently-used HCIs do not adequately address limitations on the typical target user's input (e.g., limited eye movement leading to poor spatial vision) or output (e.g. variable delays, and false positives/negatives, in "pressing the button"). In this symposium, BCI experts will present their view of the challenges arising from these limitations. The HCI community is invited to participate in a competition to provide the best solutions.
Keywords: brain-computer interfacing (BCI); electroencephalography (EEG); human-computer interaction (HCI); human factors; spelling; augmentative and alternative communication (AAC); assistive technology; competition
Trust in Human-Computer Interactions as Measured by Frustration, Surprise, and Workload BIBAKFull-Text 507-516
  Leanne M. Hirshfield; Stuart H. Hirshfield; Samuel Hincks; Matthew Russell; Rachel Ward; Tom Williams
We describe preliminary research that attempts to quantify the level of trust that exists in typical interactions between human users and their computer systems. We describe the cognitive and emotional states that are correlated to trust, and we present preliminary experiments using functional near infrared spectroscopy (fNIRS) and electroencephalography (EEG) to measure these user states. Our long term goal is to run experiments that manipulate users' level of trust in their interactions with the computer and to measure these effects via non-invasive brain measurement.
Keywords: fNIRS; EEG; electroencephalograph; near-infrared spectroscopy; workload; frustration; surprise; trust
Idea Visibility, Information Diversity, and Idea Integration in Electronic Brainstorming BIBAKFull-Text 517-524
  Elahe Javadi; Wai-Tat Fu
Despite the pervasive use of electronic media for idea generation and idea sharing, the extent and quality of idea integration and use is relatively understudied. Idea integration and use depends on information saliency but little is known about how idea integration may be facilitated by user interface features that influence information saliency. This paper examines the effect of idea visibility on idea integration and how that relationship is moderated by information diversity. Our laboratory experiment showed that although the basic level of idea integration, i.e. mere reference to partners' ideas increased when visibility increased, higher levels of idea integration decreased as visibility increased. Information diversity was found to be a significant moderator of the relationship between visibility and idea integration.
Keywords: Idea integration; visibility; information diversity; brainstorming
The Challenges of Using Scalp-EEG Input Signals for Continuous Device Control BIBAFull-Text 525-527
  Garett D. Johnson; Nicholas Waytowich; Dean J. Krusienski
Whether aiming to control a computer cursor, a robotic arm, or a wheelchair, it remains a significant challenge to achieve responsive and reliable asynchronous control via EEG signals. The most promising scalp-recorded EEG signals for this task are sensorimotor rhythms and steady-state visual evoked potentials, which have both been demonstrated to be viable for continuous device operation in controlled laboratory settings. Several issues, such as handling signal nonstationarity and identifying reliable asynchronous modes of operation, must be addressed before these scalp-EEG signals can become practical for controlling devices outside of the laboratory.
Modeling Pharmacokinetics and Pharmacodynamics on a Mobile Device to Help Caffeine Users BIBAKFull-Text 528-535
  Frank E. Ritter; Kuo-Chuan (Martin) Yeh
We introduce a mobile device application that displays key information about caffeine: the pharmacokinetics (time course of drug levels) and pharmacodynamics (the effects of caffeine level) visually on the iPhone, iPod Touch, and iPad. This application, Caffeine Zone, is based on an existing model of caffeine physiology using user inputs, including caffeine dose, start time, and consumption speed. It calculates the caffeine load in a user for the next twenty-four hours and displays it using a line chart. In addition, it shows whether the user is currently in the "cognitive alert zone" (the range of caffeine where a normal person might benefit most from caffeine) or the "possible sleep zone" (the range of caffeine where sleep is presumed not affected by caffeine level.) Understanding the pharmacokinetics and pharmacodynamics of caffeine can help people using caffeine to improve alertness, including in operational environments. Caffeine Zone may also help users create a mental model of caffeine levels when the device is not available. We argue that this app will both teach users the complex absorption/elimination process of caffeine and help monitor users' daily caffeine usage. The model, with additional validation, can be part of a system that predict cognitive state of users and provide assistances in critical conditions.
Keywords: pharmacokinetics; pharmacodynamics; caffeine; mobile app; modeling
Designing Consumer Health Information Systems: What Do User-Generated Questions Tell Us? BIBAKFull-Text 536-545
  Yan Zhang; Wai-Tat Fu
Searching for health information has become a prevalent activity on the web. The information found online has a significant impact on people's decisions on whether to seek medical care and what treatments to undergo. However, existing studies consistently suggest that general consumers have various difficulties in formulating search queries using existing search engines and the queries were often not effective in retrieving personal- and situational-relevant information. Understanding users' information needs is a gateway to designing effective information retrieval (IR) systems. In this study, we examined the types of information requested by users, the characteristics of consumers' expressions of their information needs, and their expectations for results by analyzing the questions that general users posted on Yahoo! Answers, a popular social Q&A site. Based on the results, we proposed design recommendations for facilitating users' ability to articulate their health information needs and recommendations for the presentation of information in health-related IR systems.
Keywords: Consumer health informatics; information retrieval; social Q&A; health information searching

Augmented Cognition in Complex Environments

Estimation of Cognitive Workload during Simulated Air Traffic Control Using Optical Brain Imaging Sensors BIBAKFull-Text 549-558
  Hasan Ayaz; Ben Willems; Scott C. Bunce; Patricia A. Shewokis; Kurtulus Izzetoglu; Sehchang Hah; Atul Deshmukh; Banu Onaral
Deployment of portable neuroimaging technologies to operating settings could help assess cognitive states of personnel assigned to perform critical tasks and thus help improve efficiency and safety of human machine systems. Functional Near Infrared Spectroscopy (fNIR) is an emerging noninvasive brain imaging technology that relies on optical techniques to detect brain hemodynamics within the prefrontal cortex in response to sensory, motor, or cognitive activation. Collaborating with the FAA William J. Hughes Technical Center, fNIR has been used to monitor twenty four certified professional controllers as they manage realistic Air Traffic Control (ATC) scenarios under typical and emergent conditions. We have implemented a normalization procedure to estimate cognitive workload levels from fNIR signals during ATC by developing linear regression models that were informed by the respective participants' prior n-back data. This normalization can account for oxygenation variance due to inter-personal physiological differences. Results indicate that fNIR is sensitive task loads during ATC.
Keywords: Optical Brain Imaging; Air Traffic Control; Cognitive Workload; Functional Near Infrared Spectroscopy; fNIR
Distributed Logging and Synchronization of Physiological and Performance Measures to Support Adaptive Automation Strategies BIBAKFull-Text 559-566
  Daniel Barber; Irwin Hudson
As advances in physiological sensors make them more minimally intrusive and easier to use, there is a clear desire by researchers in the fields of Augmented Cognition and Neuroergonomics to incorporate them as much as possible. To best support use of multiple measures, the data from each sensor must be accurately synchronized across all devices and tied to performance and environment events. However, each sensor provides different sampling frequencies, local timing information, and timing accuracy making data synchronization in logs or real time systems difficult. In this paper, a modular architecture is presented to address the issue of how to synchronize data to support analysis of physiological and performance measures. Specific design requirements are presented to ensure the ability to accurately measure raw sensor data and compute metrics in a distributed computing environment to support adaptive automation strategies in a research environment. Finally, an example system is described which combines multiple minimally invasive physiological sensors.
Keywords: Adaptive Automation; Closed-Loop Training System; Data Synchronization
Augmenting Robot Behaviors Using Physiological Measures BIBAKFull-Text 567-572
  Daniel Barber; Lauren Reinerman-Jones; Stephanie J. Lackey; Irwin Hudson
In recent years, advancements in Unmanned Systems have allowed Human Robot Interaction (HRI) to transition from direct remote control to autonomous systems capable of self-navigation. However, these new technologies do not yet support true mixed-initiative solider-robot teaming where soldiers work with another agent as if it were another human being. In order to achieve this goal, researchers must explore new types of multi-modal and natural communication strategies and methods to provide robots improved understanding of their human counterparts' thought process. Physiological sensors are continuously becoming more portable and affordable leading to the possibility of providing new insight of team member state to a robot team member. However, steps need to be taken to improve how affective and cognitive states are measured and how these new metrics can be used to augment the decision making process for a robot team member. This paper describes current state of the art and next steps needed for accurate profile creation for improved human robot team performance.
Keywords: Multi-Modal Communion; Implicit Communication; Human Robot Interaction; Physiological Measures for State Measurement
Operational Neuroscience: Neuroscience Research and Tool Development to Support the Warfighter BIBAKFull-Text 573-577
  Monique E. Beaudoin; Dylan Schmorrow
This paper provides a summary of the presentations presented in the Operational Neuroscience session during Augmented Cognition International 2011 at Human Computer Interaction International 2011 in Orlando, Florida, July, 2011.
Keywords: Neuroscience; military operations; warfighter support; Cognitive readiness; Neurotechnology; hemodynamics; pharmacokinetics; brain bio-markers
Performance Measures to Enable Agent-Based Support in Demanding Circumstances BIBAFull-Text 578-587
  Fiemke Both; Mark Hoogendoorn; Rianne van Lambalgen; Rogier Oorburg; Michael de Vos
In this paper, an evaluation of measurements that can be used by a personal support agent to measure the quality of human task performance is addressed. Such measurements are important in order for a support agent to give effective and personalized support during the performance of demanding tasks. Hereby, the performance quality measurement is addressed from two perspectives, namely the human's perspective as well as the task perspective. The former represents the idea the human has about the current task performance, whereas the latter measures the actual task performance compared to the goals set for the task at hand. Criteria have been identified to compare the various measurements, and an experiment has been conducted for evaluation. Based on these evaluation results, the most useful measurements are identified to be adopted within personal support agents.
Cognitive Adaptive Man Machine Interfaces for the Firefighter Commander: Design Framework and Research Methodology BIBAKFull-Text 588-597
  Maurits de Graaf; Michel Varkevisser; Masja Kempen; Nicolas Jourden
The ARTEMIS CAMMI project aims at developing a joint-cognitive system to optimise human operator's performance under demanding labour conditions. The CAMMI domain applications concern avionics, automotive, and civil emergencies. In this paper we address the development of a joint-cognitive system for firefighter commanders to optimise situational and team awareness by reducing the workload through mitigation strategies and an adaptive HMI. A general framework and a research methodology are presented to explore the possibilities of applying the CAMMI building blocks in the development of systems to support the handling of firefighter emergencies.
Keywords: HMI; Situational Awareness; Team Awareness; Mental Load; Mitigation Strategies
An Intelligent Infrastructure for In-Flight Situation Awareness of Aviation Pilots BIBAKFull-Text 598-607
  Alessandro G. Di Nuovo; Rosario Bruno Cannavò; Santo Di Nuovo
This paper presents an infrastructure that integrates intelligent agents in order to monitor, in real time, the attention of aviation pilots during training/operative flight missions. The primary goal of this infrastructure is to make the decision process easier and increase Situation Awareness, thus to increase flight safety pro-actively. The proposed hardware/software platform could be able to anticipate the onset of problems which can lead to incidents, and to make easier the decision making process toward a positive solution of the problem. To attain the goal, a multi-agent system is designed using the most recent technology in the field of artificial vision and of the measurement of psychophysical parameters, starting from the most recent knowledge of visual attention to arrive at the development of an original and innovative model of Augmented Reality. Finally it is provided a case study based on an event actually occurred to prove effectiveness of the proposed platform.
Keywords: Situation Awareness; Intelligent Agents; Augmented Reality
Applications of Functional Near Infrared Imaging: Case Study on UAV Ground Controller BIBAKFull-Text 608-617
  Kurtulus Izzetoglu; Hasan Ayaz; Justin Menda; Meltem Izzetoglu; Anna C. Merzagora; Patricia A. Shewokis; Kambiz Pourrezaei; Banu Onaral
Functional Near-Infrared (fNIR) spectroscopy is an emerging optical brain imaging technology that enables assessment of brain activity through the intact skull in human subjects. fNIR systems developed during the last decade allow for a rapid, non-invasive method of measuring the brain activity of a subject while conducting tasks in realistic environments. This paper introduces underlying principles and various fNIR designs currently applied to real-time settings, such as monitoring Unmanned Aerial Vehicle (UAV) operator's expertise development and cognitive workload during simulated missions.
Keywords: Near-infrared spectroscopy; optical brain imaging; fNIR; human performance assessment
Augmented Phonocardiogram Acquisition and Analysis BIBAFull-Text 618-627
  Nancy E. Reed; Todd R. Reed
Heart auscultation (the interpretation of heart sounds by a physician) is a widely used screening method for heart disease. It is well documented, however, that with the exception of expert cardiologists, physicians' auscultation skills are limited. It has also been shown that standard training methods do little to improve these skills. In this paper, we propose an architecture for a phonocardiogram analysis system that can augment a physician's auscultation abilities and serve as a training aid to improve those abilities.
Today's Competitive Objective: Augmenting Human Performance BIBAKFull-Text 628-635
  Kay M. Stanney; Kelly S. Hale
Gaining competitive advantage requires acquiring or developing a capability that allows an organization or individual to outperform its competitors. In today's technology-driven environment, where human capabilities are struggling to keep up with technology offerings, techniques for augmenting human performance are becoming the critical gap that is precluding realizing the full benefits that these technology advances have to offer. The challenge is thus to develop tools and techniques that augment the human potential in order to best couple it to advancing complex interactive systems. In this void, those who are developing the capability to support real-time measurement, diagnosis, and augmentation of human performance may be the first to gain the competitive edge.
Keywords: Augmented cognition; Adaptive systems; human performance
Measuring the Effectiveness of Stress Prevention Programs in Military Personnel BIBAKFull-Text 636-646
  Andrea H. Taylor; Sae Lynne Schatz
The effects of stress on military personnel are a pervasive concern. To mitigate stress's negative impacts, Defense agencies employ stress inoculation training and, more recently, have begun to provide stress resilience instruction. However, such pre-deployment programs suffer from measurement limitations, rendering their assessment difficult. Novel application of objective, individual, repeated measures, conducted under realistically stressful settings, may help address this gap. Towards that end, we reviewed common neurophysiological techniques and examined their usefulness for measuring stress reactions. These techniques include: 1) cortisol in the blood or saliva, 2) adrenaline in the blood or urine, 3) skin conductivity, 4) EEG, 5) Skin conductance, and 6) Heart rate.
Keywords: Stress; Training; Resilience; Inoculation; Physiological Measurement
Adaptive Attention Allocation Support: Effects of System Conservativeness and Human Competence BIBAFull-Text 647-656
  Peter-Paul van Maanen; Teun Lucassen; Kees van Dongen
Naval tactical picture compilation is a task for which allocation of attention to the right information at the right time is crucial. Performance on this task can be improved if a support system assists the human operator. However, there is evidence that benefits of support systems are highly dependent upon the systems' tendency to support. This paper presents a study into the effects of different levels of support conservativeness (i.e., tendency to support) and human competence on performance and on the human's trust in the support system. Three types of support are distinguished: fixed, liberal and conservative support. In fixed support, the system calculates an estimated optimal decision and suggests this to the human. In the liberal and conservative support types, the system estimated the important information in the problem space in order to make a correct decision and directs the human's attention to this information. In liberal support, the system attempts to direct the human's attention using only the assessed task requirements, whereas in conservative support, the this attempt is done provided that it has been estimated that the human is not already paying attention (more conservative). Overall results do not confirm our hypothesis that adaptive conservative support leads to the best performances. Furthermore, especially high-competent humans showed more trust in a system when delivered support was adapted to their specific needs.
A Dynamic Approach to the Physiological-Based Assessment of Resilience to Stressful Conditions BIBAKFull-Text 657-666
  Mikhail Zotov; Chris Forsythe; Alexey Voyt; Inga Akhmedova; Vladimir Petrukovich
In the presented research, a new algorithm of detection and analysis of non-stationary phases (NSPh), characterizing sudden changes in heart rate variability (HRV) parameters was used. Physiological reactions of air traffic controllers during the performance of training scenario were estimated. 39 participants -- 14 experienced air traffic controllers and 25 students performed a 40-minute scenario, which included 3 stressful incidents: a rapid increase in air traffic density, low fuel level and plane engine failure. Students also performed the scenario after brief training. The results have shown that as expertise grows respondents show a significant decrease in duration and change in patterns of non-stationary phases of heart rate arising in response to the stressful incidents. These changes of parameters of non-stationary phases are connected with increased efficiency of air traffic controllers' cognitive performance in stressful conditions. The research has illustrated that the analysis of non-stationary phase parameters complements classical HRV measures and may be used for assessment of physiological responses of operators in Augmented Cognition applications.
Keywords: heart rate variability; cognitive workload; simulation-based training
Note: Best Paper Award