HCI Bibliography Home | HCI Conferences | FAC Archive | Detailed Records | RefWorks | EndNote | Hide Abstracts
FAC Tables of Contents: 070911131415

FAC 2013: 7th International Conference on Foundations of Augmented Cognition

Fullname:FAC 2013: 7th International Conference on Foundations of Augmented Cognition
Note:Volume 24 of HCI International 2013
Editors:Dylan D. Schmorrow; Cali M. Fidopiastis
Location:Las Vegas, Nevada
Dates:2013-Jul-21 to 2013-Jul-26
Publisher:Springer Berlin Heidelberg
Series:Lecture Notes in Computer Science 8027
Standard No:DOI: 10.1007/978-3-642-39454-6 hcibib: FAC13; ISBN: 978-3-642-39453-9 (print), 978-3-642-39454-6 (online)
Links:Online Proceedings | Conference Webpage
  1. Augmented Cognition in Training and Education
  2. Team Cognition
  3. Brain Activity Measurement
  4. Understanding and Modelling Cognition
  5. Cognitive Load, Stress and Fatigue
  6. Applications of Augmented Cognition

Augmented Cognition in Training and Education

Intuitive Sensemaking: From Theory to Simulation Based Training BIBAKFull-Text 3-10
  Kathleen Bartlett; Margaret Nolan; Andrea Marraffino
The concept of sensemaking has become a prominent component of military operations in ambiguous environments. Sensemaking, in general, describes the process of pattern recognition, semantic formulation, anticipation, and holistic understanding and supports sociocultural situation assessment, anomaly detection, and anticipatory thinking. This skill enables intuitive experts to rapidly draw accurate conclusions based on cues that others cannot discern or to attend to the most important cues, based on experience. Simulation-based training can enhance and accelerate the ability to recognize and analyze cues and patterns by translating the unconscious, automatic monitoring and integration practiced by experts into a conscious cognitive process that we call intuitive sensemaking. We describe an Office of Naval Research project, currently in development, intended to effectively train previously ambiguous advanced cognitive skills such as intuition-informed sensemaking. With training, teams of military personnel should see increases in cohesiveness, sociocultural situation assessment, anomaly detection, and anticipatory thinking.
Keywords: Sensemaking; Intuition; Simulation-Based Training; Human-Computer Interaction (HCI); Expertise; Implicit Learning
Using Simulation Based Training Methods for Improved Warfighter Decision Making BIBAKFull-Text 11-20
  Perakath Benjamin; Paul Koola; Kumar Akella; Michael Graul; Michael Painter
Few efforts have greater significance to our warfighting capability than those aimed at dramatically improving the skills, knowledge, and experience of military decision makers. The research and technology ideas presented in this paper are motivated by need to improve the quality of decision makers through the design of innovative training technology for military decision makers. This paper describes an adaptive simulation based training approach to improve the effectiveness of warfighter decision making. The paper describes (i) a method for adaptive simulation based training; (ii) a mission-driven approach to measure trainee performance based on carefully designed metrics; and (iii) an automation support architecture for adaptive simulation based training. Examples are provided throughout the paper to illustrate key research ideas.
Keywords: Simulation Based Training; Warfighter Decision Making; Adaptive Training; Mission Driven Performance Measurement
Enhancing HMD-Based F-35 Training through Integration of Eye Tracking and Electroencephalography Technology BIBAKFull-Text 21-30
  Meredith Carroll; Glenn Surpris; Shayna Strally; Matthew Archer; Frank Hannigan; Kelly Hale; Wink Bennett
The ever increasing complexity of knowledge, skills and abilities (KSAs) demanded of Department of Defense (DoD) personnel has created the need to develop tools to increase the efficiency and effectiveness of training. This is especially true for the F-35, the first 5th-generation aircraft to use an HMD as the primary instrument display. Additionally, the F-35 can perform operations previously performed by multiple operators, which potentially places incredible strain on the pilot's cognitive resources by exposing him to large amounts of data from disparate sources. It is critical to ensure training results in pilots learning optimal strategies for operating in this information rich environment. This paper discusses current efforts to develop and evaluate a performance monitoring and assessment system which integrates eye tracking and Electroencephalography (EEG) technology into an HMD enabled F-35 training environment to extend traditional behavioral metrics and better understand how a pilot interacts with data presented in the HMD.
Keywords: Training; Performance Assessment; Eye tracking; EEG; Helmet-mounted display; Heads-up display; F-35
Bio-reckoning: Perceptual User Interface Design for Military Training BIBAKFull-Text 31-40
  Tami Griffith; Deanna Rumble; Pankaj Mahajan; Cali M. Fidopiastis
Simulation based training is one way to attain operational realism for training complex military tasks in a safe, task relevant manner. For successful transfer of knowledge, skills, and abilities to the dynamically changing military environment, the human-computer interface should minimally support learning during the training process and provide congruent action plans that facilitate understanding of the overall training goal. While there are emerging controller technologies, simulators still rely on such input devices as mouse and keyboard. These devices potentially cause information and training bottlenecks as they limit naturalistic interactivity within the more advanced serious gaming platforms. Given the shortcomings of current interface design, we suggest a human-computer interface framework that includes perceptual user interface components and an open source serious game testbed. We discuss a multimodal framework called bio-reckoning that integrates brain-computer interface techniques, eye tracking, and facial recognition within EDGE, the U.S. Army's newest serious game based training tool.
Keywords: simulation based training; perceptual user interfaces; brain-computer interfaces; serious games; military training; augmented cognition
Taiwanese EFLs' Metacognitive Awareness of Reading Strategy and Reading Comprehension BIBAKFull-Text 41-49
  Yen-ju Hou
The study aims to identify the types of metacognitive awareness of reading strategies that Taiwanese EFLs (English as Foreign Language) used at medical junior colleges. In addition, metacognitive awareness of reading strategies were investigated to discover whether or not it affects students' English reading performance, specifically in reading comprehension. A total of 454 junior college students participated in the study. The results indicated that problem-solving reading strategies were used the most, followed by globe reading strategies, whereas support reading strategies were used the least. Regarding of the effects of variables on English reading performance, overall reading strategy use, and problem-solving reading strategies each significantly predicted students' reading comprehension. It's hoped that the finding could be helpful for further study as well as teaching.
Keywords: Metacognitive awareness; reading strategy; reading comprehension
Automated Camera Selection and Control for Better Training Support BIBAFull-Text 50-59
  Adrian Ilie; Greg Welch
Physical training ranges have been shown to be critical in helping trainees integrate previously-perfected skills. There is a growing need for streamlining the feedback participants receive after training. This need is being met by two related research efforts: approaches for automated camera selection and control, and computer vision-based approaches for automated extraction of relevant training feedback information.
   We introduce a framework for augmenting the capabilities present in training ranges that aims to help in both domains. Its main component is ASCENT (Automated Selection and Control for ENhanced Training), an automated camera selection and control approach for operators that also helps provide better training feedback to trainees.
   We have tested our camera control approach in simulated and laboratory settings, and are pursuing opportunities to deploy it at training ranges. In this paper we outline the elements of our framework and discuss its application for better training support.
A Hierarchical Behavior Analysis Approach for Automated Trainee Performance Evaluation in Training Ranges BIBAFull-Text 60-69
  Saad Khan; Hui Cheng; Rakesh Kumar
In this paper we present a closed loop mixed reality training system that provides automatic assessment of trainee performance during kinetic military exercises. At the core of our system is a hierarchical behavior analysis approach that integrates a number of data sensor modalities including Audio/Video, RFID and IMUs to automatically capture trainee actions in a comprehensive manner. Our behavior analysis and performance evaluation framework uses a finite state machine (FSM) model in which trainee behaviors are the states of the training scenario and the transitions of states are caused by stimuli that we refer to as trigger events. The goal of behavior analysis is to estimate the states of the trainees with respect to the training scenario and quantify trainee performance. To robustly detect each state, we build classifiers for each behavioral state and trigger event. At a given time, based on the state estimation, a set of related classifiers are activated for detecting trigger events and states that can be transitioned to and from the current states. The overall structure of the FSM and trigger events is determined by a Training Ontology that is specific to the training scenario.
Augmenting Instructional Design with State-Based Assessment BIBAKFull-Text 70-79
  Kevin Oden
The Trainee Engagement Management System (TEMS) is a technology-enabled instructional design concept that leverages state-based assessment techniques to improve training processes and outcomes. Specifically, the concept is designed to support military instructors in the delivery of empirically-supported instructional prompts to foster trainee engagement within a Computer Based Training (CBT) environment. The central theme of the concept is to augment, not replace, an instructor's abilities. By reducing workload demands on an instructor, the approach enables the delivery of personalized instruction in a one (instructor) to many (trainees) context. The TEMS concept embraces a human-system philosophy and is designed to mitigate risks typically associated with the transition of advanced technologies and concepts to field settings. In this paper we discuss those challenges and describe the basic TEMS architecture.
Keywords: Instructional System Design; Augmented Cognition; Human Systems; Computer Based Training
Instrumenting Competition-Based Exercises to Evaluate Cyber Defender Situation Awareness BIBAFull-Text 80-89
  Theodore Reed; Kevin Nauer; Austin Silva
Cyber defense exercises create simulated attack and defense scenarios used to train and evaluate incident responders. The most pervasive form of competition-based exercise is comprised of jeopardy-style challenges, which compliment a fictional cyber-security event. Multiple competitions were instrumented to collect usage statistics on a per-challenge basis. The competitions use researcher-developed challenges containing over twenty attack techniques, which generate forensic evidence and observable second-order effects. The following observations were made: (1) a group of defenders performs better than an individual; (2) situation awareness of the fictional event may be measured; (3) challenge complexity does not imply difficulty. This research introduces a novel application of system instrumentation on competition-based exercises and describes an exercise development methodology for effective challenge and competition creation. Effective challenges correctly represent difficulty and reward competitors with objective points and optional forensic clues. Effective competitions compliment training goals and appropriately improve the knowledge and skill of a competitor.
Enhanced Training for Cyber Situational Awareness BIBAKFull-Text 90-99
  Susan Stevens-Adams; Armida Carbajal; Austin Silva; Kevin Nauer; Benjamin Anderson; Theodore Reed; Chris Forsythe
A study was conducted in which participants received either tool-based or narrative-based training and then completed challenges associated with network security threats. Three teams were formed: (1) Tool-Based, for which each participant received tool-based training; (2) Narrative-Based, for which each participant received narrative-based training and (3) Combined, for which three participants received tool-based training and two received narrative-based training. Results showed that the Narrative-Based team recognized the spatial-temporal relationship between events and constructed a timeline that was a reasonable approximation of ground truth. In contrast, the Combined team produced a linear sequence of events that did not encompass the relationships between different adversaries. Finally, the Tool-Based team demonstrated little appreciation of either the spatial or temporal relationships between events. These findings suggest that participants receiving Narrative-Based training were able to use the software tools in a way that allowed them to gain a greater level of situation awareness.
Keywords: cyber security; training; situational awareness
Instrumenting a Perceptual Training Environment to Support Dynamic Tailoring BIBAKFull-Text 100-109
  Robert E. Wray; Jeremiah T. Folsom-Kovarik; Angela Woods
Simulation-based practice environments would be more valuable for learning if they supported adaptive, targeted responses to students as they proceed thru the experiences afforded by the environment. However, many adaptation strategies require a richer interpretation of the student's actions and attitudes than is available thru the typical simulation interface. Further, creating extended interfaces for a single application solely to support adaptation is often cost-prohibitive. In response, we are developing "learner instrumentation middleware" that seeks to provide a generalized representation of learner state via reusable algorithms, design patterns, and software.
Keywords: Perceptual learning; adaptive training; learner modeling

Team Cognition

Improving Tool Support for Software Reverse Engineering in a Security Context BIBAKFull-Text 113-122
  Brendan Cleary; Christoph Treude; Fernando Figueira Filho; Margaret-Anne Storey; Martin Salois
Illegal cyberspace activities are increasing rapidly and many software engineers are using reverse engineering methods to respond to attacks. The security-sensitive nature of these tasks, such as the understanding of malware or the decryption of encrypted content, brings unique challenges to reverse engineering: work has to be done offline, files can rarely be shared, time pressure is immense, and there is a lack of tool and process support for capturing and sharing the knowledge obtained while trying to understand assembly code. To help us gain an understanding of this reverse engineering work, we conducted an exploratory study at a government research and development organization to explore their work processes, tools, and artifacts [1]. We have been using these findings to improve visualization and collaboration features in assembly reverse engineering tools. In this talk, we will present a review of the findings from our study, and present prototypes we have developed to improve capturing and sharing knowledge while analyzing security concerns.
Keywords: malware; reverse engineering; empirical study
Brain Biomarkers of Neural Efficiency during Cognitive-Motor Performance: Performing under Pressure BIBAKFull-Text 123-132
  Michelle E. Costanzo; Bradley D. Hatfield
The concept of neural efficiency provides a powerful framework to assess the underlying mechanisms of brain dynamics during cognitive-motor performance. Electroencephalography (EEG) studies have revealed that as cognitive-motor performance improves non-essential brain processes are progressively disengaged resulting in brain dynamics leading to a state of neural efficiency. Multiple factors such as practice, genetics, mental stress, physical fitness and social interaction (team dynamics) can influence such cortical refinements positively or negatively and translate into an enhanced or deteriorated quality of performance. This paper provides a report of brain activity, assessed via fMRI, in a group of athletes who perform well under conditions of mental stress. Better understanding of brain states associated with such groups can enhance the ability to detect and classify adaptive mental states and increase the possibility of employing field-friendly brain monitoring tools such as EEG in ecologically valid situations for assessment of cognitive-motor performance in challenging real-world settings.
Keywords: Neural efficiency; expertise; fMRI; emotion regulation
The Geometry of Behavioral and Brain Dynamics in Team Coordination BIBAFull-Text 133-142
  Silke Dodel; Emmanuelle Tognoli; J. A. Scott Kelso
Performing a task as a team requires that team members mutually coordinate their actions. It is this coordination that distinguishes the performance of a team from the same actions performed independently. Here we set out to identify signatures of team coordination in behavioral and brain dynamics. We use dual electroencephalography (EEG) to measure brain dynamics of dyadic teams performing a virtual room clearing task. Such complex tasks often exhibit high variability of behavioral and brain dynamics. Although such variability is often considered to impede identification of the behavior or brain dynamics of interest here we present a conceptual and empirical framework which explains variability in geometrical terms and classifies its sources into those that are detrimental and non-detrimental to performing the task at hand. Using our framework we found that behaviorally team coordination is reflected in terms of role dependent behavior. Furthermore we identified a low-dimensional subspace of the brain dynamics in the frequency domain which is specific for team behavior and correlated with successful team coordination. Moreover, successful team coordination was positively correlated with the inter- but not intra-brain coherence in the gamma band. Our results hence indicate that successful team coordination is associated with increased team cognition, particularly readiness to engage in the task.
Analysis of Semantic Content and Its Relation to Team Neurophysiology during Submarine Crew Training BIBAKFull-Text 143-152
  Jamie C. Gorman; Melanie J. Martin; Terri A. Dunbar; Ronald H. Stevens; Trysha Galloway
A multi-level framework for analyzing team cognition based on team communication content and team neurophysiology is described. The semantic content of team communication in submarine training crews is quantified using Latent Semantic Analysis (LSA), and their team neurophysiology is quantified using the previously described neurophysiologic synchrony method. In the current study, we validate the LSA communication metrics by demonstrating their sensitivity to variations in training segment and by showing that less experienced (novice) crews can be differentiated from more experienced crews based on the semantic relatedness of their communications. Cross-correlations between an LSA metric and a team neurophysiology metric are explored to examine fluctuations in the lead-lag relationship between team communication and team neurophysiology as a function of training segment and level of team experience. Finally, the implications of this research for team training and assessment are considered.
Keywords: Latent Semantic Analysis; Team cognition; Team communication; Team neurophysiology; Teamwork
Neurophysiological Predictors of Team Performance BIBAKFull-Text 153-161
  Robin R. Johnson; Chris Berka; David Waldman; Pierre Balthazard; Nicola Pless; Thomas Maak
Objective: To identify benchmark neurophysiological measures that predict performance at a teaming level. Advanced Brain Monitoring has a track record of success in identifying neurophysiological metrics that impact expert behavior. For example, we characterized negative and positive predictors for marksmanship skill; persons with higher HF:LF Norm metrics of Heart rate variability (HRV, an indication of anxiety) during a benchmarking auditory passive vigilance task did not achieve expert marksman performance while those with above average visuospatial processing ability achieved greater levels of expertise. In the current research, we explored the ability of benchmark neurophysiological metrics to predict team performance in two large scale studies. Significance: Identifying neurophysiological metrics of teaming ability and performance as part of a team can provide potential screening mechanisms or developmental data to help build optimal teams and improve team interactions for different types of contexts in which teams may operate.
Keywords: leadership; neurophysiology; qEEG; prediction
How Long Is the Coastline of Teamwork? BIBAKFull-Text 162-171
  John Kolm; Ronald Stevens; Trysha Galloway
A five-state Markov model is proposed for group and team operation and evolution that has a stronger basis in neurodynamics, greater descriptive accuracy and higher predictive value than many existing models. The derivation of this model from the symbolic analysis of normalized EEG activity during assigned team and group tasks is discussed, as are observations on team and group dynamics which emerge from the model. The predictive value of the model is shown when applied to independent data from submarine crew evolutions. Observations are offered on team dynamics which show the five-state model and its accompanying state transitions to be necessary and sufficient to describe both linear and non-linear team dynamics, and to begin unifying these traditional and new approaches in a straightforward way.
Keywords: nonlinear dynamics; neurodynamics teamwork; markov model; state transition; EEG symbol; tuckman
Effects of Teamwork versus Group Work on Signal Detection in Cyber Defense Teams BIBAKFull-Text 172-180
  Prashanth Rajivan; Michael Champion; Nancy J. Cooke; Shree Jariwala; Genevieve Dube; Verica Buchanan
Cyber security is critical for any modern day organization's operations. Organizational structure and reward policies not conducive for teamwork may be affecting the performance of cyber defense analysts. Past research shows that team interaction could lead to better cyber defense performance. However, the value of team work in the cyber defense context has not been demonstrated using empirical methodologies. To explore this, we conducted a study on the effects of teamwork versus group work (i.e., looking at both the team and individual levels) on signal detection performance of cyber security defense analysts using the synthetic task environment called CyberCog. The results from the preliminary analysis conducted reveal that simply encouraging analysts to work as a team and providing team-level rewards leads to better team performance in cyber defense analysis.
Keywords: Cyber Defense Performance; Team Cognition; Team Performance; Synthetic Task Environment; CyberCog
Developing Methodology for Experimentation Using a Nuclear Power Plant Simulator BIBAFull-Text 181-188
  Lauren Reinerman-Jones; Svyatoslav Guznov; Joseph Mercado; Amy D'Agostino
Many of today's most complicated systems are human-machine systems that involve extensive advanced technology and a team of highly trained operators. As these human-machine systems are so complex, it is important to understand the factors that influence operator performance, operator state (e.g., overloaded, underload, stress) and the types of errors that operators make. Thus, it is desirable to develop an experimental methodology for studying complex systems that involve team operations. This paper looks at Nuclear Power Plant (NPP) operations as a test case for building this methodology. The methodology will reference some aspects/details specific to NPPs, but the general principles are intended to extend to any complex system that involves team operations.
Modeling Complex Tactical Team Dynamics in Observed Submarine Operations BIBAKFull-Text 189-198
  Tara Smallidge; Eric Jones; Jerry Lamb; Rachel Feyre; Ronald Steed; Abaigeal Caras
Successful submarine operations -- those that accomplish the mission while maintaining security and safety -- depend on numerous factors. Among the most critical elements driving success are the effectiveness of team behavior and the ability to understand when this behavior breaks down such that this degradation can be mitigated or avoided. While underway, submarine Commanders and other leaders must be attuned and alert to potential precursors that may manifest in decreased performance. This paper describes a framework used to develop performance measures to support formative assessment of team behaviors and to examine team breakdown and degradation. Results are reported from two events: an observation of an operational exercise and a study at the Naval Submarine School concerning the validity and utility of the measures. This preliminary research captured essential aspects of performance and helped define future efforts to develop better tools for assessing team behavior and understanding team breakdown in our warfighters.
Keywords: performance measures; formative assessment; team effectiveness; team breakdown; submarine
How Tasks Help Shape the Neurodynamic Rhythms and Organizations of Teams BIBAKFull-Text 199-208
  Ronald Stevens; Trysha Galloway; Gwendolyn Campbell; Chris Berka; Pierre Balthazard
We have modeled neurophysiologic indicators of Engagement and Workload to determine the influence the task has on the resulting neurodynamic rhythms and organizations of teams. The tasks included submarine piloting and navigation and anti-submarine warfare military simulations, map navigation tasks for high school students and business case discussions for entrepreneurial / corporate teams. The team composition varied from two to six persons and all teams had teamwork experience with the tasks. For each task condition teams developed task-specific neurodynamic rhythms. These task-specific rhythms were present during much of the task but could be interrupted by exogenous or endogenous disturbances to the team or environment. The effects of these disturbances could be rapidly detected by changes in the entropy levels of the team neurodynamics symbol streams. These results suggest the possibility of performing task-specific comparisons of the rhythms and organizations across teams expanding the opportunities for rapid detection of less than successful performances and targeted interventions.
Keywords: team neurodynamics; entropy; coordination dynamics; rhythms
Neurophysiological Estimation of Team Psychological Metrics BIBAKFull-Text 209-218
  Maja Stikic; Chris Berka; David Waldman; Pierre Balthazard; Nicola Pless; Thomas Maak
The goal of this study was to explore the feasibility of continuous neurophysiological assessment of different psychological aspects of a team process. The teams consisted of the MBA students who discussed and attempted to solve a case problem dealing with corporate social responsibility (i.e. child labor). At the end of the team process, two types of psychological metrics (i.e., engagement and leadership) were assessed by team members, both at the individual and team levels. These metrics showed significant correlations with the team performance scores derived by four trained coders. Two of them rated the teams' solutions in terms of effective problem solving, decisiveness, and creativity. The other two coders rated the level of moral reasoning displayed in the solutions. The psychological metrics were then estimated based on quantitative electroencephalography (qEEG). Different modeling techniques, such as linear and quadratic discriminant function analysis (DFA) and linear regression were applied to the processed qEEG data. The models were evaluated through auto-validation, but also through cross-validation to test stability of the models in the team-independent training setting. The experimental results suggested that qEEG could be effectively used in the team settings as an estimator of individual and team engagement, as well as the leadership qualities shown by team members. Our findings suggest that qEEG can help in understanding, and perhaps building, optimal teams and team processes.
Keywords: team process; engagement; leadership; electroencephalography
Physio-behavioral Coupling as an Index of Team Processes and Performance: Overview, Measurement, and Empirical Application BIBAKFull-Text 219-228
  Adam J. Strang; Gregory J. Funke; Sheldon M. Russell; Robin D. Thomas
Research shows that teammates often exhibit similarity in their physiological and behavioral responses during cooperative task performance, a phenomenon referred to here as physio-behavioral coupling (PBC). Goals of this manuscript are to provide an overview of research examining the utility of PBC as an index of team processes (e.g., coordination) and performance, discuss applied and theoretical issues in PBC measurement, and present findings from a study using linear and nonlinear statistics to assess PBC.
Keywords: Team; Coupling; Coordination; Performance; Nonlinear

Brain Activity Measurement

Combined Linear Regression and Quadratic Classification Approach for an EEG-Based Prediction of Driver Performance BIBAKFull-Text 231-240
  Gregory Apker; Brent Lance; Scott Kerick; Kaleb McDowell
Electroencephalography (EEG) has been used to reliably and non-invasively detect fatigue in drivers. In fact, linear relationships between EEG power-spectral estimates and indices of driver performance have been found during simplified driving tasks. Here we sought to predict driver performance using linear regression in a more complex paradigm. Driver performance varied widely between participants, often varying greatly within a single driving session. We found that a non-selective linear regression model did not generalize well between periods of stable and erratic driving, yielding large errors. However, prediction errors were significantly reduced by training a linear regression model on stable driving for each participant. To provide a confidence estimate for the stable driving model, a quadratic discriminate classifier was trained to detect the transition from stable to erratic driving from the EEG power-spectra. Combined, the regression model and classifier yielded significantly lower prediction errors and provided improved discrimination of poor driving.
Keywords: EEG; Regression; Driving; Fatigue; Power Spectral Density
Differential Prefrontal Response during Natural and Synthetic Speech Perception: An fNIR Based Neuroergonomics Study BIBAKFull-Text 241-249
  Hasan Ayaz; Paul Crawford; Adrian Curtin; Mashaal Syed; Banu Onaral; Willem M. Beltman; Patricia A. Shewokis
Synthetic speech has a growing role in human computer interaction and automated systems with the emergence of ubiquitous computing such as smart phones, car multimedia control and navigation systems. Cognitive processing costs associated with comprehension of synthetic speech relative to comprehension of natural speech have been demonstrated with behavioral (reaction time, accuracy, etc.) and self-reported (ratings, etc.) measures. In this neuroergonomics study, we have used optical brain imaging (fNIR: functional near infrared spectroscopy) to capture the brain activation of participants while they were listening to speech with varied quality, as well as natural speech. Results indicated a differential hemodynamic response with speech quality. As fNIR systems are safe, portable and record brain activation in real world settings, fNIR is a practical and minimally intrusive assessment tool for user experience researchers and can provide an objective metric for the design and development of next generation synthetic speech systems.
Keywords: Optical Brain Imaging; functional near infrared spectroscopy; fNIR; synthetic speech; perception; auditory processing
Functional Near-Infrared Spectroscopy in Addiction Treatment: Preliminary Evidence as a Biomarker of Treatment Response BIBAKFull-Text 250-258
  Scott C. Bunce; Jonathan Harris; Kurtulus Izzetoglu; Hasan Ayaz; Meltem Izzetoglu; Kambiz Pourrezaei; Banu Onaral
There is growing evidence that there are functional changes in the brains of individuals with substance use disorders. Numerous studies utilizing functional magnetic resonance imaging (fMRI) have shown that drug cues elicit increased regional blood flow in reward-related brain areas among addicted participants that is not found among normal controls. This finding has prompted leading investigators to suggest fMRI might be useful as a diagnostic or prognostic biomarker of addiction severity. However, fMRI is too costly for routine use in most treatment facilities. Functional near-infrared spectroscopy (fNIRs) offers an alternative neuroimaging modality that is safe, affordable, and patient-friendly. This manuscript reviews evidence that fNIRs can be used to differentiate prefrontal cortical responses of current alcohol dependent participants from alcohol dependent patients in treatment for 90-180 days. Differential responses to both alcohol and natural reward cues in both groups suggests fNIRs might serve as a clinic-friendly neuroimaging technology to inform clinical practice.
Keywords: Addiction; alcoholism; neuroimaging; functional near infrared spectroscopy; fNIRs; functional magnetic resonance imaging; fMRI; biomarker
Towards Noise-Enhanced Augmented Cognition BIBAKFull-Text 259-268
  Alexander J. Casson
Workload classification Augmented Cognition systems aim to detect when an operator is in a high or low workload state, and then to modify their work flow and operating environment based upon this knowledge. This paper reviews state-of-the-art electroencephalography (EEG) recorders for use in such systems and investigates the impact of EEG noise on an example system performance. It is found that adding up to 15 µVRMS of artificially generated noise still leaves EEG signals that have correlations in-line with the correlations found between conventional wet EEG electrodes and new dry electrodes. The workload classification system is found to be robust in the presence of small amounts of noise, and there is initial evidence of small stochastic resonance effects whereby better performance can actually be obtained in the noisy case compared to the traditional noise-less case.
Keywords: EEG; Augmented Cognition; Workload classification; Noise-enhanced signal processing
Soft, Embeddable, Dry EEG Sensors for Real World Applications BIBAKFull-Text 269-278
  Gene Davis; Catherine McConnell; Djordje Popovic; Chris Berka; Stephanie Korszen
Over the last decade, numerous papers have presented the use of dry electrodes capable of acquiring electroencephalogram (EEG) signals through hair. A few of these dry electrode prototypes have even progressed from lab-based EEG acquisition to commercial sales. While the field has improved rapidly as of late, most dry electrodes share a number of shortcomings that limit their potential real world applications including: 1) multiple rigid prongs that require sustained pressure to penetrate hair and maintain solid scalp contact, creating higher levels of discomfort when compared to standard wet sensors; 2) cumbersome or chin-strap-type applications for maintaining electrode contact, creating barriers to end user acceptance; 3) rigid active electrodes to compensate for high input impedances that limit flexibility and placement of sensors; 4) inability to safely imbed sensors under protective headgear, restricting use in some fields where EEG metrics are most desired; and 5) expensive sensor manufacturing that drives costs high for use across subjects. Under a recent DARPA Phase 3 contract, Advanced Brain Monitoring has developed a novel semi-dry sensor that addresses the current dry electrode shortcomings, opening up the door for new real world applications without compromising subject safety or comfort. The semi-dry sensor prototype was tested during a live performance requirement at the end of Phase 3, and successfully acquired EEG across all subject hair types over a 3 day testing period. The results from the performance requirement and subsequent results for new advancements to the prototype are presented here.
Keywords: Electroencephalograms (EEG); dry-electrodes; wearable EEG; BCI; Real World Applications
Real-Time Workload Assessment as a Foundation for Human Performance Augmentation BIBAKFull-Text 279-288
  Kevin Durkee; Alexandra Geyer; Scott Pappada; Andres Ortiz; Scott Galster
While current military systems are functionally capable of adaptively aiding human operators, the effectiveness of this capability depends on the availability of timely, reliable assessments of operator states to determine when and how to augment effectively. This paper describes a response to the technical challenges associated with establishing a foundation for reliable and effective adaptive aiding technologies. The central component of this approach is a real-time, model-based classifier and predictor of operator state on a continuous high resolution (0-100) scale. Using operator workload as a test case, our approach incorporates novel methods of integrating physiological, behavioral, and contextual factors for added precision and reliability. Preliminary research conducted in the Air Force Multi Attribute Task Battery (AF_MATB) illustrates the added value of contextual and behavioral data for physiological-derived workload estimates, as well as promising trends in the classification accuracy of our approach as the basis for employing adaptive aiding strategies.
Keywords: Workload; Augmentation; Human Performance; Modeling and Simulation; Physiological Measurement
Using the EEG Error Potential to Identify Interface Design Flaws BIBAKFull-Text 289-298
  Jeff Escalante; Serena Butcher; Mark R. Costa; Leanne M. Hirshfield
There are a number of limitations to existing usability testing methods, including surveys, interviews, talk-alouds, and participant observations. These limitations include subject bias, poor recall, and inability to capture fleeting events, such as when a UI functions or behaves in a manner that contradicts user expectations. One possible solution to these problems is to use electrophysiological indicators to monitor user interaction with the UI. We propose using event related potentials (ERP), and the error potential (ErrP) more specifically, to capture moment-to-moment interactions that lead to violations in user expectations. An ERP is a response generated in the brain to stimuli, while the ErrP is a more specific signal shown to be elicited by subject error. In this experiment we monitored subjects using a 10-channel electroencephalogram (EEG) as they completed a range of simple web browsing tasks. However, roughly 1/3 of the time subjects were confronted with poor UI design features (e.g., broken links). We then used statistical and machine learning techniques to classify the data and found that we were able to accurately identify the presence of error potentials. Furthermore, the ErrP was present when the subjects encountered a UI design flaw, but only during the more 'overt' examples of our design flaws. Results support our hypothesis that ERPs and ErrPs, can be used to identify UI design flaws for a variety of systems, from web sites to video games.
Keywords: EEG; usability testing; error potential
An Effective ERP Model for Brain Computer Interface BIBAKFull-Text 299-307
  Mariko Funada; Yoshihide Igarashi; Tadashi Funada; Miki Shibukawa
The investigation of BCI (Brain Computer Interface) is particularly interesting for HCI research. Some of recent results concerning BCI have much contributed to the progress of HCI. In this paper we propose an effective ERP model that can reduce the difference among individuals in the process of repetitive tasks. Human brain reactions are quantified by ERPs (Event Related Potentials) that reflect the change of brain reactions through repetitive tasks. We discuss a method of how to even out the difference appeared in ERPs among individuals.
Keywords: BCI; EEG; ERP; Individual difference; Model
Neural Oscillatory Signature of Original Problem Solving BIBAKFull-Text 308-315
  Henk J. Haarmann; Polly O'Rourke; Timothy George; Alexei Smaliy; Kristin Grunewald; Joseph Dien
The goal of the present research was to increase understanding of the neural oscillatory signature of originality in verbal divergent thinking by determining if event-related synchronization (ERS) in frequency bands other than alpha predicts originality. EEG was recorded while participants performed the insight task in which they were presented with a brief scenario and asked to generate as many explanations as possible during a three minute period. After the EEG session, participants were asked to rate the originality of each idea they produced. Analyses revealed that high originality was associated with decreases in the high beta ERS and with hemispheric asymmetry in the low beta band, immediately prior to idea generation. These results suggest the neural signature of originality extends beyond hemispheric asymmetries in the alpha band and provide important insights into the neural underpinnings of verbal creativity.
Keywords: Divergent thinking; originality; EEG; ERS; alpha; beta
A Real-World Neuroimaging System to Evaluate Stress BIBAKFull-Text 316-325
  Bret Kellihan; Tracy Jill Doty; W. David Hairston; Jonroy Canady; Keith W. Whitaker; Chin-Teng Lin; Tzyy-Ping Jung; Kaleb McDowell
While the laboratory setting offers researchers a great deal of experimental control, this environment also limits how generalizable the results are to the real world. This is particularly true when studying the multifaceted phenomenon of stress, which often relies on personal experience, a dimension that is difficult to reproduce in the laboratory setting. This paper describes a novel, multi-aspect real-world integrated neuroimaging system (MARIN) optimized to study physiological phenomena in the real-world and particularly suited to the study of stress. This system integrates neurological data from a gel-free, wireless EEG device with physiological data from wireless cardiac and skin conductance sensors, as well as self-reports of activity and stress. Coordination of the system is managed through an Android handheld mobile device that also logs salient events and presents inventories for subjective reports of stress. The integration of these components creates a rich, multimodal dataset with minimal interference to the user's daily life, and these data will guide the further understanding of neurological mechanisms of stress.
Keywords: wireless electroencephalography; skin conductance response; electrodermal activation; heart-rate variability; wearability
Optimal Feature Selection for Artifact Classification in EEG Time Series BIBAKFull-Text 326-334
  Vernon Lawhern; W. David Hairston; Kay Robbins
Identifying artifacts or non-brain electrical signals in EEG time series is often a necessary but time-consuming preprocessing step, as many EEG analysis techniques require that the data be artifact free. Because of this, reliable and accurate techniques for automated artifact detection are desirable in practice. Previous research has shown that coefficients obtained from autoregressive (AR) models can be used as feature vectors to classify among several different artifact conditions found in EEG. However, a statistical method for identifying significant AR features has not been presented. In this work we propose a method for determining the optimal AR features that is based on penalized multinomial regression. Our results indicate that the size of the feature vector can be greatly reduced with minimal loss to classification accuracy. The features selected by this algorithm localize to specific channels and suggests a possible BCI implementation with increased computational efficiency than with using all available channels. We also show that the significant AR features produced by this approach correlate to known brain physiological properties.
Keywords: Autoregressive (AR) model; Artifacts; Electroencephalography; classification; feature selection; multinomial regression; penalized regression; machine learning
Towards a Hybrid P300-Based BCI Using Simultaneous fNIR and EEG BIBAKFull-Text 335-344
  Yichuan Liu; Hasan Ayaz; Adrian Curtin; Banu Onaral; Patricia A. Shewokis
Next generation brain computer interfaces (BCI) are expected to provide robust and continuous control mechanism. In this study, we assessed integration of optical brain imaging (fNIR: functional near infrared spectroscopy) to a P300-BCI for improving BCI usability by monitoring cognitive workload and performance. fNIR is a safe and wearable neuroimaging modality that tracks cortical hemodynamics in response to sensory, motor, or cognitive activation. Eight volunteers participated in the study where simultaneous EEG and 16 optode fNIR from anterior prefrontal cortex were recorded while participants engaged with the P300-BCI for spatial navigation. The results showed a significant response in fNIR signals during high, medium and low performance indicating a positive correlation between prefrontal oxygenation changes and BCI performance. This preliminary study provided evidence that the performance of P300-BCI can be monitored by fNIR which in turn can help improve the robustness of the BCI classification.
Keywords: BCI; P300; fNIR; Performance; Optical brain imaging; EEG
A Novel Method for Single-Trial Classification in the Face of Temporal Variability BIBAKFull-Text 345-352
  Amar R. Marathe; Anthony J. Ries; Kaleb McDowell
Machine learning techniques have been used to classify patterns of neural data obtained from electroencephalography (EEG) to increase human-system performance. This classification approach works well in controlled laboratory settings since many of the machine learning techniques used often rely on consistent neural responses and behavioral performance over time. Moving to more dynamic, unconstrained environments, however, introduces temporal variability in the neural response resulting in sub-optimal classification performance. This study describes a novel classification method that accounts for temporal variability in the neural response to increase classification performance. Specifically, using sliding windows in hierarchical discriminant component analysis (HDCA), we demonstrate a decrease in classification error by over 50% when compared to other state-of-the-art classification methods.
Keywords: Brain-Computer Interface (BCI); Rapid Serial Visual Presentation (RSVP); Electroencephalography (EEG); HDCA; Sliding HDCA; Temporal Variability; Single-trial; Real-world environment
A Translational Approach to Neurotechnology Development BIBAKFull-Text 353-360
  Kaleb McDowell; Anthony Ries
The past several decades have seen an explosion of meaningful and nuanced insights into the connection between human behavior and the nervous system; however, the translation of these insights into viable applications is a non-trivial and widely acknowledged challenge. Recent advancements in brain-computer interaction and real-world neuroimaging technologies have provided major breakthroughs that provide the underpinnings for translational neuroscience research efforts. This session focuses on building off of those advancements and specifically proposes three concepts necessary for overcoming the challenges of translation: 1) integrating aspects of knowledge of brain function that are generally separate into single analyses, 2) increasing situational complexity, and 3) continuing to develop neuroimaging tools specifically for use in real-world environments.
Keywords: Translational Neuroscience; Neurotechnology; Brain-Computer Interface (BCI); Electroencephalography (EEG); Neural Classification
Understanding Brain Connectivity Patterns during Motor Performance under Social-Evaluative Competitive Pressure BIBAKFull-Text 361-370
  Hyuk Oh; Rodolphe J. Gentili; Michelle E. Costanzo; Ronald N. Goodman; Li-Chuan Lo; Jeremy C. Rietschel; Mark Saffer; Bradley D. Hatfield
Previous studies have shown that psychological arousal impacts motor performance during social-evaluative tasks by its influence on cortical dynamics, which can translate into motor performance enhancement. Although these findings have established critical links between performance under mental stress and elevated brain activity beyond that required for performance, there is still a need to further investigate brain connectivity during cognitive motor performance under such conditions. Here both electroencephalographic (EEG) and shooting performance were obtained in a shooting task under both performance-alone and competitive conditions. Network connectivity was assessed for the localized EEG sources. The results are consistent with those previously obtained and suggest elevated statistical dependencies and causal interactions between motor and non-motor areas during the competitive condition relative to performance-alone. Such network analysis provides a complementary approach to more traditional EEG derived metrics allowing for examining brain dynamics during cognitive motor performance under varying conditions of mental stress.
Keywords: Brain connectivity; EEG Localization; Motor Cognition; Competitive Pressure
Note: Best paper award
Removal of Ocular Artifacts from EEG Using Learned Templates BIBAKFull-Text 371-380
  Max Quinn; Santosh Mathan; Misha Pavel
Electroencephalogram (EEG) data can provide information on cognitive states and processes with high temporal resolution, but to take full advantage of this temporal resolution, common transients such as blinks and eye movements must be accounted for without censoring data. This can require additional hardware, large amounts of data, or manual inspection. In this paper we introduce a greedy, template-based method for modeling and removing transient activity. The method iteratively models an input and updates a template; a process which quickly converges to a unique and efficient approximation of the input. When combined with standard source separation techniques such as Independent Component Analysis (ICA) or Principal Component Analysis (PCA), the method shows promise for the automatic and data driven removal of ocular artifacts from EEG data. In this paper we outline our method, provide evidence for its effectiveness using synthetic EEG data, and demonstrate its effect on real EEG data recorded as part of a minimally constrained cognitive task.
Keywords: EEG; EOG; ICA; PCA; BCI; matching pursuit
Brain in the Loop Learning Using Functional Near Infrared Spectroscopy BIBAKFull-Text 381-389
  Patricia A. Shewokis; Hasan Ayaz; Adrian Curtin; Kurtulus Izzetoglu; Banu Onaral
The role of practice is crucial in the skill acquisition process and for assessments of learning. In this study, we used a portable neuroimaging technique, functional near infrared (fNIR) spectroscopy for monitoring prefrontal cortex activation during learning of spatial navigation tasks throughout 11 days of training and testing. Two different tasks orders, blocked and random, were used to test the effect of the practice schedule on the acquisition and transfer of 3D computer mazes. Results indicated variable decreases in the hemodynamic response during the initial days of practice. Although there were no differences in mean oxygenation for the practice orders across acquisition the random practice order used less oxygenation than the blocked order for the more difficult tasks in the transfer phase Use of brain activation and behavioral measures provides can provide a more accurate depiction of the learning process. Since fNIR systems are safe, portable and record brain activation in ecologically valid settings, fNIR can contribute to future learning settings for assessment and personalization of the training regimen.
Keywords: Optical Brain Imaging; functional near infrared spectroscopy; fNIR; Learning; Spatial navigation; contextual interference
Brain Activity Based Assessment (BABA) BIBAKFull-Text 390-398
  Roy Stripling; Grace Chang
Event-Related Potentials (ERP) are changes in brain activity detected using electroencephalographic (EEG) methods. One well-studied ERP is the P3b, which is generally elicited by asking participants to press a key when presented a target stimulus (e.g., "T") that is intermixed with a much more commonly presented non-target stimulus (e.g., "S"). We hypothesized that we could assess knowledge by asking participants to solve a problem then press a key when they see the correct answer in a series of (mostly wrong) answers. Early pilot testing (four participants) suggests that the P3b shows promise in this regard. In a math test, P3b responses were produced when shown correct, but not incorrect answers. In a foreign-language vocabulary test (matching picture to foreign word), P3b responses were not produced when shown correct answers prior to studying the words, but did produce P3b responses after studying. Some notable deviations in individual participants are discussed.
Keywords: Evoked Potential; Electroencephalogram; EEG; Knowledge Assessment

Understanding and Modelling Cognition

Enhancing Intuitive Decision Making through Implicit Learning BIBAKFull-Text 401-409
  Joseph Cohn; Peter Squire; Ivy Estabrooke; Elizabeth O'Neill
Today's military missions pose complex time-constrained challenges, such as detecting IED emplacements while in a moving vehicle or detecting anomalous civilian behaviors indicative of impending danger. These challenges are compounded by recent doctrinal requirements that require younger and less-experienced Warfighters to make ever-more complex decisions. Current understanding of decision making, which is based on concepts developed around theories of analytic decision making (Newell and Simon, 1972), cannot effectively address these new challenges since they are based on the notion of enabling experts to apply their expertise to addressing new problems. Yet, there are actually two types of recognized decision making processes, analytical and intuitive, which appear to be mediated by different processes or systems (Ross et al, 2004; Evans, 2008; Kahneman & Klein, 2009). Analytical decision making is mediated by processes that reflect a sequential, step-by-step, methodical, and time-consuming process. To be effective, analytic decision making appears to require domain expertise. In contrast, intuitive decision making relies upon a more holistic approach to processing information at a subconscious level (Luu et al, 2010). The thesis of this paper is that unlike analytic decision making, effective intuitive decision making does not require domain expertise but, rather, can be enhanced through training methods and technologies. This paper will explore ways in which the results from a range of studies at the behavioral, cognitive and neurophysiological levels can be leveraged to provide a comprehensive approach to understanding and enabling more effective intuitive decision-making for these non-experts.
Keywords: Cognitive Modeling; Perception; Emotion and Interaction; Intuition Decision Making; Implicit Learning
Measuring Engagement to Stimulate Critical Thinking BIBAKFull-Text 410-417
  Patricia J. Donohue; Tawnya Gray; Dominic Lamboy
This research is a theoretical study of game-augmented instruction for learning and playing mathematics challenges. We wanted to extend our work with a unique Studio-Based Learning (SBL) model for peer-critiques of project designs. SBL had been used successfully in 15 universities as an approach for helping undergraduate computer science students improve their programming skills and code reviews. We piloted the model in a 9th-grade spatial studies class with some success in teaching freshmen how to critique their work and participate in peer reviews across teams. From those experiences we developed a framework for an interactive mobile application of the studio experience. Research with a group of student athletes revealed that before mobile development, we needed to consider the constraints of learner characteristics on the mobile environment. This study sets out the design for a pilot test of our finding that learning style may drive game features for instruction.
Keywords: Mobile Learning; Mathematics; Physiological Measurement; Engagement; Physical Cognition; Game Theory
Human Dimension in Cyber Operations Research and Development Priorities BIBAFull-Text 418-422
  Chris Forsythe; Austin Silva; Susan Stevens-Adams; Jeffrey Bradshaw
Within cyber security, the human element represents one of the greatest untapped opportunities for increasing the effectiveness of network defenses. However, there has been little research to understand the human dimension in cyber operations. To better understand the needs and priorities for research and development to address these issues, a workshop was conducted August 28-29, 2012 in Washington DC. A synthesis was developed that captured the key issues and associated research questions.
   Research and development needs were identified that fell into three parallel paths: (1) human factors analysis and scientific studies to establish foundational knowledge concerning factors underlying the performance of cyber defenders; (2) development of models that capture key processes that mediate interactions between defenders, users, adversaries and the public; and (3) development of a multi-purpose test environment for conducting controlled experiments that enables systems and human performance measurement.
Integration of Psycognitive States to Broaden Augmented Cognition Frameworks BIBAKFull-Text 423-432
  Karmen Guevara
Augmented Cognition technologies focus on assessing and monitoring the user to produce a composite picture of their cognitive state. This is based on the mental processes of the user involving perception, memory, judgment and reasoning. It does not include the emotional, volitional or the subconscious processes. Due to the absence of input from a core human dimension -- the subconscious -- it is inevitable that the picture to emerge from this data will be incomplete. The focus of this paper therefore, is on this subconscious dimension. The objective is to illustrate how subconscious processes can shape behaviours and determine individuals' strategic actions. We argue that in order to formulate a complete portrait of an individual's cognitive state, it is important to integrate the subconscious dimension.
Keywords: Psycognition; characterology; subconscious behaviours; character strategies; critical incident breakdown; situational appropriate behaviour; inner subjective domain
Human Performance Assessment Study in Aviation Using Functional Near Infrared Spectroscopy BIBAKFull-Text 433-442
  Joshua Harrison; Kurtulus Izzetoglu; Hasan Ayaz; Ben Willems; Sehchang Hah; Hyun Woo; Patricia A. Shewokis; Scott C. Bunce; Banu Onaral
Functional near infrared (fNIR) spectroscopy is a field-deployable optical neuroimaging technology that provides a measure of the prefrontal cortex's cerebral hemodynamics in response to the completion of sensory, motor, or cognitive tasks. Technologies such as fNIR could provide additional performance metrics directly from brain-based measures to assess safety and performance of operators in high-risk fields. This paper reports a case study utilizing a continuous wave fNIR technology deployed in a real-time air traffic control (ATC) setting to evaluate the cognitive workload of certified professional controllers (CPCs) during the deployment of one of the Federal Aviation Administration's (FAA's) Next Generation (NextGen) technologies.
Keywords: Near-infrared spectroscopy; optical brain imaging; fNIR; human performance assessment; air traffic control; workload
Robust Classification in RSVP Keyboard BIBAKFull-Text 443-449
  Matt Higger; Murat Akcakaya; Umut Orhan; Deniz Erdogmus
To use in the Rapid Serial Visual Presentation (RSVP) Keyboard™, a brain computer interface (BCI) typing system developed by our group, we propose a robust classification method of handling non-stationarity in the electroencephelography (EEG) data that is caused by artifacts and/or sensor failure. Considering the effect of these non-stationarities, we build a mixture data model to use as EEG evidence in the fusion with an n-gram language model to develop a robust classification algorithm. Using Monte Carlo simulations on the pre-recorded EEG data containing sections with or without intentionally generated artifacts we compare the typing performances of non-robust and robust classification methods in terms of speed and accuracy.
Keywords: BCI; ERP; Spelling
Real-Time Vigilance Estimation Using Mobile Wireless Mindo EEG Device with Spring-Loaded Sensors BIBAKFull-Text 450-458
  Li-Wei Ko; Chun-Hsiang Chuang; Chih-Sheng Huang; Yen-Hsuan Chen; Shao-Wei Lu; Lun-De Liao; Wan-Ting Chang; Chin-Teng Lin
Monitoring the neurophysiological activities of human brain dynamics in an operational environment poses a severe measurement challenge using current laboratory-oriented biosensor technology. The goal of this research is to design, develop and test the wearable and wireless dry-electrode EEG human-computer interface (HCI) that can allow assessment of brain activities of participants actively performing ordinary tasks in natural body positions and situations within a real operational environment. Its implications in HCI were demonstrated through a sample application: vigilance-state prediction of participants performing a realistic sustained-attention driving task. Besides, this study further developed an online signal processing for extracting EEG features and assessing cognitive performance. We demonstrated the feasibility of using dry EEG sensors and miniaturized supporting hardware/software to continuously collect EEG data recorded from hairy sites (i.e., occipital region) in a realistic VR-based dynamic driving simulator.
Keywords: Drowsy driving; Wireless and dry EEG device; Mindo; Human-computer interface
Relationship Analysis between Subjective Evaluation and NIRS-Based Index on Video Content BIBAKFull-Text 459-466
  Shinsuke Mitsui; Atsushi Maki; Toshikazu Kato
Brain activities have been investigated, and various functions of brain have been revealed recently. In our experiment, decrease of oxy-Hb change at frontal cortex was observed while subjects were watching video contents. Also, the degrees of decreases were different among the subjective evaluations about impression against the video contents. Revealing the cause of the decrease has the possibility to evaluate video content objectively. In this paper we discuss the relationship between subjective evaluation and brain activity on video content.
Keywords: Frontal cortex; Near-infrared spectroscopy; Subjective evaluation; Video content
Towards Evaluating Computational Models of Intuitive Decision Making with fMRI Data BIBAKFull-Text 467-473
  James Niehaus; Victoria Romero; Avi Pfeffer
A vast array of everyday tasks require individuals to use intuition to make decisions and act effectively, including civilian and military professional tasks such as those undertaken by firefighters, police, search and rescue, small unit leaders, and information analysts. To better understand and train intuitive decision making (IDM), we envision future training systems will represent IDM through computational models and use these models to guide IDM learning. This paper presents the first steps to the problem of validating computational models of IDM. To test if these models correlate with human performance, we examine methods to analyze functional magnetic resonance imaging (fMRI) data of human participants performing intuitive tasks. In particular, we examine the use of a new deep learning representation called sum-product networks to perform model-based fMRI analysis. Sum-product networks have been shown to be simpler, faster, and more effective than previous deep learning approaches, making them ideal candidates for this computationally demanding analysis.
Keywords: intuition; intuitive decision making; deep learning; sum-product network; functional magnetic resonance imaging; model-based fMRI
Human Memory Systems: A Framework for Understanding the Neurocognitive Foundations of Intuition BIBAKFull-Text 474-483
  Paul J. Reber; Mark Beeman; Ken A. Paller
A neurocomputational framework is described for characterizing how intuitive and deliberate processing are accomplished in the human brain. The framework is derived from memory systems theory and supported by research findings on contrasts between implicit versus explicit (nonconscious versus conscious) memory. Implicit intuition and deliberate deduction depend on separate types of memory supported by distinct brain networks. For optimal decision making, training should be designed to accommodate the operating characteristics of both types of memory. Furthermore, reliance on explicit memory can inhibit the use of implicit intuition, so training must facilitate effective interactions between the two types of mechanism. To aid investigations of these effects, we introduce a Mixture-of-Experts model that characterizes the interaction between memory systems -- the PINNACLE model (Parallel Interacting Neural Networks Competing in Learning). This model captures the separate neural networks that reflect implicit and explicit processing, as well as their interaction, and it can thus guide the development of training approaches to maximize the benefits of concurrent use of both intuition and deliberation in decision making.
Keywords: Intuition; decision making; implicit; explicit; memory systems; cognitive neuroscience; cognitive modeling
Modeling Cues for Intuitive Sensemaking Simulations BIBAKFull-Text 484-491
  Sae Schatz; Kathleen Bartlett
Modern military personnel must not only possess typical warfighting abilities; they must also be able to rapidly perceive, understand, and then respond to a range of ambiguous behavioral, social, and cultural stimuli. In other words, personnel must have sociocultural sensemaking skills -- preferably intuitive sensemaking skills that allow them to act with the utmost agility. This paper begins by discussing sensemaking, sociocultural pattern recognition, and expertise-based intuition. It briefly describes training approaches for these constructs, as well as training for the integrated concept. Instructional simulations could facilitate such training. However, for simulations to effectively support this subject matter, they must be able to replicate realistic patterns of life, from the subtle characteristics of human body language to the emergent behaviors of crowds. That is, they must provide accurate, nuanced cues to which the trainees can react. This paper closes by discussing our ongoing work to address this gap by modeling realistic cues in a simulation.
Keywords: sensemaking; intuition; patterns of life; simulation; military training; cognitive readiness
Evaluating Classifiers for Emotion Recognition Using EEG BIBAFull-Text 492-501
  Ahmad Tauseef Sohaib; Shahnawaz Qureshi; Johan Hagelbäck; Olle Hilborn; Petar Jercic
There are several ways of recording psychophysiology data from humans, for example Galvanic Skin Response (GSR), Electromyography (EMG), Electrocardiogram (ECG) and Electroencephalography (EEG). In this paper we focus on emotion detection using EEG. Various machine learning techniques can be used on the recorded EEG data to classify emotional states. K-Nearest Neighbor (KNN), Bayesian Network (BN), Artificial Neural Network (ANN) and Support Vector Machine (SVM) are some machine learning techniques that previously have been used to classify EEG data in various experiments. Five different machine learning techniques were evaluated in this paper, classifying EEG data associated with specific affective/emotional states. The emotions were elicited in the subjects using pictures from the International Affective Picture System (IAPS) database. The raw EEG data were processed to remove artifacts and a number of features were selected as input to the classifiers. The results showed that it is difficult to train a classifier to be accurate over large datasets (15 subjects) but KNN and SVM with the proposed features were reasonably accurate over smaller datasets (5 subjects) identifying the emotional states with an accuracy up to 77.78%.
From Explicit to Implicit Speech Recognition BIBAFull-Text 502-511
  Chad M. Spooner; Erik Viirre; Bradley Chase
We consider the problem of determining the word or concept that a subject holds in their mind prior to the act of speech using only a scalp-recorded electroencephalogram (EEG). Such speech acts are called covert, silent, or implicit speech acts in the literature. We consider a binary-tree classifier that uses one of a number of candidate feature types, including temporal correlation coefficients, spectral correlation, and time-gated raw voltages. The particular features and binary-tree parameters are blindly determined using the local discriminant basis (LDB) technique. The experiments involve sequential presentation of words and numbers on a computer screen. The subject wears an EEG scalp cap and is instructed to first consider the stimulus, then speak it. Later, the subject is instructed to perform the same task without the actual utterance, resulting in implicit speech. We present performance results for the various obtained classifiers, which show that the approach has significant merit.
Cognitive-Affective Interactions in Strategic Decision Making BIBAKFull-Text 512-520
  Yanlong Sun; Hongbin Wang
While making a decision to maximize the expected utility is among the prime examples of human intelligence, the ultimatum game showcases a social dilemma where people sacrifice their economic self-interest in the presence of negative emotions. In the present study, we explore human cognitive-affective interactions in strategic thinking from an integrated neurocomputational perspective. We manipulated participants' emotions by inducing incidental affective states in the ultimatum game. We found that participants' rejection rates of unfair offers were significantly lower in positive valence emotions ("happy" and "calm") than in negative valence emotions ("sad" and "anxious"). In addition, the reduction of rejection rates appeared to be independent of the arousal level (high arousal in "happy" and "anxious" versus low arousal in "calm" and "sad"). Our results suggested that positive valence emotions, by broadening people's evaluations of decision perspectives and alleviating the perception of unfairness, may help people regain focus on their economic self-interest.
Keywords: Decision making; social dilemma; ultimatum game; affective induction; fairness preference; valence; arousal
Translation of EEG-Based Performance Prediction Models to Rapid Serial Visual Presentation Tasks BIBAKFull-Text 521-530
  Jon Touryan; Gregory Apker; Scott Kerick; Brent Lance; Anthony J. Ries; Kaleb McDowell
Brain wave activity is known to correlate with decrements in behavior brought on by fatigue, boredom or low levels of alertness. Being able to predict these behavioral changes from the neural activity via electroencephalography (EEG) is an area of ongoing interest. In this study we used an established approach to predict time-on-task decrements in behavior for both a realistic driving simulator and a difficult perceptual discrimination task, utilized in many brain-computer interface applications. The goal was to quantify how well EEG-based models of behavior, developed for a driving paradigm, extend to this non-driving task. Similar to previous studies, we were able to predict time-on-task behavioral effects from the EEG power spectrum for a number of participants in both the driving and perception tasks.
Keywords: EEG; Fatigue; Power Spectral Density; Driving; RSVP
Adult Neurogenesis: Implications on Human And Computational Decision Making BIBAKFull-Text 531-540
  Craig M. Vineyard; Stephen J. Verzi; Thomas P. Caudell; Michael L. Bernard; James B. Aimone
Adult neurogenesis is the incorporation of new neurons into established, functioning neural circuits. Current theoretical work in the neurogenesis field has suggested that new neurons are of greatest importance in the encoding of new memories, particularly in the ability to fully capture features which are entirely novel or being experienced in a unique way. We present two models of neurogenesis (a spiking, biologically realistic model as well as a basic growing feedforward model) to investigate possible functional implications. We use an information theoretic computational complexity measure to quantitatively analyze the information content encoded with and without neurogenesis in our spiking model. And neural encoding capacity (as a function of neuron maturation) is examined in our simple feedforward network. Finally, we discuss potential functional implications for neurogenesis in high risk environments.
Keywords: Neurogenesis; Dentate Gyrus; Information Theoretic Complexity; Neural Network Modeling
The Effects of Spatial Attention on Face Processing: An ERPs Study BIBAKFull-Text 541-550
  Liang Zhang; Kan Zhang
Faces are one of the most biologically and socially significant objects in the human environment, and therefore believed to be processed automatically. To investigate whether the processing of faces is modulated by attention, event-related potentials (ERPs) were recorded in response to the dynamic facial stimuli. Spatial attention were manipulated by directing participants which location to be attended. The results showed that face-sensitive component N170 was not influenced by spatial attention, suggesting that the processing of faces was not modulated by the attention during the early stage. But the late-latency components were influenced by spatial attention. It indicated that the automatic processing of faces is more like to be partial rather than complete. These findings on dynamically real face processing by ERPs are expected to be used in the development of human-computer interactions.
Keywords: Face processing; Spatial attention; ERPs

Cognitive Load, Stress and Fatigue

The Information Exoskeleton: Augmenting Human Interaction with Information Systems BIBAKFull-Text 553-561
  James P. Allen; Susan Harkness Regli; Kathleen M. Stibler; Patrick Craven; Peter Gerken; Patrice D. Tremoulet
In the military intelligence cycle the warfighter acts as both a receiver and a producer of information. As a receiver the warfighter must be able to readily assimilate disparate mission-relevant information. As a producer the warfighter must be cognizant of both the current information requirements and the ability to meet them. Both of these tasks are exacerbated by the heat of battle and, in the case of the receiver, the ever-increasing amount of available information. To address these challenges Lockheed Martin Advanced Technology Laboratories (LM ATL) is creating a suite of capabilities to augment warfighter interaction with intelligence services. Much like a powered exoskeleton augments human interaction with the physical environment, our Information Exoskeleton augments the warfighter's interaction with intelligence, providing greater situational awareness with minimal operational overhead. This paper describes our vision for the Information Exoskeleton, the capabilities required to realize it, and related research efforts.
Keywords: Information Exoskeleton; Information Needs Assessment; Context Awareness; Information Alignment; Cognitive Alignment
QEEG Biomarkers: Assessment and Selection of Special Operators, and Improving Individual Performance BIBAKFull-Text 562-571
  Donald R. DuRousseau
Future military special operator selection and education programs will take advantage of state-of-the-art neuroimaging and normative statistical tools in the creation of a customized database of EEG patterns gathered from top performing specialists over their careers. Such a quantitative EEG Normative Database (qEND) will function as the benchmark for screening, assessment, selection and even training of targeted individuals required to work effectively as operators under extreme stresses and for extended periods. This assumption implies that an improved warfighter selection and training pedagogy will embrace the concept of a "model" brain activity pattern (BAP) that represents a warfighter at peak potential and in a highly focused and resilient state of mind. It also implies that this model BAP can be used to: 1) identify biomarkers of positive traits in candidates for specialized training programs, and 2) reduce stress and improve sleep and training performance of program selectees using guided EEG neurofeedback to maintain an optimal BAP. One such statistical qEND (NeuroGuide) is used clinically in the assessment and diagnosis of EEG imbalances specifically related to neurological and behavioral disorders, as well as for guiding individual brain pattern changes through the use of neurofeedback training (NT).
   To evaluate qEEG for monitoring an individual's BAP changes and potentially improving mood and work performance, two military specialists with leadership experience underwent a program of pre- and post-EEG recordings and 20 neurofeedback training (NT) sessions. Here, the NeuroGuide database was used to determine how each participant's BAP differed from the age-matched group norms, and it was also used during the NT process to inform the software of the differences from the norms at each of the 4 training sites used to adjust the trainees EEG towards the direction of "normal".
   Changes from the NT program were assessed pre- and post-intervention using seven neuropsychological assessments of mood, anxiety, sleep, work performance and life satisfaction. In addition, one subject had a series of blood draws taken over the course of the NT program to evaluate changes in his plasma Cortisol; a reliable biomarker of stress level. Both subjects reported reduced levels of anxiety, impulsivity and anger, and improved mood and life satisfaction after the 20-session NT intervention.
Keywords: Assessment and Selection; Biomarkers; Quantitative EEG; Neurofeedback; Normative Statistics; Training Technologies; Training Policy
Ecological Momentary Storytelling: Bringing Down Organizational Stress through Qualifying Work Life Stories BIBAKFull-Text 572-581
  Lisbeth Højbjerg Kappelgaard; Katja Lund
The purpose of this article is to examine ways in which a combination of ecological momentary assessments and reflective dialogues can provide a methodological framework for qualifying work-life stories in the process of reducing organizational stress. The article is based on two hypotheses: 1) a general as well as a work-related sense of coherence can mobilize resistance to stressors and 2) a sense of coherence can occur through self-reflective narratives which clarify patterns of action for oneself and for others. Focusing on hearing impaired people in the Danish work force as well as primary school teachers, the authors create a stress tracking method based on HRV-measurements coupled with mobile questionnaires and reflective dialogues. Findings in the user-test indicated that the method is a tool that creates a story-based foundation on which it is possible to start a process of talking about own experiences, stress and stressors, strategies, contexts etc. when dealing with organizational stress.
Keywords: Ecological Momentary Assessments (EMA); organizational stress; Experience Sampling Method (ESM); Heart Rate Variability (HRV); Sense of Coherence (SOC)
The Development and Application of a Novel Physiological Metric of Cognitive Workload BIBAKFull-Text 582-590
  Jeremy C. Rietschel; Matthew W. Miller
An objective assessment of the cognitive burden imposed by a task (cognitive workload) is of fundamental interest in that it would provide a "window" into one's current allocation of cognitive resources. Such insight would have tremendous implications in maximizing human performance through a multitude of applications including human-computer interaction. The authors propose a novel, electroencephalographic (EEG)-derived metric, which relies on the event-related potential (ERP) component, novelty-P3. A theoretical rationale and experimental evidence supporting the metric's utility are provided, followed by future directions.
Keywords: Cognitive workload; human performance; EEG; novelty-P3
Controlling Attention in the Face of Threat: A Method for Quantifying Endogenous Attentional Control BIBAKFull-Text 591-598
  Bartlett A. H. Russell; Bradley D. Hatfield
It is well established that anxiety causes attentional narrowing and increases distractibility, yet metrics are lacking for measuring these phenomena during performance. Attention Control Theory (ACT) postulates that anxiety consumes limited executive resources that are necessary for maintaining goal-oriented, "top-down" attentional control and for suppressing stimulus-driven, "bottom-up" distraction. While previous work has quantified the effect of anxious states and traits on bottom-up distraction, it is far more difficult to measure endogenous top-down attention. Here we briefly review theories and previous findings regarding anxiety's affect on attention control and discuss an ongoing study examining sustained attention under neutral and anxiogenic conditions. The study employs a combination of established Electroencephalographic (EEG) methods that together may offer a way to measure top-down sustained attention. If successful, the method could help build a more complete theoretical picture of attention control, and provide a way for HCI platforms to monitor user states in changing contexts.
Keywords: Attention control; Steady State Visual Evoked Potentials; anxiety
Developing Visualization Techniques for Improved Information Comprehension and Reduced Cognitive Workload BIBAKFull-Text 599-607
  Scott Scheff; Tristan Plank; John Wilson; Angelia Sebok
In today's data rich environments, enormous quantities of digital information can now be collected and made available to end-users in a wide variety of domains. With so much information now readily accessible, effective display methods that integrate and make sense of the data are needed; otherwise end-users may quickly become overwhelmed. HF Designworks, Inc. and Alion Science & Technology have developed tools that leverage large quantities of information to provide useful visualizations to the warfighter. This paper describes the approach and results of two related projects, iWarrior and My Heat Maps, where we provide end-users with deep data comprehension without imposing cognitive overload.
Keywords: Applications of Augmented Cognition
Development of Fatigue-Associated Measurement to Determine Fitness for Duty and Monitor Driving Performance BIBAKFull-Text 608-617
  Ying Ying Tan; Sheng Tong Lin; Frederick Tey
Long distance driving has been a major factor leading to road accidents [1-2]. With the lack of reliable validation on driver fatigue technology systems [3], the aim of this study is to correlate the measurements of two cognitive tests: Psychomotor Vigilance Task Tester-PVT [4] and PenScreen-PS [5] to establish the threshold levels of fatigued driving performance that will form the basis to prevent fatigued drivers from handling vehicles. PVT is recommended to be the first line of defense against putting fatigued drivers on duty. Drowsiness can be detected by SmartEye Anti-Sleep-AS, acting as a monitoring tool. Eye closure analysis on AS's eyelid opening data showed that AS is a feasible system for real-time monitoring of fatigue while driving. The results also suggested a simpler and more economical way of monitoring fatigue using AS system. PS could be used in conjunction with PVT to detect for any malingering intent.
Keywords: Fatigue; Fitness for Duty; Driving Performance
Novel Tools for Driving Fatigue Prediction: (1) Dry Eeg Sensor and (2) Eye Tracker BIBAKFull-Text 618-627
  Frederick Tey; Sheng Tong Lin; Ying Ying Tan; Xiao Ping Li; Andrea Phillipou; Larry Abel
National Sleep Foundation's Sleep in America (2005) reported 60% of adult drivers driving a vehicle while feeling drowsy in the past year, and more than 37% have actually fallen asleep at the wheel [1]. This paper presented the findings of two novel fatigue prediction tools. The first study presents a 4-channel dry EEG under simulated driving being able to predict when the driver will develop microsleep in the next 10 minutes using only 3 minutes data of collected, with an accuracy of more than 80%. The second study uses an eye tracker to assess the percentage of time that the eyelids were closed (PERCLOS) as a potential marker for fatigue. Results showed that the average magnitude of oscillation (amount of pupil fluctuation), known as Coefficient Magnitude (CM), is generated from real-time wavelet analysis, has the potential to predict fatigue 8-12 minutes ahead with 84% accuracy ahead of compromised driving behavior.
Keywords: Fatigue; dry EEG; eye tracker; microsleep
Quantifying Resilience to Enhance Individualized Training BIBAKFull-Text 628-636
  Brent Winslow; Meredith Carroll; David Jones; Frank Hannigan; Kelly Hale; Kay Stanney; Peter Squire
Resilience is the human ability to adapt in the face of tragedy, trauma, adversity, hardship, and ongoing life stressors. To date, experimental reports on this subject have focused on long-term trajectories (weeks to months) of resilience, with little or no focus on whether significant changes to resilience could be achieved by short-term interventions. Currently, an individual's resilience is defined either by self-report or by behavioral changes such as the development of depression, post-traumatic stress disorder, or suicide. We propose that the quantification of an individual's physiological and behavioral response to stress under controlled conditions is an indication of the individual's level of resilience. To address such real-time resilience, we propose the first in a series of studies to evaluate real-time human resilience by exposing participants to controlled stressors while assessing the stress response. Activation of the hypothalamus-pituitary-adrenal cortex axis and sympathetic branch of the autonomic nervous system via monitoring of the pupil constriction, heart and respiration rate, muscle tonicity, salivary cortisol, and electrodermal activity will be assessed. Stress exposure will consist of virtual stressors presented using Virtual Battlespace 2 software-based scenarios, such as noise exposure, time pressure, and emotion-induction tasks, as well as external stressors such as socio-evaluative stress via the Trier social stress task, while evaluating decision-making and performance. The relationship between performance and the physiological stress response will be quantified, including the creation of a series of stress-performance trajectories based upon individual differences. Such an analysis is similar to probing for resilience in material testing, in which a load is applied to a candidate material, and the resulting forces and observable changes in dimension are quantified and reported via stress-strain curves. Ongoing studies will examine how this resilience measure may be integrated into a closed-loop training system to provide appropriate coping strategies to optimize resilience training. Such training programs, which take into account individual perceptions of stressors and physiological responses, are expected to be effective in helping trainees develop resiliency during high-stress operations.
Keywords: Resilience; Stress; Adaptation; Training; Autonomic Nervous System

Applications of Augmented Cognition

So Fun It Hurts -- Gamifying an Engineering Course BIBAKFull-Text 639-648
  Gabriel Barata; Sandra Gama; Joaquim Jorge; Daniel Gonçalves
Good games are good motivators by nature, as they make players feel rewarded and fulfilled, which pushes them forward to persist and resist frustration. Gamification is a novel technique that uses game elements like points and badges, to motivated and engage users into embracing new behaviors, such as improving one's health condition, finances or productivity. In this paper, we present an experiment in which an MSc college course was gamified to improve student interest and engagement. The gamified course led to better learning results and participation. However, there were several negative side effects that detracted from the overall experience. We will describe them, identifying their causes and describe possible alternatives to better tailor the gamified experience, stemming from the analysis of the data gathered so far.
Keywords: Education Gamification; Perils; Student engagement; Motivation
A Practical Mobile Dry EEG System for Human Computer Interfaces BIBAFull-Text 649-655
  Yu M. Chi; Yijun Wang; Yu-Te Wang; Tzyy-Ping Jung; Trevor Kerth; Yuchen Cao
A complete mobile electroencephalogram (EEG) system based on a novel, flexible dry electrode is presented. The wireless device features 32-channels in a soft, adjustable headset. Integrated electronics enable high resolution (24-bit, 250 samples/sec) acquisition electronics and can acquire operate for more than four hours on a single AAA battery. The system weighs only 140 g and is specifically optimized for ease of use. After training users can self-don the headset in around three minutes. Test data on multiple subjects with simultaneously acquired EEGs from a traditional wet, wired system show a very high degree of signal correlation in AEP and P300 tasks.
Gamification for Measuring Cyber Security Situational Awareness BIBAKFull-Text 656-665
  Glenn Fink; Daniel Best; David Manz; Viatcheslav Popovsky; Barbara Endicott-Popovsky
Cyber defense competitions arising from U.S. service academy exercises, offer a platform for collecting data that can inform research that ranges from characterizing the ideal cyber warrior to describing behaviors during certain challenging cyber defense situations. This knowledge could lead to better preparation of cyber defenders in both military and civilian settings. This paper describes how one regional competition, the PRCCDC, a participant in the national CCDC program, conducted proof of concept experimentation to collect data during the annual competition for later analysis. The intent is to create an ongoing research agenda that expands on this current work and incorporates augmented cognition and gamification methods for measuring cybersecurity situational awareness under the stress of cyber attack.
Keywords: Cyber Defense Competitions; CCDC; cyber defender; cyberwarrior
Human-Robotic Collaborative Intelligent Control for Reaching Performance BIBAKFull-Text 666-675
  Rodolphe J. Gentili; Hyuk Oh; Isabelle M. Shuggi; Ronald N. Goodman; Jeremy C. Rietschel; Bradley D. Hatfield; James A. Reggia
In most human-robot interfaces, the user completely controls the robot that operates as a passive tool without adaptation capabilities. However, a synergetic human-robot interface where both agents collaborate could improve the user's performance while reducing the cognitive and physical workload. Specifically, when considering this framework applied to rehabilitation, we examined a shared collaborative control between a human user and an adaptive biologically inspired neurocontroller in order to perform reaching movements with a simulated prosthetic arm. When this neurocontroller was enabled, it progressively learned from the user to control the prosthetic arm, increasing its role in the shared performance and facilitating the user's reaching movements. This resulted in the user's performance enhancement and in a reduction of his/her cognitive workload. The long term goal of this work is to contribute to the development of the next generation of intelligent human-robotic interfaces for rehabilitation.
Keywords: Human-machine/robot collaborative performance; intelligent control; adaptive systems; arm reaching; assistive technology; prosthetic arm; rehabilitation
Combining Augmented Cognition and Gamification BIBAKFull-Text 676-684
  Curtis S. Ikehara; Martha E. Crosby; Paula Alexandra Silva
The strategic goal of augmented cognition is to increase task performance capacity by using physiological sensor feedback to adjust or modify the activity for the user. Gamification has been shown to increase performance by using certain combinations of game elements. Both augmented cognition and gamification address increased task performance capacity. Gamification adds to augmented cognition by directly addressing the motivation of the user to remain engaged in the activity. This has also been referred to as flow, or the optimal experience. This paper describes an example of a gamified activity in which the physiological sensors of augmented cognition are used to foster the optimal experience desired in gamification. Also, discussed is how the strategic goals of augmented cognition and gamification overlap through the use of a gamified example that describes how the components of augmented cognition and elements of gamification can be used together to better achieve the goal of increased task performance capacity.
Keywords: augmented cognition; gamification; physiological sensors
Issues in Implementing Augmented Cognition and Gamification on a Mobile Platform BIBAKFull-Text 685-694
  Curtis S. Ikehara; Jiecai He; Martha E. Crosby
There are two major trends in computing that will impact augmented cognition. The first is the shift in computing platform from the desktop to mobile computing (e.g., smartphone and tablet) because the user wants to be able to do computing tasks where ever they are. The second trend is the gamification of computer applications to keep the user engaged and motivated. Compared to a workstation, the mobile computing environment is a challenge because of limited computing power, storage capacity, internet connectivity and battery capacity. This paper discusses the issues involved in implementing augmented cognition activities on a mobile platform and the tradeoffs of gamifying augmented cognition activities. These issues are discussed in terms of two example mobile platform applications that implement internal and external sensors.
Keywords: mobile computing; augmented cognition; gamification; physiological sensors
Visual Analysis and Filtering to Augment Cognition BIBAKFull-Text 695-702
  Mathias Kölsch; Juan Wachs; Amela Sadagic
We built and demonstrated a system that augments instructors' sensing abilities and augments their cognition through analysis and filtering of visual information. Called BASE-IT, our system helps US Marine instructors provide excellent training despite the challenging environment, hundreds of trainees and high trainee-to-instructor ratios, non-stop action, and diverse training objectives. To accomplish these objectives, BASE-IT widens the sensory input in multiple dimensions and filters relevant information: BASE-IT a) establishes omnipresence in a large training area, b) supplies continuous evaluation during multi-day training, c) pays specific attention to every individual, d) is specially equipped to identify dangerous situations, and e) maintains virtual vantage points for improved situational awareness. BASE-IT also augments and personalizes the after-action review information available to trainees.
   This paper focuses on the automated data analysis component, how it supplements the information available to instructors, and how it facilitates understanding of individual and team performances on the training range.
Keywords: Augmented cognition; information analysis; training range instrumentation
A Novel HCI System Based on Real-Time fMRI Using Motor Imagery Interaction BIBAKFull-Text 703-708
  Xiaofei Li; Lele Xu; Li Yao; Xiaojie Zhao
Real-time functional resonance imaging (rtfMRI) provides an emerging human-computer interaction (HCI) technology with relatively high spatial resolution. The motor imagery is widely used for sports training of athletes and motor ability rehabilitation of patients, which is a common interaction approach for EEG-based and fMRI-based BCI. An appropriate method of interaction can improve the performance of BCI. In this paper, we implemented a novel HCI system based on rtfMRI using motor imagery interaction. The user interacted with the system by regulating blood oxygenation level dependent (BOLD) signal intensity of the region of interest (ROI) in motor areas using motor imagery, which was presented by the running speed of a virtual human in an animation. The ROI was chosen according to the motor network resulted from the real-time independent component analysis (rtICA). Through the interaction with the HCI system, the user could learn the effectiveness of his motor imagery.
Keywords: HCI system; real-time fMRI; motor imagery; animation interaction
Guided Learning Algorithms: An Application of Constrained Spectral Partitioning to Functional Magnetic Resonance Imaging (fMRI) BIBAKFull-Text 709-716
  Henry L. Phillips; Peter B. Walker; Carrie H. Kennedy; Owen Carmichael; Ian N. Davidson
Innovations in neuro-technology have created a potential gap in our ability to measure human performance and decision making in dynamic environments. Therefore, a need exists to create more reliable testing methodologies and data analytic solutions. The primary aim of this paper is to describe work to integrate subject matter expertise with algorithms designed to measure human brain activity in real time. Specifically, Guided Learning using constrained spectral partitioning to increase the reliability and interpretability of fMRI data is explicated and applied as a test case to the Default Mode Network in the elderly population. How Guided Learning can be further applied to other neuro-imaging technologies that may be more conducing to furthering the field of augmented cognition is discussed.
Keywords: augmented cognition; functional connectivity; fMRI
Next Generation of Physical Training Environments: Bringing in Sensor Systems and Virtual Reality Technologies BIBAKFull-Text 717-726
  Amela Sadagic
Training on physical training ranges is immensely important to any military unit, as many aspects of individual and team skills still need to be trained there. Nevertheless, the overall cost of training on physical ranges, the required unit throughput, as well as a need to maximize the training potential that such precious environments have, are ever increasing, and leveraging emerging technologies to make the training more effective becomes a necessity. This paper reviews several novel efforts in the research domain that could be used as a guide to the types of emerging technical solutions that may be employed to augment the capabilities of physical training ranges, including the capabilities of humans engaged in orchestrating and executing the training events (range operators and instructors), and to support the global objective of acquiring more effective training solutions.
Keywords: physical training ranges; sensor systems; virtual reality (VR); augmented reality (AR); automated behavior analysis; performance evaluation
A Study on Application of RB-ARQ Considering Probability of Occurrence and Transition Probability for P300 Speller BIBAFull-Text 727-733
  Eri Samizo; Tomohiro Yoshikawa; Takeshi Furuhashi
Brain-Computer Interfaces (BCIs) control a computer or a machine based on the information of the signal of human's brain. P300 speller is one of the BCI communication tools, which uses P300 as the feature quantity and allows users to select letters just by thinking. Because of the low signal-to-noise ratio of the P300, signal averaging is often performed to improve the spelling accuracy instead of the degradation of the spelling speed. In texts, there is variability in occurrence probabilities and transition probabilities between letters. This paper proposes P300 speller considering the occurrence probabilities and the transition probabilities as the prior probabilities in RB-ARQ. It shows that the spelling speed and then the Utility were improved by the proposed method comparing with the conventional method.
Improvement of Sensory Stabilization and Repeatability of Vibration Interface for Distance Presentation BIBAFull-Text 734-743
  Yuki Sampei; Takayuki Tanaka; Yuki Mori; Shun'ichi Kaneko
We have developed a vibration alert interface (VAI) that provides information through various vibration patterns. In our previous studies, we designed the VAI and its vibration patterns to provide analog-like information to users such as distance to obstacles. Precise information recognition requires correct perception of vibration patterns. However, various disturbances can affect perception of vibrations, causing users to perceive similar vibrations as being different. We therefore proposed the relative vibration sense presentation method to avoid disruption of the vibration sense. In this paper, we experimentally show that this method improves the repeatability of vibration sensation. We also propose a vibration presentation model for drivers to correct perception gaps due to the application and surroundings of VAI. We evaluate the proposed model through experimentation.
Effect of Light Priming and Encouraging Feedback on the Behavioral and Neural Responses in a General Knowledge Task BIBAKFull-Text 744-753
  Andreea Ioana Sburlea; Tsvetomira Tsoneva; Gary Garcia-Molina
The increase of cognitive demands in society nowadays requires new ways to deal with problems, such as burnout and mental fatigue. Lately, more and more scientifically-based rigorous research in the area of brain-computer interfaces has been done in the quest for restoring and augmenting cognition. The current research work investigates light-based priming and positive reinforcement as possible mediators of cognitive enhancement.
Keywords: priming with light; cognitive enhancement; positive feedback
Using the Smartphone Accelerometer to Monitor Fall Risk while Playing a Game: The Design and Usability Evaluation of Dance! Don't Fall BIBAKFull-Text 754-763
  Paula Alexandra Silva; Francisco Nunes; Ana Vasconcelos; Maureen Kerwin; Ricardo Moutinho; Pedro Teixeira
Falls are dangerous, and unfortunately common for older adults. Dance! Don't Fall is a game that assesses the quality of the user's locomotion based on data from the accelerometer of a smartphone. By providing a form of exercise, the game may actually reduce fall risk as well as monitoring it. In this paper, we document the development of the prototype and a usability study with ten seniors that suggested the game is well suited to its primary users.
Keywords: Fall risk assessment; older adults; mobile applications; physical activity; dance games
Augmented Interaction: Applying the Principles of Augmented Cognition to Human-Technology and Human-Human Interactions BIBAKFull-Text 764-773
  Anna Skinner; Lindsay Long; Jack Vice; John Blitch; Cali M. Fidopiastis; Chris Berka
The field of Augmented Cognition (AugCog) has evolved over the past decade from its origins in the Defense Advanced Research Projects Agency (DARPA)-funded research program, emphasizing modulation of closed-loop human-computer interactions within operational environments, to address a broader scope of domains, contexts, and science and technology (S&T) challenges. Among these are challenges related to the underlying theoretical and empirical research questions, as well as the application of advances in the field within contexts such as training and education. This paper summarizes a series of ongoing research and development (R&D) efforts aimed at applying an AugCog-inspired framework to enhance both human-technology and human-human interactions within a variety of training and operational domains.
Keywords: Augmented Cognition; Training; Simulation; Human-Robot Interaction; Adaptive Automation; Neuroscience; Psychophysiological Measures; EEG
Integration of Automated Neural Processing into an Army-Relevant Multitasking Simulation Environment BIBAKFull-Text 774-782
  Jon Touryan; Anthony J. Ries; Paul Weber; Laurie Gibson
Brain-computer interface technology has experienced a rapid evolution over recent years. Recent studies have demonstrated the feasibility of detecting the presence or absence of targets in visual imagery from the neural response alone. Classification accuracy persists even when the imagery is presented rapidly. While this capability offers significant promise for applications that require humans to process large volumes of imagery, it remains unclear how well this approach will translate to more real-world scenarios. To explore the viability of automated neural processing in an Army-relevant operational context, we designed and built a simulation environment based on a ground vehicle crewstation. Here, we describe the process of integrating and testing the automated neural processing capability within this simulation environment. Our results indicate the potential for significant benefits to be realized by incorporating brain-computer interface technology into future Army systems.
Keywords: Simulator; Brain-Computer Interface (BCI); Visual Search
Behavioral Biometric Identification on Mobile Devices BIBAKFull-Text 783-791
  Matt Wolff
We show that accelerometers, touch screens and software keyboards, which are standard components of modern mobile phones, can be used to differentiate different test subjects based on the unique interaction characteristics of each subject. This differentiation ability can be applied to authenticate individuals under a continuous authentication scheme. Based on six 15 minute data sets collected from the test subjects utilizing our data collection platform, we extract multiple features from the data and show an ability to accurately identify individuals at a rate of 83 percent using a simple normal distribution of each feature.
Keywords: identification; security