publications

Neuroimaging

2025

1. AdaptiveCoPilot: Design and Testing of a NeuroAdaptive LLM Cockpit Guidance System in both Novice and Expert Pilots

   Shaoyue Wen, Michael Middleton, Songming Ping, and 12 more authors

   In 2025 IEEE Conference Virtual Reality and 3D User Interfaces (VR), Mar 2025

   ISSN: 2642-5254

   Abs

   Pilots operating modern cockpits often face high cognitive demands due to complex interfaces and multitasking requirements, which can lead to overload and decreased performance. This study introduces AdaptiveCoPilot, a neuroadaptive guidance system that adapts visual, auditory, and textual cues in real time based on the pilot’s cognitive workload, measured via functional Near-Infrared Spectroscopy (fNIRS). A formative study with expert pilots (N=3) identified adaptive rules for modality switching and information load adjustments during preflight tasks. These insights informed the design of AdaptiveCoPilot, which integrates cognitive state assessments, behavioral data, and adaptive strategies within a context-aware Large Language Model (LLM). The system was evaluated in a virtual reality (VR) simulated cockpit with licensed pilots (N=8), comparing its performance against baseline and random feedback conditions. The results indicate that the pilots using AdaptiveCoPilot exhibited higher rates of optimal cognitive load states on the facets of working memory and perception, along with reduced task completion times. Based on the formative study, experimental findings, qualitative interviews, we propose a set of strategies for future development of neuroadaptive pilot guidance systems and highlight the potential of neuroadaptive systems to enhance pilot performance and safety in aviation environments.

2024

1. HuBar: A Visual Analytics Tool to Explore Human Behaviour based on fNIRS in AR guidance systems

   Sonia Castelo, Joao Rulff, Parikshit Solunke, and 11 more authors

   Jul 2024

   arXiv:2407.12260 [cs]

   Abs

   The concept of an intelligent augmented reality (AR) assistant has significant, wide-ranging applications, with potential uses in medicine, military, and mechanics domains. Such an assistant must be able to perceive the environment and actions, reason about the environment state in relation to a given task, and seamlessly interact with the task performer. These interactions typically involve an AR headset equipped with sensors which capture video, audio, and haptic feedback. Previous works have sought to facilitate the development of intelligent AR assistants by visualizing these sensor data streams in conjunction with the assistant’s perception and reasoning model outputs. However, existing visual analytics systems do not focus on user modeling or include biometric data, and are only capable of visualizing a single task session for a single performer at a time. Moreover, they typically assume a task involves linear progression from one step to the next. We propose a visual analytics system that allows users to compare performance during multiple task sessions, focusing on non-linear tasks where different step sequences can lead to success. In particular, we design visualizations for understanding user behavior through functional near-infrared spectroscopy (fNIRS) data as a proxy for perception, attention, and memory as well as corresponding motion data (acceleration, angular velocity, and gaze). We distill these insights into embedding representations that allow users to easily select groups of sessions with similar behaviors. We provide two case studies that demonstrate how to use these visualizations to gain insights about task performance using data collected during helicopter copilot training tasks. Finally, we evaluate our approach by conducting an in-depth examination of a think-aloud experiment with five domain experts.

2. Analysis of Pre-Flight and Monitoring Tasks Using Cognitive Performance Modeling

   Chihab Nadri, Yunmei Liu, Maryam Zahabi, and 4 more authors

   In Human Factors in Design, Engineering, and Computing, Jul 2024

   ISSN: 27710718, Issue: 159

   Abs

   Pilot cognitive workload and errors significantly contribute to aviation incidents. Evaluations of piloting tasks in both simulators and real-world settings, along with computational models of cognitive task performance can help to identify cognitively challenging tasks early in the system design process and enhance user interface designs. This study applied cognitive performance modeling (CPM) to assess pilot task demands in pre-flight and monitoring using a UH-60V Black Hawk helicopter flight simulator. The objective was to propose potential flight checklist and subtask interface redesigns to reduce pilot working memory load and improve operational effectiveness. Initial analysis involved reviewing pilot instructions and logs for pre-flight checks, monitoring activities, and emergency responses. Actions, such as button presses, task errors and the duration between tasks were recorded. A Hierarchical Task Analysis (HTA) was applied to identify sub-task interdependencies. CPMs were developed using Cogulator and a variation on the GOMS language detailing cognitive, perceptual, and motor processes. Models focused on task sequences and cognitive process durations, revealing task time estimates, working memory load, and cognitive workload. Demanding subtasks were identified based on longer durations and/or higher workloads. Cogulator model outcomes for workload assessment were compared with pilot opinions on task difficulty for model validation. Recommendations for cockpit interface enhancements were formulated with the aim of streamlining sub-task operation sequences, reducing cognitive load, and improving pre-flight and monitoring efficiency. Key suggestions included redesigning checklists, providing auto-text completion options for data entry tasks, and implementing temporary shutdowns of displays (irrelevant to the primary flight task) under emergency conditions. The study methodology was validated through expert interviews and findings inform the design of current and future piloting procedures, potentially contributing to improved aviation safety and efficiency.

PCG

2025

1. Moonshine: Distilling Game Content Generators into Steerable Generative Models

   Yuhe Nie, Michael Middleton, Tim Merino, and 4 more authors

   Proceedings of the AAAI Conference on Artificial Intelligence, Apr 2025

   Abs

   Procedural Content Generation via Machine Learning (PCGML) has enhanced game content creation, yet challenges in controllability and limited training data persist. This study addresses these issues by distilling a constructive PCG algorithm into a controllable PCGML model. We first generate a large amount of content with a constructive algorithm and label it using a Large Language Model (LLM). We use these synthetic labels to condition two PCGML models for content-specific generation, a diffusion model and the five-dollar model. This neural network distillation process ensures that the generation aligns with the original algorithm while introducing controllability through plain text. We define this text-conditioned PCGML as a Text-to-game-Map (T2M) task, offering an alternative to prevalent text-to-image multi-modal tasks. We compare our distilled models with the baseline constructive algorithm. Our analysis of the variety, accuracy, and quality of our generation demonstrates the efficacy of distilling constructive methods into controllable text-conditioned PCGML models.

AR/VR

2025

1. Design and Implementation of the Transparent, Interpretable, and Multimodal (TIM) AR Personal Assistant

   Erin McGowan, Joao Rulff, Sonia Castelo, and 11 more authors

   IEEE Computer Graphics and Applications, Jan 2025

   Abs

   The concept of an artificial intelligence (AI) assistant for task guidance is rapidly shifting from a science fiction staple to an impending reality. Such a system is inherently complex, requiring models for perceptual grounding, attention, and reasoning, an intuitive interface that adapts to the performer’s needs, and the orchestration of data streams from many sensors. Moreover, all data acquired by the system must be readily available for posthoc analysis to enable developers to understand performer behavior and quickly detect failures. We introduce TIM, the first end-to-end AI-enabled task guidance system in augmented reality, which is capable of detecting both the user and scene as well as providing adaptable, just-in-time feedback. We discuss the system challenges and propose design solutions. We also demonstrate how TIM adapts to domain applications with varying needs, highlighting how the system components can be customized for each scenario.

2024

1. HuBar: A Visual Analytics Tool to Explore Human Behaviour based on fNIRS in AR guidance systems

   Sonia Castelo, Joao Rulff, Parikshit Solunke, and 11 more authors

   Jul 2024

   arXiv:2407.12260 [cs]

   Abs

   The concept of an intelligent augmented reality (AR) assistant has significant, wide-ranging applications, with potential uses in medicine, military, and mechanics domains. Such an assistant must be able to perceive the environment and actions, reason about the environment state in relation to a given task, and seamlessly interact with the task performer. These interactions typically involve an AR headset equipped with sensors which capture video, audio, and haptic feedback. Previous works have sought to facilitate the development of intelligent AR assistants by visualizing these sensor data streams in conjunction with the assistant’s perception and reasoning model outputs. However, existing visual analytics systems do not focus on user modeling or include biometric data, and are only capable of visualizing a single task session for a single performer at a time. Moreover, they typically assume a task involves linear progression from one step to the next. We propose a visual analytics system that allows users to compare performance during multiple task sessions, focusing on non-linear tasks where different step sequences can lead to success. In particular, we design visualizations for understanding user behavior through functional near-infrared spectroscopy (fNIRS) data as a proxy for perception, attention, and memory as well as corresponding motion data (acceleration, angular velocity, and gaze). We distill these insights into embedding representations that allow users to easily select groups of sessions with similar behaviors. We provide two case studies that demonstrate how to use these visualizations to gain insights about task performance using data collected during helicopter copilot training tasks. Finally, we evaluate our approach by conducting an in-depth examination of a think-aloud experiment with five domain experts.

Misc

2022

1. ParticleChromo3D: a Particle Swarm Optimization algorithm for chromosome 3D structure prediction from Hi-C data

   David Vadnais, Michael Middleton, and Oluwatosin Oluwadare

   BioData Mining, Sep 2022

   Abs

   The three-dimensional (3D) structure of chromatin has a massive effect on its function. Because of this, it is desirable to have an understanding of the 3D structural organization of chromatin. To gain greater insight into the spatial organization of chromosomes and genomes and the functions they perform, chromosome conformation capture (3C) techniques, particularly Hi-C, have been developed. The Hi-C technology is widely used and well-known because of its ability to profile interactions for all read pairs in an entire genome. The advent of Hi-C has greatly expanded our understanding of the 3D genome, genome folding, gene regulation and has enabled the development of many 3D chromosome structure reconstruction methods.