Wearable brain-to-human communication

Every 90 minutes, someone is diagnosed with Amyotrophic Lateral Sclerosis (ALS), and someone passes away from it ¹.

Amyotrophic Lateral Sclerosis (ALS) is a relentlessly progressive disorder; muscle deterioration is slow and, currently, absolute. During the later stages, complete muscle failure is common. Current treatments (such as riluzole ²) only slow the progression, which can mean a longer time in the worst stages of the disease.

Muscle-based computer interfaces provide a stop-gap measure once speech is impossible - known as a locked-in state (LIS) - often providing a binary signal that can be used for basic communication (estimated prevalence of locked-in state: ~1 in 100,000 ³).

However, the usability of muscle-based communication is often fatiguing or unreliable, and worse, temporary. At the inevitable point of total muscle failure - a complete locked-in state (CLIS) - individuals lose the ability to communicate in any way.

Loss of communication occurs after the need for artificial respiration to live. 42% of those who risk losing the ability to be in touch with others say they would prefer to die rather than be unable to communicate ⁴; communication loss is consistently cited as a primary reason for refusing care ⁵.

Throughout this process the brain remains largely unaffected, as does, in most cases, the ability to think, imagine and feel emotions. Brain-computer interfaces (BCIs) can provide a communication channel that can endure through all stages of this disease, and restore a degree of agency when all other methods fail. Even the simplest BCI-based communication has a high impact on the perceived quality of life ⁶.

The feasibility of BCIs for communication is established with acceptable performance for a wide variety of applications:

• Spelling devices ⁷
• Finger control ⁸
• Web browsing ⁹
• Emotion detection ¹⁰
• Control for virtual reality environments ¹¹.

Yet, despite over 22,000 BCI research studies ¹², few acknowledge and adapt to the restrictions of real-life settings. For example, Wolpaw et al. ¹³ found only 3 case studies that describe independent home use of a BCI, and Craik et al. ¹⁴ surveyed that “more than a quarter of the studies (26 of 90)” removed artifacts manually making them unusable for real-time.

To scale BCIs for widespread application there is a need to reconsider laboratory-controlled setups, which don’t reflect use in the home, where there is no expert supervisor on hand to assist.

As we discovered, this is a step that is sorely needed. Existing algorithms with high accuracy rates (over 70% forming a baseline for usability) in literature cannot be applied without significant performance penalties.

Beyond the Medical

Many of the applications, the infrastructure and algorithms needed for real-world usage can be applied and expanded to consumer contexts, and more extreme environments. The benefits of usability ( from the wearable form-factor and wireless flexibility) are universal.

Contributions

RealityEXG

To measure the impact of these assumptions, we establish a platform for realistic brain signal data collection. This generated our Real-Life Dataset (“RealityEXG”), a substantial dataset of 67.2 hours across 16 participants, 9 devices, 488 disjointed sessions, and 122 days. This data is purposefully poor: it has a high proportion of artifacts; it is collected on commercially viable devices (with limited data collection); uses Bluetooth for transmission.

Mental Mathematics Model

Having established a mechanism to collect challenging data. We then proposed a novel BCI for communication using unstructured mental mathematics to enable basic communication without sensory stimuli. Our model runs in real-time and simply requires a mobile device and consumer-level headband/glasses. By removing the carefully applied EEG cap we can expand usage beyond the lab, and outside the norm.

Unstructured mental mathematics is simply the reaction to being told “perform mental maths”; we do not provide equations to solve, which would require a working visual system. This lack of structure makes detection both challenging, and very personal but also widely applicable. One person may focus heavily on numerical imagination and recall, while another may focus on working memory multiplying large numbers.

Despite this, the produced BCI can be used immediately by any new user without individual training, although accuracy jumps with tuning (76% accuracy on RealityEXG). By contrast, existing implementations ¹⁶ give over 90% accuracies on single-session lab environment inference, but fail to exceed chance when applied to noisy, consumer-grade datasets.

Further details on the initial ‘BrainSwitch’ model can be found in the full thesis text, or you can ask me directly to see the latest work on this topic.

Potential

Despite wishing we could achieve fluid, speech-level communication with our study participants, that goal is not yet realizable. Ethically, however, we must strive to achieve non-invasive, affordable BCIs; and despite our desire for perfection, a device that brings back an urgently needed function, that supplies a modicum of agency and dignity, or provides a back-up to existing approaches, is possibly a good start.

Providing individuals with control over basic but humanizing activities (requesting eye drops, asking for food, or starting music) allows for some dignity and agency against a disease that is constantly eroding their humanity.

Challenges

With the goal of fluid, speech-level communication in mind, I leave my list of challenges and limitations that should be overcome:

• Challenge 1: Efficiency. The potential for improvement in decoding algorithms is evident as we still require a large comparison window (>15s) to detect a mental state, resulting in a non-ideal information transfer rate of $.1bps$.
• Challenge 2: Supervised Algorithms do not transfer to noisy, low-channel count environments. The application of successfully supervised machine learning algorithms applied to computer vision and natural language processing tasks has proven ineffective. We attribute this to the poor signal-to-noise ratio common to EEG that is absent in other use cases. We suspect that appropriate inference requires further modifications to existing architectures to exploit the assumed second-order stationary nature of signals and the de-biasing for recently processed data. We do expect supervised strategies to exceed bio-physiological inspired strategies in the long term but are yet to determine the appropriate combination of feature extraction and network architecture to realize this performance.
• Challenge 3: Lack of free-text entry. One option would be to add a grid-speller similar to P300 to allow for the binary signal to create free-form text by submitting a character by encoding a row and column selection similar to the “Query” task for choosing environment requests. However, the speed of such an approach would be challenging, instead applying multi-modal contrastive algorithms, that relate EXG data to text, may allow for language-aware text models that combat the slow processing speed of signals.
• Challenge 4: Mental initiation. Currently, a question has to be asked before a response can be given. Allowing for proactive communication, where, for example, the mental math task can be used to turn on the “BrainSwitch” model would be far more effective. We have started to approach this using change detection of the ratios (rapid changes in ratio suggest the start of a mental task) but further work is needed.
• Challenge 5: Public distribution of the application suite Currently the applications are distributed to participants via “TestFlight”/manual installation. To allow for scaling up we would need to release the app to the App Store, move inference to occur locally, and allow data collection to be opted-in rather than mandatory.
• Challenge 6: Small electrode count causing low signal-to-noise-ratio. A large contributor to the difficulty of the decoding process was the lack of additional channels to perform noise reduction on alternative form factors that have either additional electrodes or allow for different EEG channel montages (e.g. such as over the head with a headphone form factor) that may provide channel locations particularly conducive to different mental states.
• Challenge 7: Lack of training data. Despite small sample sizes being commonplace in EEG studies, it likely doesn’t aid the supervised learning algorithms. Collecting more data would help and the application, as well as the infrastructure, is designed to support that; but additionally, the use of pre-text tasks on a large, unlabelled corpus of EEG data such as the TUH EEG Corpus which contains over a year of data, should help in overcoming the limitations. Examples considered include jigsaw puzzles in time-series segments (un-shuffling should learn time-series trends), transformation reversal (to denoise more effectively), and prediction of spectral decomposition (to learn spectral relationships).

Findings

The end-to-end system is ready for home-use, and provides basic communication functionality achieving several goals:

1. Effective decoding. Our decoding algorithm, with subject-dependent tuning, reached $76.7\pm11$% cross-session, cross-day accuracy, exceeding the $70$% requirement for BCI accuracy. Subject independence in real-time settings was also possible at a performance penalty achieving $65\pm19$% accuracy.
2. Satisfying training times. The desired accuracy can be reached without user training or calibration time.
3. Usable for long durations. The dry electrodes, long battery life (5-7 hours), comfortable form factor, and the use of a mobile phone base station allow for BCI to remain passive, waiting to be used when needed.
4. Satisfying setup times. The combination of form factor, mobile app, and infrastructure all combine to allow for a yes/no question to be asked from no-setup within 60 seconds, rather than $30\pm21$ minutes ¹³ for an industry-standard home-use BCI.