A new artificial intelligence (AI) and augmented reality (AR) device which recognises and reads the emotional cues of those around us could be a welcome step in combating mental health disorders experienced as a result of COVID-19.

Researchers from Monash University and RMIT have developed a prototype called Neo-Noumena - which is a communicative neuro responsive system that uses brain-computer interfacing to read one’s emotional state and represents them to others through head-mounted displays.

Social isolation has been a necessary measure to protect public health against COVID-19 but it has had a detrimental impact on people’s mental health and their personal relationships whilst at home.

Recognising facial expressions, a person’s tone of voice or their body language can explain a lot about the emotions of another person. Understanding these emotions can help us all recognise and appropriately respond to how our family members or housemates are feeling in isolation.

Project lead, Professor Florian ‘Floyd’ Mueller, Director of the Exertion Games Lab in Monash University’s Faculty of Information Technology (IT), says this technology has the potential to expand our capabilities in emotional communication.

“There is currently a lack of research which supports face-to-face emotional communication in everyday contexts. By using advanced technologies such as AI, AR, mobile electroencephalography (EEG) and mixed reality, we’re able to assess neurogenic emotion in real-time and display this digital content in a physical space,” Professor Mueller said.

“Neo-Noumena also has the possibility to assist people with autism to support their emotional communication ability as well as patients in therapy or rehabilitation, to better understand how they’re responding to treatment.”

Neo-Noumena uses EEG to interpret the user’s emotional state and produces a visual representation using AI technology. The visual representations are then digitally transmitted through two mixed reality head-mounted displays which provide a real-time dynamic representation of emotion.

Exertion Games Lab PhD Candidate, Nathan Semertzidis, explains how Neo-Noumena can also help users become more in tune with their emotions.

“During our field research, volunteers wore the AI-driven system for several days and began to use the system to talk to one another about how they were feeling,” he said.

“Over time participants also reported being able to better understand how their partner was feeling by associating different environmental factors, such as music, media, and the presence of stress. This also prompted participants to respond more appropriately to their partner’s emotions. In turn, we found that measures of emotional competence demonstrated a significant increase in the participants’ ability to regulate their own emotions.”

This study has the potential to benefit fellow researchers in further examining, operationalising and assessing experiences of augmented emotion communication beyond traditional settings.

Project collaborator, Associate Professor, Fabio Zambetta, Discipline Leader of Computer Science and Software Engineering at RMIT University, says that current applications of AI are changing and Neo-Noumena offers an alternative way of capturing this technology. 

"There is a growing interest in applying AI to human-centric computing, putting people at the centre of what we investigate. I believe that Neo-Noumena has great potential to contribute to research that will be of societal impact and benefit,” says Associate Professor Zambetta.

Neo-Noumena can also assist practitioners in informing the design of future augmented emotion communication systems and lead to a better understanding of appropriately designing emotion communication tools.