Openly accessible

Automatic emotion recognition using temporal multimodal deep learning

Nakisa, Bahareh, Rastgoo, Mohammad Naim, Rakotonirainy, Andry, Maire, Frederic and Chandran, Vinod 2020, Automatic emotion recognition using temporal multimodal deep learning, IEEE Access, no. Early Access, doi: 10.1109/access.2020.3027026.

Attached Files
Name Description MIMEType Size Downloads

Title Automatic emotion recognition using temporal multimodal deep learning
Author(s) Nakisa, BaharehORCID iD for Nakisa, Bahareh orcid.org/0000-0001-8583-4773
Rastgoo, Mohammad Naim
Rakotonirainy, Andry
Maire, Frederic
Chandran, Vinod
Journal name IEEE Access
Issue number Early Access
Total pages 13
Publisher Institute of Electrical and Electronics Engineers (IEEE)
Place of publication Piscataway, N.J.
Publication date 2020-09-28
ISSN 2169-3536
Keyword(s) emotion recognition
electroencephalography
blood volume pulse
convolutional neural network
long short-term memory
temporal multimodal fusion
Summary Emotion recognition using miniaturised wearable physiological sensors has emerged as a revolutionary technology in various applications. However, detecting emotions using the fusion of multiple physiological signals remains a complex and challenging task. When fusing physiological signals, it is essential to consider the ability of different fusion approaches to capture the emotional information contained within and across modalities. Moreover, since physiological signals consist of time-series data, it becomes imperative to consider their temporal structures in the fusion process. In this study, we propose a temporal multimodal fusion approach with a deep learning model to capture the non-linear emotional correlation within and across electroencephalography (EEG) and blood volume pulse (BVP) signals and to improve the performance of emotion classification. The performance of the proposed model is evaluated using two different fusion approaches – early fusion and late fusion. Specifically, we use a convolutional neural network (ConvNet) long short-term memory (LSTM) model to fuse the EEG and BVP signals to jointly learn and explore the highly correlated representation of emotions across modalities, after learning each modality with a single deep network. The performance of the temporal multimodal deep learning model is validated on our dataset collected from smart wearable sensors and is also compared with results of recent studies. The experimental results show that the temporal multimodal deep learning models, based on early and late fusion approaches, successfully classified human emotions into one of four quadrants of dimensional emotions with an accuracy of 71.61% and 70.17%, respectively.
Language eng
DOI 10.1109/access.2020.3027026
Indigenous content off
Field of Research 08 Information and Computing Sciences
09 Engineering
10 Technology
HERDC Research category C1 Refereed article in a scholarly journal
Free to Read? Yes
Persistent URL http://hdl.handle.net/10536/DRO/DU:30144498

Connect to link resolver
 
Unless expressly stated otherwise, the copyright for items in DRO is owned by the author, with all rights reserved.

Every reasonable effort has been made to ensure that permission has been obtained for items included in DRO. If you believe that your rights have been infringed by this repository, please contact drosupport@deakin.edu.au.

Versions
Version Filter Type
Citation counts: TR Web of Science Citation Count  Cited 0 times in TR Web of Science
Scopus Citation Count Cited 0 times in Scopus
Google Scholar Search Google Scholar
Access Statistics: 63 Abstract Views, 1 File Downloads  -  Detailed Statistics
Created: Mon, 26 Oct 2020, 17:05:48 EST

Every reasonable effort has been made to ensure that permission has been obtained for items included in DRO. If you believe that your rights have been infringed by this repository, please contact drosupport@deakin.edu.au.