Deakin University
Browse

File(s) under permanent embargo

Driver Emotion Recognition with a Hybrid Attentional Multimodal Fusion Framework

journal contribution
posted on 2023-03-21, 02:12 authored by L Mou, Y Zhao, C Zhou, Bahareh NakisaBahareh Nakisa, MN Rastgoo, L Ma, T Huang, B Yin, R Jain, W Gao
Negative emotions may induce dangerous driving behaviors leading to extremely serious traffic accidents. Therefore, it is necessary to establish a system that can automatically recognize driver emotions so that some actions can be taken to avoid traffic accidents. Existing studies on driver emotion recognition have mainly used facial data and physiological data. However, there are fewer studies on multimodal data with contextual characteristics of driving. In addition, fully fusing multimodal data in the feature fusion layer to improve the performance of emotion recognition is still a challenge. To this end, we propose to recognize driver emotion using a novel multimodal fusion framework based on convolutional long-short term memory network (ConvLSTM), and hybrid attention mechanism to fuse non-invasive multimodal data of eye, vehicle, and environment. In order to verify the effectiveness of the proposed method, extensive experiments have been carried out on a dataset collected using an advanced driving simulator. The experimental results demonstrate the effectiveness of the proposed method. Finally, a preliminary exploration on the correlation between driver emotion and stress is performed.

History

Journal

IEEE Transactions on Affective Computing

Volume

PP

Pagination

1-12

Location

Piscataway, N.J.

ISSN

2371-9850

eISSN

1949-3045

Language

eng

Issue

99

Publisher

Institute of Electrical and Electronics Engineers (IEEE)