A review on automatic facial expression recognition systems assisted by multimodal sensor data

Samadiani, Najmeh, Huang, Guangyan, Cai, Borui, Luo, Wei, Chi, Chi-Hung, Xiang, Yong and He, Jing 2019, A review on automatic facial expression recognition systems assisted by multimodal sensor data, Sensors, vol. 19, no. 8, doi: 10.3390/s19081863.

Attached Files
Name Description MIMEType Size Downloads

Title A review on automatic facial expression recognition systems assisted by multimodal sensor data
Author(s) Samadiani, Najmeh
Huang, GuangyanORCID iD for Huang, Guangyan orcid.org/0000-0002-1821-8644
Cai, Borui
Luo, WeiORCID iD for Luo, Wei orcid.org/0000-0002-4711-7543
Chi, Chi-Hung
Xiang, YongORCID iD for Xiang, Yong orcid.org/0000-0003-3545-7863
He, Jing
Journal name Sensors
Volume number 19
Issue number 8
Total pages 27
Publisher MDPI
Place of publication Basel, Switzerland
Publication date 2019
ISSN 1424-8220
Keyword(s) emotional expression recognition
facial expression recognition (FER)
multimodal sensor data
real-world conditions
spontaneous expression
Summary Facial Expression Recognition (FER) can be widely applied to various research areas, such as mental diseases diagnosis and human social/physiological interaction detection. With the emerging advanced technologies in hardware and sensors, FER systems have been developed to support real-world application scenes, instead of laboratory environments. Although the laboratory-controlled FER systems achieve very high accuracy, around 97%, the technical transferring from the laboratory to real-world applications faces a great barrier of very low accuracy, approximately 50%. In this survey, we comprehensively discuss three significant challenges in the unconstrained real-world environments, such as illumination variation, head pose, and subject-dependence, which may not be resolved by only analysing images/videos in the FER system. We focus on those sensors that may provide extra information and help the FER systems to detect emotion in both static images and video sequences. We introduce three categories of sensors that may help improve the accuracy and reliability of an expression recognition system by tackling the challenges mentioned above in pure image/video processing. The first group is detailed-face sensors, which detect a small dynamic change of a face component, such as eye-trackers, which may help differentiate the background noise and the feature of faces. The second is non-visual sensors, such as audio, depth, and EEG sensors, which provide extra information in addition to visual dimension and improve the recognition reliability for example in illumination variation and position shift situation. The last is target-focused sensors, such as infrared thermal sensors, which can facilitate the FER systems to filter useless visual contents and may help resist illumination variation. Also, we discuss the methods of fusing different inputs obtained from multimodal sensors in an emotion system. We comparatively review the most prominent multimodal emotional expression recognition approaches and point out their advantages and limitations. We briefly introduce the benchmark data sets related to FER systems for each category of sensors and extend our survey to the open challenges and issues. Meanwhile, we design a framework of an expression recognition system, which uses multimodal sensor data (provided by the three categories of sensors) to provide complete information about emotions to assist the pure face image/video analysis. We theoretically analyse the feasibility and achievability of our new expression recognition system, especially for the use in the wild environment, and point out the future directions to design an efficient, emotional expression recognition system.
Language eng
DOI 10.3390/s19081863
Field of Research 0301 Analytical Chemistry
0906 Electrical and Electronic Engineering
HERDC Research category C1 Refereed article in a scholarly journal
Copyright notice ©2019, The Authors
Persistent URL http://hdl.handle.net/10536/DRO/DU:30121101

Connect to link resolver
Unless expressly stated otherwise, the copyright for items in DRO is owned by the author, with all rights reserved.

Version Filter Type
Citation counts: TR Web of Science Citation Count  Cited 2 times in TR Web of Science
Scopus Citation Count Cited 2 times in Scopus
Google Scholar Search Google Scholar
Access Statistics: 65 Abstract Views, 0 File Downloads  -  Detailed Statistics
Created: Mon, 29 Apr 2019, 09:36:16 EST

Every reasonable effort has been made to ensure that permission has been obtained for items included in DRO. If you believe that your rights have been infringed by this repository, please contact drosupport@deakin.edu.au.