Research of the methods on facial expression recognition

Research Article
Open access

Research of the methods on facial expression recognition

Xuanyi Chen 1 , Yuyao Ding 2 , Zhaoheng Li 3*
  • 1 School of Artificial Intelligence, Beijing University of Posts and Telecommunications, No. 10, Xitucheng Road, Beijing, China    
  • 2 Department of Telecommunications, Xi’an Jiaotong University, No. 28, West Xianning Road, Xi'an, China    
  • 3 College of Physical Science and Technology, Hebei University, No. 180, Wusi East Road, Lianchi District, Baoding, China    
  • *corresponding author 20201304003@stumail.hbu.edu.cn
Published on 14 June 2023 | https://doi.org/10.54254/2755-2721/6/20230902
ACE Vol.6
ISSN (Print): 2755-273X
ISSN (Online): 2755-2721
ISBN (Print): 978-1-915371-59-1
ISBN (Online): 978-1-915371-60-7

Abstract

As traditional machine learning and deep learning have developed recently, the recognition of facial expressions has been paid more attention by domestic and foreign scholars. First of all, this paper introduced the common data sets of single mode, traditional feature extraction methods and more common expression classification methods in detail, pointing out that the present single-mode expression recognition does not perform well in practical application scenarios, and cannot obtain good recognition results in complex environments. At the same time, the data sets are relatively simple and the number is small. Then, three recognition methods based on multimodality are introduced: Fusion at the feature, decision, and hybrid levels. The advantages and disadvantages of the three measures are minute described respectively. Finally, the thesis is summarized. In addition, the future development of more general and richer high-quality expression datasets is prospected and the improvement of current multimodal fusion technology are prospected.

Keywords:

facial expression recognition, single mode, multi-mode, modal fusion technology

Chen,X.;Ding,Y.;Li,Z. (2023). Research of the methods on facial expression recognition. Applied and Computational Engineering,6,608-619.
Export citation

References

[1]. Liu Bowen, Shuai Jianwei, Cao Yuping. Application of Facial expression recognition technology in the diagnosis and treatment of mental diseases. 2021 Chinese J. Be. Med. Br. Sci., 30 (10): 955- 960.

[2]. Lai Dongsheng. Research and Application of Light scale Situation Recognition Algorithm based on Multi-feature fusion. 2022 Guangdong Univ. Tech.

[3]. Xu Xiaokang. Research and Application of Expression Recognition Based on Deep Learning 2022, Donghua Univ.

[4]. Wang Jin, Huang Xiaohua, Li Hang, Hong Jie. Application Research of Microexpression Recognition System in Low resolution Environment. 2022, Compute. Knowle. Tech., 18(20): 81-82+85.

[5]. Lyons M J, Akamatsu S, Kama M, et al. Coding facial expressions with gabor wavelets 1998, Inter. Conf. Face & Gest. Rec., 14-16: 200-205.

[6]. Lucey P, Cohn J F, Kande T, et al. The extended Cohn-Kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression, 2010 Conf. Compute Vis. Pat. Rec., San Francisco, Jun 13-18: 94-101.

[7]. Susskind J M, Anderson A K, Hinton G E. The Toronto face database, 2010, Toronto: Univ. Toronto.

[8]. Zhao G, Huang X, Taini M, et al. Facial expression recognition from near-infrared videos, 2011, Ima. Vis. Comput. 2(9): 607-619.

[9]. Yin L J, Wei X Z, Sun Y, et al. A 3D facial expression database for facial behavior research, 2006 7th IEEE Inter. Conf. Auto. Face Gest. Rec., 211-216.

[10]. Savran A，Ala，Dibeklion H，et al. Bos phorus database for 3D face analysis, 2008 Euro. Biomet. Ident. Man.，Heidelberg：Springer, 47-56.

[11]. Gross R, Matthews I, Conhn J, et al. Multi- PIE, Imag. Vis. Com., 2010, 28(5): 807-813.

[12]. Dhall A,Goecke R, Lucey S, et al. Static facial expression analysis in tough conditions: data, evaluation protocol and benchmark 2011 IEEE Inter. Conf. Comp. Vis., Barcelona, Nov 6- 13, 2011: 2106- 2112.

[13]. Mistassini A, Hasa B, Mahoor M H. AffectN: a database for facial expression, valence, and arousal computing in the wild, 2019, IEEE Trans. Affi. Com., 10(1): 18-31.

[14]. Goodfflow I J, Erhand D, Carrier P L, et al. Challenges in representation learning: a report on three machine learning contests, 2015, Neu. Net., 64: 59-63.

[15]. Cootes T F，Taylo R C J，Coope R D H，et al. Active shape models-their training and application, 1995 Comp. vis. Image. Under. 61(1): 38-59．

[16]. Tie Yun, Guan Ling. A deformable 3-D facial expression model for dynamic human emotional state recognition, 2013, IEEE trans. Cir. Sys. Vid. Tech, 23(1): 142-157．

[17]. Lee T S. Image representation using 2D Gabor wavelets, 1996, IEEE trans. Pat. Anal. Mac. Intel., 18(10): 959-971．

[18]. Zhu Y N, Li X Wu G H. Face expression recognition based on equable principal component analysis and linear regression classification, 2016 Inter. Conf. Sys. Infor., Nov 19-21: 876-880.

[19]. Jiang Bo, Xie Lun, Liu Xin, et al. Microexpression Capture Based on Optical Flow Modulus Estimation, 2017, J. Zhejiang Univ., 51(3): 577-583, 589．

[20]. Ahonet T, Hadida A, PietikInen M. Face recognition with local binary patterns. 2004, Compute. Vis., 469-481．

[21]. Zhang F，Zhang T，mao Q，et al. Joint pose and expression modeling for facial expression recognition 2018, Conf. compute vis. Pat. Rec, 3359-3368.

[22]. Xuchao，Dong C，Feng Zhi, et al. Facial expression pervasive analysis based on Haar-like features and SVM 2012 Berlin Heidelberg: Springer, 521-529

[23]. Viola P, Jones M. Rapid object detection using a boosted cascade of simple feature., 2011 Conf. Compute Vis. Pat. Rec.:511-518.

[24]. Xie Lun, Lu Yannan, Jiang Bo, et al. Automatic Expression Recognition Based on Facial Motion Unit and Expression Relation Model, 2016, J. Beijing Ins. Tech., 36(2): 163-169．

[25]. Gir R, Dong A J, Darr Llt T，et al. Rich feature hierarchies for accurate object detection and semantic segmentation 2014, Conf. compute Vis. Pat. Rec, 580-587.

[26]. Hinton GE, Osinde R S, Teh YW. A fast-learning algorithm for deep belief nets. 2006, Neur. Comput., 18(7): 1527- 1554.

[27]. Gong Qu, Ye Jianying, HUA TaoTao. Facial expression recognition based on improved LBP and LDP, 2013 Compute. Eng. Appl., 49(22):197-200.

[28]. Wang S, Song J, Wang Meng, Wu S, Guan. Multi-feature fusion expression recognition algorithm based on referenced facial expression, 2021, Mod. Elec. Tech., 44(7):77-81.

[29]. Xu Luhui. Facial Expression Recognition Based on the Fusion of ASM Different Texture Features and LDP Features.2015 Guangxi Normal Univ.

[30]. YAO Lisha, XU Guoming, Zhao Feng. Expression Recognition Based on Local Feature Fusion of Convolutional Neural Network, 2017 Conf. Compute Vis. Pat. Rec 3259-3269.

[31]. Chen Xinyi. Research on Multi-modal Fusion Emotion Recognition for Online Learning Scenarios. GuiLin Univ. Tech., 2022.

Cite this article

Chen,X.;Ding,Y.;Li,Z. (2023). Research of the methods on facial expression recognition. Applied and Computational Engineering,6,608-619.

Data availability

The datasets used and/or analyzed during the current study will be available from the authors upon reasonable request.

Disclaimer/Publisher's Note

The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of EWA Publishing and/or the editor(s). EWA Publishing and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

About volume

Volume title: Proceedings of the 3rd International Conference on Signal Processing and Machine Learning

ISBN：978-1-915371-59-1(Print) / 978-1-915371-60-7(Online)
Editor：Omer Burak Istanbullu
Conference website: http://www.confspml.org
Conference date: 25 February 2023
Series: Applied and Computational Engineering
Volume number: Vol.6
ISSN：2755-2721(Print) / 2755-273X(Online)

© 2024 by the author(s). Licensee EWA Publishing, Oxford, UK. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license. Authors who publish this series agree to the following terms:
1. Authors retain copyright and grant the series right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this series.
2. Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the series's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this series.
3. Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See Open access policy for details).

References

[1]. Liu Bowen, Shuai Jianwei, Cao Yuping. Application of Facial expression recognition technology in the diagnosis and treatment of mental diseases. 2021 Chinese J. Be. Med. Br. Sci., 30 (10): 955- 960.

[2]. Lai Dongsheng. Research and Application of Light scale Situation Recognition Algorithm based on Multi-feature fusion. 2022 Guangdong Univ. Tech.

[3]. Xu Xiaokang. Research and Application of Expression Recognition Based on Deep Learning 2022, Donghua Univ.

[4]. Wang Jin, Huang Xiaohua, Li Hang, Hong Jie. Application Research of Microexpression Recognition System in Low resolution Environment. 2022, Compute. Knowle. Tech., 18(20): 81-82+85.

[5]. Lyons M J, Akamatsu S, Kama M, et al. Coding facial expressions with gabor wavelets 1998, Inter. Conf. Face & Gest. Rec., 14-16: 200-205.

[6]. Lucey P, Cohn J F, Kande T, et al. The extended Cohn-Kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression, 2010 Conf. Compute Vis. Pat. Rec., San Francisco, Jun 13-18: 94-101.

[7]. Susskind J M, Anderson A K, Hinton G E. The Toronto face database, 2010, Toronto: Univ. Toronto.

[8]. Zhao G, Huang X, Taini M, et al. Facial expression recognition from near-infrared videos, 2011, Ima. Vis. Comput. 2(9): 607-619.

[9]. Yin L J, Wei X Z, Sun Y, et al. A 3D facial expression database for facial behavior research, 2006 7th IEEE Inter. Conf. Auto. Face Gest. Rec., 211-216.

[10]. Savran A，Ala，Dibeklion H，et al. Bos phorus database for 3D face analysis, 2008 Euro. Biomet. Ident. Man.，Heidelberg：Springer, 47-56.

[11]. Gross R, Matthews I, Conhn J, et al. Multi- PIE, Imag. Vis. Com., 2010, 28(5): 807-813.

[12]. Dhall A,Goecke R, Lucey S, et al. Static facial expression analysis in tough conditions: data, evaluation protocol and benchmark 2011 IEEE Inter. Conf. Comp. Vis., Barcelona, Nov 6- 13, 2011: 2106- 2112.

[13]. Mistassini A, Hasa B, Mahoor M H. AffectN: a database for facial expression, valence, and arousal computing in the wild, 2019, IEEE Trans. Affi. Com., 10(1): 18-31.

[14]. Goodfflow I J, Erhand D, Carrier P L, et al. Challenges in representation learning: a report on three machine learning contests, 2015, Neu. Net., 64: 59-63.

[15]. Cootes T F，Taylo R C J，Coope R D H，et al. Active shape models-their training and application, 1995 Comp. vis. Image. Under. 61(1): 38-59．

[16]. Tie Yun, Guan Ling. A deformable 3-D facial expression model for dynamic human emotional state recognition, 2013, IEEE trans. Cir. Sys. Vid. Tech, 23(1): 142-157．

[17]. Lee T S. Image representation using 2D Gabor wavelets, 1996, IEEE trans. Pat. Anal. Mac. Intel., 18(10): 959-971．

[18]. Zhu Y N, Li X Wu G H. Face expression recognition based on equable principal component analysis and linear regression classification, 2016 Inter. Conf. Sys. Infor., Nov 19-21: 876-880.

[19]. Jiang Bo, Xie Lun, Liu Xin, et al. Microexpression Capture Based on Optical Flow Modulus Estimation, 2017, J. Zhejiang Univ., 51(3): 577-583, 589．

[20]. Ahonet T, Hadida A, PietikInen M. Face recognition with local binary patterns. 2004, Compute. Vis., 469-481．

[21]. Zhang F，Zhang T，mao Q，et al. Joint pose and expression modeling for facial expression recognition 2018, Conf. compute vis. Pat. Rec, 3359-3368.

[22]. Xuchao，Dong C，Feng Zhi, et al. Facial expression pervasive analysis based on Haar-like features and SVM 2012 Berlin Heidelberg: Springer, 521-529

[23]. Viola P, Jones M. Rapid object detection using a boosted cascade of simple feature., 2011 Conf. Compute Vis. Pat. Rec.:511-518.

[24]. Xie Lun, Lu Yannan, Jiang Bo, et al. Automatic Expression Recognition Based on Facial Motion Unit and Expression Relation Model, 2016, J. Beijing Ins. Tech., 36(2): 163-169．

[25]. Gir R, Dong A J, Darr Llt T，et al. Rich feature hierarchies for accurate object detection and semantic segmentation 2014, Conf. compute Vis. Pat. Rec, 580-587.

[26]. Hinton GE, Osinde R S, Teh YW. A fast-learning algorithm for deep belief nets. 2006, Neur. Comput., 18(7): 1527- 1554.

[27]. Gong Qu, Ye Jianying, HUA TaoTao. Facial expression recognition based on improved LBP and LDP, 2013 Compute. Eng. Appl., 49(22):197-200.

[28]. Wang S, Song J, Wang Meng, Wu S, Guan. Multi-feature fusion expression recognition algorithm based on referenced facial expression, 2021, Mod. Elec. Tech., 44(7):77-81.

[29]. Xu Luhui. Facial Expression Recognition Based on the Fusion of ASM Different Texture Features and LDP Features.2015 Guangxi Normal Univ.

[30]. YAO Lisha, XU Guoming, Zhao Feng. Expression Recognition Based on Local Feature Fusion of Convolutional Neural Network, 2017 Conf. Compute Vis. Pat. Rec 3259-3269.

[31]. Chen Xinyi. Research on Multi-modal Fusion Emotion Recognition for Online Learning Scenarios. GuiLin Univ. Tech., 2022.

For authors
Guide for authors Policies for authors Aims and scope Subscription Open access policy
For editors
Policies for editors Editorial board
For reviewers
Policies for reviewers Privacy policy

Cookies are used by this site.
Copyright © 2024 EWA Publishing, its licensors, and contributors. All rights reserved, including text and data mining, AI training, and similar technologies.