Face Modeling Based on Deep Learning and Traditional Methods: A Survey

Yuqian Zhang

doi:10.54254/2755-2721/81/20241149

Research Article

Open access

Published on 8 November 2024

Download pdf

Zhang,Y. (2024). Face Modeling Based on Deep Learning and Traditional Methods: A Survey. Applied and Computational Engineering,81,190-209.

Export citation

Face Modeling Based on Deep Learning and Traditional Methods: A Survey

Yuqian Zhang *^,1,

¹ Shijiazhuang Tiedao University, Shijiazhuang, China

* Author to whom correspondence should be addressed.

https://doi.org/10.54254/2755-2721/81/20241149

Abstract

3D face modeling, as an important research field in computer graphics and vision, has undergone remarkable development in recent years. The technology is widely used in various scenarios, like film and television production, virtual reality, augmented reality, facial recognition, security monitoring, and medical diagnosis. This paper aims to systematically explore the main technologies of 3D face modeling, especially the application and development of deep learning methods in it, review traditional face modeling methods, introduce in detail the latest technical advances based on deep learning, and analyze the advantages and disadvantages of these methods in practical applications.

Keywords

3D facial modeling, deep learning methods, facial recognition, Virtual Reality.

View pdf

References

[1]. Zhongjin Luo, Dong Du, Heming Zhu, Yizhou Yu, Hongbo Fu, Xiaoguang Han: SketchMetaFace: A Learning-Based Sketching Interface for High-Fidelity 3D Character Face Modeling. IEEE Transactions on Visualization and Computer Graphics ( Volume: 30, Issue: 8, August 2024).

[2]. Yajie Gu, Nick E. Pears: Parts-Based Implicit 3D Face Modeling. Proceedings of the 19th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 2 VISAPP: VISAPP, 201-212, 2024 , Rome, Italy.

[3]. Guohao Li, Hongyu Yang, Di Huang, Yunhong Wang:3D Face Modeling via Weakly-supervised Disentanglement Network joint Identity-consistency Prior. 2024 IEEE 18th International Conference on Automatic Face and Gesture Recognition (FG) (2024): 1-10.

[4]. Zhenyu Zhang, Renwang Chen, Weijian Cao, Ying Tai, Chengjie Wang: Learning Neural Proto-Face Field for Disentangled 3D Face Modeling in the Wild. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada, 2023, pp. 382-393, doi: 10.1109/CVPR52729.2023.00045.

[5]. Kai Yang, Hong Shang, Tianyang Shi, Xinghan Chen, Jingkai Zhou, Zhongqian Sun, Wei Yang: ASM: Adaptive Skinning Model for High-Quality 3D Face Modeling. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), Paris, France, 2023, pp. 20651-20660, doi: 10.1109/ICCV51070.2023.01893.

[6]. Stathis Galanakis, Baris Gecer, Alexandros Lattas, Stefanos Zafeiriou: 3DMM-RF: Convolutional Radiance Fields for 3D Face Modeling. 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), Waikoloa, HI, USA, 2023, pp. 3525-3536, doi: 10.1109/WACV56688.2023.00353.

[7]. Andrew Z. Hou, Feng Liu, Zhiyuan Ren, Michel Sarkis, Ning Bi, Yiying Tong, Xiaoming Liu: INFAMOUS-NeRF: ImproviNg FAce MOdeling Using Semantically-Aligned Hypernetworks with Neural Radiance Fields. ArXiv abs/2312.16197 (2023): n. pag.

[8]. Yuta Kawanaka, Syuhei Sato, Kaisei Sakurai, Shangce Gao, Zheng Tang: Human Face Modeling based on Deep Learning through Line-drawing. 2022 The Eurographics Association, doi:10.2312/pg.20221239.

[9]. Ziyang Weng, Ziyu Zhang, Yinger Liang, Biyi Dai: Generalized Perceptual Modeling: Virtual Human Face Modeling and Expression Recognition Considering Emotion Understanding. 2022 IEEE 22nd International Conference on Software Quality, Reliability, and Security Companion (QRS-C), Guangzhou, China, 2022, pp. 417-424, doi: 10.1109/QRS-C57518.2022.00067.

[10]. Fengquan Zhang, Pingzhe Li, Yahui Gao, Liuqing Xu, Duo Cao: A real-time personalized face modeling method for peking opera with depth vision device. Computers & Electrical Engineering, Volume 93, 2021, 107236.

[11]. Cheng Di, Jing Peng, Yihua Di, Siwei Wu: 3D Face Modeling Algorithm for Film and Television Animation Based on Lightweight Convolutional Neural Network. Complexity, 2021, 6752120, 10 pages.

[12]. Feng-Ju Chang, Anh Tuan Tran, Tal Hassner, Iacopo Masi, Ram Nevatia, Gérard G. Medioni: Deep, Landmark-Free FAME: Face Alignment, Modeling, and Expression Estimation. International Journal of Computer Vision 127, 930–956 (2019). https://doi.org/10.1007/s11263-019-01151-x.

[13]. Chi Nhan Duong, Khoa Luu, Kha Gia Quach, Tien D. Bui: Deep Appearance Models: A Deep Boltzmann Machine Approach for Face Modeling. International Journal of Computer Vision 127, 437–455 (2019). https://doi.org/10.1007/s11263-018-1113-3.

[14]. F. Liu, L. Tran and X. Liu: 3D Face Modeling from Diverse Raw Scan Data. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Korea (South), 2019, pp. 9407-9417, doi: 10.1109/ICCV.2019.00950.

[15]. Son Thai Ly, Nhu-Tai Do, Gueesang Lee, Soo-Hyung Kim, Hyung-Jeong Yang: A 3d Face Modeling Approach for in-The-Wild Facial Expression Recognition on Image Datasets. 2019 IEEE International Conference on Image Processing (ICIP), Taipei, Taiwan, 2019, pp. 3492-3496, doi: 10.1109/ICIP.2019.8803434.

[16]. Xiaoguang Han, Chang Gao, and Yizhou Yu: DeepSketch2Face: a deep learning based sketching system for 3D face and caricature modeling. AACM Transactions on Graphics (TOG), Volume 36, Issue 4, Article 126 (August 2017), 12 pages. https://doi.org/10.1145/3072959.3073629.

[17]. Chi Nhan Duong; Khoa Luu; Kha Gia Quach; Tien D. Bui, "Longitudinal Face Modeling via Temporal Deep Restricted Boltzmann Machines," 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 2016, pp. 5772-5780, doi: 10.1109/CVPR.2016.622.

[18]. Donghyun Kim; Jongmoo Choi; Jatuporn Toy Leksut; Gérard Medioni: "Accurate 3D face modeling and recognition from RGB-D stream in the presence of large pose changes," 2016 IEEE International Conference on Image Processing (ICIP), Phoenix, AZ, USA, 2016, pp. 3011-3015, doi: 10.1109/ICIP.2016.7532912.

[19]. Donghyun Kim; Jongmoo Choi; Jatuporn Toy Leksut; Gérard Medioni: "Accurate 3D face modeling and recognition from RGB-D stream in the presence of large pose changes," 2016 IEEE International Conference on Image Processing (ICIP), Phoenix, AZ, USA, 2016, pp. 3011-3015, doi: 10.1109/ICIP.2016.7532912.

[20]. Sucontphunt, Tanasai, A Practical Approach for Identity-Embodied 3D Artistic Face Modeling, International Journal of Computer Games Technology, 2014, 781950, 10 pages, 2014. https://doi.org/10.1155/2014/781950.

[21]. Vytautas Kaminskas, Egidijus Vaškevičius, Aušra Vidugirienė, Modeling Human Emotions as Reactions to a Dynamical Virtual 3D Face, Informatica 25(2014), no. 3, 425-437, DOI 10.15388/Informatica.2014.22.

[22]. Noranart Vesdapunt, Mitch Rundle, HsiangTao Wu, Baoyuan Wang: JNR: Joint-based Neural Rig Representation for Compact 3D Face Modeling. Computer Vision – ECCV 2020. ECCV 2020. Lecture Notes in Computer Science(), vol 12363. Springer, Cham. https://doi.org/10.1007/978-3-030-58523-5_23.

[23]. Bindita Chaudhuri, Noranart Vesdapunt, Linda G. Shapiro, Baoyuan Wang: Personalized Face Modeling for Improved Face Reconstruction and Motion Retargeting. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, JM. (eds) Computer Vision – ECCV 2020. ECCV 2020. Lecture Notes in Computer Science(), vol 12350. Springer, Cham. https://doi.org/10.1007/978-3-030-58558-7_9.

[24]. Changwei Luo, Juyong Zhang, Jun Yu, Chang Wen Chen, Shengjin Wang: Real-Time Head Pose Estimation and Face Modeling from a Depth Image. IEEE Transactions on Multimedia, vol. 21, no. 10, pp. 2473-2481, Oct. 2019, doi: 10.1109/TMM.2019.2903724.

[25]. Shan Liu, Guanghong Gong, Luhao Xiao, Mengyuan Sun, Zhengliang Zhu: Study of rapid face modeling technology based on Kinect. International Journal of Modeling, Simulation, and Scientific Computing Vol. 09, No. 01, 1750054 (2018).

[26]. Weilong Peng, Yong Su, Keke Tang, Chao Xu, Zhiyong Feng, Meie Fang: Geometrical Consistency Modeling on B-Spline Parameter Domain for 3D Face Reconstruction From Limited Number of Wild Images. Frontiers Neurorobotics 15: 652562 (2021).

[27]. Fuqing Duan, Donghua Huang, Yun Tian, Ke Lu, Zhongke Wu, Mingquan Zhou, 3D face reconstruction from skull by regression modeling in shape parameter spaces, Neurocomputing, Volume 151, Part 2, 2015, Pages 674-682, https://doi.org/10.1016/j.neucom.2014.04.089.

[28]. Yu, Z., Ma, G., & Huang, Q. (2014). Modeling and design of a humanoid robotic face based on an active drive points model. Advanced Robotics, 28(6), 379–388. https://doi.org/10.1080/01691864.2013.867290.

[29]. Chun-Yang Tseng; I-Jan Wang; Chih-Hsing Chu: "Parametric modeling of 3D human faces using anthropometric data," 2014 IEEE International Conference on Industrial Engineering and Engineering Management, Selangor, Malaysia, 2014, pp. 491-495, doi: 10.1109/IEEM.2014.7058686.

[30]. Muhammad Aurangzeb Khan; Costas Xydeas; Hassan Ahmed, "Multi-Component/Multi-Model AAM framework for face image modeling," 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, Vancouver, BC, Canada, 2013, pp. 2124-2128, doi: 10.1109/ICASSP.2013.6638029.

[31]. Yan Luximon, Roger Ball, Lorraine Justice, The 3D Chinese head and face modeling, Computer-Aided Design, Volume 44, Issue 1, 2012, Pages 40-47, ISSN 0010-4485, https://doi.org/10.1016/j.cad.2011.01.011.

[32]. Xin Geng; Kate Smith-Miles; Zhi-Hua Zhou; Liang Wang, "Face Image Modeling by Multilinear Subspace Analysis with Missing Values," in IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), vol. 41, no. 3, pp. 881-892, June 2011, doi: 10.1109/TSMCB.2010.2097588.

[33]. Song, Jg., Kim, S. (2011). Mapping Based on Three Cameras for 3D Face Modeling. In: Kim, Th., Adeli, H., Robles, R.J., Balitanas, M. (eds) Advanced Communication and Networking. ACN 2011. Communications in Computer and Information Science, vol 199. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-23312-8_5.

[34]. Ying Zheng, Stan Z. Li, Jianglong Chang, and Zengfu Wang: 3D modeling of faces from near infrared images using statistical learning. International Journal of Pattern Recognition and Artificial Intelligence 2010 24:01, 55-71.

[35]. Alexandru-Eugen Ichim, Petr Kadleček, Ladislav Kavan, and Mark Pauly. 2017.Phace: physics-based face modeling and animation. ACM Transactions on Graphics (TOG), Volume 36, Issue 4 Article No.:153, Pages 1 – 14 (August 2017). https://doi.org/10.1145/3072959.3073664.

[36]. Wei Jiang; Zhengxing Sun (2010). Sketch-Based 3D Face Modeling for Virtual Character. In: Pan, Z., Cheok, A.D., Müller, W., Zhang, X., Wong, K. (eds) Transactions on Edutainment IV. Lecture Notes in Computer Science, vol 6250. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-14484-4_18.

[37]. Zhenyu Zhang, Yanhao Ge, Ying Tai, Weijian Cao, Renwang Chen, Kunlin Liu, Hao Tang, Xiaoming Huang, Chengjie Wang, Zhifeng Xie, Dongjin Huang: Physically-guided Disentangled Implicit Rendering for 3D Face Modeling. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, 2022, pp.20321-20331,doi:10.1109/CVPR52688.2022.01971.

[38]. Weilong Peng, Chao Xu, Zhiyong Feng: 3D face modeling based on structure optimization and surface reconstruction with B-Spline, Neurocomputing, Volume 179, 2016, Pages 228-237, https://doi.org/10.1016/j.neucom.2015.11.090.

[39]. Dan Zhang, Chenlei Lv, Na Liu, Zhongke Wu, Xingce Wang: 3D face modeling from single image based on discrete shape space. Computer Animation and Virtual Worlds. 2020; 31:e1943. https://doi.org/10.1002/cav.1943.

[40]. Abdallah Dib, Gaurav Bharaj, Junghyun Ahn, Cédric Thébault, Philippe-Henri Gosselin, Louis Chevallier: Face Reflectance and Geometry Modeling via Differentiable Ray Tracing. ArXiv abs/1910.05200 (2019): n. pag.

[41]. Hai Jin, Xun Wang, Zichun Zhong, Jing Hua: Robust 3D face modeling and reconstruction from frontal and side images. Computer Aided Geometric Design,Volume 50,2017,Pages 1-13.

[42]. Hu Han; Anil K. Jain, "3D face texture modeling from uncalibrated frontal and profile images," 2012 IEEE Fifth International Conference on Biometrics: Theory, Applications and Systems (BTAS), Arlington, VA, USA, 2012, pp. 223-230, doi: 10.1109/BTAS.2012.6374581.

[43]. Li Li, Fei Liu, Congbo Li, Guoan Chen, Realistic wrinkle generation for 3D face modeling based on automatically extracted curves and improved shape control functions, Computers & Graphics, Volume 35, Issue 1,2011,Pages 175-184,ISSN 0097-8493.

[44]. Jingu Heo; Marios Savvides, "Rapid 3D face modeling using a frontal face and a profile face for accurate 2D pose synthesis," 2011 IEEE International Conference on Automatic Face & Gesture Recognition (FG), Santa Barbara, CA, USA, 2011, pp. 632-638, doi: 10.1109/FG.2011.5771470.

[45]. Yong Sun Kim et al., "Realistic 3D face modeling using feature-preserving surface registration," 2010 IEEE International Conference on Image Processing, Hong Kong, China, 2010, pp. 1821-1824, doi: 10.1109/ICIP.2010.5650893.

[46]. S. Chung, J. -C. Bazin and I. Kweon, "2D/3D virtual face modeling," 2011 18th IEEE International Conference on Image Processing, Brussels, Belgium, 2011, pp. 1097-1100, doi: 10.1109/ICIP.2011.6115617.

[47]. Hongjun Zhao, Bin Wu: Three-dimensional face modeling technology based on 5G virtual reality binocular stereo vision. International Journal of Communication Systems. 2022; 35(5):e4651. doi:10.1002/dac.4651.

[48]. Hong Song,Jie Lv,Yanming Wang: Rapid 3D Face Modeling from Video. In: Ho, YS., Sang, J., Ro, Y., Kim, J., Wu, F. (eds) Advances in Multimedia Information Processing -- PCM 2015. PCM 2015. Lecture Notes in Computer Science(), vol 9315. Springer, Cham. https://doi.org/10.1007/978-3-319-24078-7_20.

[49]. Nguyen, Hoang Thanh (2012). Individual 3D Face Modeling and Recognition in a Video Network. UC Riverside. ProQuest ID: Nguyen_ucr_0032D_11113. Merritt ID: ark:/13030/m5wd43h2. https://escholarship.org/uc/item/4wj6q3mp.

[50]. Chengchao Qu, Hua Gao & Hazım Kemal Ekenel: Rotation update on manifold in probabilistic NRSFM for robust 3D face modeling. EURASIP Journal on Image and Video Processing. 2015, 45 (2015). https://doi.org/10.1186/s13640-015-0101-6.

[51]. Vandana D. Kaushik, V. K. Pathak, and P. Gupta, " Geometric Modeling of 3D-Face Features and Its Applications," Journal of Computers vol. 5, no. 9, pp. 1305-1314, 2010.

[52]. Chih-Hsing Chu, I-Jan Wang, Jeng-Bang Wang, Yuan-Ping Luh: 3D parametric human face modeling for personalized product design: Eyeglasses frame design case. Advanced Engineering Informatics, Volume 32, 2017, 202-223.

[53]. Hwang Jinkyu, Sunjin Yu, Joongrock Kim, and Sangyoun Lee. 2012. "3D Face Modeling Using the Multi-Deformable Method" Sensors 12, no. 10: 12870-12889. https://doi.org/10.3390/s121012870.

[54]. Shaun Canavan; Xing Zhang; Lijun Yin; Yong Zhang, "3D face sketch modeling and assessment for Component based face recognition," 2011 International Joint Conference on Biometrics (IJCB), Washington, DC, USA, 2011, pp. 1-6, doi: 10.1109/IJCB.2011.6117501.

[55]. Changwei Luo, Juyong Zhang, Changcun Bao, Yali Li, Jing Huang, Shengjin Wang: Robust 3D face modeling and tracking from RGB-D images. Multimedia Systems 28, 1657–1666 (2022).https://doi.org/10.1007/s00530-022-00925-7.

[56]. Jia Li; Yue Zhang; Pin Xu; Shanzhen Lan; Shaobin Li: 3D Personalized Face Modeling Based on KINECT2. 2016 9th International Symposium on Computational Intelligence and Design (ISCID), Hangzhou, China, 2016, pp. 197-201, doi: 10.1109/ISCID.2016.1052.

[57]. Yanhui Huang; Yangyu Fan; Weihua Liu; Liangliang Wang: "3D human face modeling for facial animation using regional adaptation," 2014 IEEE China Summit & International Conference on Signal and Information Processing (ChinaSIP), Xi'an, China, 2014, pp. 394-398, doi: 10.1109/ChinaSIP.2014.6889271.

[58]. Pavan Kumar Anasosalu; Diego Thomas; Akihiro Sugimoto, "Compact and Accurate 3-D Face Modeling Using an RGB-D Camera: Let's Open the Door to 3-D Video Conference," 2013 IEEE International Conference on Computer Vision Workshops, Sydney, NSW, Australia, 2013, pp. 67-74, doi: 10.1109/ICCVW.2013.16.

[59]. Qi Sun, Yanlong Tang, Ping Hu, "Modeling 3D faces from samplings via compressive sensing," Proc. SPIE 8878, Fifth International Conference on Digital Image Processing (ICDIP 2013), 88781X (19 July 2013); https://doi.org/10.1117/12.2030533.

[60]. Gregory P. Meyer; Minh N. Do, "Real-time 3D face modeling with a commodity depth camera," 2013 IEEE International Conference on Multimedia and Expo Workshops (ICMEW), San Jose, CA, USA, 2013, pp. 1-4, doi: 10.1109/ICMEW.2013.6618317.

[61]. Matthias Hernandez; Jongmoo Choi; Gérard Medioni, "Laser scan quality 3-D face modeling using a low-cost depth camera," 2012 Proceedings of the 20th European Signal Processing Conference (EUSIPCO), Bucharest, Romania, 2012, pp. 1995-1999.

[62]. Jongmoo Choi; Yann Dumortier; Sang-Il Choi; Muhammad Bilal Ahmad; Gérard Medioni, "Real-time 3-D face tracking and modeling from a webcam," 2012 IEEE Workshop on the Applications of Computer Vision (WACV), Breckenridge, CO, USA, 2012, pp. 33-40, doi: 10.1109/WACV.2012.6163031.

[63]. Won Beom Lee; Man Hee Lee; In Kyu Park, "Photorealistic 3D face modeling on a smartphone," CVPR 2011 WORKSHOPS, Colorado Springs, CO, USA, 2011, pp. 163-168, doi: 10.1109/CVPRW.2011.5981841.

Cite this article

Zhang,Y. (2024). Face Modeling Based on Deep Learning and Traditional Methods: A Survey. Applied and Computational Engineering,81,190-209.

Data availability

The datasets used and/or analyzed during the current study will be available from the authors upon reasonable request.

Disclaimer/Publisher's Note

The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of EWA Publishing and/or the editor(s). EWA Publishing and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

About volume

Volume title: Proceedings of the 2nd International Conference on Machine Learning and Automation

Conference website: https://2024.confmla.org/

ISBN：978-1-83558-563-4(Print) / 978-1-83558-564-1(Online)

Conference date: 21 November 2024

Editor：Mustafa ISTANBULLU, Xinqing Xiao

Series: Applied and Computational Engineering

Volume number: Vol.81

ISSN：2755-2721(Print) / 2755-273X(Online)

© 2024 by the author(s). Licensee EWA Publishing, Oxford, UK. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license. Authors who publish this series agree to the following terms:
1. Authors retain copyright and grant the series right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this series.
2. Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the series's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this series.
3. Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See Open access policy for details).