1. Introduction

The rapid evolution of deep learning technology has positioned talking head synthesis at the forefront of digital communication innovation [1]. This technology holds immense potential across various domains, including enhancing accessibility for individuals with communicative impairments [2], revolutionizing educational practices through AI-driven interactive tutoring [3], and providing therapeutic support and social interaction in healthcare settings [4].

Talking head synthesis comprises two primary stages: driving mechanisms and portrait rendering. The driving mechanism stage is crucial for imbuing static images with lifelike qualities. These mechanisms utilize either video or audio inputs to orchestrate facial movements and expressions. Video-driven approaches replicate facial movements and expressions from reference footage, while audio-driven methods synchronize lip movements and facial expressions with corresponding audio tracks. The portrait rendering stage focuses on generating photorealistic facial images using advanced generative models. These include Generative Adversarial Networks (GANs) [5], Diffusion Models [6], Neural Radiance Fields (NeRF) [7], and 3D Gaussian Splatting (3DGS) [8]. These models excel in producing high-resolution static images, which serve as the foundation for subsequent animation processes.

This paper explores the primary mechanisms driving talking head synthesis, categorized into video-driven and audio-driven methods., highlighting the unique contributions and strengths of various approaches. Additionally, the paper discusses the advantages, limitations, existing challenges, and potential solutions for future research. Future research directions focusing on overcoming challenges like handling large-angle poses, ensuring temporal consistency, and improving multilingual performance are elucidated.

2. Methods of driven mechanisms and portrait rendering

Driven mechanisms are essential in pushing the boundaries of talking head synthesis, facilitating the generation of realistic and expressive talking head videos. By utilizing different input modalities, these mechanisms effectively control the facial movements and expressions of synthesized characters, ensuring that the output appears natural and coherent. The approaches in this field are generally categorized into two main types: video-driven methods and audio-driven methods.

2.1. Video-driven Methods

Video-driven talking head synthesis involves manipulating defined features such as keypoints, meshes, and latent spaces, making these methods highly interpretable.

Keypoints-based warping methods generate motion flow by learning the correspondence between keypoints, thereby warping the features of the source image. Siarohin et al. developed the First Order Motion Model (FOMM), a method that animates objects in images using a set of learned keypoints along with their local affine transformations, generating a dense motion field and occlusion masks to deform the source image at the encoder’s feature layer and restore it through the decoder [9]. Zhao et al. improved upon FOMM by using multi-resolution occlusion masks to achieve more effective feature fusion and incorporating an affine transformation for background prediction [10]. Hong et al. further enhanced FOMM by utilizing depth information to improve the precision of warps and reduce artifacts [11]. Hong & Xu propose a novel implicit identity representation conditioned memory compensation network for talking head video generation, in which an implicit identity representation conditioned memory module and a facial memory compensation module are designed to respectively perform the meta-memory query and feature compensation [12]. Wang et al. introduced dynamic 3D keypoints for unsupervised learning from single-source images, enabling one-shot free-view avatar synthesis [13]. Drobyshev et al. utilize dynamic 3D keypoints and propose a novel contrastive loss to achieve higher degrees of disentanglement between the latent motion and appearance representations, adding a gaze loss that increases the realism and accuracy of eye animation [14]. Drobyshev et al. improve MegaPortraits model to transfer intense expressions correctly through careful latent facial expression space development and employ the expression-enhanced loss and a minimal amount of domain-specific data [15]. Guo et al. introduce a series of significant enhancements including high-quality data curation, a mixed image and video training strategy, an upgraded network architecture, scalable motion transformation, landmark-guided implicit keypoints optimization, and cascaded loss terms [16].

In contrast to keypoints-based methods, mesh-based rendering methods depend on 3D head reconstruction models. 3DMM models like DeepFaceReconstruction[17], DECA [18], Emoca [19] are employed. Yao et al. proposed a method that uses 3D mesh reconstruction to guide optical flow learning for facial reenactment[20]. Guan et al. proposed a new Style-based generator using 3D mesh and re-configure the information insertion mechanisms within the noise and style space [21]. Hong et al. presented HeadNeRF, a parametric head model using NeRFs to render high-fidelity human head images, incorporating 2D neural rendering and improving detail accuracy [22]. Xu et al. utilized controllable 3D Gaussian models for high-fidelity head avatar modeling. By incorporating high-frequency dynamic details through a fully learned MLP-based deformation field, it effectively simulates a wide range of extreme expressions [23].

Latent space-based methods represent images as embedding codes. Wang et al. proposed the LIA model, achieving animation by learning orthogonal motion directions within the latent space and linearly combining them, avoiding complex processing based on structural representation [24]. Liu et al. enhances motion representation by employing metric learning, mutual information disentanglement, and Hierarchical Aggregation Layer [25]. Li et al. introduced HiDe-NeRF, which decomposes 3D dynamic scenes into canonical appearance and implicit deformation fields, maintaining facial shape and details through multi-scale volumetric features [26].

Video-driven methods have advanced the generation of realistic talking head animations by preserving the identity of a still image while replicating the motion from a driving video. However, they face challenges such as occlusion, maintaining identity integrity, and handling large pose variations.

2.2. Audio-driven Methods

Audio-driven methods leverage audio signals to synchronize lip movements and facial expressions with spoken content, thereby enhancing the realism and naturalness of talking head animations. This multimodal task faces several challenges due to the inherent differences between audio and visual modalities, such as lip synchronization. To address these challenges, existing approaches are classified into two main categories: explicit-based and implicit-based.

Implicit-based methods represent audio as latent space-based features. Prajwal et al. propose to use a pre-trained expert lip-sync discriminator to penalize the generator for inaccurate generation [27]. Liu et al. utilized a combination of diffusion and variance adapters to predict motion latent [25]. Xu et al. proposed diffusion transformer to model the motion distribution and generate the motion latent codes in the test time given audio and other conditions [28].

Explicit-based represent audio as 3D facial mesh and 3DMM parameters. Fan et al. leveraged self-supervised pre-trained speech representations and a transformer-based autoregressive model with a biased attention mechanism to capture long-term audio context, aligning audio-motion modalities, and accurately forecasting animated 3D facial mesh sequences for enhanced lip movement precision [29]. Huang et al. used a transformer-based encoder to map audio signals to 3DMM parameters, guiding the generation of high-quality avatars by predicting long-term audio context [30]. Zhang et al. introduce SadTalker, which synthesizes 3D motion coefficients from audio inputs and integrates them with advanced 3D perception rendering technology to accurately map audio-motion correlations, capturing detailed facial expressions and head movements [31]. Cho et al. proposed GaussianTalker, a real-time pose-controllable talking head model that integrates 3D Gaussian attributes with audio features into a shared implicit feature space, leveraging 3DGS for rapid rendering, resulting in enhanced facial fidelity, lip-sync accuracy, and superior rendering speed compared to existing models [32].

Audio-driven methods heavily rely on high-quality training data, which can cause overfitting or poor generalization when the data is insufficient or biased. Moreover, their stochastic nature often leads to inconsistent animations, particularly over longer sequences.

3. Applications

Talking head synthesis technology has a wide range of applications across various sectors, revolutionizing how we interact with digital content. This section explores some of the most promising application areas where talking head synthesis is making significant impacts.

Smart Customer Service: Talking head synthesis revolutionizes customer service by deploying virtual assistants that provide consistent, empathetic interactions. These digital agents can handle customer queries with lifelike expressions and real-time responses, improving customer satisfaction and operational efficiency.

Online Education: Talking head generation enhances online education by creating engaging virtual tutors that deliver interactive lessons with realistic facial expressions and real-time responses. This personalized approach improves student engagement, making complex concepts more accessible and enjoyable, particularly in remote and self-paced learning environments.

Telemedicine: In telemedicine, talking head technology provides virtual medical professionals that interact empathetically with patients. These lifelike avatars can guide patients through medical consultations, providing a human touch in remote diagnostics and care, enhancing patient comfort and trust.

Video Creation: Talking head technology streamlines short video content creation, allowing creators to produce engaging, lifelike characters without needing live actors. This opens up new creative possibilities for influencers and brands, reducing production time and costs while maintaining high-quality visual appeal. Sections, subsections and subsubsections

The use of sections to divide the text of the paper is optional and left as a decision for the author. Where the author wishes to divide the paper into sections the formatting shown in table 2 should be used.

4. Challenges and Future Directions

4.1. Pretrained large models

Current frameworks for talking head synthesis often rely on pretrained large models, such as diffusion models, which are designed with highly specific capabilities tailored to particular tasks. While these models offer impressive performance in certain areas, their rigid architecture can hinder innovation and adaptability for more diverse or specialized use cases. Over-reliance on a single pretrained model restricts flexibility and may result in models that struggle to generalize to new tasks or novel domains. A promising avenue for future research is to explore training individual components or modules on large-scale, diverse datasets. This approach allows for more modular and adaptable architectures that can be combined or fine-tuned to suit specific tasks without being overly dependent on one core model. Furthermore, hybrid architectures that integrate pretrained models with task-specific layers or modules can offer the benefits of both pretrained expertise and targeted adaptability, potentially leading to more efficient and versatile systems for a wide range of applications.

4.2. Handling Large-Angle Poses

Large-angle poses present a significant challenge in talking head synthesis due to the limited availability of training data for extreme angles, often leading to suboptimal performance. These angles introduce substantial changes in facial geometry and can obscure parts of the face, compounding the issue. To mitigate this, expanding datasets to include a wider range of large-angle poses, as well as incorporating multi-view training strategies, can help models handle such poses more effectively.

4.3. Temporal consistency

One of the most important challenges in talking head synthesis is maintaining temporal consistency, particularly when producing video outputs that need to look smooth and natural. Inadequate handling of time-series data can result in visual discontinuities such as jitter, sudden jumps between frames, or unnatural changes in facial expressions and movements. These artifacts disrupt the fluidity of the synthesized video, reducing its overall realism. The key to addressing this issue lies in both the quality of the datasets used and the methods employed for processing sequential data. High-quality datasets with strong frame-to-frame continuity are essential to train models that can accurately capture the natural flow of movements over time. Additionally, developing improved algorithms that better account for temporal dependencies in time-series data is crucial. Future research should prioritize the integration of advanced temporal processing techniques, such as recurrent neural networks (RNNs) or temporal attention mechanisms, to enhance the model's ability to produce consistent, lifelike sequences across extended periods of time.

4.4. Multilingual Challenges

Most existing audio-driven talking head synthesis models are primarily designed and optimized for English, creating challenges when extending the technology to other languages. This limitation stems from a lack of annotated, high-quality datasets for a wide range of languages, especially those with different phonetic structures, lip movements, or cultural nuances. When models are trained exclusively on English data, their ability to accurately generate lip movements, expressions, and speech dynamics in other languages is significantly diminished, leading to less accurate and natural outputs. Addressing this issue requires the collection and curation of diverse multilingual datasets, which capture the unique linguistic and phonetic features of different languages. Furthermore, employing advanced techniques like self-supervised learning, where models can learn useful representations from unlabelled data, and transfer learning, where knowledge gained from English can be adapted to other languages, can help improve the performance and flexibility of talking head synthesis across various linguistic contexts. These approaches can enhance the scalability and global applicability of such systems, making them more inclusive and adaptable to a wider audience.

5. Conclusion

This review provided an in-depth analysis of talking head synthesis, examining both video-driven and audio-driven mechanisms and their impact on generating realistic head animations. While significant progress has been made in improving facial movement accuracy and synchronization, challenges remain, particularly in managing large-angle poses, maintaining temporal consistency, and expanding multilingual support. Applications of this technology are already transforming fields such as customer service, education, and telemedicine. Future research should focus on creating more adaptable models, expanding datasets, and developing hybrid architectures to overcome current limitations and enhance the versatility and realism of talking head synthesis.