Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/Curated-Awesome-Lists/awesome-ai-talking-heads

A curated list of 'Talking Head Generation' resources. Features influential papers, groundbreaking algorithms, crucial GitHub repositories, insightful videos, and more. Ideal for AI enthusiasts, researchers, and graphics professionals
https://github.com/Curated-Awesome-Lists/awesome-ai-talking-heads

List: awesome-ai-talking-heads

ai-art awesome-list machine-learning state-of-the-art talking-face-generation talking-head-videos talking-heads

Last synced: 3 months ago
JSON representation

A curated list of 'Talking Head Generation' resources. Features influential papers, groundbreaking algorithms, crucial GitHub repositories, insightful videos, and more. Ideal for AI enthusiasts, researchers, and graphics professionals

Awesome Lists containing this project

README 

        
# Awesome talking-head



Welcome to the Awesome List for Talking Head Generation! This curated collection of resources focuses on the intriguing domain of 'Talking Head Generation' - an area of computer graphics and artificial intelligence that strives to create lifelike digital recreations of human heads and faces. These 'talking heads' can be used in a variety of applications, from realistic video content and virtual reality, to advanced communication tools and beyond. This list aims to gather key research papers, state-of-the-art algorithms, seminal GitHub repositories, educational videos, inspiring blogs, and more. Whether you are an AI researcher, computer graphics professional, or an AI enthusiast, this list is your one-stop destination to dive into the world of Talking Head Generation. Happy exploring!



## Table of Contents



- [GitHub projects](#github-projects)

- [Articles & Blogs](#articles-&-blogs)

- [Online Courses](#online-courses)

- [Research Papers](#research-papers)

- [Tools & Software](#tools-&-software)

- [Slides & Presentations](#slides-&-presentations)



## GitHub projects



- [AudioGPT](https://github.com/AIGC-Audio/AudioGPT) : Understanding and Generating Speech, Music, Sound, and Talking Head. 🗣️🎵

- [SadTalker](https://github.com/OpenTalker/SadTalker) : Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animation. 🎭🎶

- [Thin-Plate-Spline-Motion-Model](https://github.com/yoyo-nb/Thin-Plate-Spline-Motion-Model) : Thin-Plate Spline Motion Model for Image Animation. 🖼️

- [GeneFace](https://github.com/yerfor/GeneFace) : Generalized and High-Fidelity 3D Talking Face Synthesis; ICLR 2023; Official code. 👤💬

- [CVPR2022-DaGAN](https://github.com/harlanhong/CVPR2022-DaGAN) : Official code for CVPR2022 paper: Depth-Aware Generative Adversarial Network for Talking Head Video Generation. 👥📹

- [sd-wav2lip-uhq](https://github.com/numz/sd-wav2lip-uhq) : Wav2Lip UHQ extension for Automatic. 👄

- [Text2Video](https://github.com/sibozhang/Text2Video) : ICASSP 2022: "Text2Video: text-driven talking-head video synthesis with phonetic dictionary". 🔤🎞️

- [OTAvatar](https://github.com/theEricMa/OTAvatar) : This is the official repository for OTAvatar: One-shot Talking Face Avatar with Controllable Tri-plane Rendering [CVPR2023]. 👤🎭

- [Audio2Head](https://github.com/wangsuzhen/Audio2Head) : Code for paper "Audio2Head: Audio-driven One-shot Talking-head Generation with Natural Head Motion" in the conference of IJCAI 2021. 🗣️👤

* [IP_LAP](https://github.com/Weizhi-Zhong/IP_LAP) : CVPR2023 talking face implementation for Identity-Preserving Talking Face Generation With Landmark and Appearance Priors. 🔥🤖

* [Wunjo AI](https://github.com/wladradchenko/wunjo.wladradchenko.ru) : Synthesize & clone voices in English, Russian & Chinese, real-time speech recognition, deepfake face & lips animation, face swap with one photo, change video by text prompts, segmentation, and retouching. Open-source, local & free. 🗣️👤💬

* [LIHQ](https://github.com/johnGettings/LIHQ) : Long-Inference, High Quality Synthetic Speaker (AI avatar/ AI presenter). 🎙️👤

* [Co-Speech-Motion-Generation](https://github.com/TheTempAccount/Co-Speech-Motion-Generation) : Freeform Body Motion Generation from Speech. 🗣️🚶

* [Neural Head Reenactment with Latent Pose Descriptors](https://github.com/shrubb/latent-pose-reenactment) : The authors' implementation of the "Neural Head Reenactment with Latent Pose Descriptors" (CVPR 2020) paper. 🤖👤

* [NED](https://github.com/foivospar/NED) : PyTorch implementation for NED (CVPR 2022). It can be used to manipulate the facial emotions of actors in videos based on emotion labels or reference styles. 😃🎭🎥

* [WACV23_TSNet](https://github.com/nihaomiao/WACV23_TSNet) : The pytorch implementation of our WACV23 paper "Cross-identity Video Motion Retargeting with Joint Transformation and Synthesis". 🎬✨

* [ICCV2023-MCNET](https://github.com/harlanhong/ICCV2023-MCNET) : The official code of our ICCV2023 work: Implicit Identity Representation Conditioned Memory Compensation Network for Talking Head video Generation. 🎥🤖

* [Speech2Video](https://github.com/sibozhang/Speech2Video) : Code for ACCV 2020 "Speech2Video Synthesis with 3D Skeleton Regularization and Expressive Body Poses". 🗣️🎥💃

* [StyleLipSync](https://github.com/TaekyungKi/StyleLipSync) : Official pytorch implementation of "StyleLipSync: Style-based Personalized Lip-sync Video Generation". 💋🎥



## Articles & Blogs



- [How to Create Fake Talking Head Videos With Deep Learning (Code Tutorial)](https://medium.com/swlh/how-to-create-fake-talking-head-videos-with-deep-learning-code-tutorial-f9cfc0c19ab5): An article explaining the process of generating fake talking head videos using deep learning techniques.

- [AudioGPT: Understanding and Generating Speech, Music, Sound](https://arxiv.org/abs/2304.12995): A research paper introducing AudioGPT, a multi-modal AI system that can process complex audio information and understand and generate speech, music, sound, and talking head content.

- [Text-based Editing of Talking-head Video](https://arxiv.org/pdf/1906.01524): An academic publication discussing the editing of talking-head videos using text-based instructions.

- [Few-Shot Adversarial Learning of Realistic Neural Talking Head](https://arxiv.org/abs/1905.08233): A research paper presenting a system capable of learning personalized talking head models from just a few image views of a person, using adversarial training techniques.

- [DisCoHead: Audio-and-Video-Driven Talking Head Generation](https://arxiv.org/abs/2303.07697): A paper describing DisCoHead, a method that disentangles and controls head pose and facial expressions in talking head generation, without supervision.

- [Microsoft's 3D Photo Realistic Talking Head](https://blogs.microsoft.com/ai/microsofts-3d-photo-realistic-talking-head/): A blog post showcasing Microsoft's 3D talking head technology, which combines photorealistic video with a 3D mesh model.

- [Depth-Aware Generative Adversarial Network for Talking Head](https://arxiv.org/abs/2203.06605): A research paper proposing a GAN-based approach that leverages dense 3D facial geometry to generate realistic and accurate talking head videos.

- [Talking-head Generation with Rhythmic Head Motion](https://arxiv.org/abs/2007.08547): This article presents a method for generating realistic talking-head videos with natural head movements, addressing the challenge of generating lip-synced videos while incorporating natural head motion. The proposed approach utilizes a 3D-aware generative network along with a hybrid embedding module and a non-linear composition module, resulting in controllable and photo-realistic talking-head videos with natural head movements.

- [Learned Spatial Representations for Few-shot Talking-Head Synthesis](https://arxiv.org/abs/2104.14557): This article introduces a novel approach for few-shot talking-head synthesis by factorizing the representation of a subject into its spatial and style components. The proposed method predicts a dense spatial layout for the target image and utilizes it for synthesizing the target frame, achieving improved preservation of the subject's identity in the source images.

- [Efficient Emotional Adaptation for Audio-Driven Talking-Head](https://arxiv.org/abs/2309.04946): This article proposes the Emotional Adaptation for Audio-driven Talking-head (EAT) method, which transforms emotion-agnostic talking-head models into emotion-controllable ones in a cost-effective and efficient manner. The approach utilizes lightweight adaptations to enable precise and realistic emotion controls, achieving state-of-the-art performance on widely-used benchmarks.

- [High-Fidelity and Freely Controllable Talking Head Video Generation](https://arxiv.org/abs/2304.10168): This article addresses the challenges faced by current methods in generating high-quality and controllable talking-head videos. It introduces a novel model that leverages self-supervised learned landmarks and 3D face model-based landmarks to model the motion, along with a motion-aware multi-scale feature alignment module. The proposed method produces high-fidelity talking-head videos with free control over head pose and expression.

- [Implicit Identity Representation Conditioned Memory Compensation](https://arxiv.org/abs/2307.09906): This article proposes a global facial representation space and a novel implicit identity representation conditioned memory compensation network for high-fidelity talking head generation. The network module learns a unified spatial facial meta-memory bank, which compensates warped source facial features to overcome limitations due to complex motions in the driving video, resulting in improved generation quality.

- [Avatar Fingerprinting for Authorized Use of Synthetic Talking-Head](https://arxiv.org/abs/2305.03713): This article focuses on the task of avatar fingerprinting, which verifies the trustworthiness of rendered talking-head videos. It proposes an embedding that groups the motion signatures of one identity together, allowing the identification of synthetic videos using the appearance of a specific individual driving the expressions.

- [Style Transfer for 2D Talking Head Animation](https://arxiv.org/abs/2303.09799): This article presents a method for generating talking head animation with learnable style references. It reconstructs 2D talking head animation based on a single input image and an audio stream, utilizing facial landmarks motion, style-pattern construction, and a style-aware image generator. The method achieves better results than recent state-of-the-art methods in generating photo-realistic and fidelity 2D animation.

- [One-Shot Free-View Neural Talking-Head Synthesis for Video](https://arxiv.org/abs/2011.15126): This article proposes a neural talking-head video synthesis model that learns to synthesize videos using a source image containing the target person's appearance and a driving video for motion. The model achieves high visual quality and bandwidth efficiency, outperforming competing methods on benchmark datasets.

- [Progressive Disentangled Representation Learning for Fine](https://arxiv.org/abs/2211.14506): This article presents a one-shot talking head synthesis method that achieves disentangled control over lip motion, eye gaze & blink, head pose, and emotional expression. It utilizes a progressive disentangled representation learning strategy to isolate each motion factor, allowing for fine-grained control and high-quality speech and lip-motion synchronization.

- [VideoReTalking: Audio-based Lip Synchronization for Talking Head](https://arxiv.org/abs/2211.14758): This article introduces VideoReTalking, a system for editing real-world talking head videos according to input audio. It disentangles the editing task into face video generation, audio-driven lip-sync, and face enhancement, ultimately producing a high-quality and lip-syncing output video. The system utilizes learning-based approaches in a sequential pipeline, without requiring user intervention.



## Online Courses



- [Video Production: You Can Make Simple Talking Head Video | Udemy](https://www.udemy.com/course/how-to-make-simple-talking-head-video/) 🎥: This Video Production course teaches you how to create professional and engaging talking head videos.

- [The Complete Talking Head Video Production Masterclass | Udemy](https://www.udemy.com/course/the-complete-talking-head-video-production-masterclass/) 🎥: Dive into the world of talking head video production with this comprehensive and in-depth course.

- [Video Production - Inexpensive Talking Head Video - Business | Udemy](https://www.udemy.com/course/video-production-inexpensive-talking-head-video-business/) 🎥: Learn how to create simple and effective talking head videos for various business communication needs.

- [How to Create a Talking Head Video | Udemy](https://www.udemy.com/course/how-to-create-a-talking-head-video/) 🎥: A beginner-friendly course that covers the technicalities of filming and creating talking head videos.



## Research Papers



- [Talking-Heads Attention](https://arxiv.org/abs/2003.02436): Introducing "talking-heads attention," a variation on multi-head attention that improves language modeling and comprehension tasks.

- [Few-Shot Adversarial Learning of Realistic Neural Talking Head](https://arxiv.org/abs/1905.08233): Presenting a system that enables the generation of realistic talking head models from a few image views of a person.

- [MakeItTalk: Speaker-Aware Talking-Head Animation](https://arxiv.org/abs/2004.12992): Proposing a method for generating expressive talking heads from a single facial image with audio as the only input.

- [StyleTalk: One-shot Talking Head Generation with Controllable](https://arxiv.org/abs/2301.01081): Introducing a framework for generating one-shot talking heads with diverse personalized speaking styles.

- [DiffTalk: Crafting Diffusion Models for Generalized](https://arxiv.org/abs/2301.03786): Modeling talking head generation as an audio-driven denoising process using Latent Diffusion Models.

- [One-Shot High-Fidelity Talking-Head Synthesis with Deformable](https://arxiv.org/abs/2304.05097): Proposing a method for generating high-fidelity talking heads by employing explicit 3D structural representations.

- [AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head](https://arxiv.org/abs/2103.11078): Presenting a framework for generating high-fidelity talking head videos directly from input audio using neural scene representation networks.

- [Depth-Aware Generative Adversarial Network for Talking Head](https://arxiv.org/abs/2203.06605): Introducing a self-supervised geometry learning method and leveraging dense 3D facial geometry for accurate talking head video generation.

- [What comprises a good talking-head video generation?: A Survey](https://arxiv.org/abs/2005.03201): This paper presents a benchmark and evaluation metrics for talking-head video generation, addressing the limitations of subjective evaluations. It explores desired properties such as identity preservation, lip synchronization, high video quality, and natural-spontaneous motion.

- [Text-based Editing of Talking-head Video](https://arxiv.org/abs/1906.01524): The paper proposes a method to edit talking-head videos based on their transcript, allowing for modifications in speech content while maintaining a seamless audio-visual flow. It utilizes annotations of facial features and a parametric face model for realistic video output.

- [MetaPortrait: Identity-Preserving Talking Head Generation with Fast Personalized Adaptation](https://arxiv.org/abs/2212.08062): This work introduces an identity-preserving talking head generation framework that utilizes dense landmarks for accurate geometry-aware flow fields. It also proposes adaptive fusion of source identity during synthesis and a fast adaptation model using meta-learning for personalized fine-tuning.

- [Learning Dynamic Facial Radiance Fields for Few-Shot Talking Head Synthesis](https://arxiv.org/abs/2207.11770): The paper introduces Dynamic Facial Radiance Fields (DFRF) for few-shot talking head synthesis, which generalizes to unseen identities with limited training data. It conditions face radiance field on 2D appearance images, allowing flexible adjustment to new identities with few reference images.

- [Audio2Head: Audio-driven One-shot Talking-head Generation with Natural Head Motions](http://arxiv.org/abs/2107.09293): This work proposes an audio-driven talking-head method that produces photo-realistic videos from a single reference image. It addresses challenges of producing natural head motions matching speech prosody while maintaining appearance during large head motions. It utilizes a head pose predictor and a motion field generator.

- [Efficient Emotional Adaptation for Audio-Driven Talking-Head Synthesis](https://arxiv.org/abs/2309.04946): The paper introduces the Emotional Adaptation for Audio-driven Talking-head (EAT) method, which transforms emotion-agnostic talking-head models into emotion-controllable ones through parameter-efficient adaptations. It utilizes adaptations from different perspectives to enable realistic emotion controls.

- [Talking Head Generation with Probabilistic Audio-to-Visual Diffusion](https://arxiv.org/abs/2212.04248): This paper introduces a framework for one-shot audio-driven talking head generation using a probabilistic approach. It probabilistically generates facial motions matching input audio while maintaining audio-lip synchronization and overall photo-realism. It avoids the need for additional driving sources for controlled synthesis.

- [One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing](https://arxiv.org/abs/2011.15126): The paper proposes a neural talking-head video synthesis model for video conferencing. It utilizes a source image and a driving video to synthesize the talking-head video. It outperforms competing methods on benchmark datasets and enables video conferencing with high visual quality and low bandwidth usage.

- [Write-a-speaker: Text-based Emotional and Rhythmic Talking-head Video Generation](https://arxiv.org/abs/2104.07995): This paper presents a text-based talking-head video generation framework that synthesizes facial expressions and head motions according to contextual sentiments and speech rhythm. It consists of a speaker-independent stage and a speaker-specific stage, allowing tailored video synthesis for different individuals.

- [Style Transfer for 2D Talking Head Animation](https://arxiv.org/abs/2303.09799): The paper introduces a method for generating talking-head animation with learnable style references. It reconstructs 2D animation based on a single input image and an audio stream by extracting facial landmarks motion and incorporating style patterns from reference images.



## Tools & Software



- [LUCIA](https://sourceforge.net/projects/lucia/): Development of a MPEG-4 Talking Head Engine. 💻

- [Yepic Studio](https://www.g2.com/products/yepic-studio/reviews): Create and dub talking head-style videos in minutes without expensive equipment. 🎥

- [Mel McGee's Talkbots](https://sourceforge.net/projects/talkbots/): A complete multi-browser, multi-platform talking head application in SVG suitable for web sites or as an avatar. 🗣️

- [face3D_chung](https://sourceforge.net/projects/face3dchung/): Create 3D character avatar head objects with texture from a single photo for your games. 🎮

- [CrazyTalk](https://www.g2.com/products/crazytalk/reviews): Exciting features for 3D head creation and automation. 🤪

- [tts avatar free download - SourceForge](https://sourceforge.net/directory/?q=tts%20avatar) : Mel McGee's Talkbots is a complete multi-browser, multi-platform talking head. (🔧👄)

- [Verbatim AI - Product Information, Latest Updates, and Reviews 2023](https://www.producthunt.com/products/verbatim-ai) : A simple yet powerful API to generate AI "talking head" videos in near real-time with Verbatim AI. Add interest, intrigue, and dynamism to your chat bots! (🔧👄)

- [Best Open Source BASIC 3D Modeling Software](https://sourceforge.net/directory/3d-modeling/basic/) : Includes talk3D_chung, a small example using obj models created with face3D_chung, and speak3D_chung_dll, a dll to load and display face3D_chung talking avatars. (🛠️🎭)

- [DVDStyler / Discussion / Help: ffmpeg-vbr or internal](https://sourceforge.net/p/dvdstyler/discussion/318795/thread/82dcb647/) : Talking heads would get a bitrate which is unnecessarily high while using DVDStyler. (🛠️👄)

- [puffin web browser free download - SourceForge](https://sourceforge.net/directory/lisp/?q=puffin+web+browser) : Mel McGee's Talkbots is a complete multi-browser, multi-platform talking head. (🔧👄)

- [12 best AI video generators to use in 2023 [Free and paid] | Product ...](https://www.producthunt.com/stories/best-ai-video-generators-free) : Whether you’re an entrepreneur, small business owner, or run a large company, AI video generators make it super easy to create high-quality videos from scratch. (🔧🎥)



## Slides & Presentations



- [(Paper Review)Few-Shot Adversarial Learning of Realistic Neural Talking Head Models](https://www.slideshare.net/ssuserc9d82a/paper-reviewfewshot-adversarial-learning-of-realistic-neural-talking-head-models): Presentation reviewing the few-shot adversarial learning of realistic neural talking head models.

- [Nethania Michelle's Character | PPT](https://www.slideshare.net/ZULHICZARARIETINARBU/nethania-michelles-character): Presentation discussing the improvement of a 3D talking head for use in an avatar of a virtual meeting room.

- [Presenting you: Top tips on presenting with Prezi Video – Prezi](https://support.prezi.com/hc/en-us/articles/360036679953-Presenting-you-Top-tips-on-presenting-with-Prezi-Video): Article providing top tips for presenting with Prezi Video.

- [Research Presentation | PPT](https://pt.slideshare.net/willg_36/research-presentation-presentation-956726): Resident Research Presentation Slide Deck.

- [Adding narration to your presentation (using Prezi Video) – Prezi ...](https://support.prezi.com/hc/en-us/articles/360038281894-Adding-narration-to-your-presentation-using-Prezi-Video-): Learn how to add narration to your Prezi presentation with Prezi Video.



---



This initial version of the Awesome List was generated with the help of the [Awesome List Generator](https://github.com/alialsaeedi19/GPT-Awesome-List-Maker). It's an open-source Python package that uses the power of GPT models to automatically curate and generate starting points for resource lists related to a specific topic.