{"id":17439348,"url":"https://github.com/lipku/LiveTalking","last_synced_at":"2025-03-01T14:30:45.037Z","repository":{"id":214428510,"uuid":"733283939","full_name":"lipku/LiveTalking","owner":"lipku","description":"Real time interactive streaming digital human","archived":false,"fork":false,"pushed_at":"2025-02-25T12:05:46.000Z","size":46543,"stargazers_count":4700,"open_issues_count":266,"forks_count":688,"subscribers_count":54,"default_branch":"main","last_synced_at":"2025-02-26T05:08:33.081Z","etag":null,"topics":["aigc","digihuman","digital-human","er-nerf","lip-sync","metahuman-stream","musetalk","nerf","realtime","streaming","talking-head","virtualhumans","wav2lip"],"latest_commit_sha":null,"homepage":"https://livetalking-doc.readthedocs.io/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lipku.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-12-19T01:32:46.000Z","updated_at":"2025-02-26T04:33:23.000Z","dependencies_parsed_at":"2024-03-09T03:20:02.568Z","dependency_job_id":"f08169e1-d63e-490a-aa43-24f027d7c3ea","html_url":"https://github.com/lipku/LiveTalking","commit_stats":null,"previous_names":["lipku/metahuman-stream","lipku/livetalking"],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lipku%2FLiveTalking","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lipku%2FLiveTalking/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lipku%2FLiveTalking/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lipku%2FLiveTalking/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lipku","download_url":"https://codeload.github.com/lipku/LiveTalking/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241377752,"owners_count":19953159,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aigc","digihuman","digital-human","er-nerf","lip-sync","metahuman-stream","musetalk","nerf","realtime","streaming","talking-head","virtualhumans","wav2lip"],"created_at":"2024-10-17T13:02:17.624Z","updated_at":"2025-03-01T14:30:45.027Z","avatar_url":"https://github.com/lipku.png","language":"Python","funding_links":[],"categories":["Python","🤖 AI \u0026 Machine Learning"],"sub_categories":[],"readme":"Real time interactive streaming digital human， realize audio video synchronous dialogue. It can basically achieve commercial effects.  \n实时交互流式数字人，实现音视频同步对话。基本可以达到商用效果\n\n[ernerf效果](https://www.bilibili.com/video/BV1PM4m1y7Q2/)  [musetalk效果](https://www.bilibili.com/video/BV1gm421N7vQ/)  [wav2lip效果](https://www.bilibili.com/video/BV1Bw4m1e74P/)\n\n## 为避免与3d数字人混淆，原项目metahuman-stream改名为livetalking，原有链接地址继续可用\n\n## News\n- 2024.12.8 完善多并发，显存不随并发数增加\n- 2024.12.21 添加wav2lip、musetalk模型预热，解决第一次推理卡顿问题。感谢@heimaojinzhangyz\n- 2024.12.28 添加数字人模型Ultralight-Digital-Human。 感谢@lijihua2017\n- 2025.2.7 添加fish-speech tts\n- 2025.2.21 添加wav2lip256开源模型 感谢@不蠢不蠢\n\n## Features\n1. 支持多种数字人模型: ernerf、musetalk、wav2lip、Ultralight-Digital-Human\n2. 支持声音克隆\n3. 支持数字人说话被打断\n4. 支持全身视频拼接\n5. 支持rtmp和webrtc\n6. 支持视频编排：不说话时播放自定义视频\n7. 支持多并发\n\n## 1. Installation\n\nTested on Ubuntu 20.04, Python3.10, Pytorch 1.12 and CUDA 11.3\n\n### 1.1 Install dependency\n\n```bash\nconda create -n nerfstream python=3.10\nconda activate nerfstream\n#如果cuda版本不为11.3(运行nvidia-smi确认版本)，根据\u003chttps://pytorch.org/get-started/previous-versions/\u003e安装对应版本的pytorch \nconda install pytorch==1.12.1 torchvision==0.13.1 cudatoolkit=11.3 -c pytorch\npip install -r requirements.txt\n#如果需要训练ernerf模型，安装下面的库\n# pip install \"git+https://github.com/facebookresearch/pytorch3d.git\"\n# pip install tensorflow-gpu==2.8.0\n# pip install --upgrade \"protobuf\u003c=3.20.1\"\n``` \n安装常见问题[FAQ](https://livetalking-doc.readthedocs.io/en/latest/faq.html)  \nlinux cuda环境搭建可以参考这篇文章 https://zhuanlan.zhihu.com/p/674972886\n\n\n## 2. Quick Start\n- 下载模型  \n百度云盘\u003chttps://pan.baidu.com/s/1yOsQ06-RIDTJd3HFCw4wtA\u003e 密码: ltua  \nGoogleDriver \u003chttps://drive.google.com/drive/folders/1FOC_MD6wdogyyX_7V1d4NDIO7P9NlSAJ?usp=sharing\u003e  \n将wav2lip256.pth拷到本项目的models下, 重命名为wav2lip.pth;  \n将wav2lip256_avatar1.tar.gz解压后整个文件夹拷到本项目的data/avatars下\n- 运行  \npython app.py --transport webrtc --model wav2lip --avatar_id wav2lip256_avatar1  \n用浏览器打开http://serverip:8010/webrtcapi.html , 先点‘start',播放数字人视频；然后在文本框输入任意文字，提交。数字人播报该段文字  \n\n\u003cfont color=red\u003e服务端需要开放端口 tcp:8010; udp:1-65536 \u003c/font\u003e  \n如果需要商用高清wav2lip模型，可以与我联系购买\n\n如果访问不了huggingface，在运行前\n```\nexport HF_ENDPOINT=https://hf-mirror.com\n``` \n\n\n## 3. More Usage\n使用说明: \u003chttps://livetalking-doc.readthedocs.io/\u003e\n  \n## 4. Docker Run  \n不需要前面的安装，直接运行。\n```\ndocker run --gpus all -it --network=host --rm registry.cn-beijing.aliyuncs.com/codewithgpu2/lipku-metahuman-stream:vjo1Y6NJ3N\n```\n代码在/root/metahuman-stream，先git pull拉一下最新代码，然后执行命令同第2、3步 \n\n提供如下镜像\n- autodl镜像: \u003chttps://www.codewithgpu.com/i/lipku/metahuman-stream/base\u003e   \n[autodl教程](https://livetalking-doc.readthedocs.io/en/latest/autodl/README.html)\n- ucloud镜像: \u003chttps://www.compshare.cn/images-detail?ImageID=compshareImage-18tpjhhxoq3j\u0026referral_code=3XW3852OBmnD089hMMrtuU\u0026ytag=GPU_livetalking1.3\u003e  \n可以开放任意端口，不需要另外部署srs服务.  \n[ucloud教程](https://livetalking-doc.readthedocs.io/en/latest/ucloud/ucloud.html) \n\n\n## 5. TODO\n- [x] 添加chatgpt实现数字人对话\n- [x] 声音克隆\n- [x] 数字人静音时用一段视频代替\n- [x] MuseTalk\n- [x] Wav2Lip\n- [x] Ultralight-Digital-Human\n\n---\n如果本项目对你有帮助，帮忙点个star。也欢迎感兴趣的朋友一起来完善该项目.\n* 知识星球: https://t.zsxq.com/7NMyO 沉淀高质量常见问题、最佳实践经验、问题解答  \n* 微信公众号：数字人技术  \n![](https://mmbiz.qpic.cn/sz_mmbiz_jpg/l3ZibgueFiaeyfaiaLZGuMGQXnhLWxibpJUS2gfs8Dje6JuMY8zu2tVyU9n8Zx1yaNncvKHBMibX0ocehoITy5qQEZg/640?wxfrom=12\u0026tp=wxpic\u0026usePicPrefetch=1\u0026wx_fmt=jpeg\u0026amp;from=appmsg)  \n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flipku%2FLiveTalking","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flipku%2FLiveTalking","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flipku%2FLiveTalking/lists"}