Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/aigc-audio/audiogpt
AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head
https://github.com/aigc-audio/audiogpt
audio gpt music sound speech talking-head
Last synced: 10 days ago
JSON representation
AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head
- Host: GitHub
- URL: https://github.com/aigc-audio/audiogpt
- Owner: AIGC-Audio
- License: other
- Created: 2023-03-16T07:12:18.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2024-07-06T21:35:18.000Z (4 months ago)
- Last Synced: 2024-10-29T00:32:21.130Z (11 days ago)
- Topics: audio, gpt, music, sound, speech, talking-head
- Language: Python
- Homepage: https://huggingface.co/spaces/AIGC-Audio/AudioGPT
- Size: 23 MB
- Stars: 10,016
- Watchers: 136
- Forks: 862
- Open Issues: 50
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-ChatGPT-repositories - AudioGPT - AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head (Others)
README
# AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head
[](https://arxiv.org/abs/2304.12995)
[](https://github.com/AIGC-Audio/AudioGPT)
[](https://huggingface.co/spaces/AIGC-Audio/AudioGPT)
We provide our implementation and pretrained models as open source in this repository.
## Get Started
Please refer to [run.md](run.md)
## Capabilities
Here we list the capability of AudioGPT at this time. More supported models and tasks are coming soon. For prompt examples, refer to [asset](assets/README.md).
Currently not every model has repository.
### Speech
| Task | Supported Foundation Models | Status |
|:--------------------------:|:-------------------------------:|:------:|
| Text-to-Speech | [FastSpeech](https://github.com/ming024/FastSpeech2), [SyntaSpeech](https://github.com/yerfor/SyntaSpeech), [VITS](https://github.com/jaywalnut310/vits) | Yes (WIP) |
| Style Transfer | [GenerSpeech](https://github.com/Rongjiehuang/GenerSpeech) | Yes |
| Speech Recognition | [whisper](https://github.com/openai/whisper), [Conformer](https://github.com/sooftware/conformer) | Yes |
| Speech Enhancement | [ConvTasNet]() | Yes (WIP) |
| Speech Separation | [TF-GridNet](https://arxiv.org/pdf/2211.12433.pdf) | Yes (WIP) |
| Speech Translation | [Multi-decoder](https://arxiv.org/pdf/2109.12804.pdf) | WIP |
| Mono-to-Binaural | [NeuralWarp](https://github.com/fdarmon/NeuralWarp) | Yes |
### Sing
| Task | Supported Foundation Models | Status |
|:-------------------------:|:-------------------------------:|:------:|
| Text-to-Sing | [DiffSinger](https://github.com/MoonInTheRiver/DiffSinger), [VISinger](https://github.com/jerryuhoo/VISinger) | Yes (WIP) |
### Audio
| Task | Supported Foundation Models | Status |
|:----------------------:|:---------------------------:|:------:|
| Text-to-Audio | [Make-An-Audio]() | Yes |
| Audio Inpainting | [Make-An-Audio]() | Yes |
| Image-to-Audio | [Make-An-Audio]() | Yes |
| Sound Detection | [Audio-transformer](https://github.com/RetroCirce/HTS-Audio-Transformer) | Yes |
| Target Sound Detection | [TSDNet](https://github.com/gy65896/TSDNet) | Yes |
| Sound Extraction | [LASSNet](https://github.com/liuxubo717/LASS) | Yes |
### Talking Head
| Task | Supported Foundation Models | Status |
|:-------------------------:|:-------------------------------:|:----------:|
| Talking Head Synthesis | [GeneFace](https://github.com/yerfor/GeneFace) | Yes (WIP) |
## Acknowledgement
We appreciate the open source of the following projects:
[ESPNet](https://github.com/espnet/espnet)
[NATSpeech](https://github.com/NATSpeech/NATSpeech)
[Visual ChatGPT](https://github.com/microsoft/visual-chatgpt)
[Hugging Face](https://github.com/huggingface)
[LangChain](https://github.com/hwchase17/langchain)
[Stable Diffusion](https://github.com/CompVis/stable-diffusion)