https://github.com/intel/openvino-plugins-ai-audacity
A set of AI-enabled effects, generators, and analyzers for Audacity®.
https://github.com/intel/openvino-plugins-ai-audacity
Last synced: about 1 month ago
JSON representation
A set of AI-enabled effects, generators, and analyzers for Audacity®.
- Host: GitHub
- URL: https://github.com/intel/openvino-plugins-ai-audacity
- Owner: intel
- License: gpl-3.0
- Created: 2023-09-06T21:10:29.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2024-04-12T18:53:58.000Z (about 1 year ago)
- Last Synced: 2024-04-13T22:57:39.840Z (about 1 year ago)
- Language: C++
- Homepage:
- Size: 4.08 MB
- Stars: 539
- Watchers: 21
- Forks: 25
- Open Issues: 36
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE.txt
- Code of conduct: CODE_OF_CONDUCT.md
- Security: security.md
Awesome Lists containing this project
- awesome-openvino - OpenVINO™ AI Plugins for Audacity® - A set of AI-enabled effects, generators, and analyzers for Audacity® such as Music Stem Separation, Noise Suppression, Transcription, and Music Generation. (Table of content / AI Audio)
README
# OpenVINO™ AI Plugins for Audacity* :metal:
[](https://www.gnu.org/licenses/gpl-3.0) [](https://github.com/openvinotoolkit/awesome-openvino)
A set of AI-enabled effects, generators, and analyzers for [Audacity®](https://www.audacityteam.org/). These AI features run 100% locally on your PC :computer: -- no internet connection necessary! [OpenVINO™](https://github.com/openvinotoolkit/openvino) is used to run AI models on supported accelerators found on the user's system such as CPU, GPU, and NPU.
- [**Music Separation**](doc/feature_doc/music_separation/README.md):musical_note: -- Separate a mono or stereo track into individual stems -- Drums, Bass, Vocals, & Other Instruments.
- [**Noise Suppression**](doc/feature_doc/noise_suppression/README.md):broom: -- Removes background noise from an audio sample.
- [**Music Generation & Continuation**](doc/feature_doc/music_generation/README.md):notes: -- Uses MusicGen LLM to generate snippets of music, or to generate a continuation of an existing snippet of music.
- [**Whisper Transcription**](doc/feature_doc/whisper_transcription/README.md):microphone: -- Uses [whisper.cpp](https://github.com/ggerganov/whisper.cpp) to generate a label track containing the transcription or translation for a given selection of spoken audio or vocals.
- [**Super Resolution**](doc/feature_doc/super_resolution/README.md):sparkles: -- Upscales and enriches audio for improved clarity and detail.
## Installation :floppy_disk:
Go [here](https://github.com/intel/openvino-plugins-ai-audacity/releases) to find installation packages & instructions for the latest Windows release.
## Build Instructions :hammer:
- [Windows Build Instructions](doc/build_doc/windows/README.md)
- [Linux Build Instructions](doc/build_doc/linux/README.md)
## Help, Feedback, & Bug Reports 🙋♂️
We welcome you to submit an issue [here](https://github.com/intel/openvino-plugins-ai-audacity/issues) for
* Questions
* Bug Reports
* Feature Requests
* Feedback of any kind -- how can we improve this project?
## Contribution :handshake:
Your contributions are welcome and valued, no matter how big or small. Feel free to submit a pull-request!
## Acknowledgements & Citations :pray:
* Audacity® development team & Muse Group-- Thank you for your support!
* Audacity® GitHub -- https://github.com/audacity/audacity
* Whisper transcription & translation analyzer uses whisper.cpp (with OpenVINO™ backend): https://github.com/ggerganov/whisper.cpp
* Music Generation & Continuation use MusicGen model, from Meta.
- We currently have support for [MusicGen-Small](https://huggingface.co/facebook/musicgen-small), and [MusicGen-Small-Stereo](https://huggingface.co/facebook/musicgen-stereo-small)
- The txt-to-music pipelines were ported from python to C++, referencing logic from the Hugging Face transformers project: https://github.com/huggingface/transformers
* Music Separation effect uses Meta's Demucs v4 model (https://github.com/facebookresearch/demucs), which has been ported to work with OpenVINO™
* Noise Suppression:
* noise-suppression-denseunet-ll: from OpenVINO™'s Open Model Zoo: https://github.com/openvinotoolkit/open_model_zoo
* DeepFilterNet2 & DeepFilterNet3:
* Ported the models & pipeline from here: https://github.com/Rikorose/DeepFilterNet
* We also made use of @grazder's fork / branch (https://github.com/grazder/DeepFilterNet/tree/torchDF-changes) to better understand the Rust implementation, and so we also based some of our C++ implementation on ```torch_df_offline.py``` found here.
* Citations:
```bibtex
@inproceedings{schroeter2022deepfilternet2,
title = {{DeepFilterNet2}: Towards Real-Time Speech Enhancement on Embedded Devices for Full-Band Audio},
author = {Schröter, Hendrik and Escalante-B., Alberto N. and Rosenkranz, Tobias and Maier, Andreas},
booktitle={17th International Workshop on Acoustic Signal Enhancement (IWAENC 2022)},
year = {2022},
}
@inproceedings{schroeter2023deepfilternet3,
title = {{DeepFilterNet}: Perceptually Motivated Real-Time Speech Enhancement},
author = {Schröter, Hendrik and Rosenkranz, Tobias and Escalante-B., Alberto N. and Maier, Andreas},
booktitle={INTERSPEECH},
year = {2023},
}
```
* Super Resolution feature is a port (python -> C++, pytorch -> OpenVINO IR) of this project: https://github.com/haoheliu/versatile_audio_super_resolution
* Citations:
```bibtex
@article{liu2023audiosr,
title={{AudioSR}: Versatile Audio Super-resolution at Scale},
author={Liu, Haohe and Chen, Ke and Tian, Qiao and Wang, Wenwu and Plumbley, Mark D},
journal={arXiv preprint arXiv:2309.07314},
year={2023}
}
```
* [OpenVINO™ Notebooks](https://github.com/openvinotoolkit/openvino_notebooks) -- We have learned a lot from this awesome set of python notebooks, and are still using it to learn latest / best practices for implementing AI pipelines using OpenVINO™!