Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/tensorsofthewall/vidtune

VidTune: Tailored Music For Your Videos
https://github.com/tensorsofthewall/vidtune

audiocraft generative-ai generative-audio google-gemini meta-ai music-generation musicgen streamlit transformers video-to-audio vidtune

Last synced: 3 months ago
JSON representation

VidTune: Tailored Music For Your Videos

Host: GitHub
URL: https://github.com/tensorsofthewall/vidtune
Owner: tensorsofthewall
License: other
Created: 2024-06-06T01:42:57.000Z (8 months ago)
Default Branch: main
Last Pushed: 2024-10-20T18:31:29.000Z (4 months ago)
Last Synced: 2024-10-20T22:56:20.650Z (4 months ago)
Topics: audiocraft, generative-ai, generative-audio, google-gemini, meta-ai, music-generation, musicgen, streamlit, transformers, video-to-audio, vidtune
Language: Python
Homepage: https://huggingface.co/spaces/sandeshb/VidTune-Gradio
Size: 4.61 MB
Stars: 2
Watchers: 2
Forks: 1
Open Issues: 0
Metadata Files:
- Readme: README.md
- License: LICENSE

Awesome Lists containing this project

README

        ---

title: VidTune

colorFrom: indigo

colorTo: pink

sdk: streamlit

python_version: 3.9.19

sdk_version: 1.36.0

suggested_hardware: t4-medium

suggested_storage: small

app_file: main.py

pinned: true

preload_from_hub:

- facebook/musicgen-small

- facebook/musicgen-medium

- facebook/musicgen-large

short_description: Generate tailored soundtracks for your videos.

---



[![Contributors][contributors-shield]][contributors-url]

[![Forks][forks-shield]][forks-url]

[![Stargazers][stars-shield]][stars-url]

[![Issues][issues-shield]][issues-url]

[![AGPL License][license-shield]][license-url]

[![Sandesh-LinkedIn][sandesh-linkedin-shield]][sandesh-linkedin-url]

[![Animikh-LinkedIn][animikh-linkedin-shield]][animikh-linkedin-url]






  

    

  

  
VidTune


  


    Tailored soundtracks for your videos

    


    


    Report Bug

    ·

    Request Feature

  



  Table of Contents

  


    

      About The Project

      

        Built With

      

    

    

      Hardware Requirements

      

        Hardware used for Development and Testing

      

    

    See VidTune in action!

    Roadmap

    Contributing

    License

    Contact

    Acknowledgments

  


## About The Project

![startup_screen]

**VidTune** is a generative AI application designed to create custom music tailored to your video content. By leveraging advanced AI models for video analysis and music creation, **VidTune** provides an intuitive and seamless experience for generating and integrating music into your videos.

**VidTune** employs two state-of-the-art models for video understanding and music generation:

1. [**Google Gemini**](https://ai.google.dev/gemini-api) - Google's largest and most capable multimodal AI model.

2. [**MusicGen**](https://huggingface.co/facebook/musicgen-large) - Meta's text-to-music model, capable of generating high-quality music conditioned on text or audio prompts.

(back to top)


### Built With

[![Transformers][Transformers-shield]][Transformers-url]

[![Google Gemini][Google-Gemini-shield]][Google-Gemini-url]

[![AudioCraft][AudioCraft-shield]][AudioCraft-url]

[![MusicGen][MusicGen-shield]][MusicGen-url]

[![Streamlit][Streamlit-shield]][Streamlit-url]

(back to top)


## Hardware Requirements

### Hardware used for Development and Testing

- **CPU:** AMD Ryzen 7 3700X - 8 Cores 16 Threads

- **GPU:** Nvidia GeForce RTX 4060 Ti 16 GB

- **RAM:** 64 GB DDR4 @ 3200 MHz

- **OS:** Linux (WSL | Ubuntu 22.40)

The above is just used for development and by no means is necessary to run this application. The Minimum Hardware Requirements are given in the next section

While VidTune is supported on CPU-only machines, we recommend using a GPU with minimum 16GB of memory for faster results.

## See VidTune in action!

[![Watch the video](https://img.youtube.com/vi/knbQjWZtL3Y/maxresdefault.jpg)](https://youtu.be/knbQjWZtL3Y)

## Running VidTune

First, clone the repository:

```sh

git clone https://github.com/sandesh-bharadwaj/VidTune.git

cd VidTune

```

### Using conda

If you're using conda as your virtual environment manager, do the following:

```

conda env create -f environment.yml

conda activate vidtune

streamlit run main.py

```

### Using python / pip

```

pip install -r requirements.txt

streamlit run main.py

```

### Using Docker

- [Docker](https://docs.docker.com/engine/install/)

- [Nvidia Docker](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html#installing-with-apt)

Docker Hub Image: https://hub.docker.com/r/animikhaich/vidtune

```

docker run --rm -it --gpus all -p 8003:8003 animikhaich/vidtune

```

## Roadmap

- [x] Customized Prompt for Gemini Prompting

- [x] Flutter version of app for proof-of-concept

- [x] MusicGen integration

- [x] Audio Mixing

- [x] Streamlit app

- [x] Docker image

- [ ] OpenVINO-optimized versions of MusicGen for CPU-Only use.

- [ ] Support for music generation duration > 30 seconds.

- [ ] Add more settings for controlling generation.

- [ ] Option to edit music prompts before music generation.

See the [open issues](https://github.com/sandesh-bharadwaj/VIdTune/issues) for a full list of proposed features (and known issues).

(back to top)


## Contributing

If you have a suggestion that would improve this, please **open an issue** with the tag *"enhancement"*.You can also **fork the repo** and create a pull request. Your feedback is greatly appreciated!

Don't forget to give the project a star! Thanks again!

1. Fork the Project

2. Create your Feature Branch (`git checkout -b feature/AmazingFeature`)

3. Commit your Changes (`git commit -m 'Add some AmazingFeature'`)

4. Push to the Branch (`git push origin feature/AmazingFeature`)

5. Open a Pull Request

(back to top)


## License

Distributed under the CC BY-NC 4.0 License. See [`LICENSE`](./LICENSE) for more information.

(back to top)


## Contact

Sandesh Bharadwaj - [email protected]

Animikh Aich - [email protected]

Project Link: [https://github.com/sandesh-bharadwaj/VidTune](https://github.com/sandesh-bharadwaj/VidTune)

(back to top)


## Acknowledgments

* Google.

* Meta.

(back to top)


[contributors-shield]: https://img.shields.io/github/contributors/sandesh-bharadwaj/VidTune.svg?style=for-the-badge

[contributors-url]: https://github.com/sandesh-bharadwaj/VidTune/graphs/contributors

[forks-shield]: https://img.shields.io/github/forks/sandesh-bharadwaj/VidTune.svg?style=for-the-badge

[forks-url]: https://github.com/sandesh-bharadwaj/VidTune/network/members

[stars-shield]: https://img.shields.io/github/stars/sandesh-bharadwaj/VidTune.svg?style=for-the-badge

[stars-url]: https://github.com/sandesh-bharadwaj/VidTune/stargazers

[issues-shield]: https://img.shields.io/github/issues/sandesh-bharadwaj/VidTune.svg?style=for-the-badge

[issues-url]: https://github.com/sandesh-bharadwaj/VidTune/issues

[license-shield]: https://img.shields.io/github/license/sandesh-bharadwaj/VidTune.svg?style=for-the-badge

[license-url]: https://github.com/sandesh-bharadwaj/VidTune/blob/main/LICENSE

[llama-3-shield]: https://img.shields.io/badge/License-Llama%203-purple.svg?style=for-the-badge

[llama-3-license]: https://github.com/sandesh-bharadwaj/VidTune/blob/main/LLAMA-3-LICENSE

[sandesh-linkedin-shield]: https://img.shields.io/badge/-Sandesh_Bharadwaj-black.svg?style=for-the-badge&logo=linkedin&colorB=555

[sandesh-linkedin-url]: https://linkedin.com/in/sandeshbharadwaj97

[animikh-linkedin-shield]: https://img.shields.io/badge/-Animikh_Aich-black.svg?style=for-the-badge&logo=linkedin&colorB=555

[animikh-linkedin-url]: https://linkedin.com/in/animikh-aich

[startup_screen]: assets/homepage.png

[Python-url]: https://img.shields.io/badge/python-3670A0?style=for-the-badge&logo=python&logoColor=ffdd54

[Langchain-shield]: https://img.shields.io/badge/LangChain-0.2.12-1C3C3C?style=for-the-badge&logo=langchain

[Langchain-url]: https://github.com/langchain-ai/langchain

[Transformers-shield]: https://img.shields.io/badge/Transformers-4.42.4-blue?style=for-the-badge

[Transformers-url]: https://github.com/huggingface/transformers

[Optimum-shield]: https://img.shields.io/badge/Optimum-1.21.2-blue?style=for-the-badge

[Optimum-url]: https://github.com/huggingface/optimum

[OpenVINO-shield]: https://img.shields.io/badge/OpenVINO-2024.3-purple?style=for-the-badge

[OpenVINO-url]: https://github.com/openvinotoolkit/openvino

[Chroma-shield]: https://img.shields.io/badge/Chroma-0.5.5-blue?style=for-the-badge

[Chroma-url]: https://github.com/chroma-core/chroma

[Google-Gemini-shield]: https://img.shields.io/badge/Google%20Gemini-886FBF?style=for-the-badge&logo=googlegemini&logoColor=fff

[Google-Gemini-url]: https://ai.google.dev/gemini-api

[Streamlit-shield]: https://img.shields.io/badge/-Streamlit-FF4B4B?style=for-the-badge&logo=streamlit&logoColor=white

[Streamlit-url]: https://streamlit.io/

[AudioCraft-shield]: https://img.shields.io/badge/-AudioCraft-blue?style=for-the-badge&logo=Meta

[AudioCraft-url]: https://audiocraft.metademolab.com/

[MusicGen-shield]:https://img.shields.io/badge/-MusicGen-blue?style=for-the-badge&logo=Meta

[MusicGen-url]: https://musicgen.com/