https://github.com/srivathsanp23/transart-multimodal-application
Deep Learning multimodel application to generate image from prompt and audio
https://github.com/srivathsanp23/transart-multimodal-application
chatbot deep-learning feature-en huggingface image-translation pipeline
Last synced: 7 months ago
JSON representation
Deep Learning multimodel application to generate image from prompt and audio
- Host: GitHub
- URL: https://github.com/srivathsanp23/transart-multimodal-application
- Owner: SrivathsanP23
- Created: 2024-10-13T10:14:55.000Z (about 1 year ago)
- Default Branch: main
- Last Pushed: 2024-10-24T06:53:25.000Z (12 months ago)
- Last Synced: 2025-01-28T23:30:06.319Z (9 months ago)
- Topics: chatbot, deep-learning, feature-en, huggingface, image-translation, pipeline
- Language: Jupyter Notebook
- Homepage: https://huggingface.co/spaces/Sriv890/Multimodal-Application
- Size: 9.08 MB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# TransArt-Multimodal-Application
```
🚀🚀🚀
TransArt seamlessly transcribes audio, translates it into multiple languages, and generates stunning visuals
based on the content. Additionally,
it offers an interactive chatbot for engaging conversations and image generation, all in one platform.
```
## to install requirements
pip install -r requirements.txt
```
Here are the key steps you followed to build the **TransArt Gradio application**:
1. **Imported Libraries and Set Up Environment**:
Imported essential libraries like `gradio`, `whisper`, `GoogleTranslator`, `StableDiffusionXLPipeline`, and others.
Set up API keys for Hugging Face and Groq
for model integration.
2. **Created Audio Processing Function**:
Developed a function (`process_audio`) to handle audio uploads, transcribe speech using Whisper,
translate it from Tamil to English using `GoogleTranslator`,
and optionally generate an image based on the translation.
3. **Implemented Image Generation from Prompt**:
Created a function (`generate_image_from_prompt`) to query a custom Stable Diffusion model,
generating images based on user-provided text prompts.
4. **Integrated Chatbot with Image Generation**:
Added a chatbot feature (`chatbox`) using the Groq API, where the user enters a prompt and receives
both a text-based chatbot response and a generated image.
5. **Designed Gradio Interface**:
Built the Gradio app using `gr.Blocks()` with multiple tabs:
- **Audio to Text Tab**: For audio transcription, translation, and optional image generation.
- **Prompt to Image Tab**: For generating images directly from user-entered text prompts.
- **Chatbot Tab**: Allows users to interact with a chatbot and generate images based on prompts.
6. **Custom CSS Styling**:
Customized the app’s appearance with CSS, defining background colors, button styles, and text color for a
polished user interface.
7. **Launched the Application**:
Launched the Gradio app to be hosted on the specified server, making it accessible for live testing and
demonstrations.
These steps encapsulate the entire process of developing the **TransArt Gradio application**.
```
