https://github.com/sitamgithub-msit/paligemma2-docci-litserve

Leverage PaliGemma 2's DOCCI fine-tuned variant capabilities using LitServe.
https://github.com/sitamgithub-msit/paligemma2-docci-litserve

artificial-intelligence deep-learning fastapi image-captioning lightning-ai litserve paligemma python transformers vision-language-models

Last synced: 7 months ago
JSON representation

Leverage PaliGemma 2's DOCCI fine-tuned variant capabilities using LitServe.

• Host: GitHub
• URL: https://github.com/sitamgithub-msit/paligemma2-docci-litserve
• Owner: sitamgithub-MSIT
• License: apache-2.0
• Created: 2025-01-02T09:58:34.000Z (9 months ago)
• Default Branch: main
• Last Pushed: 2025-02-20T23:56:39.000Z (8 months ago)
• Last Synced: 2025-02-21T00:32:56.542Z (8 months ago)
• Topics: artificial-intelligence, deep-learning, fastapi, image-captioning, lightning-ai, litserve, paligemma, python, transformers, vision-language-models
• Language: Python
• Homepage: https://lightning.ai/sitammeur/studios/deploy-docci-fine-tuned-paligemma-2-vision-language-model
• Size: 468 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# PaliGemma2-DOCCI LitServe

[![Open In Studio](https://pl-bolts-doc-images.s3.us-east-2.amazonaws.com/app-2/studio-badge.svg)](https://lightning.ai/sitammeur/studios/deploy-docci-fine-tuned-paligemma-2-vision-language-model)

[PaliGemma 2](https://huggingface.co/collections/google/paligemma-2-release-67500e1e1dbfdd4dee27ba48) is an updated vision-language model that leverages **Gemma 2** and **SigLIP** for superior performance on various vision-language tasks. It processes images and text to generate multilingual text outputs. This project shows how to create a self-hosted, private API that deploys PaliGemma 2's [fine-tuned variant](https://huggingface.co/google/paligemma2-3b-ft-docci-448) on the DOCCI dataset with LitServe, an easy-to-use, flexible serving engine for AI models built on FastAPI.

## Project Structure

The project is structured as follows:

- `server.py`: The file containing the main code for the web server.

- `client.py`: The file containing the code for client-side requests.

- `LICENSE`: The license file for the project.

- `README.md`: The README file that contains information about the project.

- `assets`: The folder containing screenshots for working on the application.

- `images`: The folder containing images for testing purposes.

- `.env.example`: The example file for environment variables.

- `.gitignore`: The file containing the list of files and directories to be ignored by Git.

## Tech Stack

- Python (for the programming language)

- PyTorch (for the deep learning framework)

- Hugging Face Transformers Library (for the model)

- LitServe (for the serving engine)

## Getting Started

To get started with this project, follow the steps below:

1. Run the server: `python server.py`

2. Upon running the server successfully, you will see uvicorn running on port 8000.

3. Open a new terminal window.

4. Run the client: `python client.py`

Now, you can see the model's output based on the input request. The model will generate captions in the selected language for the images provided in the `images` folder.

**Note**: You need a Hugging Face access token to run the application. You can get the token by signing up on the Hugging Face website and creating a new token from the settings page. After getting the token, you can set it as an environment variable `ACCESS_TOKEN` in your system by creating a `.env` file in the project's root directory. Check the `.env.example` file for reference.

## Usage

The project can be used to serve the PaliGemma 2 family of models using LitServe. It particularly allows you to input an image and select a language to generate a caption, suggesting potential use cases in semantic tagging, visual question answering, and more.

## Contributing

Contributions are welcome! If you would like to contribute to this project, please raise an issue to discuss the changes you want to make. Once the changes are approved, you can create a pull request.

## License

This project is licensed under the [Apache-2.0 License](LICENSE).

## Contact

If you have any questions or suggestions about the project, feel free to contact me on my GitHub profile.

Happy coding! 🚀