{"id":16277918,"url":"https://github.com/adithya-s-k/yologemma","last_synced_at":"2025-04-30T09:26:01.894Z","repository":{"id":240961517,"uuid":"803918892","full_name":"adithya-s-k/YoloGemma","owner":"adithya-s-k","description":"Testing and evaluating the capabilities of Vision-Language models (PaliGemma) in performing computer vision tasks such as object detection and segmentation.","archived":false,"fork":false,"pushed_at":"2024-05-29T17:55:50.000Z","size":12344,"stargazers_count":80,"open_issues_count":1,"forks_count":5,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-30T14:51:11.862Z","etag":null,"topics":["gemma","paligemma","vlm"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/adithya-s-k.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-21T15:55:31.000Z","updated_at":"2025-02-18T07:21:13.000Z","dependencies_parsed_at":"2024-05-21T17:31:42.258Z","dependency_job_id":"c22198c2-5dc1-4377-b109-01ec6b319703","html_url":"https://github.com/adithya-s-k/YoloGemma","commit_stats":null,"previous_names":["adithya-s-k/yologemma"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adithya-s-k%2FYoloGemma","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adithya-s-k%2FYoloGemma/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adithya-s-k%2FYoloGemma/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/adithya-s-k%2FYoloGemma/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/adithya-s-k","download_url":"https://codeload.github.com/adithya-s-k/YoloGemma/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251675749,"owners_count":21625882,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gemma","paligemma","vlm"],"created_at":"2024-10-10T18:56:45.871Z","updated_at":"2025-04-30T09:26:01.867Z","avatar_url":"https://github.com/adithya-s-k.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# YoloGemma\n\n**YoloGemma** is a project showcasing the capabilities of Vision-Language models in performing computer vision tasks such as object detection and segmentation. At the heart of this experiment lies [**PaliGemma**](https://huggingface.co/google/paligemma-3b-mix-224), a state-of-the-art model that bridges the gap between Language and Vision. Through YoloGemma, we aim to explore whether Vision-Language models can match conventional methods of computer vision.\n\n## Outputs\n\nYoloGemma generates outputs by processing images and videos to identify and segment objects within them. The results are visualized as annotated images or videos, highlighting detected objects with bounding boxes or segmentation masks.\n\n\u003ctable\u003e\n    \u003ctr\u003e\n        \u003ctd\u003e\n            Detect Big Cat:\n            \u003cvideo width=\"100%\" src=\"https://github.com/adithya-s-k/YoloGemma/assets/27956426/7647388d-4dff-4c15-b1a1-12c2be546c08\"\u003e\n            \u003c/td\u003e\n        \u003ctd\u003e\n            Detect Small Cat:\n            \u003cvideo width=\"100%\" src=\"https://github.com/adithya-s-k/YoloGemma/assets/27956426/3c912f39-4a14-407b-a0d3-59f917ddf413\"\u003e\n        \u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003e\n            Detect Gojo:\n            \u003cvideo width=\"100%\" src=\"https://github.com/adithya-s-k/YoloGemma/assets/27956426/1fe68517-0705-47b2-a32b-c574a5c6756f\"\u003e\n        \u003c/td\u003e\n        \u003ctd\u003e\n            Detect Short Person:\n            \u003cvideo width=\"100%\" src=\"https://github.com/adithya-s-k/YoloGemma/assets/27956426/5766b75d-4b9a-4a0b-8228-57d2b866b550\"\u003e\n        \u003c/td\u003e\n    \u003c/tr\u003e\n\u003c/table\u003e\n\n## Installation\n\nTo get started with YoloGemma, follow these simple installation steps:\n\n1. Clone the repository:\n    ```bash\n    git clone https://github.com/your-username/YoloGemma.git\n    cd YoloGemma\n    ```\n\n2. Install the required dependencies:\n    ```bash\n    conda create -n YoloGemma-venv python=3.10\n    conda activate YoloGemma-venv\n    pip install -e .\n    ```\n\n## How to Run\n\n### Model Download\nYou can download the model by running the following command:\n```bash\npython download.py\n```\nThis command will download and quantize the model.\n\nYoloGemma provides three main scripts to facilitate various tasks. Below are instructions on how to run each script:\n\n### Main Script for Object Detection\n```bash\npython main.py --prompt \"Detect 4 people\" --vid_path ./people.mp4 --vid_start 1 --vid_end 12 --max_new_tokens 10\n```\n\n#### Command Line Arguments\n\n- **`--prompt`** (type: `str`, default: `\"detect cat\"`): The prompt specifying what to detect in the video.\n- **`--vid_path`** (type: `str`, default: `\"\"`): The path to the input MP4 video file.\n- **`--vid_start`** (type: `int`, default: `0`): The start time in seconds where the detection should begin.\n- **`--vid_end`** (type: `int`, default: `10`): The end time in seconds where the detection should stop.\n- **`--max_new_tokens`** (type: `int`, default: `15`): Maximum number of new tokens.\n\n#### Additional Parameters\n\n- **`--interactive`** (action: `store_true`): Launch the application in interactive mode.\n- **`--top_k`** (type: `int`, default: `200`): Top-k sampling for generating new tokens.\n- **`--temperature`** (type: `float`, default: `0.8`): Sampling temperature.\n- **`--checkpoint_path`** (type: `Path`, default: `Path(\"checkpoints/google/paligemma-3b-mix-224/modelint8.pth\")`): Path to the model checkpoint file.\n- **`--compile`** (action: `store_true`, default: `True`): Whether to compile the model.\n- **`--compile_prefill`** (action: `store_true`): Whether to compile the prefill for improved performance.\n- **`--profile`** (type: `Path`, default: `None`): Path to the profile.\n- **`--speculate_k`** (type: `int`, default: `5`): Speculative execution depth.\n- **`--draft_checkpoint_path`** (type: `Path`, default: `None`): Path to the draft checkpoint.\n- **`--device`** (type: `str`, default: `\"cuda\"`): Device to use for running the model (e.g., `\"cuda\"` for GPU).\n\n## Example\n\n```bash\npython main.py --prompt \"Detect 4 people\" --vid_path ./people.mp4 --vid_start 1 --vid_end 12 --max_new_tokens 10\n```\n\nThis command will start the detection process for the prompt \"Detect 4 people\" on the video located at `./people.mp4`, beginning at 1 second and ending at 12 seconds into the video. It will use a maximum of 10 new tokens during processing.\n\n### Gradio Interface (Coming Soon)\n\n```bash\npython demo.py\n```\nThis command will launch a Gradio interface, providing an interactive web application to perform object detection and segmentation.\n\n## Troubleshooting\n\nIf you encounter any issues, please ensure that:\n- The video file path is correct and the file is accessible.\n- The required dependencies are installed.\n- Your system has the necessary hardware (e.g., a compatible GPU if using CUDA).\n\nFor further assistance, please refer to the project's [issues page](https://github.com/adithyas-s-k/YoloGemma/issues) or contact the maintainers.\n\n## Acknowledgements\n\nSpecial thanks to [PaliGemma](https://huggingface.co/blog/paligemma) for their groundbreaking work in Vision-Language models, which serves as the foundation for this project. The project was inspired by this repository - [loopvlm](https://github.com/sumo43/loopvlm).\n\n---\n\nYoloGemma is an exciting experimental step towards the future of vision-language model-based computer vision, blending the strengths of language models with visual understanding.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fadithya-s-k%2Fyologemma","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fadithya-s-k%2Fyologemma","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fadithya-s-k%2Fyologemma/lists"}