{"id":28903463,"url":"https://github.com/mouleshgs/yolo-dense-captioning","last_synced_at":"2026-04-29T10:04:39.634Z","repository":{"id":300225347,"uuid":"1005165982","full_name":"mouleshgs/yolo-dense-captioning","owner":"mouleshgs","description":"Real-time video captioning using YOLOv8 for object detection, GIT for initial captions, and LLaMA 3 for natural language enhancement — all running locally with Ollama.","archived":false,"fork":false,"pushed_at":"2025-06-20T13:34:52.000Z","size":10,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-06-20T14:33:05.606Z","etag":null,"topics":["dense-captioning","image-to-text","llama3","yolov8"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mouleshgs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-19T19:26:20.000Z","updated_at":"2025-06-20T13:37:39.000Z","dependencies_parsed_at":"2025-06-20T14:34:05.484Z","dependency_job_id":"08f42a28-a660-432f-bd11-7d184dc9a251","html_url":"https://github.com/mouleshgs/yolo-dense-captioning","commit_stats":null,"previous_names":["mouleshgs/yolo-dense-captioning"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/mouleshgs/yolo-dense-captioning","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mouleshgs%2Fyolo-dense-captioning","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mouleshgs%2Fyolo-dense-captioning/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mouleshgs%2Fyolo-dense-captioning/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mouleshgs%2Fyolo-dense-captioning/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mouleshgs","download_url":"https://codeload.github.com/mouleshgs/yolo-dense-captioning/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mouleshgs%2Fyolo-dense-captioning/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261119462,"owners_count":23112204,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dense-captioning","image-to-text","llama3","yolov8"],"created_at":"2025-06-21T12:02:42.294Z","updated_at":"2026-04-29T10:04:39.623Z","avatar_url":"https://github.com/mouleshgs.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# YOLO-Based Dense Captioning in Real-Time\n\nThis project performs real-time video analysis by combining object detection using YOLOv8, image captioning using GIT, and caption enhancement through a local LLaMA 3 language model.\n\nThe result is a more descriptive and natural understanding of the scene captured from a live video stream.\n\n---\n\n## Features\n\n- Real-time video capture from webcam\n- Object detection using YOLOv8 (via Ultralytics)\n- Caption generation using GIT (`microsoft/git-base`)\n- Caption enhancement using LLaMA 3 (via Ollama)\n- Visual display of bounding boxes and enhanced captions on video frames\n\n---\n\n## Project Structure\n\n```\nyolo-dense-captioning/\n├── main.py                   # Main script: handles video capture, detection, captioning, and enhancement\n├── detector.py              # YOLOv8 detector class\n├── caption_model.py         # GIT-based caption generator\n├── caption_enhancer.py      # LLaMA 3 caption enhancer using Ollama's local API\n├── utils/\n│   ├── preprocessor.py      # Image/frame preprocessing helpers\n│   └── visualizer.py        # Functions to draw bounding boxes and captions on frames\n├── assets/                  # Optional: sample frames or icons\n├── requirements.txt         # Python dependencies\n└── README.md                # Project documentation\n```\n\n---\n\n## Installation\n\n1. Clone the repository:\n\n```bash\ngit clone https://github.com/mouleshgs/yolo-dense-captioning.git\ncd yolo-dense-captioning\n```\n\n2. Set up a virtual environment:\n\n```bash\npython -m venv .venv\nsource .venv/bin/activate  # On Windows: .venv\\Scripts\\activate\n```\n\n3. Install the required packages:\n\n```bash\npip install -r requirements.txt\n```\n\n4. Install and run Ollama for LLaMA 3:\n\n- Download Ollama: https://ollama.com\n- Pull the model:\n\n```bash\nollama pull llama3\n```\n\n---\n\n## Running the Project\n\nTo start real-time captioning:\n\n```bash\npython main.py\n```\n\nThe webcam will open, and you'll see live object detection with an enhanced caption displayed at the top of the screen. Press `q` to exit.\n\n---\n\n## Notes\n\n- The LLaMA 3 model runs locally via Ollama. Ensure Ollama is properly installed and accessible from your system PATH.\n- Caption generation is based on `microsoft/git-base` and may vary depending on image clarity.\n- For faster inference, you can use a optimized model like `llama3.1` or `llama3.2`.\n\n---\n\n## License\n\nThis project is for educational and research purposes. For commercial use, please check the licenses of Meta’s LLaMA 3, Microsoft's GIT, and Ultralytics YOLOv8.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmouleshgs%2Fyolo-dense-captioning","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmouleshgs%2Fyolo-dense-captioning","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmouleshgs%2Fyolo-dense-captioning/lists"}