{"id":24835702,"url":"https://github.com/raj-tyagi/4clip-image-captioning","last_synced_at":"2026-05-16T08:02:55.970Z","repository":{"id":272153523,"uuid":"915676565","full_name":"raj-tyagi/4CLIP-Image-Captioning","owner":"raj-tyagi","description":"This repository presents 4CLIP, a novel approach to image captioning that enhances traditional models by dividing images into four quadrants and processing them individually. By leveraging a pretrained ViT-GPT2 model from Hugging Face, 4CLIP generates more detailed and comprehensive captions, making it suitable for fine-grained visual tasks.","archived":false,"fork":false,"pushed_at":"2025-01-27T16:58:04.000Z","size":295,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-08-07T09:30:54.725Z","etag":null,"topics":["computer-vision","deep-learning","image-captioning","natural-language-processing","pretrained-models","pytorch","quadrant-based-analysis","research-based-approaches","transformers","vision-transformers"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/raj-tyagi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-01-12T14:11:34.000Z","updated_at":"2025-01-27T16:58:07.000Z","dependencies_parsed_at":"2025-01-12T15:30:41.761Z","dependency_job_id":"752e2590-5da7-4203-875a-f975e6724e83","html_url":"https://github.com/raj-tyagi/4CLIP-Image-Captioning","commit_stats":null,"previous_names":["raj-tyagi/4clip-image-captioning"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/raj-tyagi/4CLIP-Image-Captioning","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/raj-tyagi%2F4CLIP-Image-Captioning","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/raj-tyagi%2F4CLIP-Image-Captioning/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/raj-tyagi%2F4CLIP-Image-Captioning/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/raj-tyagi%2F4CLIP-Image-Captioning/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/raj-tyagi","download_url":"https://codeload.github.com/raj-tyagi/4CLIP-Image-Captioning/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/raj-tyagi%2F4CLIP-Image-Captioning/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273752967,"owners_count":25161910,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-05T02:00:09.113Z","response_time":402,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","deep-learning","image-captioning","natural-language-processing","pretrained-models","pytorch","quadrant-based-analysis","research-based-approaches","transformers","vision-transformers"],"created_at":"2025-01-31T04:51:23.348Z","updated_at":"2025-10-05T15:09:37.938Z","avatar_url":"https://github.com/raj-tyagi.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 4CLIP: Enhanced Image Captioning using Quadrant-Based Feature Extraction\r\n\r\n4CLIP is a research-driven image captioning project that enhances traditional image captioning models by dividing images into four quadrants and processing them individually. This quadrant-based approach allows for better image understanding and more comprehensive captions. The project utilizes a pretrained VisionEncoderDecoderModel from the Hugging Face Transformers library, specifically the ViT-GPT2 model. \r\n\r\n## Repository Name\r\n`4CLIP-Image-Captioning`\r\n\r\n## Project Structure\r\n```\r\n├── README.md                      # Detailed project description (this file)\r\n├── requirements.txt               # List of necessary packages for the project\r\n├── src/                           # Source code directory containing Python scripts\r\n│   ├── image_captioning.py        # Core script containing image captioning functions\r\n│   └── utils.py                   # Utility functions for image processing\r\n└── LICENSE                        # Project license\r\n```\r\n\r\n## Installation\r\nTo run this project, ensure you have Python installed. Then, install the necessary packages:\r\n```bash\r\npip install transformers torch requests pillow matplotlib tqdm\r\n```\r\n\r\n## Usage\r\nYou can run the project directly through the Jupyter Notebook provided (`4clip_image_captioning.ipynb`).\r\n\r\n```python\r\n# Example Usage:\r\nfrom src.image_captioning import compare_captions\r\ncompare_captions(url=\"https://example.com/sample.jpg\", greedy=True)\r\n```\r\n\r\n### Functionality\r\n1. **Full Image Captioning:** Captions generated for the entire image.\r\n2. **Quadrant Captioning:** The image is split into four quadrants, and captions are generated for each.\r\n3. **Final Caption (4CLIP):** Captions generated using combined quadrant features.\r\n\r\n## How It Works\r\n### Steps Implemented:\r\n1. **Image Splitting:** The image is divided into four quadrants.\r\n2. **Feature Extraction:** Features are extracted using the ViT processor.\r\n3. **Caption Generation:**\r\n   - Traditional Caption: Generated using the full image.\r\n   - Quadrant Captions: Each quadrant is captioned separately.\r\n   - Final Caption: Quadrant features are combined and passed into the model for a comprehensive caption.\r\n\r\n## Example Results\r\n**Traditional Caption:**\r\n\u003e \"A flag waving in the wind.\"\r\n\r\n**4CLIP Captions:**\r\n- Quadrant 1: \"A red and yellow flag.\"\r\n- Quadrant 2: \"Blue sky in the background.\"\r\n- Quadrant 3: \"A waving flag pole.\"\r\n- Quadrant 4: \"A bright outdoor scene.\"\r\n\r\n**Final Caption (4CLIP):**\r\n\u003e \"A red and yellow flag waving in the wind with a bright blue sky.\"\r\n\r\n## Research Objective\r\nThe aim of this project is to explore whether splitting an image into multiple parts and combining their features can generate more detailed and descriptive captions. This could be particularly useful in domains requiring fine-grained visual understanding, such as medical imaging and assistive technologies.\r\n\r\n## Contributing\r\nContributions are welcome! Please fork the repository, create a feature branch, and submit a pull request.\r\n\r\n## License\r\nThis project is licensed under the MIT License. See the `LICENSE` file for more details.\r\n\r\n## Acknowledgments\r\n- Hugging Face for the pretrained ViT-GPT2 model.\r\n- Python libraries: PyTorch, Transformers, Pillow, Matplotlib.\r\n\r\n---\r\n\r\n**Raj Tyagi**: [LinkedIn](https://www.linkedin.com/in/raj-tyagi-83765b21b/) \r\n**Tushar Sood**:  [LinkedIn](https://www.linkedin.com/in/tushar-sood-03927b238/)\r\n\r\n\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fraj-tyagi%2F4clip-image-captioning","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fraj-tyagi%2F4clip-image-captioning","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fraj-tyagi%2F4clip-image-captioning/lists"}