{"id":32465226,"url":"https://github.com/umair-hassan2/coco-urdu","last_synced_at":"2026-06-17T16:32:04.443Z","repository":{"id":314492291,"uuid":"1053742314","full_name":"umair-hassan2/COCO-Urdu","owner":"umair-hassan2","description":"Large-Scale Urdu Image-Caption Dataset with Multimodal Quality Estimation","archived":false,"fork":false,"pushed_at":"2025-09-12T18:17:12.000Z","size":5794,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-10-26T13:52:23.874Z","etag":null,"topics":["clip","dataset","image-captioning","multimodal-ai","natual-language-processing","qwen3","seamlessm4t","vision-transformer"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/umair-hassan2.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-09T21:38:49.000Z","updated_at":"2025-09-12T18:17:15.000Z","dependencies_parsed_at":"2025-09-12T21:03:30.615Z","dependency_job_id":null,"html_url":"https://github.com/umair-hassan2/COCO-Urdu","commit_stats":null,"previous_names":["umair-hassan2/coco-urdu"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/umair-hassan2/COCO-Urdu","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/umair-hassan2%2FCOCO-Urdu","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/umair-hassan2%2FCOCO-Urdu/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/umair-hassan2%2FCOCO-Urdu/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/umair-hassan2%2FCOCO-Urdu/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/umair-hassan2","download_url":"https://codeload.github.com/umair-hassan2/COCO-Urdu/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/umair-hassan2%2FCOCO-Urdu/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34454695,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-17T02:00:05.408Z","response_time":127,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clip","dataset","image-captioning","multimodal-ai","natual-language-processing","qwen3","seamlessm4t","vision-transformer"],"created_at":"2025-10-26T13:52:13.566Z","updated_at":"2026-06-17T16:32:04.438Z","avatar_url":"https://github.com/umair-hassan2.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# COCO-Urdu: Large-Scale Urdu Image-Caption Dataset with Multimodal Quality Estimation\n\nThis repository contains scripts and instructions to reproduce the **COCO-Urdu** dataset, a large-scale Urdu image-caption dataset derived from MS COCO. The dataset is accompanied by a hybrid multimodal quality estimation (QE) pipeline for translation, validation, and iterative refinement.\n\n- **Research Paper:** https://www.arxiv.org/abs/2509.09014\n- **Dataset:** [COCO-Urdu on Hugging Face](https://huggingface.co/datasets/umairhassan02/urdu-translated-coco-captions-subset)\n\n---\n\n## Overview\n\nUrdu, spoken by over 250 million people, remains critically under-served in multimodal and vision-language research. COCO-Urdu addresses this gap by providing 59K images and 319K high-quality Urdu captions. Captions were generated via zero-shot translation using SeamlessM4T v2, validated with a hybrid QE pipeline combining COMET-Kiwi, CLIP-based visual grounding, and BERTScore with back-translation, and low-scoring captions were iteratively refined using open-source LLMs.\n\nThis repository contains scripts for preprocessing, translation, quality estimation, refinement, benchmarking, and dataset assembly to reproduce the full pipeline.\n\n---\n\n## Reproduction Steps\n\nFollow these steps to reproduce the COCO-Urdu dataset:\n\n1. **Dataset Splitting**  \n   Load the MS COCO dataset and preprocess it using the `dataset_split` script to generate the desired training and validation splits.\n\n2. **Translation with Hybrid QE**  \n   Generate Urdu translations and validate them with the hybrid multimodal quality estimation pipeline using the `translation_pipeline` script.  \n   **Requirements:** GPU (≥32GB recommended, e.g., A100 or RTX 5090). The pipeline supports distributed execution across multiple GPUs for faster processing.\n\n3. **Reference Translation Generation**  \n   Produce reference translations for evaluation using the `nllb_reference_generation` script. GPU execution is recommended.\n\n4. **Iterative Refinement of Low-Scoring Captions**  \n   Refine captions identified as low-quality by the hybrid QE pipeline using the `targeted_refinements` script.\n\n5. **Benchmarking**  \n   Evaluate captions before and after refinement with the `benchmarking` script. This handles all metrics and ensures consistency across iterations.\n\n6. **Final Dataset Assembly**  \n   Collate all processed translations and generate the final version of COCO-Urdu using the `generate_final_version` script.\n\n**Note:** Depending on hardware and compute resources, full reproduction may take up to ~40 hours.\n\n---\n\n**Consider citing the work if you find this helpful**\n\n```bibtex\n@inproceedings{Hassan2025COCOUrduAL,\n  title={COCO-Urdu: A Large-Scale Urdu Image-Caption Dataset with Multimodal Quality Estimation},\n  author={Umair Hassan},\n  year={2025},\n  url={https://api.semanticscholar.org/CorpusID:281252320}\n}\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fumair-hassan2%2Fcoco-urdu","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fumair-hassan2%2Fcoco-urdu","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fumair-hassan2%2Fcoco-urdu/lists"}