{"id":50412638,"url":"https://github.com/Karine-Huang/T2I-CompBench","last_synced_at":"2026-06-16T20:00:40.380Z","repository":{"id":181166254,"uuid":"664145835","full_name":"Karine-Huang/T2I-CompBench","owner":"Karine-Huang","description":"[Neurips 2023 \u0026 TPAMI] T2I-CompBench (++) for Compositional Text-to-image Generation Evaluation","archived":false,"fork":false,"pushed_at":"2026-05-07T07:04:30.000Z","size":81175,"stargazers_count":339,"open_issues_count":20,"forks_count":21,"subscribers_count":3,"default_branch":"main","last_synced_at":"2026-05-07T09:10:03.518Z","etag":null,"topics":["benchmark","compositionality","dataset","text-to-image"],"latest_commit_sha":null,"homepage":"https://arxiv.org/pdf/2307.06350.pdf","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Karine-Huang.png","metadata":{"files":{"readme":"Readme.md","changelog":null,"contributing":null,"funding":null,"license":"License.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":"NOTICE.md","maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2023-07-09T03:34:23.000Z","updated_at":"2026-05-07T07:04:36.000Z","dependencies_parsed_at":"2025-04-10T08:32:45.578Z","dependency_job_id":null,"html_url":"https://github.com/Karine-Huang/T2I-CompBench","commit_stats":null,"previous_names":["karine-huang/t2i-compbench"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Karine-Huang/T2I-CompBench","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Karine-Huang%2FT2I-CompBench","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Karine-Huang%2FT2I-CompBench/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Karine-Huang%2FT2I-CompBench/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Karine-Huang%2FT2I-CompBench/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Karine-Huang","download_url":"https://codeload.github.com/Karine-Huang/T2I-CompBench/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Karine-Huang%2FT2I-CompBench/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34421326,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-16T02:00:06.860Z","response_time":126,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmark","compositionality","dataset","text-to-image"],"created_at":"2026-05-31T04:30:35.937Z","updated_at":"2026-06-16T20:00:40.375Z","avatar_url":"https://github.com/Karine-Huang.png","language":"Python","funding_links":[],"categories":["Code Availability Summary"],"sub_categories":["With Official Code ✅"],"readme":"\r\n# T2I-CompBench(++): Benchmark for Compositonal Text-to-Image Generation\r\n\r\n\u003ca href='https://karine-h.github.io/T2I-CompBench-new/'\u003e\u003cimg src='https://img.shields.io/badge/Project-Page-Green'\u003e\u003c/a\u003e\r\n\u003ca href='https://ieeexplore.ieee.org/abstract/document/10847875'\u003e\u003cimg src='https://img.shields.io/badge/T2I--CompBench++-Paper-red'\u003e\u003c/a\u003e \r\n\u003ca href='https://arxiv.org/pdf/2307.06350v2'\u003e\u003cimg src='https://img.shields.io/badge/T2I--CompBench-Arxiv-red'\u003e\u003c/a\u003e \r\n\u003ca href='https://connecthkuhk-my.sharepoint.com/:f:/g/personal/huangky_connect_hku_hk/Er_BhrcMwGREht6gnKGIErMB4egvvKM5ouhmkc0u5ZIKPw'\u003e\u003cimg src='https://img.shields.io/badge/Dataset-T2I--CompBench++-blue'\u003e\u003c/a\u003e \r\n\u003ca href='https://connecthkuhk-my.sharepoint.com/:u:/g/personal/huangky_connect_hku_hk/EXEFBTzE6khPlsx2qPMjF9EBQYkE4WC2Z_XQGIjRUevjRQ'\u003e\u003cimg src='https://img.shields.io/badge/Dataset-Human eval images-purple'\u003e\u003c/a\u003e \r\n\r\nThis repository contains the following papers:\r\n\u003e **T2I-CompBench: A Comprehensive Benchmark for Open-world Compositional Text-to-image Generation**\u003cbr\u003e\r\n\u003e [Kaiyi Huang](https://scholar.google.com/citations?user=dB86D_cAAAAJ\u0026hl=zh-CN\u0026oi=sra)\u003csup\u003e1\u003c/sup\u003e, [Kaiyue Sun](https://scholar.google.com/citations?user=mieuBzUAAAAJ\u0026hl=zh-CN\u0026oi=sra)\u003csup\u003e1\u003c/sup\u003e, [Enze Xie](https://xieenze.github.io/)\u003csup\u003e2\u003c/sup\u003e, [Zhenguo Li](https://scholar.google.com.sg/citations?user=XboZC1AAAAAJ\u0026hl=en)\u003csup\u003e2\u003c/sup\u003e, [Xihui Liu](https://xh-liu.github.io/)\u003csup\u003e1+\u003c/sup\u003e\u003cbr\u003e\r\n\u003e \u003csup\u003e1\u003c/sup\u003eThe University of Hong Kong, \u003csup\u003e2\u003c/sup\u003eHuawei Noah’s Ark Lab\u003cbr\u003e\r\n\u003e Conference on Neural Information Processing Systems (**Neurips**), 2023\r\n\r\n\u003e **T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-image Generation**\u003cbr\u003e\r\n\u003e [Kaiyi Huang](https://scholar.google.com/citations?user=dB86D_cAAAAJ\u0026hl=zh-CN\u0026oi=sra)\u003csup\u003e1\u003c/sup\u003e, [Chengqi Duan](https://scholar.google.com/citations?user=r9qb4ZwAAAAJ\u0026hl=zh-CN\u0026oi=sra)\u003csup\u003e1,3\u003c/sup\u003e, [Kaiyue Sun](https://scholar.google.com/citations?user=mieuBzUAAAAJ\u0026hl=zh-CN\u0026oi=sra)\u003csup\u003e1\u003c/sup\u003e, [Enze Xie](https://xieenze.github.io/)\u003csup\u003e2\u003c/sup\u003e, [Zhenguo Li](https://scholar.google.com.sg/citations?user=XboZC1AAAAAJ\u0026hl=en)\u003csup\u003e2\u003c/sup\u003e, [Xihui Liu](https://xh-liu.github.io/)\u003csup\u003e1+\u003c/sup\u003e\u003cbr\u003e\r\n\u003e \u003csup\u003e1\u003c/sup\u003eThe University of Hong Kong, \u003csup\u003e2\u003c/sup\u003eHuawei Noah’s Ark Lab, \u003csup\u003e3\u003c/sup\u003eTsinghua University\u003cbr\u003e\r\n\u003e IEEE Transactions on Pattern Analysis and Machine Intelligence (**TPAMI**), 2025\r\n\r\n## 🚩 **New Features/Updates**\r\n- ✅ Jan. 11, 2025. 💥 T2I-Compbench++ accepted to TPAMI.\r\n- ✅ Nov. 06, 2024. Include evaluation results on [**FLUX.1**](https://github.com/black-forest-labs/flux).\r\n- ✅ Jun. 27, 2024. Release [**human evaluation**](https://connecthkuhk-my.sharepoint.com/:u:/g/personal/huangky_connect_hku_hk/EXEFBTzE6khPlsx2qPMjF9EBo6VR2scOo0KRY1M9g4CMkg?e=o5YT2k) of image-score pairs.\r\n- ✅ Mar. 14, 2024. Release a more comprehensive version of compositional benchmark T2I-CompBench++.\r\n- ✅ Mar. 05, 2024. 💥 Evaluation metric adopted by 🧨 [**Stable Diffusion 3**](https://arxiv.org/pdf/2403.03206).\r\n- ✅ Dec. 02, 2023. Release the inference code for generating images in metric evaluation.\r\n- ✅ Oct. 20, 2023. 💥 Evaluation metric adopted by 🧨 [**DALL-E 3**](https://cdn.openai.com/papers/dall-e-3.pdf) as the evaluation metric for compositionality.\r\n- ✅ Sep. 30, 2023. 💥 Evaluation metric adopted by 🧨 [**PixArt-α**](https://arxiv.org/pdf/2310.00426.pdf) as the evaluation metric for compositionality.\r\n- ✅ Sep. 22, 2023. 💥 Paper accepted to Neurips 2023.\r\n- ✅ Jul. 9, 2023. Release the dataset, training and evaluation code.\r\n\r\n\r\n\r\n## **Installing the dependencies**\r\n\r\nBefore running the scripts, make sure to install the library's training dependencies:\r\n\r\n**Important**\r\n\r\nWe recommend using the **latest code** to ensure consistency with the results presented in the paper. To make sure you can successfully run the example scripts, execute the following steps in a new virtual environment.\r\nWe use the **diffusers version** as **0.15.0.dev0**\r\nYou can either install the development version from PyPI: \r\n```bash\r\npip install diffusers==0.15.0.dev0\r\n```\r\nor install from the provided source:\r\n```bash\r\nunzip diffusers.zip\r\ncd diffusers\r\npip install .\r\n```\r\n\r\nThen cd in the example folder  and run\r\n```bash\r\npip install -r requirements.txt\r\n```\r\n\r\nAnd initialize an [🤗Accelerate](https://github.com/huggingface/accelerate/) environment with:\r\n\r\n```bash\r\naccelerate config\r\n```\r\n\r\n\r\n\r\n\r\n## **Evaluation**\r\n\r\n\r\nFor evaluation, the input images files are stored in the directory \"examples/samples/\", with the following structures:\r\n\r\n```\r\nexamples/samples/\r\n        ├── a green bench and a blue bowl_000000.png\r\n        ├── a green bench and a blue bowl_000001.png\r\n        └──...\r\n\r\n```\r\nThe evaluation result directory should include a json file named \"vqa_result.json\", and the json file should be a dictionary that maps from\r\n`{\"question_id\", \"answer\"}`, e.g.,\r\n\r\n```\r\n[{\"question_id\": 0, \"answer\": \"0.6900\"},\r\n {\"question_id\": 1, \"answer\": \"0.7110\"},\r\n ...]\r\n```\r\nThe question_id is the _last 6 digits_ of the image file name, for example, _a green bench and a blue bowl_000000.png_ is with question_id _0_, and the evaluation score _0.6900_.\r\n\r\nYou need to generate images for each category from this [dataset](https://github.com/Karine-Huang/T2I-CompBench/tree/main/examples/dataset) as described above separately, and get the evaluation results below. \r\n\r\n\r\n#### 1. BLIP-VQA for Attribute Binding:\r\n```\r\nexport project_dir=\"BLIPvqa_eval/\"\r\ncd $project_dir\r\nout_dir=\"../examples/\"\r\npython BLIP_vqa.py --out_dir=$out_dir\r\n```\r\nor run\r\n```\r\ncd T2I-CompBench\r\nbash BLIPvqa_eval/test.sh\r\n```\r\nThe output files are formatted as a json file named \"vqa_result.json\" in \"examples/annotation_blip/\" directory.\r\n\r\n\r\n#### 2. UniDet for 2D/3D-Spatial Relationships and Numeracy evaluation:\r\n\r\ndownload weight and put under repo experts/expert_weights:\r\n```\r\nmkdir -p UniDet_eval/experts/expert_weights\r\ncd UniDet_eval/experts/expert_weights\r\nwget https://huggingface.co/shikunl/prismer/resolve/main/expert_weights/Unified_learned_OCIM_RS200_6x%2B2x.pth\r\nwget https://huggingface.co/lllyasviel/ControlNet/resolve/main/annotator/ckpts/dpt_hybrid-midas-501f0c75.pt\r\npip install gdown\r\ngdown https://docs.google.com/uc?id=1C4sgkirmgMumKXXiLOPmCKNTZAc3oVbq\r\n```\r\n\r\nfor **2D-spatial** evaluation, run:\r\n```\r\nexport project_dir=UniDet_eval\r\ncd $project_dir\r\n\r\npython 2D_spatial_eval.py\r\n```\r\nTo calculate prompts from the **\"complex\" category**, set the **\"--complex\" parameter to True**; otherwise, set it to False. The output files are formatted as a json file named \"vqa_result.json\" in \"examples/labels/annotation_obj_detection_2d\" directory.\r\n\r\nfor **numeracy** evaluation, run:\r\n```\r\nexport project_dir=UniDet_eval\r\ncd $project_dir\r\n\r\npython numeracy_eval.py\r\n```\r\nThe output files are formatted as a json file named \"vqa_result.json\" in \"examples/annotation_num\" directory.\r\n\r\nfor **3D spatial** evaluation, run:\r\n```\r\nexport project_dir=UniDet_eval\r\ncd $project_dir\r\npython 3D_spatial_eval.py \r\n```\r\nThe output files are formatted as a json file named \"vqa_result.json\" in \"examples/labels/annotation_obj_detection_3d\" directory.\r\n\r\n#### 3. CLIPScore for Non-Spatial Relationships:\r\n```\r\noutpath=\"examples/\"\r\npython CLIPScore_eval/CLIP_similarity.py --outpath=${outpath}\r\n```\r\nor run\r\n```\r\ncd T2I-CompBench\r\nbash CLIPScore_eval/test.sh\r\n```\r\nTo calculate prompts from the **\"complex\" category**, set the **\"--complex\" parameter to True**; otherwise, set it to False. \r\nThe output files are formatted as a json file named \"vqa_result.json\" in \"examples/annotation_clip\" directory.\r\n\r\n\r\n#### 4. 3-in-1 for Complex Compositions:\r\n```\r\nexport project_dir=\"3_in_1_eval/\"\r\ncd $project_dir\r\noutpath=\"../examples/\"\r\npython \"3_in_1.py\" --outpath=${outpath}\r\n```\r\nThe output files are formatted as a json file named \"vqa_result.json\" in \"examples/annotation_3_in_1\" directory.\r\n\r\n#### 5. MLLM_eval:\r\nIf you want to use ShareGPT4V or GPT-4V as evaluation metrics as in TABLE XIII in [T2I-CompBench++](https://github.com/Karine-Huang/T2I-CompBench/blob/main/paper/T2I_CompBench%2B%2B.pdf), you can evaluate as follows.\r\nPrepare the environment for ShareGPT4V.\r\n\r\nShareGPT4V is based on its repositories, please refer to the link for environment dependencies and weights: \r\n```\r\nhttps://github.com/InternLM/InternLM-XComposer/tree/main/projects/ShareGPT4V\r\n```\r\n\r\nFor convenience, you can try the following commands to install ShareGPT4V's environment and download the required weights. \r\n```\r\nexport project_dir=MLLM_eval/ShareGPT4V-CoT_eval/\r\ncd $project_dir\r\nconda create -n share4v python=3.10 -y\r\nconda activate share4v\r\npip install --upgrade pip\r\npip install -e .\r\npip install -e \".[train]\"\r\npip install flash-attn --no-build-isolation\r\npip install spacy\r\npython -m spacy download en_core_web_sm\r\nmkdir -p Lin-Chen/\r\ncd Lin-Chen/\r\ngit lfs install\r\ngit clone https://huggingface.co/Lin-Chen/ShareGPT4V-7B_Pretrained_vit-large336-l12\r\n```\r\n\r\n##### GPT-4V (OpenAI):\r\n\r\nSet your OpenAI API key via the `OPENAI_API_KEY` environment variable (or edit line 13 of `gpt4v_eval.py` directly).\r\n\r\n```bash\r\nexport OPENAI_API_KEY=\"your-openai-api-key\"\r\nexport project_dir=MLLM_eval\r\ncd $project_dir\r\npython gpt4v_eval.py --category \"color\" --start 0 --step 10\r\n```\r\n\r\n##### MiniMax M2.7 (alternative to GPT-4V):\r\n\r\n[MiniMax M2.7](https://www.minimax.io) is a multimodal LLM with vision support and an OpenAI-compatible API, providing a cost-effective alternative to GPT-4V for evaluation. Use the `--provider minimax` flag:\r\n\r\n```bash\r\nexport MINIMAX_API_KEY=\"your-minimax-api-key\"\r\nexport project_dir=MLLM_eval\r\ncd $project_dir\r\npython gpt4v_eval.py --provider minimax --category \"color\" --start 0 --step 10\r\n```\r\n\r\nThe `--provider` argument accepts `openai` (default, uses GPT-4V) or `minimax` (uses MiniMax-M2.7). The API key is read from `OPENAI_API_KEY` or `MINIMAX_API_KEY` respectively.\r\n\r\nThe output files are formatted as a json file named \"gpt4v_result\\_{start}\\_{step}.json\" in \"examples/gpt4v\" directory.\r\n\r\nIn the paper we test 600 images, setting {start=0, step=10}, and {start=1, step=10} from existing 3000 images each category.\r\n\r\n##### ShareGPT4V-CoT:\r\nFor ShareGPT4V evaluation, run the following commands:\r\n```\r\nexport project_dir=MLLM_eval/ShareGPT4V-CoT_eval/\r\ncd $project_dir\r\ncategory=\"color\"\r\noutput_path=\"../../examples/\"\r\npython Share_eval.py --category ${category} --file-path ${output_path} --cot\r\n```\r\nThe output files are formatted as a json file named \"vqa_result.json\" in \"examples/sharegpt4v\" directory.\r\n\r\n\r\n\r\n\r\n## **Finetuning**\r\nIf you want to use GORS method to finetune your diffusion model, please follow the steps to prepare the training code and training images.\r\n\r\nThe training images are stored in the directory \"examples/samples/\", with the format the same as the evaluation data.\r\n\r\n1. LoRA finetuning\r\n\r\nUse LoRA finetuning method, please refer to the link for downloading \"lora_diffusion\" directory: \r\n\r\n```\r\nhttps://github.com/cloneofsimo/lora/tree/master\r\n```\r\n2. Example usage\r\n\r\n\r\n```\r\nexport project_dir=/T2I-CompBench\r\ncd $project_dir\r\n\r\nexport train_data_dir=\"examples/samples/\"\r\nexport output_dir=\"examples/output/\"\r\nexport reward_root=\"examples/reward/\"\r\nexport dataset_root=\"examples/dataset/color.txt\"\r\nexport script=GORS_finetune/train_text_to_image.py\r\n\r\naccelerate launch --multi_gpu --mixed_precision=fp16 \\\r\n--num_processes=8 --num_machines=1 \\\r\n--dynamo_backend=no \"${script}\" \\\r\n--train_data_dir=\"${train_data_dir}\" \\\r\n--output_dir=\"${output_dir}\" \\\r\n--reward_root=\"${reward_root}\" \\\r\n--dataset_root=\"${dataset_root}\"\r\n\r\n```\r\nor run\r\n```\r\ncd T2I-CompBench\r\nbash GORS_finetune/train.sh\r\n```\r\n\r\n\r\n\r\n\r\nThe image directory should be a directory containing the images, e.g.,\r\n\r\n\r\n```\r\nexamples/samples/\r\n        ├── a green bench and a blue bowl_000000.png\r\n        ├── a green bench and a blue bowl_000001.png\r\n        └──...\r\n\r\n```\r\nThe reward directory should include a json file named \"vqa_result.json\", and the json file should be a dictionary that maps from\r\n`{\"question_id\", \"answer\"}`, e.g.,\r\n\r\n```\r\n[{\"question_id\": 0, \"answer\": \"0.6900\"},\r\n {\"question_id\": 1, \"answer\": \"0.7110\"},\r\n ...]\r\n```\r\n\r\nThe dataset should be placed in the directory \"examples/dataset/\".\r\n\r\n\r\n### Inference\r\nRun the inference.py to visualize the image.\r\n```\r\nexport pretrained_model_path=\"checkpoint/color/lora_weight_e357_s124500.pt.pt\"\r\nexport prompt=\"A bathroom with green tile and a red shower curtain\"\r\npython inference.py --pretrained_model_path \"${pretrained_model_path}\" --prompt \"${prompt}\"\r\n```\r\n**Generate images for metric calculation.** Run the inference_eval.py to generate images in the test set. As stated in the paper, 10 images are generated per prompt for **metric calculation**, and we use the fixed seed across all methods.\r\nYou can specify the test set by changing the \"from_file\" parameter among {color_val.txt, shape_val.txt, texture_val.txt, spatial_val.txt, non_spatial_val.txt, complex_val.txt}.\r\n```\r\nexport from_file=\"../examples/dataset/color_val.txt\"\r\npython inference_eval.py  --from_file \"${from_file}\"\r\n```\r\n\r\n### Citation\r\nIf you're using T2I-CompBench(++) in your research or applications, please cite using this BibTeX:\r\n```bibtex\r\n@article{huang2023t2icompbench,\r\n  title={T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation},\r\n  author={Huang, Kaiyi and Sun, Kaiyue and Xie, Enze and Li, Zhenguo and Liu, Xihui},\r\n  journal={Advances in Neural Information Processing Systems},\r\n  volume={36},\r\n  pages={78723--78747},\r\n  year={2023}\r\n}\r\n@article{huang2025t2icompbench++,\r\nauthor={Huang, Kaiyi and Duan, Chengqi and Sun, Kaiyue and Xie, Enze and Li, Zhenguo and Liu, Xihui},\r\n        journal={ IEEE Transactions on Pattern Analysis Machine Intelligence },\r\n        title={{ T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-Image Generation }},\r\n        year={5555},\r\n        number={01},\r\n        ISSN={1939-3539},\r\n        pages={1-17},\r\n        url = {https://doi.ieeecomputersociety.org/10.1109/TPAMI.2025.3531907},\r\n        publisher={IEEE Computer Society},\r\n        address={Los Alamitos, CA, USA},\r\n        month=jan,\r\n}\r\n```\r\n\r\n\r\n  ### License\r\n\r\nThis project is licensed under the MIT License. See the \"License.txt\" file for details. For detailed information on third-party licenses, please see the [NOTICE.md](https://github.com/Karine-Huang/T2I-CompBench/edit/main/NOTICE.md) file.\r\n\r\n\r\n\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FKarine-Huang%2FT2I-CompBench","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FKarine-Huang%2FT2I-CompBench","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FKarine-Huang%2FT2I-CompBench/lists"}