{"id":23529428,"url":"https://github.com/tiger-ai-lab/imagenhub","last_synced_at":"2025-04-12T04:53:59.144Z","repository":{"id":198129275,"uuid":"698463055","full_name":"TIGER-AI-Lab/ImagenHub","owner":"TIGER-AI-Lab","description":"A one-stop library to standardize the inference and evaluation of all the conditional image generation models. (ICLR 2024)","archived":false,"fork":false,"pushed_at":"2025-04-11T03:39:25.000Z","size":111341,"stargazers_count":167,"open_issues_count":2,"forks_count":18,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-04-12T04:53:08.465Z","etag":null,"topics":["deep-learning","diffusion-models","generative-ai","generative-art","image-editing","image-generation","pytorch","stable-diffusion"],"latest_commit_sha":null,"homepage":"https://tiger-ai-lab.github.io/ImagenHub/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/TIGER-AI-Lab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-09-30T02:09:22.000Z","updated_at":"2025-04-11T03:39:29.000Z","dependencies_parsed_at":"2025-04-11T04:34:32.911Z","dependency_job_id":null,"html_url":"https://github.com/TIGER-AI-Lab/ImagenHub","commit_stats":null,"previous_names":["tiger-ai-lab/imagenhub"],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TIGER-AI-Lab%2FImagenHub","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TIGER-AI-Lab%2FImagenHub/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TIGER-AI-Lab%2FImagenHub/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TIGER-AI-Lab%2FImagenHub/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/TIGER-AI-Lab","download_url":"https://codeload.github.com/TIGER-AI-Lab/ImagenHub/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248519472,"owners_count":21117757,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","diffusion-models","generative-ai","generative-art","image-editing","image-generation","pytorch","stable-diffusion"],"created_at":"2024-12-25T21:10:33.859Z","updated_at":"2025-04-12T04:53:59.123Z","avatar_url":"https://github.com/TIGER-AI-Lab.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 🖼️ ImagenHub \n[![arXiv](https://img.shields.io/badge/arXiv-2310.01596-b31b1b.svg)](https://arxiv.org/abs/2310.01596)\n\n[![Docs](https://img.shields.io/badge/docs-latest-blue)](https://imagenhub.readthedocs.io/en/latest/)\n\n[![contributors](https://img.shields.io/github/contributors/TIGER-AI-Lab/ImagenHub)](https://github.com/TIGER-AI-Lab/ImagenHub/graphs/contributors)\n[![license](https://img.shields.io/github/license/TIGER-AI-Lab/ImagenHub.svg)](https://github.com/TIGER-AI-Lab/ImagenHub/blob/main/LICENSE)\n[![GitHub](https://img.shields.io/github/stars/TIGER-AI-Lab/ImagenHub?style=social)](https://github.com/TIGER-AI-Lab/ImagenHub)\n[![Hits](https://hits.seeyoufarm.com/api/count/incr/badge.svg?url=https%3A%2F%2Fgithub.com%2FTIGER-AI-Lab%2FImagenHub\u0026count_bg=%23C83DB9\u0026title_bg=%23555555\u0026icon=\u0026icon_color=%23E7E7E7\u0026title=visitors\u0026edge_flat=false)](https://hits.seeyoufarm.com)\n\nImagenHub: Standardizing the evaluation of conditional image generation models \n\u003cbr\u003e\nICLR 2024\n\n\u003cdiv align=\"center\"\u003e\n\u003cimg src=\"https://github.com/TIGER-AI-Lab/ImagenHub/blob/gh-pages/static/images/banner.png\" width=\"40%\"\u003e\n\u003c/div\u003e\n\n\nImagenHub is a one-stop library to standardize the inference and evaluation of all the conditional image generation models.\n* We define 7 prominent tasks and curate 7 high-quality evaluation datasets for each task. \n* We built a unified inference pipeline to ensure fair comparison. We currently support over 50 models.\n* We designed two human evaluation scores, i.e. Semantic Consistency and Perceptual Quality, along with comprehensive guidelines to evaluate generated images. \n* We provide code for visualization, autometrics and Amazon mechanical turk templates.\n\n\u003cdiv align=\"center\"\u003e\n \u003ca href = \"https://tiger-ai-lab.github.io/ImagenHub/\"\u003e[🌐 Project Page]\u003c/a\u003e \u003ca href = \"https://imagenhub.readthedocs.io/en/latest/index.html\"\u003e[📘 Documentation]\u003c/a\u003e \u003ca href = \"https://arxiv.org/abs/2310.01596\"\u003e[📄 Arxiv]\u003c/a\u003e \u003ca href = \"https://huggingface.co/ImagenHub\"\u003e[💾 Datasets]\u003c/a\u003e \u003ca href = \"https://chromaica.github.io/#imagen-museum\"\u003e[🏛️ ImagenMuseum]\u003c/a\u003e \u003ca href = \"https://huggingface.co/spaces/TIGER-Lab/GenAI-Arena\"\u003e[⚔️ GenAI-Arena]\u003c/a\u003e\n\u003c/div\u003e\n\n## 📰 News\n* 2024 Jun 10: [GenAI-Arena](https://arxiv.org/abs/2406.04485v1) Paper is out. It is featured on [Huggingface Daily Papers](https://huggingface.co/papers?date=2024-06-10).\n* 2024 Jun 07: ImagenHub is finally on PyPI! Check: [https://pypi.org/project/imagen-hub/](https://pypi.org/project/imagen-hub/)\n* 2024 Apr 07: We released [Human evaluation ratings](https://github.com/TIGER-AI-Lab/ImagenHub/tree/main/eval/human_ratings) According to our latest Arxiv paper.\n* 2024 Feb 14: Checkout [⚔️ GenAI-Arena ⚔️ : Benchmarking Visual Generative Models in the Wild](https://huggingface.co/spaces/TIGER-Lab/GenAI-Arena)! \n* 2024 Jan 15: Paper accepted to ICLR 2024! See you in Vienna! \n* 2024 Jan 7: We updated [Human Evaluation Guideline](https://imagenhub.readthedocs.io/en/latest/Guidelines/humaneval.html), [ImagenMuseum Submission](https://imagenhub.readthedocs.io/en/latest/Guidelines/imagenmuseum.html)! Now we welcome researchers to submit their method on ImagenMuseum with minimal effort.\n* 2023 Oct 23: Version 0.1.0 released! [ImagenHub’s documentation](https://imagenhub.readthedocs.io/en/latest/index.html) now available!\n* 2023 Oct 19: Code Released. Docs under construction.\n* 2023 Oct 13: We released [Imagen Museum](https://chromaica.github.io/#imagen-museum), a visualization page for all models from ImagenHub!\n* 2023 Oct 4: Our paper is featured on [Huggingface Daily Papers](https://huggingface.co/papers?date=2023-10-04)!\n* 2023 Oct 2: Paper available on [Arxiv](https://arxiv.org/abs/2310.01596). Code coming Soon!\n\n## 📄 Table of Contents\n\n- [🛠️ Installation](#%EF%B8%8F-installation-)\n- [👨‍🏫 Get Started](#-get-started-)\n- [📘 Documentation](#-documentation-)\n- [🧠 Philosophy](#-philosophy-)\n- [🙌 Contributing](#-contributing-)\n- [🖊️ Citation](#%EF%B8%8F-citation-)\n- [🤝 Acknowledgement](#-acknowledgement-)\n- [🎫 License](#-license-)\n\n## 🛠️ Installation [🔝](#-table-of-contents)\n\nInstall from PyPI:\n```\npip install imagen-hub\n```\n\nOr build from source:\n```python\ngit clone https://github.com/TIGER-AI-Lab/ImagenHub.git\ncd ImagenHub\nconda env create -f env_cfg/imagen_environment.yml\nconda activate imagen\npip install -e .\n```\n\nFor models like Dall-E, DreamEdit, and BLIPDiffusion, please see [Extra Setup](https://imagenhub.readthedocs.io/en/latest/Guidelines/install.html#environment-management)\n\nFor some models (Stable Diffusion, SDXL, CosXL, etc.), you need to login through `huggingface-cli`.\n```shell\nhuggingface-cli login\n```\n\n## 👨‍🏫 Get Started [🔝](#-table-of-contents)\n\n### Benchmarking\nTo reproduce our experiment reported in the paper:\n\nExample for text-guided image generation:\n```shell\npython3 benchmarking.py -cfg benchmark_cfg/ih_t2i.yml\n```\n\nNote that the expected output structure would be:\n```shell\nresult_root_folder\n└── experiment_basename_folder\n    ├── input (If applicable)\n    │   └── image_1.jpg ...\n    ├── model1\n    │   └── image_1.jpg ...\n    ├── model2\n    │   └── image_1.jpg ...\n    ├── ...\n```\n\nThen after running the experiment, you can run\n```shell\npython3 visualize.py --cfg benchmark_cfg/ih_t2i.yml\n```\nto produce a `index.html` file for visualization.\n\nThe file would look like something like this. We hosted our experiment results on [Imagen Museum](https://chromaica.github.io/#imagen-museum).\n\u003cimg src=\"https://i.imgur.com/0uOMhtT.png\" width=\"512\" /\u003e\n\n\n### Infering one model\n```python\nimport imagen_hub\n\nmodel = imagen_hub.load(\"SDXL\")\nimage = model.infer_one_image(prompt=\"people reading pictures in a museum, watercolor\", seed=1)\nimage\n```\n\u003cimg src=\"https://i.imgur.com/ruU0BJ0.jpg\" width=\"256\" /\u003e\n\n### Running Metrics\n```python\nfrom imagen_hub.metrics import MetricLPIPS\nfrom imagen_hub.utils import load_image, save_pil_image, get_concat_pil_images\n\ndef evaluate_one(model, real_image, generated_image):\n  score = model.evaluate(real_image, generated_image)\n  print(\"====\u003e Score : \", score)\n\nimage_I = load_image(\"https://chromaica.github.io/Museum/ImagenHub_Text-Guided_IE/input/sample_102724_1.jpg\")\nimage_O = load_image(\"https://chromaica.github.io/Museum/ImagenHub_Text-Guided_IE/DiffEdit/sample_102724_1.jpg\")\nshow_image = get_concat_pil_images([image_I, image_O], 'h')\n\nmodel = MetricLPIPS()\nevaluate_one(model, image_I, image_O) # ====\u003e Score :  0.11225218325853348\n\nshow_image\n```\n\u003cimg src=\"https://i.imgur.com/af8CB4c.jpg\" width=\"512\" /\u003e\n\n\n\n\n## 📘 Documentation [🔝](#-table-of-contents)\nThe tutorials and API documentation are hosted on [imagenhub.readthedocs.io](https://imagenhub.readthedocs.io/en/latest/index.html).\n\n## 🧠 Philosophy [🔝](#-philosophy-)\nBy streamlining research and collaboration, ImageHub plays a pivotal role in propelling the field of Image Generation and Editing.\n\n* Purity of Evaluation: We ensure a fair and consistent evaluation for all models, eliminating biases.\n* Research Roadmap: By defining tasks and curating datasets, we provide clear direction for researchers. \n* Open Collaboration: Our platform fosters the exchange and cooperation of related technologies, bringing together minds and innovations.\n\n### Implemented Models\nWe included more than 30 Models in image synthesis. See the full list here:\n* Supported Models: https://github.com/TIGER-AI-Lab/ImagenHub/issues/1\n* Supported Metrics: https://github.com/TIGER-AI-Lab/ImagenHub/issues/6\n\n|        Method     \t         |   Venue  \t    |            Type           \t|\n|:---------------------------:|:-------------:|:-------------------------:\t|\n|       Stable Diffusion   \t        |  - \t   | Text-To-Image Generation \t|\n|       Stable Diffusion XL   \t        |  arXiv'23 \t   | Text-To-Image Generation \t|\n|       DeepFloyd-IF   \t        |  - \t   | Text-To-Image Generation \t|\n|       OpenJourney   \t        |  - \t   | Text-To-Image Generation \t|\n|       Dall-E   \t        |  - \t   | Text-To-Image Generation \t|\n|       Kandinsky  \t        |  - \t   | Text-To-Image Generation \t|\n|       MagicBrush   \t        |  arXiv'23 \t   | Text-guided Image Editing \t|\n|      InstructPix2Pix \t      |   CVPR'23 \t   | Text-guided Image Editing \t|\n|        DiffEdit    \t        |  ICLR'23 \t   | Text-guided Image Editing \t|\n|         Imagic    \t         |   CVPR'23\t   | Text-guided Image Editing \t|\n|     CycleDiffusion    \t     |  ICCV'23 \t   | Text-guided Image Editing \t|\n|         SDEdit    \t         |   ICLR'22 \t   | Text-guided Image Editing \t|\n|    Prompt-to-Prompt    \t    |   ICLR'23 \t   | Text-guided Image Editing \t|\n|          Text2Live          |   ECCV'22 \t   | Text-guided Image Editing \t|\n|        Pix2PixZero \t        | SIGGRAPH'23 \t | Text-guided Image Editing \t|\n|         GLIDE    \t          |   ICML'22 \t   | Mask-guided Image Editing \t|\n|      Blended Diffusion      |   CVPR'22 \t   | Mask-guided Image Editing \t|\n| Stable Diffusion Inpainting |      - \t      | Mask-guided Image Editing \t|\n| Stable Diffusion XL Inpainting |      - \t      | Mask-guided Image Editing \t|\n|     TextualInversion        | ICLR'23  | Subject-driven Image Generation|\n|       BLIP-Diffusion     |   arXiv'23    | Subject-Driven Image Generation|\n|         DreamBooth(+ LoRA)          |    CVPR'23    | Subject-Driven Image Generation|\n|       Photoswap    \t        |  arXiv'23 \t   | Subject-Driven Image Editing \t|\n|       DreamEdit    \t        |  arXiv'23 \t   | Subject-Driven Image Editing \t|\n|      Custom Diffusion       |    CVPR'23    | Multi-Subject-Driven Generation|\n|         ControlNet          |   arXiv'23    | Control-guided Image Generation|\n|         UniControl          |   arXiv'23    | Control-guided Image Generation|\n\n### Comprehensive Functionality\n* [X] Common Metrics for GenAI\n* [X] Visualization tool\n* [ ] Amazon Mechanical Turk Templates (Coming Soon)\n\n### High quality software engineering standard.\n\n* [X] Documentation\n* [X] Type Hints\n* [ ] Code Coverage (Coming Soon)\n\n## 🙌 Contributing [🔝](#-table-of-contents)\n\n### For the Community\n_**Community contributions are encouraged!**_\n\nImagenHub is still under development. More models and features are going to be added and we always welcome contributions to help make ImagenHub better. If you would like to contribute, please check out [CONTRIBUTING.md](CONTRIBUTING.md). \n\nWe believe that everyone can contribute and make a difference. Whether it's writing code 💻, fixing bugs 🐛, or simply sharing feedback 💬, your contributions are definitely welcome and appreciated 🙌\n\n\u003e And if you like the project, but just don't have time to contribute, that's fine. There are other easy ways to support the project and show your appreciation, which we would also be very happy about:\n\u003e - Star the project\n\u003e - Tweet about it\n\u003e - Refer this project in your project's readme\n\u003e - Mention the project at local meetups and tell your friends/colleagues\n\n### For the Researchers:\n\n* Q: How can I use your evaluation method for my method?\n* A: Please Refer to [https://imagenhub.readthedocs.io/en/latest/Guidelines/humaneval.html](https://imagenhub.readthedocs.io/en/latest/Guidelines/humaneval.html)\n\n* Q: How can I add my method to ImagenHub codebase?\n* A: Please Refer to [https://imagenhub.readthedocs.io/en/latest/Guidelines/custommodel.html](https://imagenhub.readthedocs.io/en/latest/Guidelines/custommodel.html)\n\n* Q: I want to feature my method on [ImagenMuseum](https://chromaica.github.io/#imagen-museum)!\n* A: Please Refer to [https://imagenhub.readthedocs.io/en/latest/Guidelines/imagenmuseum.html](https://imagenhub.readthedocs.io/en/latest/Guidelines/imagenmuseum.html)\n\n\n## 🖊️ Citation [🔝](#-table-of-contents)\n\nPlease kindly cite our paper if you use our code, data, models or results:\n\n```bibtex\n@inproceedings{\nku2024imagenhub,\ntitle={ImagenHub: Standardizing the evaluation of conditional image generation models},\nauthor={Max Ku and Tianle Li and Kai Zhang and Yujie Lu and Xingyu Fu and Wenwen Zhuang and Wenhu Chen},\nbooktitle={The Twelfth International Conference on Learning Representations},\nyear={2024},\nurl={https://openreview.net/forum?id=OuV9ZrkQlc}\n}\n```\n\n```bibtex\n@article{ku2023imagenhub,\n  title={ImagenHub: Standardizing the evaluation of conditional image generation models},\n  author={Max Ku and Tianle Li and Kai Zhang and Yujie Lu and Xingyu Fu and Wenwen Zhuang and Wenhu Chen},\n  journal={arXiv preprint arXiv:2310.01596},\n  year={2023}\n}\n```\n\n## 🤝 Acknowledgement [🔝](#-table-of-contents)\n\nPlease refer to [ACKNOWLEDGEMENTS.md](ACKNOWLEDGEMENTS.md)\n\n## 🎫 License [🔝](#-table-of-contents)\n\nThis project is released under the [License](LICENSE).\n\n\n## ⭐ Star History [🔝](#-table-of-contents)\n\n[![Star History Chart](https://api.star-history.com/svg?repos=TIGER-AI-Lab/ImagenHub\u0026type=Date)](https://star-history.com/#TIGER-AI-Lab/ImagenHub\u0026Date)\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftiger-ai-lab%2Fimagenhub","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftiger-ai-lab%2Fimagenhub","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftiger-ai-lab%2Fimagenhub/lists"}