{"id":13488130,"url":"https://github.com/conceptbed/evaluations","last_synced_at":"2025-03-27T23:32:56.095Z","repository":{"id":171485393,"uuid":"647111365","full_name":"ConceptBed/evaluations","owner":"ConceptBed","description":"[AAAI 2024] ConceptBed Evaluations for Personalized Text-to-Image Diffusion Models","archived":false,"fork":false,"pushed_at":"2023-06-01T00:52:53.000Z","size":1514,"stargazers_count":23,"open_issues_count":1,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-10-30T23:36:25.610Z","etag":null,"topics":["concept-learning","text-to-image"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2306.04695","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ConceptBed.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2023-05-30T04:54:44.000Z","updated_at":"2024-10-24T01:20:40.000Z","dependencies_parsed_at":"2024-01-16T09:02:37.411Z","dependency_job_id":"6bef3d67-5c1d-463a-ba6f-287a302705d3","html_url":"https://github.com/ConceptBed/evaluations","commit_stats":null,"previous_names":["conceptbed/evaluations"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ConceptBed%2Fevaluations","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ConceptBed%2Fevaluations/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ConceptBed%2Fevaluations/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ConceptBed%2Fevaluations/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ConceptBed","download_url":"https://codeload.github.com/ConceptBed/evaluations/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245944041,"owners_count":20697946,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["concept-learning","text-to-image"],"created_at":"2024-07-31T18:01:10.107Z","updated_at":"2025-03-27T23:32:51.029Z","avatar_url":"https://github.com/ConceptBed.png","language":"Python","funding_links":[],"categories":["New Concept Learning"],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n# ConceptBed: Evaluations\n\n[\\[Project\\]](https://conceptbed.github.io/)\n[\\[Paper\\]](#)\n[\\[Data\\]](https://conceptbed.github.io/data.html)\n[\\[Results Explorer\\]](https://conceptbed.github.io/explorer.html)\n\n\n\n\u003c/div\u003e\n\n## Description\n\nThis repository is the part of the \"ConceptBed: Evaluating Concept Learning Abilities of Text-to-Image Diffusion Models\" paper.\n\nThis project provides the evaluation framework for the Personalized Text-to-Image Generative models (also known as Concept Leanrers).\n\n\n## Getting Started\n\nInstall dependencies\n\n```bash\n# clone project\ngit clone https://github.com/ConceptBed/evaluations\ncd evaluations\n\n# [OPTIONAL] create conda environment\nconda create -p ./venv python=3.8\nconda activate ./venv\n\n# install pytorch according to instructions\n# https://pytorch.org/get-started/\n\n# install requirements\npip install -r requirements.txt\n```\n\nDownload the data and model checkpoints\n\n```bash\nbash ./scripts/download_data.sh\nbash ./scripts/download_checkpoints.sh\n```\n\nFollow the below guidelines to store the generated images:\n\n    \u003cyour-global-path\u003e\n    ├── ...\n    ├── outputs                    # put the images anywhere\n    │   ├── imagenet               # follow the downloaded conceptbed subset of imagenet folder\n    │   │   ├── n01530575\n    │   │   │   ├── 0.jpeg\n    │   │   │   ├── 1.jpeg\n    │   │   │   └── ...\n    │   │   └── ...\n    │   ├── pacs                   # follow the same folder structure as PACs\n    │   │   ├── art_painting\n    │   │   │   ├── dog\n    │   │   │   │   ├── 0.jpeg\n    │   │   │   │   ├── 1.jpeg\n    │   │   │   │   ├── 2.jpeg\n    │   │   │   │   └── ...\n    │   │   │   └── ...\n    │   │   └── ...\n    │   ├── cub                    # follow the same folder structure as CUB\n    │   │   ├── 001.Black_footed_Albatross\n    │   │   │   ├── 0.jpeg\n    │   │   │   ├── 1.jpeg\n    │   │   │   └── ...\n    │   │   └── ...\n    │   └── compositions           # follow the downloaded conceptbed subset of imagenet folder\n    │   │   ├── n01530575\n    │   │   │   ├── 0_0.jpeg\n    │   │   │   ├── 1_0.jpeg\n    │   │   │   └── ...\n    │   │   └── ...\n    │   └── ...\n    └── ...\n\n\nPerform the ConceptBed (uncertainty-aware) evaluations:\n\n1. Measure Concept Alignment\n\n```bash\n# evaluate on PACs subset\nbash ./scripts/concept_alignment/test_pacs.sh\n\n# evaluate on imagenet subset\nbash ./scripts/concept_alignment/test_imagenet.sh\n\n# evaluate on imagenet composition subset\nbash ./scripts/concept_alignment/test_imagenet_composition.sh\n```\n\n2. Measure Composition Alignment\n\n```bash\n# evaluate the compositions \nbash ./scripts/composition_alignment/test_composition.sh\n```\n\nAdditional instructions:\n* Define the `gen_datapath` and `gen_name` in each bash script based on your setup.\n* `test_imagenet_composition.sh` and `test_imagenet.sh` are essentially the same but you have to provide the path to the composition based generated images.\n\n\n## Benchmark Results\n\n### Human Performance\n\u003c!-- ![Human Evaluations](./assets/human_eval.png) --\u003e\n![Human Evaluations](./assets/human_eval_plot.png)\n\n### Concept Alignment\n![Concept Alignment](./assets/concept_alignment.png)\n\n#### Uncertainty Sensitivity\n![Uncertainty Sensitivity](./assets/everything_boxplot.png)\n\n\n### Composition Alignment\n![Composition Alignment](./assets/composition_alignment.png)\n\n\n\n\n## Citation\n\nIf you our work helpful, please consider citing:\n\n```\n@article{patel2023conceptbed,\n  author    = {Patel, Maitreya and Gokhale, Tejas and Baral, Chitta and Yang, Yezhou},\n  title     = {ConceptBed: Evaluating Concept Learning Abilities of Text-to-Image Diffusion Models},\n  journal={arXiv},\n  year      = {2023},\n}\n```\n\n## Acknowledgement\nWe would like to acknowledge the [PyTorch](https://pytorch.org), [timm](https://github.com/huggingface/pytorch-image-models), [transformers](https://github.com/huggingface/transformers), and [lightning+hydra](https://github.com/ashleve/lightning-hydra-template).\n\nThis work was supported by NSF RI grants #1750082 and #2132724, and a grant from Meta AI Learning Alliance. The views and opinions of the authors expressed herein do not necessarily state or reflect those of the funding agencies and employers.\n\nIf you have any questions or suggestions, please feel free to reach out to us.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fconceptbed%2Fevaluations","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fconceptbed%2Fevaluations","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fconceptbed%2Fevaluations/lists"}