{"id":13488335,"url":"https://github.com/kfirgoldberg/ConceptLab","last_synced_at":"2025-03-28T00:33:38.921Z","repository":{"id":186027859,"uuid":"673639775","full_name":"kfirgoldberg/ConceptLab","owner":"kfirgoldberg","description":"Official Implementation for \"ConceptLab: Creative Generation using Diffusion Prior Constraints\"","archived":false,"fork":false,"pushed_at":"2023-12-19T09:32:06.000Z","size":143177,"stargazers_count":243,"open_issues_count":0,"forks_count":19,"subscribers_count":34,"default_branch":"main","last_synced_at":"2024-10-31T00:39:48.887Z","etag":null,"topics":["creative-generation","diffusion-models","diffusion-prior","personalization","text-to-image"],"latest_commit_sha":null,"homepage":"https://kfirgoldberg.github.io/ConceptLab/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kfirgoldberg.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2023-08-02T05:04:38.000Z","updated_at":"2024-10-02T02:38:13.000Z","dependencies_parsed_at":"2024-01-16T09:02:44.673Z","dependency_job_id":"604faf2a-5606-41df-8f14-734615cb5960","html_url":"https://github.com/kfirgoldberg/ConceptLab","commit_stats":null,"previous_names":["kfirgoldberg/conceptlab"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kfirgoldberg%2FConceptLab","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kfirgoldberg%2FConceptLab/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kfirgoldberg%2FConceptLab/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kfirgoldberg%2FConceptLab/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kfirgoldberg","download_url":"https://codeload.github.com/kfirgoldberg/ConceptLab/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245949271,"owners_count":20698911,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["creative-generation","diffusion-models","diffusion-prior","personalization","text-to-image"],"created_at":"2024-07-31T18:01:13.931Z","updated_at":"2025-03-28T00:33:33.808Z","avatar_url":"https://github.com/kfirgoldberg.png","language":"Python","funding_links":[],"categories":["T2I Diffusion Model augmentation","\u003cspan id=\"image\"\u003eImage\u003c/span\u003e"],"sub_categories":["\u003cspan id=\"tool\"\u003eLLM (LLM \u0026 Tool)\u003c/span\u003e"],"readme":"# ConceptLab: Creative Concept Generation using VLM-Guided Diffusion Prior Constraints\n\u003e Elad Richardson, Kfir Goldberg, Yuval Alaluf, Daniel Cohen-Or  \n\u003e Tel Aviv University  \n\u003e Recent text-to-image generative models have enabled us to transform our words into vibrant, captivating imagery. The surge of personalization techniques that has followed has also allowed us to imagine unique concepts in new scenes. However, an intriguing question remains: How can we generate a \u003ci\u003enew\u003c/i\u003e, imaginary concept that has never been seen before? In this paper, we present the task of \u003ci\u003ecreative text-to-image generation\u003c/i\u003e, where we seek to generate new members of a broad category  (e.g., generating a pet that differs from all existing pets). We leverage the under-studied Diffusion Prior models and show that the creative generation problem can be formulated as an optimization process over the output space of the diffusion prior, resulting in a set of \"prior constraints\". To keep our generated concept from converging into existing members, we incorporate a question-answering Vision-Language Model (VLM) that adaptively adds new constraints to the optimization problem, encouraging the model to discover increasingly more unique creations. Finally, we show that our prior constraints can also serve as a strong mixing mechanism allowing us to create hybrids between generated concepts, introducing even more flexibility into the creative process.\n\n\u003ca href=\"https://arxiv.org/abs/2308.02669\"\u003e\u003cimg src=\"https://img.shields.io/badge/arXiv-2308.02669-b31b1b.svg\" height=22.5\u003e\u003c/a\u003e\n\u003ca href=\"https://kfirgoldberg.github.io/ConceptLab/\"\u003e\u003cimg src=\"https://img.shields.io/static/v1?label=Project\u0026message=Website\u0026color=red\" height=20.5\u003e\u003c/a\u003e \n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"docs/teaser.jpg\" width=\"800px\"/\u003e  \n\u003cbr\u003e\nNew pets generated using ConceptLab. Each pair depicts a learned concept that was optimized to be novel and not match existing members of the pet category. Running our method with different seeds allows us to generate a variety of different brand-new concepts.\n\u003c/p\u003e\n\n## Description\nOfficial implementation of our ConceptLab paper.\n\n## Setup\nTo create the conda environment needed to run the code, run the following command:\n\n```\nconda env create -f environment/env.yaml\nconda activate ConceptLab\n```\n\nAlternatively, install the requirements from `requirements.txt`\n\n## Usage\n### Creative Generation\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"docs/sealrat.jpg\" width=\"800px\"/\u003e  \n\u003cimg src=\"docs/suncrest_lizard.jpg\" width=\"800px\"/\u003e  \n\u003cimg src=\"docs/250_214.jpg\" width=\"800px\"/\u003e  \n\u003cimg src=\"docs/250_211.jpg\" width=\"800px\"/\u003e  \n\u003cbr\u003e\nSample text-guided creative generation results and edits obtained with ConceptLab.\n\u003c/p\u003e\n\n#### Training a New Concept\n\nTo train a new concept of the `pet` category using the Adaptive Negatives scheme, run the following command:\n\n```python -m scripts.train --config configs/new_pet.yaml --output_dir=\u003coutput_dir\u003e```\n\n#### Generating Images with a Trained Concept\nTo generate images from a trained concept, using some prompts e.g. `['a photo of a {}','a plush toy {}']` run the following command:\n\n```python -m scripts.infer --prompts=\"['a photo of a {}','a plush toy {}']\" --output_dir \u003coutput_dir\u003e --learned_embeds_path \u003clearned_embeds_path.bin\u003e```\n\n#### Creating Art Styles\nTo create an art style instead of an object use the following command\n```python -m scripts.train --config configs/new_art.yaml --output_dir=\u003coutput_dir\u003e```\n\n\n### Evolutionary Generation\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"docs/mix_tree.jpg\" width=\"800px\"/\u003e  \n\u003cbr\u003e\nConceptLab can be used to mix up generated concepts to iteratively learn new unique creations. This process can be repeated to create further \"Generations\", each one being a hybrid between the previous two.\n\u003c/p\u003e\n\n#### Creating Parents Gallery\nIn order to define each parent in the evolution process, we first need to create a few (3-5) images of each.\nTo do so, the following command can be used\n\n```python -m scripts.infer --prompts=\"['a photo of a {}']\" --output_dir \u003coutput_dir\u003e --learned_embeds_path \u003clearned_embeds_path.bin\u003e --samples_per_prompt=32```\n\nSelect 3-5 examples that you like for each parent and place them in `\u003cparent1_dir\u003e` and `\u003cparent2_dir\u003e`.\n\nTo train the resulting concept, run the following command:\n\n```python -m scripts.train_evolution --parents_images_dirs=\"['\u003cparent1_dir\u003e','\u003cparent2_dir\u003e']\" --initializer_token \u003cinitializer_token\u003e --output_dir \u003coutput_dir\u003e``` --mix_weights=\"[0.4,0.6]\"\n\n## Concept Mixing\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"docs/concept_mixing.jpg\" width=\"800px\"/\u003e  \n\u003cbr\u003e\nWith ConceptLab, we can also form hybrid concepts by merging unique traits across multiple real concepts. This can be done by defining multiple positive concepts, allowing us to create unique creations such as a lobs-turtle, pine-melon, and more!\n\u003c/p\u003e\n\nTo mix 2 or more concepts, run the training script with the positives constraints, and no negative constraints:\n\n```python -m scripts.train --config configs/new_pet.yaml --output_dir=\u003coutput_dir\u003e --live_negatives=False --positive_classes=\"['fish','panda']\"```\n\n\n## Acknowledgements \nThis code is builds on the code from the [kandinsky2](https://github.com/ai-forever/Kandinsky-2)  library as well as the [diffusers](https://github.com/huggingface/diffusers) codebase.\n\n\n## Citation\nIf you use this code for your research, please cite the following work: \n```\n@misc{richardson2023conceptlab,\n      title={ConceptLab: Creative Generation using Diffusion Prior Constraints}, \n      author={Elad Richardson and Kfir Goldberg and Yuval Alaluf and Daniel Cohen-Or},\n      year={2023},\n      eprint={2308.02669},\n      archivePrefix={arXiv},\n      primaryClass={cs.CV}\n}\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkfirgoldberg%2FConceptLab","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkfirgoldberg%2FConceptLab","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkfirgoldberg%2FConceptLab/lists"}