{"id":14977153,"url":"https://github.com/ai-forever/kandinsky-2","last_synced_at":"2025-05-15T03:04:49.142Z","repository":{"id":63673584,"uuid":"551603272","full_name":"ai-forever/Kandinsky-2","owner":"ai-forever","description":"Kandinsky 2 — multilingual text2image latent diffusion model","archived":false,"fork":false,"pushed_at":"2024-05-01T17:03:31.000Z","size":39071,"stargazers_count":2796,"open_issues_count":83,"forks_count":311,"subscribers_count":47,"default_branch":"main","last_synced_at":"2025-05-15T03:03:56.074Z","etag":null,"topics":["diffusion","image-generation","image2image","inpainting","ipython-notebook","kandinsky","outpainting","text-to-image","text2image"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ai-forever.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"license","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-10-14T18:13:08.000Z","updated_at":"2025-05-07T09:29:10.000Z","dependencies_parsed_at":"2023-01-22T15:15:56.914Z","dependency_job_id":"d26c7304-eb19-430d-bcb6-7c20b842bc2b","html_url":"https://github.com/ai-forever/Kandinsky-2","commit_stats":{"total_commits":154,"total_committers":8,"mean_commits":19.25,"dds":0.5584415584415585,"last_synced_commit":"aeefc1ce3a989eefe7c99d6a02cce44318c4d210"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ai-forever%2FKandinsky-2","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ai-forever%2FKandinsky-2/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ai-forever%2FKandinsky-2/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ai-forever%2FKandinsky-2/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ai-forever","download_url":"https://codeload.github.com/ai-forever/Kandinsky-2/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254264765,"owners_count":22041793,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["diffusion","image-generation","image2image","inpainting","ipython-notebook","kandinsky","outpainting","text-to-image","text2image"],"created_at":"2024-09-24T13:55:12.170Z","updated_at":"2025-05-15T03:04:49.088Z","avatar_url":"https://github.com/ai-forever.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Kandinsky 2.2\n\n[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1MfN9dfmejT8NjXhR353NeP5RzbruHgo7?usp=sharing) — Inference example\n\n[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1lUWfe4CWhPJhUZYjMAE7g4ciHX4764rN?usp=sharing) — Fine-tuning with LoRA\n\n+ [Habr post](https://habr.com/ru/companies/sberbank/articles/747446/)\n+ [Demo fusionbrain.ai](https://fusionbrain.ai/diffusion)\n+ [Telegram-bot](https://t.me/kandinsky21_bot)\n\n\u003cp align=\"left\"\u003e\n\u003cimg src=\"./content/kand_22.png\" width=\"60%\"\u003e\n\u003c/p\u003e\n\n**Description:**\n\nKandinsky 2.2 brings substantial improvements upon its predecessor, Kandinsky 2.1, by introducing a new, more powerful image encoder - CLIP-ViT-G and the ControlNet support.\n\nThe switch to CLIP-ViT-G as the image encoder significantly increases the model's capability to generate more aesthetic pictures and better understand text, thus enhancing the model's overall performance.\n\nThe addition of the ControlNet mechanism allows the model to effectively control the process of generating images. This leads to more accurate and visually appealing outputs and opens new possibilities for text-guided image manipulation.\n\n**Architecture details:**\n\n+ Text encoder (XLM-Roberta-Large-Vit-L-14) - 560M\n+ Diffusion Image Prior — 1B\n+ CLIP image encoder (ViT-bigG-14-laion2B-39B-b160k) - 1.8B\n+ Latent Diffusion U-Net - 1.22B\n+ MoVQ encoder/decoder - 67M\n\n**Сheckpoints:**\n\n+ [Prior](https://huggingface.co/kandinsky-community/kandinsky-2-2-prior): A prior diffusion model mapping text embeddings to image embeddings\n+ [Text-to-Image / Image-to-Image](https://huggingface.co/kandinsky-community/kandinsky-2-2-decoder): A decoding diffusion model mapping image embeddings to images\n+ [Inpainting](https://huggingface.co/kandinsky-community/kandinsky-2-2-decoder-inpaint): A decoding diffusion model mapping image embeddings and masked images to images\n+ [ControlNet-depth](https://huggingface.co/kandinsky-community/kandinsky-2-2-controlnet-depth): A decoding diffusion model mapping image embedding and additional depth condition to images\n\n### Inference regimes\n\n\u003cp align=\"left\"\u003e\n\u003cimg src=\"./content/kand_22_setting.png\" width=\"60%\"\u003e\n\u003c/p\u003e\n\n## How to use:\n\nCheck our jupyter notebooks with examples in `./notebooks` folder\n### 1. text2image\n\n```python\nfrom kandinsky2 import get_kandinsky2\nmodel = get_kandinsky2('cuda', task_type='text2img', model_version='2.2')\nimages = model.generate_text2img(\n    \"red cat, 4k photo\", \n    decoder_steps=50,\n    batch_size=1, \n    h=1024,\n    w=768,\n)\n```\n\n\n# Kandinsky 2.1\n\n[![Framework: PyTorch](https://img.shields.io/badge/Framework-PyTorch-orange.svg)](https://pytorch.org/) [![Huggingface space](https://img.shields.io/badge/🤗-Huggingface-yello.svg)](https://huggingface.co/sberbank-ai/Kandinsky_2.1)\n[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1xSbu-b-EwYd6GdaFPRVgvXBX_mciZ41e?usp=sharing)\n\n\n[Habr post](https://habr.com/ru/company/sberbank/blog/725282/)\n\n[Demo](https://fusionbrain.ai/diffusion)\n\n`pip install \"git+https://github.com/ai-forever/Kandinsky-2.git\"`\n\n## Model architecture:\n\n\u003cp align=\"left\"\u003e\n\u003cimg src=\"./content/kandinsky21.png\" width=\"80%\"\u003e\n\u003c/p\u003e\n\n\nKandinsky 2.1 inherits best practicies from Dall-E 2 and Latent diffusion, while introducing some new ideas.\n\nAs text and image encoder it uses CLIP model and diffusion image prior (mapping) between latent spaces of CLIP modalities. This approach increases the visual performance of the model and unveils new horizons in blending images and text-guided image manipulation.\n\nFor diffusion mapping of latent spaces we use transformer with num_layers=20, num_heads=32 and hidden_size=2048.\n\n**Other architecture parts:**\n\n+ Text encoder (XLM-Roberta-Large-Vit-L-14) - 560M\n+ Diffusion Image Prior — 1B\n+ CLIP image encoder (ViT-L/14) - 427M\n+ Latent Diffusion U-Net - 1.22B\n+ MoVQ encoder/decoder - 67M\n\n\nKandinsky 2.1 was trained on a large-scale image-text dataset LAION HighRes and fine-tuned on our internal datasets.\n\n\n## How to use:\n\n Check our jupyter notebooks with examples in `./notebooks` folder\n\n\n### 1. text2image\n\n```python\nfrom kandinsky2 import get_kandinsky2\nmodel = get_kandinsky2('cuda', task_type='text2img', model_version='2.1', use_flash_attention=False)\nimages = model.generate_text2img(\n    \"red cat, 4k photo\", \n    num_steps=100,\n    batch_size=1, \n    guidance_scale=4,\n    h=768, w=768,\n    sampler='p_sampler', \n    prior_cf_scale=4,\n    prior_steps=\"5\"\n)\n```\n\n![](./content/einstein.png)\n\nprompt: \"Einstein in space around the logarithm scheme\"\n\n### 2. image fuse\n\n```python\nfrom kandinsky2 import get_kandinsky2\nfrom PIL import Image\nmodel = get_kandinsky2('cuda', task_type='text2img', model_version='2.1', use_flash_attention=False)\nimages_texts = ['red cat', Image.open('img1.jpg'), Image.open('img2.jpg'), 'a man']\nweights = [0.25, 0.25, 0.25, 0.25]\nimages = model.mix_images(\n    images_texts, \n    weights, \n    num_steps=150,\n    batch_size=1, \n    guidance_scale=5,\n    h=768, w=768,\n    sampler='p_sampler', \n    prior_cf_scale=4,\n    prior_steps=\"5\"\n)\n```\n\n![](./content/fuse.png)\n\n### 3. inpainting\n\n```python\nfrom kandinsky2 import get_kandinsky2\nfrom PIL import Image\nimport numpy as np\n\nmodel = get_kandinsky2('cuda', task_type='inpainting', model_version='2.1', use_flash_attention=False)\ninit_image = Image.open('img.jpg')\nmask = np.ones((768, 768), dtype=np.float32)\nmask[:,:550] =  0\nimages = model.generate_inpainting(\n    'man 4k photo', \n    init_image, \n    mask, \n    num_steps=150,\n    batch_size=1, \n    guidance_scale=5,\n    h=768, w=768,\n    sampler='p_sampler', \n    prior_cf_scale=4,\n    prior_steps=\"5\"\n)\n```\n\n\n# Kandinsky 2.0\n\n[![Framework: PyTorch](https://img.shields.io/badge/Framework-PyTorch-orange.svg)](https://pytorch.org/) [![Huggingface space](https://img.shields.io/badge/🤗-Huggingface-yello.svg)](https://huggingface.co/sberbank-ai/Kandinsky_2.0) \n[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1uPg9KwGZ2hJBl9taGA_3kyKGw12Rh3ij?usp=sharing)\n\n[Habr post](https://habr.com/ru/company/sberbank/blog/701162/)\n\n[Demo](https://fusionbrain.ai/diffusion)\n\n`pip install \"git+https://github.com/ai-forever/Kandinsky-2.git\"`\n\n\n## Model architecture:\n\nIt is a latent diffusion model with two multilingual text encoders:\n* mCLIP-XLMR 560M parameters\n* mT5-encoder-small 146M parameters\n\nThese encoders and multilingual training datasets unveil the real multilingual text-to-image generation experience!\n\n**Kandinsky 2.0** was trained on a large 1B multilingual set, including samples that we used to train Kandinsky.\n\nIn terms of diffusion architecture Kandinsky 2.0 implements UNet with 1.2B parameters.\n\n**Kandinsky 2.0** architecture overview:\n\n![](./content/NatallE.png)\n\n## How to use:\n \n Check our jupyter notebooks with examples in `./notebooks` folder\n \n### 1. text2img\n\n```python\nfrom kandinsky2 import get_kandinsky2\n\nmodel = get_kandinsky2('cuda', task_type='text2img')\nimages = model.generate_text2img('A teddy bear на красной площади', batch_size=4, h=512, w=512, num_steps=75, denoised_type='dynamic_threshold', dynamic_threshold_v=99.5, sampler='ddim_sampler', ddim_eta=0.05, guidance_scale=10)\n```\n![](./content/bear.jpeg)\n\nprompt: \"A teddy bear на красной площади\"\n\n### 2. inpainting\n```python \nfrom kandinsky2 import get_kandinsky2\nfrom PIL import Image\nimport numpy as np\n\nmodel = get_kandinsky2('cuda', task_type='inpainting')\ninit_image = Image.open('image.jpg')\nmask = np.ones((512, 512), dtype=np.float32)\nmask[100:] =  0\nimages = model.generate_inpainting('Девушка в красном платье', init_image, mask, num_steps=50, denoised_type='dynamic_threshold', dynamic_threshold_v=99.5, sampler='ddim_sampler', ddim_eta=0.05, guidance_scale=10)\n```\n\n![](./content/inpainting.png)\n\nprompt: \"Девушка в красном платье\"\n\n### 3. img2img\n```python\nfrom kandinsky2 import get_kandinsky2\nfrom PIL import Image\n\nmodel = get_kandinsky2('cuda', task_type='img2img')\ninit_image = Image.open('image.jpg')\nimages = model.generate_img2img('кошка', init_image, strength=0.8, num_steps=50, denoised_type='dynamic_threshold', dynamic_threshold_v=99.5, sampler='ddim_sampler', ddim_eta=0.05, guidance_scale=10)\n```\n\n# Authors\n\n+ Arseniy Shakhmatov: [Github](https://github.com/cene555), [Blog](https://t.me/gradientdip)\n+ Anton Razzhigaev: [Github](https://github.com/razzant), [Blog](https://t.me/abstractDL)\n+ Aleksandr Nikolich: [Github](https://github.com/AlexWortega), [Blog](https://t.me/lovedeathtransformers)\n+ Vladimir Arkhipkin: [Github](https://github.com/oriBetelgeuse)\n+ Igor Pavlov: [Github](https://github.com/boomb0om)\n+ Andrey Kuznetsov: [Github](https://github.com/kuznetsoffandrey)\n+ Denis Dimitrov: [Github](https://github.com/denndimitrov)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fai-forever%2Fkandinsky-2","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fai-forever%2Fkandinsky-2","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fai-forever%2Fkandinsky-2/lists"}