{"id":14977140,"url":"https://github.com/ai-forever/deforum-kandinsky","last_synced_at":"2025-10-27T23:31:50.706Z","repository":{"id":199882566,"uuid":"699436123","full_name":"ai-forever/deforum-kandinsky","owner":"ai-forever","description":"Kandinsky x Deforum — generating short animations","archived":false,"fork":false,"pushed_at":"2024-01-22T11:17:03.000Z","size":2087,"stargazers_count":105,"open_issues_count":0,"forks_count":10,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-09-09T01:14:18.791Z","etag":null,"topics":["diffusion","ipython-notebook","kandinsky","text-to-video","video-generation"],"latest_commit_sha":null,"homepage":"https://ai-forever.github.io/deforumkandinsky.github.io/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ai-forever.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-10-02T16:21:11.000Z","updated_at":"2025-06-19T09:45:06.000Z","dependencies_parsed_at":"2024-01-22T13:25:17.338Z","dependency_job_id":"692cb497-1732-429c-b539-a894524487da","html_url":"https://github.com/ai-forever/deforum-kandinsky","commit_stats":null,"previous_names":["ai-forever/deforum-kandinsky"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ai-forever/deforum-kandinsky","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ai-forever%2Fdeforum-kandinsky","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ai-forever%2Fdeforum-kandinsky/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ai-forever%2Fdeforum-kandinsky/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ai-forever%2Fdeforum-kandinsky/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ai-forever","download_url":"https://codeload.github.com/ai-forever/deforum-kandinsky/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ai-forever%2Fdeforum-kandinsky/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":281361406,"owners_count":26487881,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-27T02:00:05.855Z","response_time":61,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["diffusion","ipython-notebook","kandinsky","text-to-video","video-generation"],"created_at":"2024-09-24T13:55:11.065Z","updated_at":"2025-10-27T23:31:50.119Z","avatar_url":"https://github.com/ai-forever.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Deforum-Kandinsky\n\u003cp align=\"left\"\u003e\n    \u003ca target=\"_blank\" href=\"https://colab.research.google.com/drive/1V0E_nM8bxOhVPBXP-J9jCbeAaNAYsI9v?usp=sharing\"\u003e\n      \u003cimg src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/\u003e\n    \u003c/a\u003e\n\u003c/p\u003e \n\n## Introduction\n\u003cp\u003e\nIn the past few years, there has been a marked increase in the popularity of generative models that utilize various data modalities. One of the most challenging undertakings in this regard is synthesizing videos from text, which is both time-consuming and resource-intensive. The core of proposed solution/animation approach is Kandinsky extension with Deforum features. This leads to new generative opportunities of text2image model.\n\u003c/p\u003e\n\n## Examples\n\u003cp\u003e\n    \u003cimg src=\"https://github.com/ai-forever/deforum-kandinsky/assets/69383296/afe7885c-fb3c-479d-9f1a-8427f2636555\"/\u003e\n    \u003cimg src=\"https://github.com/ai-forever/deforum-kandinsky/assets/69383296/6f2b74f3-d290-4b26-b7c9-ccfe239e54e6\"/\u003e\n\u003cp\u003e\n     \u003cimg src=\"https://github.com/ai-forever/deforum-kandinsky/assets/69383296/4cc93196-5146-4889-9b9d-0b87079c643f\"/\u003e\n     \u003cimg src=\"https://github.com/ai-forever/deforum-kandinsky/assets/69383296/03a0d489-7138-42fe-9089-48c63bed9911\"/\u003e\n\u003c/p\u003e\n\u003cp\u003e\n    \u003cimg src=\"https://github.com/ai-forever/deforum-kandinsky/assets/69383296/f7588587-d307-4eb6-a610-b5618c5c0be5\"\u003e\n    \u003cimg src=\"https://github.com/ai-forever/deforum-kandinsky/assets/69383296/1d58d9f6-da3b-4cca-9240-ff196478f1dd\u003e\"\u003e\n\u003c/p\u003e                                  \n\u003cp\u003e\n    \u003cimg src=\"https://github.com/ai-forever/deforum-kandinsky/assets/69383296/58f8e66d-1d42-40b5-bbd1-9f593237c6aa\"/\u003e\n    \u003cimg src=\"https://github.com/ai-forever/deforum-kandinsky/assets/69383296/725b5165-e6cf-4aaf-a51f-afc382c2a5f9\"/\u003e\n\u003c/p\u003e\n\n## Getting Started\n\n### 1. Clone repository\n```bash\ngit clone https://github.com/ai-forever/deforum-kandinsky.git\ncd deforum-kandinsky\n```\n\n### 2. Install dependencies\n```bash\npip install -r requirements.txt\n```\n\n## Running Deforum\n### 1. Import dependencies\n```python\nfrom IPython.display import Video\nfrom deforum_kandinsky import KandinskyV22Img2ImgPipeline, DeforumKandinsky\nfrom diffusers import KandinskyV22PriorPipeline\nfrom transformers import CLIPVisionModelWithProjection\nfrom diffusers.models import UNet2DConditionModel\nimport imageio.v2 as iio\nfrom PIL import Image\nimport numpy as np\nimport torch\nimport datetime\nfrom tqdm.notebook import tqdm\nimport ipywidgets as widgets\nfrom IPython import display\n```\n\n### 2. Convert list frames to mp4 video\n```python\n#  create video from generated frames\ndef frames2video(frames, output_path=\"video.mp4\", fps=24, display=False):\n    writer = iio.get_writer(output_path, fps=fps)\n    for frame in tqdm(frames):\n        writer.append_data(np.array(frame))\n    writer.close()\n    if display:\n        display.Video(url=output_path)\n```\n\n### 3. Load Kandinsky 2.1 or 2.2 \n```python\nfrom diffusers import KandinskyV22PriorPipeline\nfrom deforum_kandinsky import (\n    KandinskyV22Img2ImgPipeline, \n    DeforumKandinsky,  \n    KandinskyImg2ImgPipeline, \n    DeforumKandinsky\n)\n\n# load models\nmodel_version = 2.2\ndevice = \"cuda\"\n\nif model_version == 2.2:\n    image_encoder = CLIPVisionModelWithProjection.from_pretrained(\n        'kandinsky-community/kandinsky-2-2-prior', \n        subfolder='image_encoder'\n        ).to(torch.float16).to(device)\n\n    unet = UNet2DConditionModel.from_pretrained(\n        'kandinsky-community/kandinsky-2-2-decoder', \n        subfolder='unet'\n        ).to(torch.float16).to(device)\n\n    prior = KandinskyV22PriorPipeline.from_pretrained(\n        'kandinsky-community/kandinsky-2-2-prior', \n        image_encoder=image_encoder, \n        torch_dtype=torch.float16\n        ).to(device)\n    decoder = KandinskyV22Img2ImgPipeline.from_pretrained(\n        'kandinsky-community/kandinsky-2-2-decoder', \n        unet=unet, \n        torch_dtype=torch.float16\n        ).to(device)\n\nelif model_version == 2.1: \n\n    image_encoder = CLIPVisionModelWithProjection.from_pretrained(\n        \"kandinsky-community/kandinsky-2-1-prior\", \n        subfolder='image_encoder',\n        torch_dtype=torch.float16\n        ).to(device)\n    unet = UNet2DConditionModel.from_pretrained(\n        \"kandinsky-community/kandinsky-2-1\", \n        subfolder='unet',\n        torch_dtype=torch.float16\n        ).to(device)\n    prior = KandinskyPriorPipeline.from_pretrained(\n        \"kandinsky-community/kandinsky-2-1-prior\", \n        torch_dtype=torch.float16\n        ).to(device)\n    decoder = KandinskyImg2ImgPipeline.from_pretrained(\n        'kandinsky-community/kandinsky-2-1', \n        unet=unet, \n        torch_dtype=torch.float16\n        ).to(device)\n```\n\n### 4. Define instance of Kandinsky with Deforum\n```python\ndeforum = DeforumKandinsky(\n    prior=prior,\n    decoder_img2img=decoder,\n    device='cuda'\n)\n```\n\n### 5. Create text prompt and set up configs for animation\n```python   \nanimation = deforum(\n    prompts=[\n        \"winter forest, snowflakes, Van Gogh style\",\n        \"spring forest, flowers, sun rays, Van Gogh style\",\n        \"summer forest, lake, reflections on the water, summer sun, Van Gogh style\",\n        \"autumn forest, rain, Van Gogh style\",\n        Image.open(\"path/to/image.jpg\"),\n    ], \n    animations=['live', 'right', 'right', 'right', 'live'], \n    prompt_durations=[1, 1, 1, 1, 1],\n    H=640,\n    W=640,\n    fps=24,\n    save_samples=False,\n)\n\nframes = []\n\nout = widgets.Output()\npbar = tqdm(animation, total=len(deforum))\ndisplay.display(out)\nwith out:\n    for item in pbar:\n        frame = item.pop('image', None)\n        frames.append(frame)\n        display.clear_output(wait=True) \n        display.display(frame)\n        for key, value in item.items():\n            print(f\"{key}: {value}\")\n\n# save and display video\ndisplay.clear_output(wait=True) \nframes2video(frames, \"output_2_2.mp4\", fps=24)\ndisplay.Video(url=\"output_2_2.mp4\")\n```\n\n## References\n\u003ca href=\"https://deforum.github.io/\"\u003eDeforum web-page\u003c/a\u003e\n\u003c/br\u003e\n\u003ca href=\"https://docs.google.com/document/d/1RrQv7FntzOuLg4ohjRZPVL7iptIyBhwwbcEYEW2OfcI/edit#heading=h.7z6glzthkva2)https://docs.google.com/document/d/1RrQv7FntzOuLg4ohjRZPVL7iptIyBhwwbcEYEW2OfcI/edit#heading=h.7z6glzthkva2\"\u003eQuick Guide to Deforum v06\u003c/a\u003e\n\u003c/br\u003e\n\u003ca href=\"https://github.com/deforum-art/deforum-stable-diffusion\"\u003eGitHub repository: deforum-stable-diffusion\u003c/a\u003e\n\u003c/br\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fai-forever%2Fdeforum-kandinsky","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fai-forever%2Fdeforum-kandinsky","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fai-forever%2Fdeforum-kandinsky/lists"}