{"id":15034142,"url":"https://github.com/picsart-ai-research/text2video-zero","last_synced_at":"2025-04-13T14:00:22.076Z","repository":{"id":147846802,"uuid":"616990533","full_name":"Picsart-AI-Research/Text2Video-Zero","owner":"Picsart-AI-Research","description":"[ICCV 2023 Oral] Text-to-Image Diffusion Models are Zero-Shot Video Generators","archived":false,"fork":false,"pushed_at":"2023-05-06T22:35:28.000Z","size":94428,"stargazers_count":4160,"open_issues_count":48,"forks_count":364,"subscribers_count":67,"default_branch":"main","last_synced_at":"2025-04-06T11:04:51.402Z","etag":null,"topics":["video-editing","video-generation"],"latest_commit_sha":null,"homepage":"https://text2video-zero.github.io/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Picsart-AI-Research.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2023-03-21T13:41:40.000Z","updated_at":"2025-04-05T22:24:46.000Z","dependencies_parsed_at":"2023-05-27T17:45:44.342Z","dependency_job_id":null,"html_url":"https://github.com/Picsart-AI-Research/Text2Video-Zero","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Picsart-AI-Research%2FText2Video-Zero","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Picsart-AI-Research%2FText2Video-Zero/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Picsart-AI-Research%2FText2Video-Zero/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Picsart-AI-Research%2FText2Video-Zero/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Picsart-AI-Research","download_url":"https://codeload.github.com/Picsart-AI-Research/Text2Video-Zero/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248724585,"owners_count":21151560,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["video-editing","video-generation"],"created_at":"2024-09-24T20:24:04.188Z","updated_at":"2025-04-13T14:00:21.920Z","avatar_url":"https://github.com/Picsart-AI-Research.png","language":"Python","funding_links":[],"categories":["视频剪辑"],"sub_categories":[],"readme":"\n\n\n# Text2Video-Zero\n\nThis repository is the official implementation of [Text2Video-Zero](https://arxiv.org/abs/2303.13439).\n\n\n**[Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators](https://arxiv.org/abs/2303.13439)**\n\u003c/br\u003e\nLevon Khachatryan,\nAndranik Movsisyan,\nVahram Tadevosyan,\nRoberto Henschel,\n[Zhangyang Wang](https://www.ece.utexas.edu/people/faculty/atlas-wang), Shant Navasardyan, [Humphrey Shi](https://www.humphreyshi.com)\n\u003c/br\u003e\n\n[Paper](https://arxiv.org/abs/2303.13439) | [Video](https://www.dropbox.com/s/uv90mi2z598olsq/Text2Video-Zero.MP4?dl=0) | [![Hugging Face Spaces](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue)](https://huggingface.co/spaces/PAIR/Text2Video-Zero) | [Project](https://text2video-zero.github.io/)\n\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"__assets__/github/teaser/teaser_final.png\" width=\"800px\"/\u003e  \n\u003cbr\u003e\n\u003cem\u003eOur method Text2Video-Zero enables zero-shot video generation using (i) a textual prompt (see rows 1, 2),  (ii) a prompt combined with guidance from poses or edges (see lower right), and  (iii)  Video Instruct-Pix2Pix, i.e., instruction-guided video editing (see lower left). \n    Results are temporally consistent and follow closely the guidance and textual prompts.\u003c/em\u003e\n\u003c/p\u003e\n\n## News\n\n* [03/23/2023] Paper [Text2Video-Zero](https://arxiv.org/abs/2303.13439) released!\n* [03/25/2023] The [first version](https://huggingface.co/spaces/PAIR/Text2Video-Zero) of our huggingface demo (containing `zero-shot text-to-video generation` and  `Video Instruct Pix2Pix`) released!\n* [03/27/2023] The [full version](https://huggingface.co/spaces/PAIR/Text2Video-Zero) of our huggingface demo released! Now also included: `text and pose conditional video generation`, `text and edge conditional video generation`, and \n`text, edge and dreambooth conditional video generation`.\n* [03/28/2023] Code for all our generation methods released! We added a new low-memory setup. Minimum required GPU VRAM is currently **12 GB**. It will be further reduced in the upcoming releases. \n* [03/29/2023] Improved [Huggingface demo](https://huggingface.co/spaces/PAIR/Text2Video-Zero)! (i) For text-to-video generation, **any base model for stable diffusion** and **any dreambooth model** hosted on huggingface can now be loaded! (ii) We improved the quality of Video Instruct-Pix2Pix. (iii) We added two longer examples for Video Instruct-Pix2Pix.   \n* [03/30/2023] New code released! It includes all improvements of our latest huggingface iteration. See the news update from `03/29/2023`. In addition, generated videos (text-to-video) can have **arbitrary length**. \n* [04/06/2023] We integrated [Token Merging](https://github.com/dbolya/tomesd) into our code. When the highest compression is used and chunk size set to `2`, our code can run with **less than 7 GB VRAM**.  \n* [04/11/2023] New code and Huggingface demo released! We integrated **depth control**, based on [MiDaS](https://arxiv.org/pdf/1907.01341.pdf).\n* [04/13/2023] Our method has been integrad into 🧨 [Diffusers](https://huggingface.co/docs/diffusers/api/pipelines/text_to_video_zero)!\n\n## Contribute\nWe are on a journey to democratize AI and empower the creativity of everyone, and we believe Text2Video-Zero is a great research direction to unleash the zero-shot video generation and editing capacity of the amazing text-to-image models!\n\nTo achieve this goal, all contributions are welcome. Please check out these external implementations and extensions of Text2Video-Zero. We thank the authors for their efforts and contributions:\n* https://github.com/JiauZhang/Text2Video-Zero\n* https://github.com/camenduru/text2video-zero-colab\n* https://github.com/SHI-Labs/Text2Video-Zero-sd-webui\n\n\n\n\n\n\n## Setup\n\n\n\n1. Clone this repository and enter:\n\n``` shell\ngit clone https://github.com/Picsart-AI-Research/Text2Video-Zero.git\ncd Text2Video-Zero/\n```\n2. Install requirements using Python 3.9 and CUDA \u003e= 11.6\n``` shell\nvirtualenv --system-site-packages -p python3.9 venv\nsource venv/bin/activate\npip install -r requirements.txt\n```\n\n\n\n\n--- \n\n\n\n## Inference API\n\n\nTo run inferences create an instance of `Model` class\n\n``` python\nimport torch\nfrom model import Model\n\nmodel = Model(device = \"cuda\", dtype = torch.float16)\n```\n\n\n---\n\n\n### Text-To-Video\nTo directly call our text-to-video generator, run this python command which stores the result in `tmp/text2video/A_horse_galloping_on_a_street.mp4` :\n``` python\nprompt = \"A horse galloping on a street\"\nparams = {\"t0\": 44, \"t1\": 47 , \"motion_field_strength_x\" : 12, \"motion_field_strength_y\" : 12, \"video_length\": 8}\n\nout_path, fps = f\"./text2video_{prompt.replace(' ','_')}.mp4\", 4\nmodel.process_text2video(prompt, fps = fps, path = out_path, **params)\n```\n\nTo use a different stable diffusion base model run this python command:\n``` python\nfrom hf_utils import get_model_list\nmodel_list = get_model_list()\nfor idx, name in enumerate(model_list):\n  print(idx, name)\nidx = int(input(\"Select the model by the listed number: \")) # select the model of your choice\nmodel.process_text2video(prompt, model_name = model_list[idx], fps = fps, path = out_path, **params)\n```\n\n\n#### Hyperparameters (Optional)\n\nYou can define the following hyperparameters:\n* **Motion field strength**:   `motion_field_strength_x` = $\\delta_x$  and `motion_field_strength_y` = $\\delta_y$ (see our paper, Sect. 3.3.1). Default: `motion_field_strength_x=motion_field_strength_y= 12`.\n* $T$ and $T'$ (see our paper, Sect. 3.3.1). Define values `t0` and `t1` in the range `{0,...,50}`. Default: `t0=44`, `t1=47` (DDIM steps). Corresponds to timesteps `881` and `941`, respectively. \n* **Video length**: Define the number of frames `video_length` to be generated. Default: `video_length=8`.\n\n\n---\n\n\n### Text-To-Video with Pose Control\nTo directly call our text-to-video generator with pose control, run this python command:\n``` python\nprompt = 'an astronaut dancing in outer space'\nmotion_path = '__assets__/poses_skeleton_gifs/dance1_corr.mp4'\nout_path = f\"./text2video_pose_guidance_{prompt.replace(' ','_')}.gif\"\nmodel.process_controlnet_pose(motion_path, prompt=prompt, save_path=out_path)\n```\n\n\n---\n\n\n\n### Text-To-Video with Edge Control\nTo directly call our text-to-video generator with edge control, run this python command:\n``` python\nprompt = 'oil painting of a deer, a high-quality, detailed, and professional photo'\nvideo_path = '__assets__/canny_videos_mp4/deer.mp4'\nout_path = f'./text2video_edge_guidance_{prompt}.mp4'\nmodel.process_controlnet_canny(video_path, prompt=prompt, save_path=out_path)\n```\n\n#### Hyperparameters\n\nYou can define the following hyperparameters for Canny edge detection:\n* **low threshold**. Define value `low_threshold` in the range $(0, 255)$. Default: `low_threshold=100`.\n* **high threshold**. Define value `high_threshold` in the range $(0, 255)$. Default: `high_threshold=200`. Make sure that `high_threshold` \u003e `low_threshold`.\n\nYou can give hyperparameters as arguments to `model.process_controlnet_canny`\n\n\n---\n\n\n### Text-To-Video with Edge Guidance and Dreambooth specialization\nLoad a dreambooth model then proceed as described in `Text-To-Video with Edge Guidance`\n``` python\n\nprompt = 'your prompt'\nvideo_path = 'path/to/your/video'\ndreambooth_model_path = 'path/to/your/dreambooth/model'\nout_path = f'./text2video_edge_db_{prompt}.gif'\nmodel.process_controlnet_canny_db(dreambooth_model_path, video_path, prompt=prompt, save_path=out_path)\n```\n\nThe value `video_path` can be the path to a `mp4` file. To use one of the example videos provided, set `video_path=\"woman1\"`, `video_path=\"woman2\"`, `video_path=\"woman3\"`, or `video_path=\"man1\"`. \n \n\nThe value `dreambooth_model_path` can either be a link to a diffuser model file, or the name of one of the dreambooth models provided. To this end, set `dreambooth_model_path = \"Anime DB\"`, `dreambooth_model_path = \"Avatar DB\"`, `dreambooth_model_path = \"GTA-5 DB\"`, or `dreambooth_model_path = \"Arcane DB\"`.  The corresponding keywords are: `1girl` (for `Anime DB`), `arcane style` (for `Arcane DB`) `avatar style` (for `Avatar DB`) and `gtav style`  (for `GTA-5 DB`).\n\n\n#### Custom Dreambooth Models\n\n\nTo load custom Dreambooth models, [transfer](https://github.com/lllyasviel/ControlNet/discussions/12) control to the custom model and  [convert](https://github.com/huggingface/diffusers/blob/main/scripts/convert_original_stable_diffusion_to_diffusers.py) it to diffuser format. Then, the value of `dreambooth_model_path` must link to the folder containing the diffuser file. Dreambooth models can be obtained, for instance, from [CIVITAI](https://civitai.com). \n\n\n\n---\n\n\n\n### Video Instruct-Pix2Pix\n\nTo perform pix2pix video editing, run this python command:\n``` python\nprompt = 'make it Van Gogh Starry Night'\nvideo_path = '__assets__/pix2pix video/camel.mp4'\nout_path = f'./video_instruct_pix2pix_{prompt}.mp4'\nmodel.process_pix2pix(video_path, prompt=prompt, save_path=out_path)\n```\n\n\n---\n\n\n### Text-To-Video with Depth Control\n\nTo directly call our text-to-video generator with depth control, run this python command:\n``` python\nprompt = 'oil painting of a deer, a high-quality, detailed, and professional photo'\nvideo_path = '__assets__/depth_videos/deer.mp4'\nout_path = f'./text2video_depth_control_{prompt}.mp4'\nmodel.process_controlnet_depth(video_path, prompt=prompt, save_path=out_path)\n```\n\n\n\n---\n\n\n\n\n### Low Memory Inference\nEach of the above introduced interface can be run in a low memory setup. In the minimal setup, a GPU with **12 GB VRAM** is sufficient. \n\nTo reduce the memory usage, add `chunk_size=k` as additional parameter when calling one of the above defined inference APIs. The integer value `k` must be in the range `{2,...,video_length}`. It defines the number of frames that are processed at once (without any loss in quality). The lower the value the less memory is needed.\n\nWhen using the gradio app, set `chunk_size` in the `Advanced options`. \n\nThanks to the great work of [Token Merging](https://arxiv.org/abs/2303.17604), the memory usage can be further reduced. It can be configured using the  `merging_ratio` parameter with values in `[0,1]`. The higher the value, the more compression is applied (leading to faster inference and less memory requirements). Be aware that too high values will decrease the image quality. \n\n \nWe plan to continue optimizing our code to enable even lower memory consumption.\n\n---\n\n\n### Ablation Study\nTo replicate the ablation study, add additional parameters when calling the above defined inference APIs.\n*  To deactivate `cross-frame attention`: Add `use_cf_attn=False` to the parameter list.\n* To deactivate enriching latent codes with `motion dynamics`: Add `use_motion_field=False` to the parameter list.\n\n\nNote: Adding `smooth_bg=True` activates background smoothing. However, our  code does not include the salient object detector necessary to run that code.\n\n\n\n\n---\n\n\n\n## Inference using Gradio\n\n\n\u003cdetails closed\u003e\n\u003csummary\u003eClick to see details.\u003c/summary\u003e\n\nFrom the project root folder, run this shell command:\n``` shell\npython app.py\n```\n\nThen access the app [locally](http://127.0.0.1:7860) with a browser.\n\nTo access the app remotely, run this shell command:\n``` shell\npython app.py --public_access\n```\nFor security information about public access we refer to the documentation of [gradio](https://gradio.app/sharing-your-app/#security-and-file-access).\n\n\u003c/details\u003e\n\n\n\n---  \n\n\n\n## Results\n\n### Text-To-Video\n\u003ctable class=\"center\"\u003e\n\u003ctr\u003e\n  \u003ctd\u003e\u003cimg src=\"__assets__/github/results/t2v/cat_running.gif\" raw=true\u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"__assets__/github/results/t2v/playing.gif\"\u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"__assets__/github/results/t2v/running.gif\"\u003e\u003c/td\u003e              \n  \u003ctd\u003e\u003cimg src=\"__assets__/github/results/t2v/skii.gif\"\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n  \u003ctd width=25% align=\"center\"\u003e\"A cat is running on the grass\"\u003c/td\u003e\n  \u003ctd width=25% align=\"center\"\u003e\"A panda is playing guitar on times square\"\u003c/td\u003e\n  \u003ctd width=25% align=\"center\"\u003e\"A man is running in the snow\"\u003c/td\u003e\n  \u003ctd width=25% align=\"center\"\u003e\"An astronaut is skiing down the hill\"\u003c/td\u003e\n\u003c/tr\u003e\n\n\u003ctr\u003e\n  \u003ctd\u003e\u003cimg src=\"__assets__/github/results/t2v/panda_surfing.gif\" raw=true\u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"__assets__/github/results/t2v/bear_dancing.gif\"\u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"__assets__/github/results/t2v/bicycle.gif\"\u003e\u003c/td\u003e              \n  \u003ctd\u003e\u003cimg src=\"__assets__/github/results/t2v/horse_galloping.gif\"\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n  \u003ctd width=25% align=\"center\"\u003e\"A panda surfing on a wakeboard\"\u003c/td\u003e\n  \u003ctd width=25% align=\"center\"\u003e\"A bear dancing on times square\"\u003c/td\u003e\n  \u003ctd width=25% align=\"center\"\u003e\"A man is riding a bicycle in the sunshine\"\u003c/td\u003e\n  \u003ctd width=25% align=\"center\"\u003e\"A horse galloping on a street\"\u003c/td\u003e\n\u003c/tr\u003e\n\n\u003ctr\u003e\n  \u003ctd\u003e\u003cimg src=\"__assets__/github/results/t2v/tiger_walking.gif\" raw=true\u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"__assets__/github/results/t2v/panda_surfing_2.gif\"\u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"__assets__/github/results/t2v/horse_galloping_2.gif\"\u003e\u003c/td\u003e              \n  \u003ctd\u003e\u003cimg src=\"__assets__/github/results/t2v/cat_walking.gif\"\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n  \u003ctd width=25% align=\"center\"\u003e\"A tiger walking alone down the street\"\u003c/td\u003e\n  \u003ctd width=25% align=\"center\"\u003e\"A panda surfing on a wakeboard\"\u003c/td\u003e\n  \u003ctd width=25% align=\"center\"\u003e\"A horse galloping on a street\"\u003c/td\u003e\n  \u003ctd width=25% align=\"center\"\u003e\"A cute cat running in a beautiful meadow\"\u003c/td\u003e\n\u003c/tr\u003e\n\n\n\u003ctr\u003e\n  \u003ctd\u003e\u003cimg src=\"__assets__/github/results/t2v/horse_galloping_3.gif\" raw=true\u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"__assets__/github/results/t2v/panda_walking.gif\"\u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"__assets__/github/results/t2v/dog_walking.gif\"\u003e\u003c/td\u003e              \n  \u003ctd\u003e\u003cimg src=\"__assets__/github/results/t2v/astronaut.gif\"\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n  \u003ctd width=25% align=\"center\"\u003e\"A horse galloping on a street\"\u003c/td\u003e\n  \u003ctd width=25% align=\"center\"\u003e\"A panda walking alone down the street\"\u003c/td\u003e\n  \u003ctd width=25% align=\"center\"\u003e\"A dog is walking down the street\"\u003c/td\u003e\n  \u003ctd width=25% align=\"center\"\u003e\"An astronaut is waving his hands on the moon\"\u003c/td\u003e\n\u003c/tr\u003e\n\n\n\u003c/table\u003e\n\n### Text-To-Video with Pose Guidance\n\n\n\u003ctable class=\"center\"\u003e\n\u003ctr\u003e\n  \u003ctd\u003e\u003cimg src=\"__assets__/github/results/pose2v/img_bot_left_merged_with_input.gif\" raw=true\u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"__assets__/github/results/pose2v/img_bot_right_merged_with_input.gif\" raw=true\u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"__assets__/github/results/pose2v/img_top_left_merged_with_input.gif\" raw=true\u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"__assets__/github/results/pose2v/img_top_right_merged_with_input.gif\" raw=true\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n  \u003ctd width=25% align=\"center\"\u003e\"A bear dancing on the concrete\"\u003c/td\u003e\n  \u003ctd width=25% align=\"center\"\u003e\"An alien dancing under a flying saucer\"\u003c/td\u003e\n  \u003ctd width=25% align=\"center\"\u003e\"A panda dancing in Antarctica\"\u003c/td\u003e\n  \u003ctd width=25% align=\"center\"\u003e\"An astronaut dancing in the outer space\"\u003c/td\u003e\n\n\u003c/tr\u003e\n\u003c/table\u003e\n\n### Text-To-Video with Edge Guidance\n\n\n\n\u003ctable class=\"center\"\u003e\n\u003ctr\u003e\n  \u003ctd\u003e\u003cimg src=\"__assets__/github/results/edge2v/butterfly_merged_with_input.gif\" raw=true\u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"__assets__/github/results/edge2v/head_merged_with_input.gif\" raw=true\u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"__assets__/github/results/edge2v/jelly_merged_with_input.gif\" raw=true\u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"__assets__/github/results/edge2v/mask_merged_with_input.gif\" raw=true\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n  \u003ctd width=25% align=\"center\"\u003e\"White butterfly\"\u003c/td\u003e\n  \u003ctd width=25% align=\"center\"\u003e\"Beautiful girl\"\u003c/td\u003e\n    \u003ctd width=25% align=\"center\"\u003e\"A jellyfish\"\u003c/td\u003e\n  \u003ctd width=25% align=\"center\"\u003e\"beautiful girl halloween style\"\u003c/td\u003e\n\u003c/tr\u003e\n\n\u003ctr\u003e\n  \u003ctd\u003e\u003cimg src=\"__assets__/github/results/edge2v/fox_merged_with_input.gif\" raw=true\u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"__assets__/github/results/edge2v/head_2_merged_with_input.gif\" raw=true\u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"__assets__/github/results/edge2v/santa_merged_with_input.gif\" raw=true\u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"__assets__/github/results/edge2v/dear_merged_with_input.gif\" raw=true\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n  \u003ctd width=25% align=\"center\"\u003e\"Wild fox is walking\"\u003c/td\u003e\n  \u003ctd width=25% align=\"center\"\u003e\"Oil painting of a beautiful girl close-up\"\u003c/td\u003e\n    \u003ctd width=25% align=\"center\"\u003e\"A santa claus\"\u003c/td\u003e\n  \u003ctd width=25% align=\"center\"\u003e\"A deer\"\u003c/td\u003e\n\u003c/tr\u003e\n\n\u003c/table\u003e\n\n\n### Text-To-Video with Edge Guidance and Dreambooth specialization\n\n\n\n\n\u003ctable class=\"center\"\u003e\n\u003ctr\u003e\n  \u003ctd\u003e\u003cimg src=\"__assets__/github/results/canny_db/anime_style_merged_with_input.gif\" raw=true\u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"__assets__/github/results/canny_db/arcane_style_merged_with_input.gif\" raw=true\u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"__assets__/github/results/canny_db/gta-5_man_style_merged_with_input.gif\" raw=true\u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"__assets__/github/results/canny_db/img_bot_right_merged_with_input.gif\" raw=true\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n  \u003ctd width=25% align=\"center\"\u003e\"anime style\"\u003c/td\u003e\n  \u003ctd width=25% align=\"center\"\u003e\"arcane style\"\u003c/td\u003e\n    \u003ctd width=25% align=\"center\"\u003e\"gta-5 man\"\u003c/td\u003e\n  \u003ctd width=25% align=\"center\"\u003e\"avatar style\"\u003c/td\u003e\n\u003c/tr\u003e\n\n\u003c/table\u003e\n\n\n## Video Instruct Pix2Pix\n\n\u003ctable class=\"center\"\u003e\n\u003ctr\u003e\n  \u003ctd\u003e\u003cimg src=\"__assets__/github/results/Video_InstructPix2Pix/frame_1/up_left_merged_with_input.gif\" raw=true\u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"__assets__/github/results/Video_InstructPix2Pix/frame_1/up_mid_merged_with_input.gif\" raw=true\u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"__assets__/github/results/Video_InstructPix2Pix/frame_1/up_right_merged_with_input.gif\" raw=true\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n  \u003ctd width=25% align=\"center\"\u003e\"Replace man with chimpanze\"\u003c/td\u003e\n  \u003ctd width=25% align=\"center\"\u003e\"Make it Van Gogh Starry Night style\"\u003c/td\u003e\n    \u003ctd width=25% align=\"center\"\u003e\"Make it Picasso style\"\u003c/td\u003e\n\u003c/tr\u003e\n\n\u003ctr\u003e\n  \u003ctd\u003e\u003cimg src=\"__assets__/github/results/Video_InstructPix2Pix/frame_2/up_left_merged_with_input.gif\" raw=true\u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"__assets__/github/results/Video_InstructPix2Pix/frame_2/up_mid_merged_with_input.gif\" raw=true\u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cimg src=\"__assets__/github/results/Video_InstructPix2Pix/frame_2/up_right_merged_with_input.gif\" raw=true\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n  \u003ctd width=25% align=\"center\"\u003e\"Make it Expressionism style\"\u003c/td\u003e\n  \u003ctd width=25% align=\"center\"\u003e\"Make it night\"\u003c/td\u003e\n    \u003ctd width=25% align=\"center\"\u003e\"Make it autumn\"\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\n\n## Related Links \n\n* [High-Resolution Image Synthesis with Latent Diffusion Models (a.k.a. LDM \u0026 Stable Diffusion)](https://ommer-lab.com/research/latent-diffusion-models/)\n* [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://www.timothybrooks.com/instruct-pix2pix/)\n* [Adding Conditional Control to Text-to-Image Diffusion Models (a.k.a ControlNet)](https://github.com/lllyasviel/ControlNet)\n* [Diffusers](https://github.com/huggingface/diffusers)\n* [Token Merging for Stable Diffusion](https://github.com/dbolya/tomesd)\n\n## License\nOur code is published under the CreativeML Open RAIL-M license. The license provided in this repository applies to all additions and contributions we make upon the original stable diffusion code. The original stable diffusion code is under the CreativeML Open RAIL-M license, which can found [here](https://github.com/CompVis/stable-diffusion/blob/main/LICENSE).\n\n\n## BibTeX\nIf you use our work in your research, please cite our publication:\n```\n@article{text2video-zero,\n    title={Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators},\n    author={Khachatryan, Levon and Movsisyan, Andranik and Tadevosyan, Vahram and Henschel, Roberto and Wang, Zhangyang and Navasardyan, Shant and Shi, Humphrey},\n    journal={arXiv preprint arXiv:2303.13439},\n    year={2023}\n}\n```\n\n\n\n## Alternative ways to use Text2Video-Zero\n\nText2Video-Zero can alternatively used via \n\n* 🧨 [Diffusers](https://github.com/huggingface/diffusers) Library.\n\n\u003cdetails closed\u003e\n\u003csummary\u003eClick to see details.\u003c/summary\u003e\n\n\n\n### Text2Video-Zero in 🧨 Diffusers Library\n\nText2Video-Zero is [available](https://huggingface.co/docs/diffusers/api/pipelines/text_to_video_zero) in 🧨 Diffusers, starting from version `0.15.0`! \n\n\n\n[Diffusers](https://github.com/huggingface/diffusers) can be installed using the following command:\n\n\n``` shell\nvirtualenv --system-site-packages -p python3.9 venv\nsource venv/bin/activate\npip install diffusers torch imageio\n```\n\n\nTo generate a video from a text prompt, run the following command:\n\n``` python\nimport torch\nimport imageio\nfrom diffusers import TextToVideoZeroPipeline\n\n# load stable diffusion model weights\nmodel_id = \"runwayml/stable-diffusion-v1-5\"\n\n# create a TextToVideoZero pipeline\npipe = TextToVideoZeroPipeline.from_pretrained(model_id, torch_dtype=torch.float16).to(\"cuda\")\n\n# define the text prompt\nprompt = \"A panda is playing guitar on times square\"\n\n# generate the video using our pipeline\nresult = pipe(prompt=prompt).images\nresult = [(r * 255).astype(\"uint8\") for r in result]\n\n# save the resulting image\nimageio.mimsave(\"video.mp4\", result, fps=4)\n```\n\n\nFor more information, including how to run `text and pose conditional video generation`, `text and edge conditional video generation` and `text and edge and dreambooth conditional video generation`, please check the [documentation](https://huggingface.co/docs/diffusers/api/pipelines/text_to_video_zero).  \n\n\n\n\u003c/details\u003e\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpicsart-ai-research%2Ftext2video-zero","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpicsart-ai-research%2Ftext2video-zero","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpicsart-ai-research%2Ftext2video-zero/lists"}