{"id":13488893,"url":"https://github.com/lucidrains/video-diffusion-pytorch","last_synced_at":"2025-04-14T06:49:01.400Z","repository":{"id":37408449,"uuid":"479199061","full_name":"lucidrains/video-diffusion-pytorch","owner":"lucidrains","description":"Implementation of Video Diffusion Models, Jonathan Ho's new paper extending DDPMs to Video Generation - in Pytorch","archived":false,"fork":false,"pushed_at":"2024-05-03T16:58:59.000Z","size":1321,"stargazers_count":1297,"open_issues_count":28,"forks_count":134,"subscribers_count":28,"default_branch":"main","last_synced_at":"2025-04-07T00:16:13.552Z","etag":null,"topics":["artificial-intelligence","ddpm","deep-learning","text-to-video","video-generation"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lucidrains.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-04-08T01:08:24.000Z","updated_at":"2025-04-06T17:29:37.000Z","dependencies_parsed_at":"2024-01-15T00:07:24.428Z","dependency_job_id":"0e9cc5b5-e033-4c26-80f7-9f8464c7e233","html_url":"https://github.com/lucidrains/video-diffusion-pytorch","commit_stats":null,"previous_names":[],"tags_count":57,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2Fvideo-diffusion-pytorch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2Fvideo-diffusion-pytorch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2Fvideo-diffusion-pytorch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2Fvideo-diffusion-pytorch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lucidrains","download_url":"https://codeload.github.com/lucidrains/video-diffusion-pytorch/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248837273,"owners_count":21169373,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["artificial-intelligence","ddpm","deep-learning","text-to-video","video-generation"],"created_at":"2024-07-31T18:01:23.651Z","updated_at":"2025-04-14T06:49:01.375Z","avatar_url":"https://github.com/lucidrains.png","language":"Python","funding_links":[],"categories":["Video Generation","Python","Libraries"],"sub_categories":["Video"],"readme":"![machine imagined fireworks](./fireworks.webp)\n\n*these fireworks do not exist*\n\n## Video Diffusion - Pytorch\n\nText to video, it is happening! \u003ca href=\"https://video-diffusion.github.io/\"\u003eOfficial Project Page\u003c/a\u003e\n\nImplementation of \u003ca href=\"https://arxiv.org/abs/2204.03458\"\u003eVideo Diffusion Models\u003c/a\u003e, \u003ca href=\"http://www.jonathanho.me/\"\u003eJonathan Ho\u003c/a\u003e's new paper extending DDPMs to Video Generation - in Pytorch. It uses a special space-time factored U-net, extending generation from 2d images to 3d videos\n\n\u003cimg src=\"./3d-unet.png\" width=\"500px\"\u003e\u003c/img\u003e\n\n## Status\n\n14k for difficult moving mnist (converging much faster and better than \u003ca href=\"https://wandb.ai/lucidrains/nuwa-moving-mnist/reports/moving-mnist-nuwa--VmlldzoxNjk3MjI3?accessToken=cx03lswmr4bxj9dhrzzm5c3xebdmfq28a4dqzsoq9n89by6ppofukq7bxp19078j\"\u003eNUWA\u003c/a\u003e) - wip\n\n\u003cimg src=\"./samples/moving-mnist.gif\" width=\"250px\"\u003e\n\nThe above experiments are possible only due to resources provided by \u003ca href=\"https://stability.ai/\"\u003eStability.ai\u003c/a\u003e\n\nAny new developments for text-to-video synthesis will be centralized at \u003ca href=\"https://github.com/lucidrains/imagen-pytorch#text-to-video-ongoing-research\"\u003eImagen-pytorch\u003c/a\u003e\n\n## Install\n\n```bash\n$ pip install video-diffusion-pytorch\n```\n\n## Usage\n\n```python\nimport torch\nfrom video_diffusion_pytorch import Unet3D, GaussianDiffusion\n\nmodel = Unet3D(\n    dim = 64,\n    dim_mults = (1, 2, 4, 8)\n)\n\ndiffusion = GaussianDiffusion(\n    model,\n    image_size = 32,\n    num_frames = 5,\n    timesteps = 1000,   # number of steps\n    loss_type = 'l1'    # L1 or L2\n)\n\nvideos = torch.randn(1, 3, 5, 32, 32) # video (batch, channels, frames, height, width) - normalized from -1 to +1\nloss = diffusion(videos)\nloss.backward()\n# after a lot of training\n\nsampled_videos = diffusion.sample(batch_size = 4)\nsampled_videos.shape # (4, 3, 5, 32, 32)\n```\n\nFor conditioning on text, they derived text embeddings by first passing the tokenized text through BERT-large. Then you just have to train it like so\n\n```python\nimport torch\nfrom video_diffusion_pytorch import Unet3D, GaussianDiffusion\n\nmodel = Unet3D(\n    dim = 64,\n    cond_dim = 64,\n    dim_mults = (1, 2, 4, 8)\n)\n\ndiffusion = GaussianDiffusion(\n    model,\n    image_size = 32,\n    num_frames = 5,\n    timesteps = 1000,   # number of steps\n    loss_type = 'l1'    # L1 or L2\n)\n\nvideos = torch.randn(2, 3, 5, 32, 32) # video (batch, channels, frames, height, width)\ntext = torch.randn(2, 64)             # assume output of BERT-large has dimension of 64\n\nloss = diffusion(videos, cond = text)\nloss.backward()\n# after a lot of training\n\nsampled_videos = diffusion.sample(cond = text)\nsampled_videos.shape # (2, 3, 5, 32, 32)\n```\n\nYou can also directly pass in the descriptions of the video as strings, if you plan on using BERT-base for text conditioning\n\n```python\nimport torch\nfrom video_diffusion_pytorch import Unet3D, GaussianDiffusion\n\nmodel = Unet3D(\n    dim = 64,\n    use_bert_text_cond = True,  # this must be set to True to auto-use the bert model dimensions\n    dim_mults = (1, 2, 4, 8),\n)\n\ndiffusion = GaussianDiffusion(\n    model,\n    image_size = 32,    # height and width of frames\n    num_frames = 5,     # number of video frames\n    timesteps = 1000,   # number of steps\n    loss_type = 'l1'    # L1 or L2\n)\n\nvideos = torch.randn(3, 3, 5, 32, 32) # video (batch, channels, frames, height, width)\n\ntext = [\n    'a whale breaching from afar',\n    'young girl blowing out candles on her birthday cake',\n    'fireworks with blue and green sparkles'\n]\n\nloss = diffusion(videos, cond = text)\nloss.backward()\n# after a lot of training\n\nsampled_videos = diffusion.sample(cond = text, cond_scale = 2)\nsampled_videos.shape # (3, 3, 5, 32, 32)\n```\n\n## Training\n\nThis repository also contains a handy `Trainer` class for training on a folder of `gifs`. Each `gif` must be of the correct dimensions `image_size` and `num_frames`.\n\n```python\nimport torch\nfrom video_diffusion_pytorch import Unet3D, GaussianDiffusion, Trainer\n\nmodel = Unet3D(\n    dim = 64,\n    dim_mults = (1, 2, 4, 8),\n)\n\ndiffusion = GaussianDiffusion(\n    model,\n    image_size = 64,\n    num_frames = 10,\n    timesteps = 1000,   # number of steps\n    loss_type = 'l1'    # L1 or L2\n).cuda()\n\ntrainer = Trainer(\n    diffusion,\n    './data',                         # this folder path needs to contain all your training data, as .gif files, of correct image size and number of frames\n    train_batch_size = 32,\n    train_lr = 1e-4,\n    save_and_sample_every = 1000,\n    train_num_steps = 700000,         # total training steps\n    gradient_accumulate_every = 2,    # gradient accumulation steps\n    ema_decay = 0.995,                # exponential moving average decay\n    amp = True                        # turn on mixed precision\n)\n\ntrainer.train()\n```\n\nSample videos (as `gif` files) will be saved to `./results` periodically, as are the diffusion model parameters.\n\n## Co-training Images and Video\n\nOne of the claims in the paper is that by doing factored space-time attention, one can force the network to attend on the present for training images and video in conjunction, leading to better results.\n\nIt was not clear how they achieved this, but I furthered a guess.\n\nTo arrest attention to the present moment for a certain percentage of batch videos samples, simply pass `prob_focus_present = \u003cprob\u003e` on the diffusion forward method\n\n```python\nloss = diffusion(videos, cond = text, prob_focus_present = 0.5) # for 50% of videos, focus on the present during training\nloss.backward()\n```\n\nIf you have a better idea how this is done, just open a github issue.\n\n## Todo\n\n- [x] wire up text conditioning, use classifier free guidance\n- [x] relative positional encodings in attention (space and time) - use T5 relative positional bias instead of what they used\n- [x] add a forward keyword argument that arrests attention across time (as reported / claimed in the paper, this type of image + video simultaneous training improves results)\n- [x] consider doing a 3d version of CLIP, so one can eventually apply the lessons of DALL-E2 to video https://github.com/lucidrains/dalle2-video\n- [x] offer way for Trainer to curtail or pad frames, if gif is too long\n- [ ] find a good torchvideo-like library (torchvideo seems immature) for training on fireworks\n- [ ] project text into 4-8 tokens, and use them as memory key / values to condition both time and space in attention blocks\n- [ ] prepare a jax version for large scale TPU training\n- [ ] have Trainer take care of conditional video synthesis, with text offered as corresponding {video_filename}.txt within the same folder\n- [ ] see if ffcv or squirrel-core is a good fit\n- [ ] bring in token shifts, along time and space\n\n## Citations\n\n```bibtex\n@misc{ho2022video,\n  title   = {Video Diffusion Models}, \n  author  = {Jonathan Ho and Tim Salimans and Alexey Gritsenko and William Chan and Mohammad Norouzi and David J. Fleet},\n  year    = {2022},\n  eprint  = {2204.03458},\n  archivePrefix = {arXiv},\n  primaryClass = {cs.CV}\n}\n```\n\n```bibtex\n@misc{Saharia2022,\n    title   = {Imagen: unprecedented photorealism × deep level of language understanding},\n    author  = {Chitwan Saharia*, William Chan*, Saurabh Saxena†, Lala Li†, Jay Whang†, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho†, David Fleet†, Mohammad Norouzi*},\n    year    = {2022}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flucidrains%2Fvideo-diffusion-pytorch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flucidrains%2Fvideo-diffusion-pytorch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flucidrains%2Fvideo-diffusion-pytorch/lists"}