{"id":13493100,"url":"https://github.com/lucidrains/make-a-video-pytorch","last_synced_at":"2025-05-14T23:07:01.443Z","repository":{"id":60445867,"uuid":"543189172","full_name":"lucidrains/make-a-video-pytorch","owner":"lucidrains","description":"Implementation of Make-A-Video, new SOTA text to video generator from Meta AI, in Pytorch","archived":false,"fork":false,"pushed_at":"2024-05-03T17:34:14.000Z","size":232,"stargazers_count":1969,"open_issues_count":7,"forks_count":186,"subscribers_count":68,"default_branch":"main","last_synced_at":"2025-05-14T23:06:44.623Z","etag":null,"topics":["artificial-intelligence","attention-mechanisms","axial-convolutions","deep-learning","text-to-video"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lucidrains.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-09-29T15:17:15.000Z","updated_at":"2025-05-14T13:10:33.000Z","dependencies_parsed_at":"2024-01-13T01:39:23.442Z","dependency_job_id":"6d5f0ec6-a333-46c1-b4cb-222f767ce77a","html_url":"https://github.com/lucidrains/make-a-video-pytorch","commit_stats":{"total_commits":34,"total_committers":1,"mean_commits":34.0,"dds":0.0,"last_synced_commit":"3280b3d4ab82fe83f2704975f1ad3a133f5425cd"},"previous_names":[],"tags_count":18,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2Fmake-a-video-pytorch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2Fmake-a-video-pytorch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2Fmake-a-video-pytorch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2Fmake-a-video-pytorch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lucidrains","download_url":"https://codeload.github.com/lucidrains/make-a-video-pytorch/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254243362,"owners_count":22038046,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["artificial-intelligence","attention-mechanisms","axial-convolutions","deep-learning","text-to-video"],"created_at":"2024-07-31T19:01:12.191Z","updated_at":"2025-05-14T23:06:56.435Z","avatar_url":"https://github.com/lucidrains.png","language":"Python","funding_links":[],"categories":["Python","Uncategorized"],"sub_categories":["Uncategorized"],"readme":"\u003cimg src=\"./make-a-video.png\" width=\"400px\"\u003e\u003c/img\u003e\n\n## Make-A-Video - Pytorch (wip)\n\nImplementation of \u003ca href=\"https://makeavideo.studio/\"\u003eMake-A-Video\u003c/a\u003e, new SOTA text to video generator from Meta AI, in Pytorch. They combine pseudo-3d convolutions (axial convolutions) and temporal attention and show much better temporal fusion.\n\nThe pseudo-3d convolutions isn't a new concept. It has been explored before in other contexts, say for protein contact prediction as \u003ca href=\"https://www.biorxiv.org/content/10.1101/2022.08.04.502748v2.full\"\u003e\"dimensional hybrid residual networks\"\u003c/a\u003e.\n\nThe gist of the paper comes down to, take a SOTA text-to-image model (here they use DALL-E2, but the same learning points would easily apply to Imagen), make a few minor modifications for \u003ca href=\"https://arxiv.org/abs/2204.03458\"\u003eattention across time\u003c/a\u003e and other ways to skimp on the compute cost, do frame interpolation correctly, get a great video model out.\n\n\u003ca href=\"https://www.youtube.com/watch?v=AcvmyqGgMh8\"\u003eAI Coffee Break explanation\u003c/a\u003e\n\n## Appreciation\n\n- \u003ca href=\"https://stability.ai/\"\u003eStability.ai\u003c/a\u003e for the generous sponsorship to work on cutting edge artificial intelligence research\n\n- \u003ca href=\"http://www.jonathanho.me/\"\u003eJonathan Ho\u003c/a\u003e for bringing about a revolution in generative artificial intelligence through \u003ca href=\"https://arxiv.org/abs/2006.11239\"\u003ehis seminal paper\u003c/a\u003e\n\n- \u003ca href=\"https://github.com/arogozhnikov\"\u003eAlex\u003c/a\u003e for \u003ca href=\"https://github.com/arogozhnikov/einops\"\u003eeinops\u003c/a\u003e, an abstraction that is simply genius. No other word for it.\n\n## Install\n\n```bash\n$ pip install make-a-video-pytorch\n```\n\n## Usage\n\nPassing in video features\n\n```python\nimport torch\nfrom make_a_video_pytorch import PseudoConv3d, SpatioTemporalAttention\n\nconv = PseudoConv3d(\n    dim = 256,\n    kernel_size = 3\n)\n\nattn = SpatioTemporalAttention(\n    dim = 256,\n    dim_head = 64,\n    heads = 8\n)\n\nvideo = torch.randn(1, 256, 8, 16, 16) # (batch, features, frames, height, width)\n\nconv_out = conv(video) # (1, 256, 8, 16, 16)\nattn_out = attn(video) # (1, 256, 8, 16, 16)\n```\n\nPassing in images (if one were to pretrain on images first), both temporal convolution and attention will be automatically skipped. In other words, you can use this straightforwardly in your 2d Unet and then port it over to a 3d Unet once that phase of the training is done. The temporal modules are initialized to output identity as the paper had done.\n\n```python\nimport torch\nfrom make_a_video_pytorch import PseudoConv3d, SpatioTemporalAttention\n\nconv = PseudoConv3d(\n    dim = 256,\n    kernel_size = 3\n)\n\nattn = SpatioTemporalAttention(\n    dim = 256,\n    dim_head = 64,\n    heads = 8\n)\n\nimages = torch.randn(1, 256, 16, 16) # (batch, features, height, width)\n\nconv_out = conv(images) # (1, 256, 16, 16)\nattn_out = attn(images) # (1, 256, 16, 16)\n```\n\nYou can also control the two modules so that when fed 3-dimensional features, it only does training spatially\n\n```python\nimport torch\nfrom make_a_video_pytorch import PseudoConv3d, SpatioTemporalAttention\n\nconv = PseudoConv3d(\n    dim = 256,\n    kernel_size = 3\n)\n\nattn = SpatioTemporalAttention(\n    dim = 256,\n    dim_head = 64,\n    heads = 8\n)\n\nvideo = torch.randn(1, 256, 8, 16, 16) # (batch, features, frames, height, width)\n\n# below it will not train across time\n\nconv_out = conv(video, enable_time = False) # (1, 256, 8, 16, 16)\nattn_out = attn(video, enable_time = False) # (1, 256, 8, 16, 16)\n```\n\nFull `SpaceTimeUnet` that is agnostic to images or video training, and where even if video is passed in, time can be ignored\n\n\n```python\nimport torch\nfrom make_a_video_pytorch import SpaceTimeUnet\n\nunet = SpaceTimeUnet(\n    dim = 64,\n    channels = 3,\n    dim_mult = (1, 2, 4, 8),\n    resnet_block_depths = (1, 1, 1, 2),\n    temporal_compression = (False, False, False, True),\n    self_attns = (False, False, False, True),\n    condition_on_timestep = False,\n    attn_pos_bias = False,\n    flash_attn = True\n).cuda()\n\n# train on images\n\nimages = torch.randn(1, 3, 128, 128).cuda()\nimages_out  = unet(images)\n\nassert images.shape == images_out.shape\n\n# then train on videos\n\nvideo = torch.randn(1, 3, 16, 128, 128).cuda()\nvideo_out = unet(video)\n\nassert video_out.shape == video.shape\n\n# or even treat your videos as images\n\nvideo_as_images_out = unet(video, enable_time = False)\n```\n\n## Todo\n\n- [x] give attention the best positional embeddings research has to offer\n- [x] soup up the attention\n- [x] add flash attention\n\n- [ ] make sure dalle2-pytorch can accept `SpaceTimeUnet` for training\n\n## Citations\n\n```bibtex\n@misc{Singer2022,\n    author  = {Uriel Singer},\n    url     = {https://makeavideo.studio/Make-A-Video.pdf}\n}\n```\n\n```bibtex\n@inproceedings{rogozhnikov2022einops,\n    title   = {Einops: Clear and Reliable Tensor Manipulations with Einstein-like Notation},\n    author  = {Alex Rogozhnikov},\n    booktitle = {International Conference on Learning Representations},\n    year    = {2022},\n    url     = {https://openreview.net/forum?id=oapKSVM2bcj}\n}\n```\n\n```bibtex\n@article{Dong2021AttentionIN,\n    title   = {Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth},\n    author  = {Yihe Dong and Jean-Baptiste Cordonnier and Andreas Loukas},\n    journal = {ArXiv},\n    year    = {2021},\n    volume  = {abs/2103.03404}\n}\n```\n\n```bibtex\n@article{Zhang2021TokenST,\n    title   = {Token Shift Transformer for Video Classification},\n    author  = {Hao Zhang and Y. Hao and Chong-Wah Ngo},\n    journal = {Proceedings of the 29th ACM International Conference on Multimedia},\n    year    = {2021}\n}\n```\n\n```bibtex\n@inproceedings{shleifer2022normformer,\n    title   = {NormFormer: Improved Transformer Pretraining with Extra Normalization},\n    author  = {Sam Shleifer and Myle Ott},\n    booktitle = {Submitted to The Tenth International Conference on Learning Representations },\n    year    = {2022},\n    url     = {https://openreview.net/forum?id=GMYWzWztDx5},\n}\n```\n\n```bibtex\n@inproceedings{dao2022flashattention,\n    title   = {Flash{A}ttention: Fast and Memory-Efficient Exact Attention with {IO}-Awareness},\n    author  = {Dao, Tri and Fu, Daniel Y. and Ermon, Stefano and Rudra, Atri and R{\\'e}, Christopher},\n    booktitle = {Advances in Neural Information Processing Systems},\n    year    = {2022}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flucidrains%2Fmake-a-video-pytorch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flucidrains%2Fmake-a-video-pytorch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flucidrains%2Fmake-a-video-pytorch/lists"}