Awesome-Video-Diffusion-Models
[CSUR] A Survey on Video Diffusion Models
https://github.com/ChenHsing/Awesome-Video-Diffusion-Models
Last synced: 16 days ago
JSON representation
-
Contact
-
Video Understanding
-
-
Data
-
Caption-level
- CelebV-Text: A Large-Scale Facial Text-Video Dataset - b31b1b.svg)](https://arxiv.org/pdf/2303.14717.pdf) | [](https://github.com/CelebV-Text/CelebV-Text) | - | CVPR, 2023
- Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptions - b31b1b.svg)](https://arxiv.org/abs/2111.10337)|- |- |Nov, 2021 |
- Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval - b31b1b.svg)](https://arxiv.org/abs/2104.00650) | - | - |ICCV, 2021 |
- MSR-VTT: A Large Video Description Dataset for Bridging Video and Language - b31b1b.svg)](https://openaccess.thecvf.com/content_cvpr_2016/html/Xu_MSR-VTT_A_Large_CVPR_2016_paper.html) | -| -| CVPR, 2016|
- Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers - b31b1b.svg)](https://arxiv.org/abs/2402.19479)| [](https://github.com/snap-research/Panda-70M)| [](https://snap-research.github.io/Panda-70M/) | CVPR, 2024 |
- CelebV-Text: A Large-Scale Facial Text-Video Dataset - b31b1b.svg)](https://arxiv.org/pdf/2303.14717.pdf) | [](https://github.com/CelebV-Text/CelebV-Text) | - | CVPR, 2023
-
Category-level
- UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild - b31b1b.svg)](https://arxiv.org/abs/1212.0402) | - | - | Dec., 2012
- First Order Motion Model for Image Animation - b31b1b.svg)](https://arxiv.org/abs/2003.00196) | -|- | May, 2023 |
- Learning to Generate Time-Lapse Videos Using Multi-Stage Dynamic Generative Adversarial Networks - b31b1b.svg)](https://arxiv.org/abs/1709.07592) | -| -| CVPR,2018|
-
Metric and BenchMark
- Towards A Better Metric for Text-to-Video Generation - b31b1b.svg)](https://arxiv.org/abs/2401.07781) |-|[](https://showlab.github.io/T2VScore/) | Jan, 2024
- AIGCBench: Comprehensive Evaluation of Image-to-Video Content Generated by AI - b31b1b.svg)](https://arxiv.org/abs/2401.01651) | - | - | Jan, 2024 |
- VBench: Comprehensive Benchmark Suite for Video Generative Models - b31b1b.svg)](https://arxiv.org/abs/2311.17982) |[](https://github.com/Vchitect/VBench)|[](https://vchitect.github.io/VBench-project/) | Nov, 2023
- FETV: A Benchmark for Fine-Grained Evaluation of Open-Domain Text-to-Video Generation - b31b1b.svg)](https://arxiv.org/abs/2311.01813) | - | - | NeurIPS, 2023 |
- CVPR 2023 Text Guided Video Editing Competition - b31b1b.svg)](https://arxiv.org/abs/2310.16003) | - | - | Oct., 2023 |
- EvalCrafter: Benchmarking and Evaluating Large Video Generation Models - b31b1b.svg)](https://arxiv.org/abs/2310.11440) | [](https://github.com/evalcrafter/EvalCrafter)|[](https://huggingface.co/datasets/RaphaelLiu/EvalCrafter_T2V_Dataset) | Oct., 2023 |
- Measuring the Quality of Text-to-Video Model Outputs: Metrics and Dataset - b31b1b.svg)](https://arxiv.org/abs/2309.08009) | - | - | Sep., 2023 |
- Subjective-Aligned Dateset and Metric for Text-to-Video Quality Assessment - b31b1b.svg)](https://arxiv.org/abs/2403.11956) |-|- | Mar, 2024
- STREAM: Spatio-TempoRal Evaluation and Analysis Metric for Video Generative Models - b31b1b.svg)](https://arxiv.org/abs/2403.09669) |[](https://github.com/pro2nit/STREAM)|- | ICLR, 2024
- ChronoMagic-Bench: A Benchmark for Metamorphic Evaluation of Text-to-Time-lapse Video Generation - b31b1b.svg)](https://arxiv.org/abs/2406.18522)| [](https://github.com/PKU-YuanGroup/ChronoMagic-Bench)| [](https://pku-yuangroup.github.io/ChronoMagic-Bench/) | NeurIPS, 2024 |
- Fréchet Video Motion Distance: A Metric for Evaluating Motion Consistency in Videos - b31b1b.svg)](https://arxiv.org/abs/2407.16124)| [](https://github.com/DSL-Lab/FVMD-frechet-video-motion-distance)| - | Jul., 2024 |
- OpenS2V-Nexus: A Detailed Benchmark and Million-Scale Dataset for Subject-to-Video Generation - b31b1b.svg)](https://arxiv.org/abs/2505.20292) |[](https://github.com/PKU-YuanGroup/OpenS2V-Nexus)|[](https://pku-yuangroup.github.io/OpenS2V-Nexus) | May, 2025 |
-
-
Depth-guided Video Generation
-
Brain-guided Video Generation
- Animate-A-Story: Storytelling with Retrieval-Augmented Video Generation - b31b1b.svg)](https://arxiv.org/abs/2307.06940) | [](https://github.com/VideoCrafter/Animate-A-Story) | [](https://videocrafter.github.io/Animate-A-Story/) | Jul., 2023 |
- Make-Your-Video: Customized Video Generation Using Textual and Structural Guidance - b31b1b.svg)](https://arxiv.org/abs/2306.00943) | [](https://github.com/VideoCrafter/Make-Your-Video) | [](https://doubiiu.github.io/projects/Make-Your-Video/) | Jun., 2023 |
- StableV2V: Stablizing Shape Consistency in Video-to-Video Editing - b31b1b.svg)]([https://arxiv.org/abs/2307.06940](https://arxiv.org/pdf/2411.11045)) | [](https://github.com/AlonzoLeeeooo/StableV2V) | [](https://alonzoleeeooo.github.io/StableV2V/) | Nov., 2024 |
-
Multi-modal guided Video Generation
- CustomVideo: Customizing Text-to-Video Generation with Multiple Subjects - b31b1b.svg)](https://arxiv.org/abs/2401.09962) | - | - | Jan., 2024 |
- MoonShot: Towards Controllable Video Generation and Editing with Multimodal Conditions - b31b1b.svg)](https://arxiv.org/abs/2401.01827) | [](https://github.com/salesforce/LAVIS) | [](https://showlab.github.io/Moonshot/) | Jan., 2024 |
- PEEKABOO: Interactive Video Generation via Masked-Diffusion - b31b1b.svg)](https://arxiv.org/abs/2312.07509) | - | [](https://jinga-lala.github.io/projects/Peekaboo/) | Dec., 2023 |
- CMMD: Contrastive Multi-Modal Diffusion for Video-Audio Conditional Modeling - b31b1b.svg)](https://arxiv.org/abs/2312.05412) | - | - | Dec., 2023 |
- Fine-grained Controllable Video Generation via Object Appearance and Context - b31b1b.svg)](https://arxiv.org/abs/2312.02919) | - | [](https://hhsinping.github.io/factor/) | Nov., 2023 |
- GPT4Video: A Unified Multimodal Large Language Model for Instruction-Followed Understanding and Safety-Aware Generation - b31b1b.svg)](https://arxiv.org/abs/2311.16511) | - | [](https://gpt4video.github.io/) | Nov., 2023 |
- Panacea: Panoramic and Controllable Video Generation for Autonomous Driving - b31b1b.svg)](https://arxiv.org/abs/2311.16813) | - | [](https://panacea-ad.github.io/) | Nov., 2023 |
- SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Models - b31b1b.svg)](https://arxiv.org/abs/2311.16933) | - | [](https://guoyww.github.io/projects/SparseCtrl/) | Nov., 2023 |
- VideoComposer: Compositional Video Synthesis with Motion Controllability - b31b1b.svg)](https://arxiv.org/abs/2306.02018) | [](https://github.com/damo-vilab/videocomposer) | [](https://videocomposer.github.io/) | Jun., 2023 |
- NExT-GPT: Any-to-Any Multimodal LLM - b31b1b.svg)](https://arxiv.org/abs/2309.05519) | - | - | Sep, 2023 |
- MovieFactory: Automatic Movie Creation from Text using Large Generative Models for Language and Images - b31b1b.svg)](https://arxiv.org/pdf/2306.07257.pdf) | - | [](https://www.bilibili.com/video/BV1qj411Q76P/) | Jun, 2023 |
- Any-to-Any Generation via Composable Diffusion - b31b1b.svg)](https://arxiv.org/abs/2305.11846) | [](https://github.com/microsoft/i-Code/tree/main/i-Code-V3) | [](https://codi-gen.github.io/) | May, 2023 |
- Mm-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation - b31b1b.svg)](https://arxiv.org/abs/2212.09478) | [](https://github.com/researchmm/MM-Diffusion) | - | CVPR 2023 |
- ActAnywhere: Subject-Aware Video Background Generation - b31b1b.svg)](https://arxiv.org/abs/2401.10822) | - | [](https://actanywhere.github.io/) | Jan., 2024 |
- AnimateLCM: Accelerating the Animation of Personalized Diffusion Models and Adapters with Decoupled Consistency Learning - b31b1b.svg)](https://arxiv.org/abs/2402.00769) | - | -| Jan., 2024 |
- InteractiveVideo: User-Centric Controllable Video Generation with Synergistic Multimodal Instructions - b31b1b.svg)](https://arxiv.org/abs/2402.03040) | - | [](https://invictus717.github.io/InteractiveVideo/) | Feb., 2024 |
- Direct-a-Video: Customized Video Generation with User-Directed Camera Movement and Object Motion - b31b1b.svg)](https://arxiv.org/abs/2402.03162) | - | [](https://direct-a-video.github.io/) | Feb., 2024 |
- Boximator: Generating Rich and Controllable Motions for Video Synthesis - b31b1b.svg)](https://arxiv.org/abs/2402.01566) | - | [](https://boximator.github.io/) | Feb., 2024 |
- Magic-Me: Identity-Specific Video Customized Diffusion - b31b1b.svg)](https://arxiv.org/abs/2402.09368) | - | [](https://github.com/Zhen-Dong/Magic-Me) | Feb., 2024 |
- UniCtrl: Improving the Spatiotemporal Consistency of Text-to-Video Diffusion Models via Training-Free Unified Attention Control - b31b1b.svg)](https://arxiv.org/abs/2403.02332) | - | - | Mar., 2024 |
- MovieFactory: Automatic Movie Creation from Text using Large Generative Models for Language and Images - b31b1b.svg)](https://arxiv.org/pdf/2306.07257.pdf) | - | [](https://www.bilibili.com/video/BV1qj411Q76P/) | Jun, 2023 |
-
-
Open-source Toolboxes and Foundation Models
- VideoPoet - |
- Stable Video Diffusion - AI/generative-models.svg?style=social&label=Star)](https://github.com/Stability-AI/generative-models)|
- NeverEnds - |
- EMU-Video - |
- GEN-2 - |
- ModelScope
- ZeroScope - |
- Diffusers (T2V synthesis) - |
- Genmo - |
- Fliki - |
- Sora - |
- Morph Studio - |
- Genie - |
- Morph Studio - |
- Open-Sora-Plan - YuanGroup/Open-Sora-Plan.svg?style=social&label=Star)](https://github.com/PKU-YuanGroup/Open-Sora-Plan)|
- Open-Sora - Sora.svg?style=social&label=Star)](https://github.com/hpcaitech/Open-Sora)|
- T2V Synthesis Colab - to-video-synthesis-colab.svg?style=social&label=Star)](https://github.com/camenduru/text-to-video-synthesis-colab)|
- AnimateDiff
- Text2Video-Zero - AI-Research/Text2Video-Zero.svg?style=social&label=Star)](https://github.com/Picsart-AI-Research/Text2Video-Zero)|
- HotShot-XL - XL.svg?style=social&label=Star)](https://github.com/hotshotco/Hotshot-XL)|
- CogVideoX
- NeverEnds - |
- Pika - |
- Helios - YuanGroup/Helios.svg?style=social&label=Star)](https://github.com/PKU-YuanGroup/Helios)|
- Seedream AI Studio - |
-
Text-to-Video Generation
-
Training-based
- InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation - b31b1b.svg)](https://arxiv.org/abs/2307.06942) | [](https://github.com/OpenGVLab/InternVideo/tree/main/Data/InternVid) | - | Jul., 2023 |
- VideoFactory: Swap Attention in Spatiotemporal Diffusions for Text-to-Video Generation - b31b1b.svg)](https://arxiv.org/abs/2305.10874) | - | - | May, 2023 |
- UNIVG: TOWARDS UNIFIED-MODAL VIDEO GENERATION - b31b1b.svg)](https://arxiv.org/abs/2401.09084) |-| [](https://univg-baidu.github.io/) | Jan, 2024
- VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models - b31b1b.svg)](https://arxiv.org/abs/2401.09047) |[](https://github.com/VideoCrafter/VideoCrafter)|[](https://ailab-cvc.github.io/videocrafter/) | Jan, 2024
- 360DVD: Controllable Panorama Video Generation with 360-Degree Video Diffusion Model - b31b1b.svg)](https://arxiv.org/abs/2401.06578) |-|[](https://akaneqwq.github.io/360DVD/) | Jan, 2024
- MagicVideo-V2: Multi-Stage High-Aesthetic Video Generation - b31b1b.svg)](https://arxiv.org/abs/2401.04468) |-|[](https://magicvideov2.github.io/) | Jan, 2024
- VideoDrafter: Content-Consistent Multi-Scene Video Generation with LLM - b31b1b.svg)](https://arxiv.org/abs/2401.01256) |-|[](https://videodrafter.github.io/) | Jan, 2024
- A Recipe for Scaling up Text-to-Video Generation with Text-free Videos - b31b1b.svg)](https://arxiv.org/abs/2312.15770) |[](https://github.com/damo-vilab/i2vgen-xl)|[](https://tf-t2v.github.io/) | Dec, 2023
- InstructVideo: Instructing Video Diffusion Models with Human Feedback - b31b1b.svg)](https://arxiv.org/abs/2312.12490) |[](https://github.com/damo-vilab/i2vgen-xl)|[](https://instructvideo.github.io/) | Dec, 2023
- VideoLCM: Video Latent Consistency Model - b31b1b.svg)](https://arxiv.org/abs/2312.09109) |-|- | Dec, 2023
- Photorealistic Video Generation with Diffusion Models - b31b1b.svg)](https://arxiv.org/abs/2312.06662) |-|[](https://walt-video-diffusion.github.io/) | Dec, 2023
- Hierarchical Spatio-temporal Decoupling for Text-to-Video Generation - b31b1b.svg)](https://arxiv.org/abs/2312.04483) |[](https://github.com/damo-vilab/i2vgen-xl)|[](https://higen-t2v.github.io/) | Dec, 2023
- Delving Deep into Diffusion Transformers for Image and Video Generation - b31b1b.svg)](https://arxiv.org/abs/2312.04557) |-|[](https://www.shoufachen.com/gentron_website/) | Dec, 2023
- StyleCrafter: Enhancing Stylized Text-to-Video Generation with Style Adapter - b31b1b.svg)](https://arxiv.org/abs/2312.00330) |[](https://github.com/GongyeLiu/StyleCrafter)|[](https://gongyeliu.github.io/StyleCrafter.github.io/) | Nov, 2023
- MicroCinema: A Divide-and-Conquer Approach for Text-to-Video Generation - b31b1b.svg)](https://arxiv.org/abs/2311.18829) |- |[](https://wangyanhui666.github.io/MicroCinema.github.io/) | Nov, 2023
- ART•V: Auto-Regressive Text-to-Video Generation with Diffusion Models - b31b1b.svg)](https://arxiv.org/abs/2311.18834) |[](https://github.com/WarranWeng/ART.V) |[](https://warranweng.github.io/art.v) | Nov, 2023
- Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets - b31b1b.svg)](https://arxiv.org/abs/2311.15127) |[](https://github.com/Stability-AI/generative-models)|[](https://huggingface.co/stabilityai/stable-video-diffusion-img2vid) | Nov, 2023
- FusionFrames: Efficient Architectural Aspects for Text-to-Video Generation Pipeline - b31b1b.svg)](https://arxiv.org/abs/2311.13073) | [](https://github.com/ai-forever/KandinskyVideo) |[](https://ai-forever.github.io/kandinsky-video/)| Nov, 2023
- MoVideo: Motion-Aware Video Generation with Diffusion Models - b31b1b.svg)](https://arxiv.org/abs/2311.11325) |-|[](https://jingyunliang.github.io/MoVideo/)| Nov, 2023
- Make Pixels Dance: High-Dynamic Video Generation - b31b1b.svg)](https://arxiv.org/abs/2311.10982) |-|[](https://makepixelsdance.github.io/)| Nov, 2023
- Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioning - b31b1b.svg)](https://arxiv.org/abs/2311.10709) |-|[](https://emu-video.metademolab.com/)| Nov, 2023
- Optimal Noise pursuit for Augmenting Text-to-Video Generation - b31b1b.svg)](https://arxiv.org/abs/2311.00949) |-|-| Nov, 2023
- VideoDreamer: Customized Multi-Subject Text-to-Video Generation with Disen-Mix Finetuning - b31b1b.svg)](https://arxiv.org/abs/2311.00990) |-|[](https://videodreamer23.github.io/) | Nov, 2023
- VideoCrafter1: Open Diffusion Models for High-Quality Video Generation - b31b1b.svg)](https://arxiv.org/abs/2310.19512) |[](https://github.com/VideoCrafter/VideoCrafter)|[](https://ailab-cvc.github.io/videocrafter/) | Oct, 2023
- SEINE: Short-to-Long Video Diffusion Model for Generative Transition and Prediction - b31b1b.svg)](https://arxiv.org/abs/2310.20700) | [](https://github.com/Vchitect/SEINE) | [](https://vchitect.github.io/SEINE-project/) | Oct, 2023 |
- DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors - b31b1b.svg)](https://arxiv.org/abs/2310.12190) | [](https://github.com/AILab-CVC/VideoCrafter) | [](https://ailab-cvc.github.io/videocrafter/) | Oct., 2023 |
- LAMP: Learn A Motion Pattern for Few-Shot-Based Video Generation - b31b1b.svg)](https://arxiv.org/abs/2310.10769) | [](https://github.com/RQ-Wu/LAMP) | [](https://rq-wu.github.io/projects/LAMP/) | Oct., 2023 |
- DrivingDiffusion: Layout-Guided multi-view driving scene video generation with latent diffusion model - b31b1b.svg)](https://arxiv.org/abs/2310.07771) | [](https://github.com/shalfun/DrivingDiffusion) | [](https://drivingdiffusion.github.io/) | Oct, 2023 |
- MotionDirector: Motion Customization of Text-to-Video Diffusion Models - b31b1b.svg)](https://arxiv.org/abs/2310.08465) | [](https://github.com//showlab/MotionDirector) | [](https://showlab.github.io/MotionDirector/) | Oct, 2023 |
- VideoDirectorGPT: Consistent Multi-scene Video Generation via LLM-Guided Planning - b31b1b.svg)](https://arxiv.org/abs/2309.15091) | [](https://github.com/HL-hanlin/VideoDirectorGPT) | [](https://videodirectorgpt.github.io/) | Sep., 2023 |
- Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation - b31b1b.svg)](https://arxiv.org/abs/2309.15818) | [](https://github.com//showlab/Show-1) | [](https://showlab.github.io/Show-1/) | Sep., 2023 |
- LaVie: High-Quality Video Generation with Cascaded Latent Diffusion Models - b31b1b.svg)](https://arxiv.org/abs/2309.15103) | [](https://github.com/Vchitect/LaVie) | [](https://vchitect.github.io/LaVie-project/) | Sep., 2023 |
- Reuse and Diffuse: Iterative Denoising for Text-to-Video Generation - b31b1b.svg)](https://arxiv.org/abs/2309.03549) | [](https://github.com/anonymous0x233/ReuseAndDiffuse) | [](https://anonymous0x233.github.io/ReuseAndDiffuse/) | Sep., 2023 |
- VideoGen: A Reference-Guided Latent Diffusion Approach for High Definition Text-to-Video Generation - b31b1b.svg)](https://arxiv.org/abs/2309.00398) | - | [](https://videogen.github.io/VideoGen/) | Sep., 2023 |
- Text2Performer: Text-Driven Human Video Generation - b31b1b.svg)](https://arxiv.org/abs/2304.08483) | [](https://github.com/yumingj/Text2Performer) | [](https://yumingj.github.io/projects/Text2Performer) | Apr., 2023 |
- AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning - b31b1b.svg)](https://arxiv.org/abs/2307.04725) | [](https://github.com/guoyww/animatediff/) | [](https://animatediff.github.io/) | Jul., 2023 |
- Dysen-VDM: Empowering Dynamics-aware Text-to-Video Diffusion with Large Language Models - b31b1b.svg)](https://arxiv.org/abs/2308.13812) | - | [](https://haofei.vip/Dysen-VDM/) | Aug., 2023 |
- SimDA: Simple Diffusion Adapter for Efficient Video Generation - b31b1b.svg)](https://arxiv.org/abs/2308.09710) | [](https://github.com/ChenHsing/SimDA) | [](https://chenhsing.github.io/SimDA/) | CVPR, 2024 |
- Dual-Stream Diffusion Net for Text-to-Video Generation - b31b1b.svg)](https://arxiv.org/abs/2308.08316) | - | - | Aug., 2023 |
- ModelScope Text-to-Video Technical Report - b31b1b.svg)](https://arxiv.org/abs/2308.06571) | [](https://github.com/modelscope/modelscope) | [](https://modelscope.cn/models/damo/text-to-video-synthesis/summary) | Aug., 2023 |
- Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models - b31b1b.svg)](https://arxiv.org/abs/2305.10474) | - | [](https://research.nvidia.com/labs/dir/pyoco/) | May, 2023 |
- Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models - b31b1b.svg)](https://arxiv.org/abs/2304.08818) | - | [](https://research.nvidia.com/labs/toronto-ai/VideoLDM/) | - | CVPR 2023 |
- Latent-Shift: Latent Diffusion with Temporal Shift - b31b1b.svg)](https://arxiv.org/abs/2304.08477) | - | [](https://latent-shift.github.io/) | - | Apr., 2023 |
- Probabilistic Adaptation of Text-to-Video Models - b31b1b.svg)](https://arxiv.org/abs/2306.01872) | - | [](https://video-adapter.github.io/video-adapter/) | Jun., 2023 |
- NUWA-XL: Diffusion over Diffusion for eXtremely Long Video Generation - b31b1b.svg)](https://arxiv.org/abs/2303.12346) | - | [](https://msra-nuwa.azurewebsites.net/#/NUWAXL) | Mar., 2023 |
- ED-T2V: An Efficient Training Framework for Diffusion-based Text-to-Video Generation - | - | - | IJCNN, 2023 |
- MagicVideo: Efficient Video Generation With Latent Diffusion Models - b31b1b.svg)](https://arxiv.org/abs/2211.11018) | - | [](https://magicvideo.github.io/#) | - | Nov., 2022 |
- Imagen Video: High Definition Video Generation With Diffusion Models - b31b1b.svg)](https://arxiv.org/abs/2210.02303) | - | [](https://imagen.research.google/video/) | - | Oct., 2022 |
- VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation - b31b1b.svg)](https://arxiv.org/abs/2303.08320) | [](https://github.com/modelscope/modelscope) | [](https://modelscope.cn/models/damo/text-to-video-synthesis/summary) | - | CVPR 2023 |
- Make-A-Video: Text-to-Video Generation without Text-Video Data - b31b1b.svg)](https://openreview.net/forum?id=nJfylDvgzlq) | - | [](https://makeavideo.studio) | - | ICLR 2023 |
- Latent Video Diffusion Models for High-Fidelity Video Generation With Arbitrary Lengths - b31b1b.svg)](https://arxiv.org/abs/2211.13221) | [](https://github.com/YingqingHe/LVDM) | [](https://yingqinghe.github.io/LVDM/) | Nov., 2022 |
- Video Diffusion Models - b31b1b.svg)](https://arxiv.org/abs/2204.03458) | - | [](https://video-diffusion.github.io/) | - | Apr., 2022 |
- Lumiere: A Space-Time Diffusion Model for Video Generation - b31b1b.svg)](https://arxiv.org/abs/2401.12945) |-|[](https://lumiere-video.github.io/) | Jan, 2024
- Snap Video: Scaled Spatiotemporal Transformers for Text-to-Video Synthesis - b31b1b.svg)](https://arxiv.org/abs/2402.14797) |-|[](https://snap-research.github.io/snapvideo/) | Feb., 2024
- Genie: Generative Interactive Environments - b31b1b.svg)](https://arxiv.org/abs/2402.15391) |-|[](https://sites.google.com/view/genie-2024/home) | Feb., 2024
- MobileVidFactory: Automatic Diffusion-Based Social Media Video Generation for Mobile Devices from Text - b31b1b.svg)](https://arxiv.org/abs/2307.16371) | - | - | Jul., 2023 |
- Phenaki: Variable Length Video Generation From Open Domain Textual Description - b31b1b.svg)](https://arxiv.org/abs/2210.02399) | - | [](https://phenaki.video/) | - | Oct., 2022 |
- MAGVIT: Masked Generative Video Transformer - b31b1b.svg)](https://arxiv.org/abs/2212.05199) | - | [](https://magvit.cs.cmu.edu/) | Dec., 2022 |
- Make-A-Video: Text-to-Video Generation without Text-Video Data - b31b1b.svg)](https://arxiv.org/abs/2209.14792) | - | [](https://makeavideo.studio) | - | ICLR 2023 |
- CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers - b31b1b.svg)](https://arxiv.org/abs/2205.15868) | [](https://github.com/THUDM/CogVideo) | - | May, 2022 |
- Mora: Enabling Generalist Video Generation via A Multi-Agent Framework - b31b1b.svg)](https://arxiv.org/abs/2403.13248) |-|- | Mar., 2024
- VSTAR: Generative Temporal Nursing for Longer Dynamic Video Synthesis - b31b1b.svg)](https://arxiv.org/abs/2403.13501) |-|- | Mar., 2024
- MagicTime: Time-lapse Video Generation Models as Metamorphic Simulators - b31b1b.svg)](https://arxiv.org/abs/2404.05014) |[](https://github.com/PKU-YuanGroup/MagicTime)|[](https://pku-yuangroup.github.io/MagicTime/) | Apr., 2024
- Grid Diffusion Models for Text-to-Video Generation - b31b1b.svg)](https://arxiv.org/abs/2404.00234) |[](https://github.com/taegyeong-lee/Grid-Diffusion-Models-for-Text-to-Video-Generation)|[](https://taegyeong-lee.github.io/text2video) | CVPR, 2024
- Movie Gen - b31b1b.svg)](https://ai.meta.com/static-resource/movie-gen-research-paper) |-|[](https://ai.meta.com/research/movie-gen/) | Oct, 2024
- CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer - b31b1b.svg)](https://arxiv.org/abs/2408.06072) |[](https://github.com/THUDM/CogVideo)|-| Oct, 2024
- Enhancing Motion in Text-to-Video Generation with Decomposed Encoding and Conditioning - b31b1b.svg)](https://arxiv.org/abs/2410.24219) |[](https://github.com/PR-Ryan/DEMO)|[](https://pr-ryan.github.io/DEMO-project/)| NeurIPS 2024
- Helios: Real Real-Time Long Video Generation Model - b31b1b.svg)](https://arxiv.org/abs/2603.04379) |[](https://github.com/PKU-YuanGroup/Helios)|[](https://pku-yuangroup.github.io/Helios-Page/) | Arxiv, 2026
-
Training-free
- TRAILBLAZER: TRAJECTORY CONTROL FOR DIFFUSION-BASED VIDEO GENERATION - b31b1b.svg)](https://arxiv.org/abs/2401.00896) | [](https://github.com/hohonu-vicml/Trailblazer) | [](https://hohonu-vicml.github.io/Trailblazer.Page/) | Jan, 2024 |
- FreeInit: Bridging Initialization Gap in Video Diffusion Models - b31b1b.svg)](https://arxiv.org/abs/2312.07537) | [](https://github.com/TianxingWu/FreeInit) | [](https://tianxingwu.github.io/pages/FreeInit/) | Dec, 2023 |
- MTVG : Multi-text Video Generation with Text-to-Video Models - b31b1b.svg)](https://arxiv.org/abs/2312.04086) | - | [](https://kuai-lab.github.io/mtvg-page/) | Dec, 2023 |
- F3-Pruning: A Training-Free and Generalized Pruning Strategy towards Faster and Finer Text-to-Video Synthesis - b31b1b.svg)](https://arxiv.org/abs/2312.03459) | - | - | Nov, 2023 |
- AdaDiff: Adaptive Step Selection for Fast Diffusion - b31b1b.svg)](https://arxiv.org/abs/2311.14768) | - | - | Nov, 2023 |
- FlowZero: Zero-Shot Text-to-Video Synthesis with LLM-Driven Dynamic Scene Syntax - b31b1b.svg)](https://arxiv.org/abs/2311.15813) | [](https://github.com/aniki-ly/FlowZero) | [](https://flowzero-video.github.io/) | Nov, 2023 |
- 🏀GPT4Motion: Scripting Physical Motions in Text-to-Video Generation via Blender-Oriented GPT Planning - b31b1b.svg)](https://arxiv.org/abs/2311.12631) | [](https://github.com/jiaxilv/GPT4Motion) | [](https://gpt4motion.github.io/) | Nov, 2023 |
- FreeNoise: Tuning-Free Longer Video Diffusion Via Noise Rescheduling - b31b1b.svg)](https://arxiv.org/abs/2310.15169) | [](https://github.com/arthur-qiu/LongerCrafter) | [](http://haonanqiu.com/projects/FreeNoise.html) | Oct, 2023 |
- ConditionVideo: Training-Free Condition-Guided Text-to-Video Generation - b31b1b.svg)](https://arxiv.org/abs/2310.07697) | [](https://github.com/pengbo807/ConditionVideo) | [](https://pengbo807.github.io/conditionvideo-website/) | Oct, 2023 |
- LLM-grounded Video Diffusion Models - b31b1b.svg)](https://arxiv.org/abs/2309.17444) | [](https://github.com/TonyLianLong/LLM-groundedVideoDiffusion) | [](https://llm-grounded-video-diffusion.github.io/) | Oct, 2023 |
- Free-Bloom: Zero-Shot Text-to-Video Generator with LLM Director and LDM Animator - b31b1b.svg)](https://arxiv.org/abs/2309.14494) | [](https://github.com/SooLab/Free-Bloom) | - | NeurIPS, 2023 |
- DiffSynth: Latent In-Iteration Deflickering for Realistic Video Synthesis - b31b1b.svg)](https://arxiv.org/abs/2308.03463) | [](https://github.com/alibaba/EasyNLP/tree/master/diffusion/DiffSynth) | [](https://anonymous456852.github.io/) | Aug, 2023 |
- Large Language Models are Frame-level Directors for Zero-shot Text-to-Video Generation - b31b1b.svg)](https://arxiv.org/abs/2305.14330) | [](https://github.com/KU-CVLAB/DirecT2V) | - | May, 2023 |
- Text2video-Zero: Text-to-Image Diffusion Models Are Zero-Shot Video Generators - b31b1b.svg)](https://arxiv.org/abs/2303.13439) | [](https://github.com/Picsart-AI-Research/Text2Video-Zero) | [](https://text2video-zero.github.io/) | Mar., 2023 |
- VideoElevator: Elevating Video Generation Quality with Versatile Text-to-Image Diffusion Models - b31b1b.svg)](https://arxiv.org/abs/2403.05438) | [](https://github.com/YBYBZhang/VideoElevator) | [](https://videoelevator.github.io/) | Mar, 2024 |
- InfLVG: Reinforce Inference-Time Consistent Long Video Generation with GRPO - b31b1b.svg)](https://arxiv.org/abs/2505.17574) | [](https://github.com/MAPLE-AIGC/InfLVG) | - | May, 2025 |
- A²RD: Agentic Autoregressive Diffusion for Long Video Consistency - b31b1b.svg)](https://arxiv.org/abs/2605.06924) | - | [](https://dxlong2000.github.io/AARD/) | May, 2026 |
- VISTA: A Test-Time Self-Improving Video Generation Agent - b31b1b.svg)](https://arxiv.org/abs/2510.15831) | - | [](https://g-vista.github.io/) | CVPR, 2026 |
-
-
Uncategorized
-
Unconditional Video Generation
-
Transformer based
- Latte: Latent Diffusion Transformer for Video Generation - b31b1b.svg)](https://arxiv.org/abs/2401.03048) | [](https://github.com/Vchitect/Latte) | [](https://maxin-cn.github.io/latte_project/) | Jan., 2024 |
- VDT: An Empirical Study on Video Diffusion with Transformers - b31b1b.svg)](https://arxiv.org/abs/2305.13311) | [](https://github.com/RERV/VDT) | - | May, 2023 |
- Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformer - b31b1b.svg)](https://arxiv.org/abs/2204.03638) | [](https://github.com/SongweiGe/TATS) | [](https://songweige.github.io/projects/tats/index.html) | May, 2023 |
- Redefining Temporal Modeling in Video Diffusion: The Vectorized Timestep Approach - b31b1b.svg)](https://arxiv.org/pdf/2410.03160) | [](https://github.com/Yaofang-Liu/FVDM) |- | Oct., 2024 |
- FEAT: Full-Dimensional Efficient Attention Transformer for Medical Video Generation - b31b1b.svg)](https://arxiv.org/pdf/2506.04956) | [](https://github.com/Yaziwel/FEAT) |- | Jun., 2025 |
-
U-Net based
- Video Probabilistic Diffusion Models in Projected Latent Space - b31b1b.svg)](https://arxiv.org/abs/2302.07685) | [](https://github.com/sihyun-yu/PVDM) | [](https://sihyun.me/PVDM/) | CVPR 2023 |
- VIDM: Video Implicit Diffusion Models - b31b1b.svg)](https://arxiv.org/abs/2212.00235) | [](https://github.com/MKFMIKU/VIDM) | [](https://kfmei.page/vidm/) | AAAI 2023 |
- GD-VDM: Generated Depth for better Diffusion-based Video Generation - b31b1b.svg)](https://arxiv.org/abs/2306.11173) | [](https://github.com/lapid92/GD-VDM) | - | Jun., 2023 |
- LEO: Generative Latent Image Animator for Human Video Synthesis - b31b1b.svg)](https://arxiv.org/pdf/2305.03989.pdf) | [](https://github.com/wyhsirius/LEO) | [](https://wyhsirius.github.io/LEO-project/) | May., 2023 |
- Hybrid Video Diffusion Models with 2D Triplane and 3D Wavelet Representation - b31b1b.svg)](https://arxiv.org/abs/2402.13729) | [](https://github.com/hxngiee/HVDM) | [](https://hxngiee.github.io/HVDM/) | Feb. 2024 |
- LEO: Generative Latent Image Animator for Human Video Synthesis - b31b1b.svg)](https://arxiv.org/pdf/2305.03989.pdf) | [](https://github.com/wyhsirius/LEO) | [](https://wyhsirius.github.io/LEO-project/) | May., 2023 |
-
-
Video Completion
-
Video Enhancement and Restoration
- Towards Language-Driven Video Inpainting via Multimodal Large Language Models - b31b1b.svg)](https://arxiv.org/abs/2401.10226) | [](https://github.com/jianzongwu/Language-Driven-Video-Inpainting) | [](https://jianzongwu.github.io/projects/rovi/) | Jan., 2024 |
- Inflation with Diffusion: Efficient Temporal Adaptation for Text-to-Video Super-Resolution - | - | - | WACW, 2023 |
- Upscale-A-Video: Temporal-Consistent Diffusion Model for Real-World Video Super-Resolution - b31b1b.svg)](https://arxiv.org/abs/2312.06640) | [](https://github.com/sczhou/Upscale-A-Video) | [](https://shangchenzhou.com/projects/upscale-a-video/) | Dec., 2023 |
- AVID: Any-Length Video Inpainting with Diffusion Model - b31b1b.svg)](https://arxiv.org/abs/2312.03816) | [](https://github.com/zhang-zx/AVID) | [](https://zhang-zx.github.io/AVID/) | Dec., 2023 |
- Motion-Guided Latent Diffusion for Temporally Consistent Real-world Video Super-resolution - b31b1b.svg)](https://arxiv.org/abs/2312.00853) | [](https://github.com/IanYeung/MGLD-VSR) | -| CVPR 2023 |
- LDMVFI: Video Frame Interpolation with Latent Diffusion Models - b31b1b.svg)](https://arxiv.org/abs/2303.09508) | - |- | Mar., 2023 |
- CaDM: Codec-aware Diffusion Modeling for Neural-enhanced Video Streaming - b31b1b.svg)](https://arxiv.org/abs/2211.08428) | - | - | Nov., 2022 |
- Look Ma, No Hands! Agent-Environment Factorization of Egocentric Videos - b31b1b.svg)](https://arxiv.org/pdf/2305.16301.pdf) | - | - | May., 2023 |
- Look Ma, No Hands! Agent-Environment Factorization of Egocentric Videos - b31b1b.svg)](https://arxiv.org/pdf/2305.16301.pdf) | - | - | May., 2023 |
-
Video Prediction
- STDiff: Spatio-temporal Diffusion for Continuous Stochastic Video Prediction - b31b1b.svg)](https://arxiv.org/abs/2312.06486) | [](https://github.com/XiYe20/STDiffProject) | - | Dec, 2023 |
- Video Diffusion Models with Local-Global Context Guidance - b31b1b.svg)](https://arxiv.org/pdf/2306.02562.pdf) | [](https://github.com/exisas/LGC-VD) | - | IJCAI, 2023 |
- Seer: Language Instructed Video Prediction with Latent Diffusion Models - b31b1b.svg)](https://arxiv.org/abs/2303.14897) | - | [](https://seervideodiffusion.github.io/) | Mar., 2023 |
- Diffusion Models for Video Prediction and Infilling - b31b1b.svg)](https://arxiv.org/abs/2206.07696) | [](https://github.com/Tobi-r9/RaMViD) | [](https://sites.google.com/view/video-diffusion-prediction) | TMLR 2022 |
- McVd: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation - b31b1b.svg)](https://arxiv.org/abs/2205.09853) | [](https://github.com/voletiv/mcvd-pytorch) | [](https://mask-cond-video-diffusion.github.io) | NeurIPS 2022 |
- Diffusion Probabilistic Modeling for Video Generation - b31b1b.svg)](https://arxiv.org/abs/2203.09481) | [](https://github.com/buggyyang/RVD) | - | Mar., 2022 |
- Flexible Diffusion Modeling of Long Videos - b31b1b.svg)](https://arxiv.org/abs/2205.11495) | [](https://github.com/plai-group/flexible-video-diffusion-modeling) | [](https://fdmolv.github.io/) | May, 2022 |
- Control-A-Video: Controllable Text-to-Video Generation with Diffusion Models - b31b1b.svg)](https://arxiv.org/abs/2305.13840) | [](https://github.com/Weifeng-Chen/control-a-video) | [](https://controlavideo.github.io/) | May, 2023 |
- MaskViT: Masked Visual Pre-Training for Video Prediction - b31b1b.svg)](https://arxiv.org/abs/2206.11894) | [](https://github.com/agrimgupta92/maskvit) | [](https://maskedvit.github.io/) | Jun, 2022 |
- AID: Adapting Image2Video Diffusion Models for Instruction-guided Video Prediction - b31b1b.svg)](https://arxiv.org/abs/2406.06465) | [](https://github.com/Chenhsing/AID) | [](https://chenhsing.github.io/AID) | Jun, 2024 |
- Video Diffusion Models with Local-Global Context Guidance - b31b1b.svg)](https://arxiv.org/pdf/2306.02562.pdf) | [](https://github.com/exisas/LGC-VD) | - | IJCAI, 2023 |
- SyncVP: Joint Diffusion for Synchronous Multi-Modal Video Prediction - b31b1b.svg)](https://arxiv.org/abs/2503.18933) | [](https://github.com/PallottaEnrico/SyncVP) | [](https://syncvp.github.io/) | CVPR, 2025 |
-
-
Video Editing
-
Domain-specific Editing Model
- TRAINING-FREE SEMANTIC VIDEO COMPOSITION VIA PRE-TRAINED DIFFUSION MODEL - b31b1b.svg)](https://arxiv.org/abs/2401.09195) | - | - | Jan, 2024 |
- Generative Rendering: Controllable 4D-Guided Video Generation with 2D Diffusion Models - b31b1b.svg)](https://arxiv.org/abs/2312.01409) | - | [](https://primecai.github.io/generative_rendering) | CVPR 2023 |
- Multimodal-driven Talking Face Generation via a Unified Diffusion-based Generator - b31b1b.svg)](https://arxiv.org/pdf/2305.02594.pdf) | - | - | May, 2023 |
- DiffSynth: Latent In-Iteration Deflickering for Realistic Video Synthesis - b31b1b.svg)](https://arxiv.org/pdf/2308.03463.pdf) | - | - | Aug, 2023 |
- Style-A-Video: Agile Diffusion for Arbitrary Text-based Video Style Transfer - b31b1b.svg)](https://arxiv.org/abs/2305.05464) | [](https://github.com/haha-lisa/Style-A-Video) | - | May, 2023 |
- Instruct-Video2Avatar: Video-to-Avatar Generation with Instructions - b31b1b.svg)](https://arxiv.org/abs/2306.02903) | [](https://github.com/lsx0101/Instruct-Video2Avatar) | - | Jun, 2023 |
- Video Colorization with Pre-trained Text-to-Image Diffusion Models - b31b1b.svg)](https://arxiv.org/abs/2306.01732) | [](https://github.com/ColorDiffuser/ColorDiffuser) | [](https://colordiffuser.github.io/) | Jun, 2023 |
- Diffusion Video Autoencoders: Toward Temporally Consistent Face Video Editing via Disentangled Video Encoding - b31b1b.svg)](https://arxiv.org/abs/2212.02802) | [](https://github.com/man805/Diffusion-Video-Autoencoders) | [](https://diff-video-ae.github.io/) | CVPR 2023 |
- Diffutoon: High-Resolution Editable Toon Shading via Diffusion Models - b31b1b.svg)](https://arxiv.org/abs/2401.16224) | - | [](https://ecnu-cilab.github.io/DiffutoonProjectPage/) | Jan. 2024 |
-
Programming Languages
Categories
Sub Categories
Training-based
68
Video Understanding
36
Training-free Editing Model
30
Multi-modal guided Video Generation
21
Image-guided Video Generation
20
Training-free
18
One-shot Editing Model
17
Pose-guided Video Generation
15
General Editing Model
14
Video Prediction
12
Motion-guided Video Generation
12
Domain-specific Editing Model
12
Metric and BenchMark
12
Instruct-guided Video Editing
10
Video Enhancement and Restoration
9
Multi-modal Control Editing Model
8
U-Net based
6
Caption-level
6
Motion-guided Video Editing
6
Sound-guided Video Generation
6
Transformer based
5
Brain-guided Video Generation
5
Category-level
3
Non-diffusion Editing model
3
Sound-guided Video Editing
3
Uncategorized
2