Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
Awesome-Video-Diffusion
A curated list of recent diffusion models for video generation, editing, restoration, understanding, etc.
https://github.com/showlab/Awesome-Video-Diffusion
Last synced: 1 day ago
JSON representation
-
Table of Contents <!-- omit in toc -->
-
Video Generation
- 
- 
- ![Star - AI/generative-models)
- ![Star - to-video-synthesis-colab)
- ModelScope (Text-to-video synthesis)
- ![Star
- Diffusers (Text-to-video synthesis)
- ![Star
- Open-Sora-Plan
- Open-Sora
- Stable Video Diffusion
- Show-1
- text-to-video-synthesis-colab
- ![Star
- ![Star - YuanGroup/Open-Sora-Plan)
- ![Star - Sora)
- VideoTuna
- ![Star
- VideoCrafter: A Toolkit for Text-to-Video Generation and Editing
- ![Star - ai/Allegro)
- ![Star
- Movie Gen: A Cast of Media Foundation Models
- Hotshot-XL (text-to-GIF)
- ![Star - XL)
- zeroscope_v2
- Mochi 1
- ![Star
- Pyramidal Flow Matching for Efficient Video Generative Modeling
- ![Star - Flow)
- HunyuanVideo: A Systematic Framework For Large Video Generative Models
- ![Star
-
Evaluation Benchmarks and Metrics
- T2VScore: Towards A Better Metric for Text-to-Video Generation
- VBench: Comprehensive Benchmark Suite for Video Generative Models
- EvalCrafter: Benchmarking and Evaluating Large Video Generation Models
- Frechet Video Motion Distance: A Metric for Evaluating Motion Consistency in Videos
- ![Star - Lab/FVMD-frechet-video-motion-distance)
- T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generation
- ![Star - CompBench)
- FETV: A Benchmark for Fine-Grained Evaluation of Open-Domain Text-to-Video Generation
- ![Star
- ![Star
- StoryBench: A Multifaceted Benchmark for Continuous Story Visualization
- ![Star
- ![Star
- ![Star
- ![Star - research/Panda-70M)
- ReLight My NeRF: A Dataset for Novel View Synthesis and Relighting of Real World Objects
- ![Star - ai/rene)
- ChronoMagic-Bench: A Benchmark for Metamorphic Evaluation of Text-to-Time-lapse Video Generation
- ![Star - YuanGroup/ChronoMagic-Bench)
- VidProM: A Million-scale Real Prompt-Gallery Dataset for Text-to-Video Diffusion Models
- ![Star
- Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers
- ![Star - research/Panda-70M)
- Evaluation of Text-to-Video Generation Models: A Dynamics Perspective
- ![Star
-
New Video Generation Benchmark and Metrics
-
Video Editing
- Object-Centric Diffusion for Efficient Video Editing
- VASE: Object-Centric Shape and Appearance Manipulation of Real Videos
- ![Star
- FlowVid: Taming Imperfect Optical Flows for Consistent Video-to-Video Synthesis
- ![Star - LiangF/FlowVid)
- Fairy: Fast Parallelized Instruction-Guided Video-to-Video Synthesis
- RealCraft: Attention Control as A Solution for Zero-shot Long Video Editing
- VidToMe: Video Token Merging for Zero-Shot Video Editing
- ![Star
- A Video is Worth 256 Bases: Spatial-Temporal Expectation-Maximization Inversion for Zero-Shot Video Editing
- ![Star - Inv/stem-inv)
- Neutral Editing Framework for Diffusion-based Video Editing
- DiffusionAtlas: High-Fidelity Consistent Diffusion Video Editing
- RAVE: Randomized Noise Shuffling for Fast and Consistent Video Editing with Diffusion Models
- ![Star - lab/RAVE)
- SAVE: Protagonist Diversification with Structure Agnostic Video Editing
- MagicStick: Controllable Video Editing via Control Handle Transformations
- ![Star
- VideoSwap: Customized Video Subject Swapping with Interactive Semantic Point Correspondence
- ![Star
- DragVideo: Interactive Drag-style Video Editing
- ![Star - Official)
- Drag-A-Video: Non-rigid Video Editing with Point-based Interaction
- Motion-Conditioned Image Animation for Video Editing
- Space-Time Diffusion Features for Zero-Shot Text-Driven Motion Transfer
- Cut-and-Paste: Subject-Driven Video Editing with Attention Control
- LatentWarp: Consistent Diffusion Latents for Zero-Shot Video-to-Video Translation
- Fuse Your Latents: Video Editing with Multi-source Latent Diffusion Models
- DynVideo-E: Harnessing Dynamic NeRF for Large-Scale Motion- and View-Change Human-Centric Video Editing
- Ground-A-Video: Zero-shot Grounded Video Editing using Text-to-image Diffusion Models
- BIVDiff: A Training-Free Framework for General-Purpose Video Synthesis via Bridging Image and Video Diffusion Models
- ![Star - Motion-Customization)
- MotionEditor: Editing Video Motion via Content-Aware Diffusion
- ![Star - Rings/MotionEditor)
- ![Star - A-Video/Ground-A-Video)
- CCEdit: Creative and Controllable Video Editing via Diffusion Models
- MagicProp: Diffusion-based Video Editing via Motion-aware Appearance Propagation
- MagicEdit: High-Fidelity and Temporally Coherent Video Editing
- ![Star - research/magic-edit)
- StableVideo: Text-driven Consistency-aware Diffusion Video Editing
- ![Star
- CoDeF: Content Deformation Fields for Temporally Consistent Video Processing
- ![Star
- TokenFlow: Consistent Diffusion Features for Consistent Video Editing
- ![Star
- INVE: Interactive Neural Video Editing
- VidEdit: Zero-Shot and Spatially Aware Text-Driven Video Editing
- Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translation
- ![Star
- ControlVideo: Adding Conditional Control for One Shot Text-to-Video Editing
- ![Star - ml/controlvideo)
- Make-A-Protagonist: Generic Video Editing with An Ensemble of Experts
- ![Star - A-Protagonist/Make-A-Protagonist)
- Soundini: Sound-Guided Diffusion for Natural Video Editing
- ![Star - lab/soundini-official)
- ![Star - zero)
- FateZero: Fusing Attentions for Zero-shot Text-based Video Editing
- ![Star
- Pix2video: Video Editing Using Image Diffusion
- Video-P2P: Video Editing with Cross-attention Control
- ![Star - P2P)
- Dreamix: Video Diffusion Models Are General Video Editors
- Shape-Aware Text-Driven Layered Video Editing
- Speech Driven Video Editing via an Audio-Conditioned Diffusion Model
- ![Star
- Diffusion Video Autoencoders: Toward Temporally Consistent Face Video Editing via Disentangled Video Encoding
- ![Star - Video-Autoencoders)
- EVA: Zero-shot Accurate Attributes and Multi-Object Video Editing
- ![Star
- DreamMotion: Space-Time Self-Similarity Score Distillation for Zero-Shot Video Editing
- MaskINT: Video Editing via Interpolative Non-autoregressive Masked Transformers
- FLATTEN: optical FLow-guided ATTENtion for consistent text-to-video editing
- ![Star
- Spectral Motion Alignment for Video Motion Transfer using Diffusion Models
- Edit-A-Video: Single Video Editing with Object-Aware Consistency
- UniEdit: A Unified Tuning-Free Framework for Video Motion and Appearance Editing
- ![Star
- ![Star - LiangF/streamv2v)
- ![Star
- ![Star - AI-Lab/AnyV2V)
- Video Editing via Factorized Diffusion Distillation
- FastVideoEdit: Leveraging Consistency Models for Efficient Text-to-Video Editing
- CoCoCo: Improving Text-Guided Video Inpainting for Better Consistency, Controllability and Compatibility
- Unified Editing of Panorama, 3D Scenes, and Videos Through Disentangled Self-Attention Injection
- I2VEdit: First-Frame-Guided Video Editing via Image-to-Video Diffusion Models
- Looking Backward: Streaming Video-to-Video Translation with Feature Banks
- Slicedit: Zero-Shot Video Editing With Text-to-Image Diffusion Models Using Spatio-Temporal Slices
- ![Star - LiangF/streamv2v)
- ReVideo: Remake a Video with Motion and Content Control
- ![Star - E/ReVideo)
- ![Star
- ViViD: Video Virtual Try-on using Diffusion Models
- ![Star
- Edit-Your-Motion: Space-Time Diffusion Decoupling Learning for Video Motion Editing
- GenVideo: One-shot target-image and shape aware video editing using T2I diffusion models
- AnyV2V: A Tuning-Free Framework For Any Video-to-Video Editing Tasks
- ![Star - AI-Lab/AnyV2V)
- VMC: Video Motion Customization using Temporal Attention Adaption for Text-to-Video Diffusion Models
- MotionFlow: Attention-Driven Motion Transfer in Video Diffusion Models
- DIVE: Taming DINO for Subject-Driven Video Editing
- AniGS: Animatable Gaussian Avatar from a Single Image with Inconsistent Gaussian Reconstruction
- ![Star
- StableV2V: Stablizing Shape Consistency in Video-to-Video Editing
- ![Star
- ![Star
-
Long-form Video Generation and Completion
- ![Star - pytorch)
- NUWA-XL: Diffusion over Diffusion for eXtremely Long Video Generation
- Flexible Diffusion Modeling of Long Videos
- ![Star - group/flexible-video-diffusion-modeling)
- Streetscapes: Large-scale Consistent Street View Generation Using Autoregressive Video Diffusion
- MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation
- Redefining Temporal Modeling in Video Diffusion: The Vectorized Timestep Approach
- ![Star - Liu/FVDM)
- Long Video Diffusion Generation with Segmented Cross-Attention and Content-Rich Video Data Curation
-
Human or Subject Motion
- Avatars Grow Legs: Generating Smooth Human Motion from Sparse Tracking Inputs with Diffusion Model
- ![Star
- InterGen: Diffusion-based Multi-human Motion Generation under Complex Interactions
- ![Star
- ReMoDiffuse: Retrieval-Augmented Motion Diffusion Model
- ![Star - zhang/ReMoDiffuse)
- Human Motion Diffusion as a Generative Prior
- ![Star
- Can We Use Diffusion Probabilistic Models for 3d Motion Prediction?
- ![Star - ahn/diffusion-motion-prediction)
- Single Motion Diffusion
- ![Star
- HumanMAC: Masked Motion Completion for Human Motion Prediction
- ![Star
- DiffMotion: Speech-Driven Gesture Synthesis Using Denoising Diffusion Model
- Modiff: Action-Conditioned 3d Motion Generation With Denoising Diffusion Probabilistic Models
- PhysDiff: Physics-Guided Human Motion Diffusion Model
- BeLFusion: Latent Diffusion for Behavior-Driven Human Motion Prediction
- ![Star
- Unifying Human Motion Synthesis and Style Transfer With Denoising Diffusion Probabilistic Models
- ![Star
- Executing Your Commands via Motion Diffusion in Latent Space
- ![Star - latent-diffusion)
- Pretrained Diffusion Models for Unified Human Motion Synthesis
- Diffusion Motion: Generate Text-Guided 3d Human Motion by Diffusion Model
- Human Joint Kinematics Diffusion-Refinement for Stochastic Motion Prediction
- Human Motion Diffusion Model
- ![Star - diffusion-model)
- FLAME: Free-form Language-based Motion Synthesis & Editing
- ![Star
- MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model
- ![Star - zhang/MotionDiffuse)
- Stochastic Trajectory Prediction via Motion Indeterminacy Diffusion
- ![Star
- EchoMimic: Lifelike Audio-Driven Portrait Animations through Editable Landmark Conditions
- ![Star
- KMM: Key Frame Mask Mamba for Extended Motion Generation
- ![Star - zeyu-zhang/KMM)
- DanceFusion: A Spatio-Temporal Skeleton Diffusion Transformer for Audio-Driven Dance Motion Reconstruction
- DiffSHEG: A Diffusion-Based Approach for Real-Time Speech-driven Holistic 3D Expression and Gesture Generation
- ![Star
- ![Star
- Enhancing Motion in Text-to-Video Generation with Decomposed Encoding and Conditioning
- ![Star - Ryan/DEMO)
- AnchorCrafter: Animate CyberAnchors Saling Your Products via Human-Object Interacting Video Generation
- ![Star
- A Comprehensive Survey on Human Video Generation: Challenges, Methods, and Insights
-
Talking Head Generation
- Listen, Denoise, Action! Audio-Driven Motion Synthesis With Diffusion Models
- DAWN: Dynamic Frame Avatar with Non-autoregressive Diffusion Framework for Talking Head Video Generation
- ![Star - Cheng/DAWN-pytorch)
- Loopy: Taming Audio-Driven Portrait Avatar with Long-Term Motion Dependency
- HelloMeme: Integrating Spatial Knitting Attentions to Embed High-Level and Fidelity-Rich Conditions in Diffusion Models
- ![Star
- GestureDiffuCLIP: Gesture Diffusion Model with CLIP Latents
- From Audio to Photoreal Embodiment: Synthesizing Humans in Conversations
- Takin-ADA: Emotion Controllable Audio-Driven Animation with Canonical and Landmark Loss Optimization
- PersonaTalk: Bring Attention to Your Persona in Visual Dubbing
- Talking With Hands 16.2M: A Large-Scale Dataset of Synchronized Body-Finger Motion and Audio for Conversational Motion Analysis and Synthesis
- Learning to Listen: Modeling Non-Deterministic Dyadic Facial Motion
- ![Star
- TANGO: Co-Speech Gesture Video Reenactment with Hierarchical Audio Motion Embedding and Diffusion Interpolation
- ![Star - generative-vision/hallo)
- Hallo2: Long-Duration and High-Resolution Audio-driven Portrait Image Animation
- ![Star - generative-vision/hallo2)
- ![Star
- MimicTalk: Mimicking a personalized and expressive 3D talking face in few minutes
- IF-MDM: Implicit Face Motion Diffusion Model for High-Fidelity Realtime Talking Head Generation
- INFP: Audio-Driven Interactive Head Generation in Dyadic Conversations
- MEMO: Memory-Guided Diffusion for Expressive Talking Video Generation
- ![Star
- SINGER: Vivid Audio-driven Singing Video Generation with Multi-scale Spectral Diffusion Model
- Synergizing Motion and Appearance: Multi-Scale Compensatory Codebooks for Talking Head Video Generation
- ![Star - motion-appearance)
- Hallo3: Highly Dynamic and Realistic Portrait Image Animation with Diffusion Transformer Networks
- FLOAT: Generative Motion Latent Flow Matching for Audio-driven Talking Portrait
- EmotiveTalk: Expressive Talking Head Generation through Audio Information Decoupling and Emotional Video Diffusion
- LetsTalk: Latent Diffusion Transformer for Talking Video Synthesis
- Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animation
-
3D / NeRF
- RoomDreamer: Text-Driven 3D Indoor Scene Synthesis with Coherent Geometry and Texture
- Text2NeRF: Text-Driven 3D Scene Generation with Neural Radiance Fields
- NeuralField-LDM: Scene Generation with Hierarchical Latent Diffusion Models
- Single-Stage Diffusion NeRF: A Unified Approach to 3D Generation and Reconstruction
- ![Star
- Instruct-NeRF2NeRF: Editing 3D Scenes with Instructions
- ![Star - nerf2nerf)
- DiffusioNeRF: Regularizing Neural Radiance Fields with Denoising Diffusion Models
- ![Star
- NerfDiff: Single-image View Synthesis with NeRF-guided Distillation from 3D-aware Diffusion
- SV4D: Dynamic 3D Content Generation with Multi-Frame and Multi-View Consistency
- Shape of Motion: 4D Reconstruction from a Single Video
- ![Star - of-motion/)
- WonderWorld: Interactive 3D Scene Generation from a Single Image
- WonderJourney: Going from Anywhere to Everywhere
- ![Star
- L3DG: Latent 3D Gaussian Diffusion
- ReconX: Reconstruct Any Scene from Sparse Views with Video Diffusion Model
- ![Star
- Hi3D: Pursuing High-Resolution Image-to-3D Generation with Video Diffusion Models
- GPT-4V(ision) is a Human-Aligned Evaluator for Text-to-3D Generation
- ![Star
- MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motion
- ![Star
- MultiDiff: Consistent Novel View Synthesis from a Single Image
- ReconX: Reconstruct Any Scene from Sparse Views with Video Diffusion Model
- ![Star
- Vivid-ZOO: Multi-View Video Generation with Diffusion Model
- DiffRF: Rendering-guided 3D Radiance Field Diffusion
- Director3D: Real-world Camera Trajectory and 3D Scene Generation from Text
- ![Star
- YouDream: Generating Anatomically Controllable Consistent Text-to-3D Animals
- ![Star
-
Video Enhancement and Restoration
-
Video Understanding
- ![Star - NJU/PDPP)
- Exploring Diffusion Models for Unsupervised Video Anomaly Detection
- PDPP:Projected Diffusion for Procedure Planning in Instructional Videos
- DiffTAD: Temporal Action Detection with Proposal Denoising Diffusion
- ![Star
- DiffusionRet: Generative Text-Video Retrieval with Diffusion Model
- ![Star
- Refined Semantic Enhancement Towards Frequency Diffusion for Video Captioning
- ![Star
- A Generalist Framework for Panoptic Segmentation of Images and Videos
- ![Star - research/pix2seq)
- Diffusion Action Segmentation
- ![Star
- VLM-Grounder: A VLM Agent for Zero-Shot 3D Visual Grounding
- ![Star - Grounder)
-
Healthcare and Biology
- Annealed Score-Based Diffusion Model for Mr Motion Artifact Reduction
- Feature-Conditioned Cascaded Video Diffusion Models for Precise Echocardiogram Synthesis
- Neural Cell Video Synthesis via Optical-Flow Diffusion
- ![Website
- Artificial Intelligence for Biomedical Video Generation
- Exploring Variational Autoencoders for Medical Image Generation: A Comprehensive Study
- MedSora: Optical Flow Representation Alignment Mamba Diffusion Model for Medical Video Generation
- Medical Video Generation for Disease Progression Simulation
- ![arXiv
-
Long Video / Film Generation
- MovieDreamer: Hierarchical Generation for Coherent Long Visual Sequence
- ![Star - uofa/MovieDreamer)
- AutoAD-Zero: A Training-Free Framework for Zero-Shot Audio Description
- ![Star - Zero)
- AesopAgent: Agent-driven Evolutionary System on Story-to-Video Production
- TheaterGen: Character Management with LLM for Consistent Multi-turn Image Generation
- ![Star
- AutoStudio: Crafting Consistent Subjects in Multi-turn Interactive Image Generation
- ![Star
- DreamStory: Open-Domain Story Visualization by LLM-Guided Multi-Subject Consistent Diffusion
- VideoDirectorGPT: Consistent Multi-scene Video Generation via LLM-Guided Planning
- Story-Adapter: A Training-free Iterative Framework for Long Story Visualization
- StoryMaker: Towards consistent characters in text-to-image generation
- ![Star
- Storynizor: Consistent Story Generation via Inter-Frame Synchronized and Shuffled ID Injection
- ACDC: Autoregressive Coherent Multimodal Generation using Diffusion Correction
- ![Star - adapter)
- In-Context LoRA for Diffusion Transformers
- ![Star - vilab/In-Context-LoRA)
- SEED-Story: Multimodal Long Story Generation with Large Language Model
- ![Star - Story)
- StoryAgent: Customized Storytelling Video Generation via Multi-Agent Collaboration
- DreamFactory: Pioneering Multi-Scene Long Video Generation with a Multi-Agent Framework
- Kubrick: Multimodal Agent Collaborations for Synthetic Video Generation
- DreamCinema: Cinematic Transfer with Free Camera and 3D Character
- ![Star - wl20/DreamCinema?tab=readme-ov-file)
- SkyScript-100M: 1,000,000,000 Pairs of Scripts and Shooting Scripts for Short Drama
- ![Star - 100M)
- Anim-Director: A Large Multimodal Model Powered Agent for Controllable Animation Video Generation
- ![Star - TMG/Anim-Director)
- ![Star - wl20/DreamCinema?tab=readme-ov-file)
- CinePreGen: Camera Controllable Video Previsualization via Engine-powered Diffusion
- ![Dataset - preview-Data)
- ARLON: Boosting Diffusion Transformers With Autoregressive Models for Long Video Generation
- Unbounded: A Generative Infinite Game of Character Life Simulation
- Loong: Generating Minute-level Long Videos with Autoregressive Language Models
- DreamCinema: Cinematic Transfer with Free Camera and 3D Character
- Video Storyboarding: Multi-Shot Character Consistency for Text-to-Video Generation
- Mind the Time: Temporally-Controlled Multi-Event Video Generation
- ![Star - Huang/GenMAC)
- GenMAC: Compositional Text-to-Video Generation with Multi-Agent Collaboration
- VideoGen-of-Thought: A Collaborative Framework for Multi-Shot Video Generation
- Identity-Preserving Text-to-Video Generation by Frequency Decomposition
- ![Star - YuanGroup/ConsisID)
- MotionCharacter: Identity-Preserving and Motion Controllable Human Video Generation
- MovieBench: A Hierarchical Movie Level Dataset for Long Video Generation
- ![Star
- MotionPrompt: Optical-Flow Guided Prompt Optimization for Coherent Video Generation
- ![Star
- DreamRunner: Fine-Grained Storytelling Video Generation with Retrieval-Augmented Motion Adaptation
- ![Star
-
Video Generation with Physical Prior / 3D
- PhysDreamer: Physics-Based Interaction with 3D Objects via Video Generation
- ![Star
- ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis
- ![Star
- Compositional 3D-aware Video Generation with LLM Director
- StereoCrafter: Diffusion-based Generation of Long and High-fidelity Stereoscopic 3D from Monocular Videos
- ![Star
- Tex4D: Zero-shot 4D Scene Texturing with Video Diffusion Models
- PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generation
- ![Star
- How Far is Video Generation from World Model: A Physical Law Perspective
- ![Star
- ![Star
- PhyGenBench: Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generation
- ![Star
- Motion Dreamer: Realizing Physically Coherent Video Generation through Scene-Aware Motion Reasoning
- PhyT2V: LLM-Guided Iterative Self-Refinement for Physics-Grounded Text-to-Video Generation
- ![Star
- Phys4DGen: A Physics-Driven Framework for Controllable and Efficient 4D Content Generation from a Single Image
- PhysMotion: Physics-Grounded Dynamics From a Single Image
- IDOL: Unified Dual-Modal Latent Diffusion for Human-Centric Joint Video-Depth Generation
-
Human/AI Feedback for Video Generation
-
Human Feedback for Video Generation
-
Policy Learning with Video Generation
- Any-point Trajectory Modeling for Policy Learning
- ![Star - Trajectory-Model/ATM)
- GR-MG: Leveraging Partially Annotated Data via Multi-Modal Goal Conditioned Policy
- ![Star - MG/tree/main)
- EgoVid-5M: A Large-Scale Video-Action Dataset for Egocentric Video Generation
- ![Star
- Motion Tracks: A Unified Representation for Human-Robot Transfer in Few-Shot Imitation Learning
- Stag-1: Towards Realistic 4D Driving Simulation with Video Generation Model
- ![Star
- RT-Sketch: Goal-Conditioned Imitation Learning from Hand-Drawn Sketches
- ![Star
- Dreamitate: Real-World Visuomotor Policy Learning via Video Generation
- ![Star - columbia/dreamitate)
- This&That: Language-Gesture Controlled Video Generation for Robot Planning
- ![Star - and-that)
-
Open-World Model
- Digital Life Project: Autonomous 3D Characters with Social Intelligence
- Oasis: A Universe in a Transformer
- AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents
- ![Star - ai/open-oasis)
- ![Star
- The Matrix: Infinite-Horizon World Generation with Real-Time Moving Control
- Navigation World Models
- Genie 2: A large-scale foundation world model
- Understanding World or Predicting Future? A Comprehensive Survey of World Models
- 3D-VLA: A 3D Vision-Language-Action Generative World Model
-
World Model
-
Motion Customization
- ![Star
- MotionBank: A Large-scale Video Motion Benchmark with Disentangled Rule-based Annotations
- ![Star
- LAMP: Learn A Motion Pattern for Few-Shot-Based Video Generation
- ![Star - Research/MotionInversion)
- ![Star - vilab/VGen)
- ![Star
- ![Star - park/Spectral-Motion-Alignment)
- Motion Inversion for Video Customization
- Video Diffusion Models are Training-free Motion Interpreter and Controller
- ![Star
- ![Star
- Zero-Shot Controllable Image-to-Video Animation via Motion Decomposition
- Reenact Anything: Semantic Video Motion Transfer Using Motion-Textual Inversion
- I2VControl-Camera: Precise Video Camera Control with Adjustable Motion Strength
- ![Star
- Customizing Motion in Text-to-Video Diffusion Models
- DragAnything: Motion Control for Anything using Entity Representation
- MotionDirector: Motion Customization of Text-to-Video Diffusion Models
- ![Star - Wu/LAMP)
- DreamVideo: Composing Your Dream Videos with Customized Subject and Motion
- Direct-a-Video: Customized Video Generation with User-Directed Camera Movement and Object Motion
- Customize-A-Video: One-Shot Motion Customization of Text-to-Video Diffusion Models
- Collaborative Video Diffusion: Consistent Multi-video Generation with Camera Control
- MotionClone: Training-Free Motion Cloning for Controllable Video Generation
- ![Star - qiu/FreeTraj)
- MotionCtrl: A Unified and Flexible Motion Controller for Video Generation
- Video Motion Transfer with Diffusion Transformers
- ![Star
- Latent-Reframe: Enabling Camera Control for Video Diffusion Model without Training
- Motion Modes: What Could Happen Next?
- MoTrans: Customized Motion Transfer with Text-driven Video
- ![Star
- AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformers
- ![Star
- Tora: Trajectory-oriented Diffusion Transformer for Video Generation
- ViewExtrapolator: Novel View Extrapolation with Video Diffusion Priors
- ![Star - Liu/ViewExtrapolator)
- FreeTraj: Tuning-Free Trajectory Control in Video Diffusion Models
-
4D
- DimensionX: Create Any 3D and 4D Scenes from a Single Image with Controllable Video Diffusion
- ![Star
- ![Star
- 4Real-Video: Learning Generalizable Photo-Realistic 4D Video Diffusion
- PaintScene4D: Consistent 4D Scene Generation from Text Prompts
- CAT4D: Create Anything in 4D with Multi-View Video Diffusion Models
-
Audio Synthesis for Video
- Draw an Audio: Leveraging Multi-Instruction for Video-to-Audio Synthesis
- ![Star - an-Audio-Code)
- ![Star
- Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming
- ![Star - omni/mini-omni)
- Speech To Speech: an effort for an open-sourced and modular GPT4-o
- ![Star - to-speech)
- VMAs: Video-to-Music Generation via Semantic Alignment in Web Music Videos
- STA-V2A: Video-to-Audio Generation with Semantic and Temporal Alignment
- MuVi: Video-to-Music Generation with Semantic Alignment and Rhythmic Synchronization
- Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptation
- Video-Guided Foley Sound Generation with Multimodal Controls
- FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Sounds
- Network Bending of Diffusion Models for Audio-Visual Generation
- ![Star
- Masked Generative Video-to-Audio Transformers with Enhanced Synchronicity
- Video-to-Audio Generation with Hidden Alignment
- ![Star - ldm)
- Read, Watch and Scream! Sound Generation from Text and Video
- ![Star - ai/rewas)
-
Commercial Product
-
AI Safety for Video Generation
-
Try On with Video Generation
-
Unified Model for Generation and Understanding
-
Acceleration for Video Generation
-
Game Generation
-
Rendering with Virtual Engine
-
Efficiency for Video Generation
-
Video Generation with Physical Prior/3D
-
Programming Languages
Categories
Sub Categories
Video Generation
283
Video Editing
105
Controllable Video Generation
63
Long Video / Film Generation
51
Human or Subject Motion
47
Motion Customization
39
Open-source Toolboxes and Foundation Models
37
3D / NeRF
33
Talking Head Generation
31
Evaluation Benchmarks and Metrics
25
Video Generation with Physical Prior / 3D
21
Audio Synthesis for Video
20
Policy Learning with Video Generation
15
Video Understanding
15
Open-World Model
10
Healthcare and Biology
9
Long-form Video Generation and Completion
9
4D
6
Video Enhancement and Restoration
5
Commercial Product
5
Human/AI Feedback for Video Generation
4
Efficiency for Video Generation
4
Unified Model for Generation and Understanding
3
New Video Generation Benchmark and Metrics
3
Rendering with Virtual Engine
2
Human Feedback for Video Generation
2
World Model
2
Game Generation
1
AI Safety for Video Generation
1
Try On with Video Generation
1
Video Generation with Physical Prior/3D
1
Acceleration for Video Generation
1
Keywords
text-to-video
3
ai
3
colab
1
colab-notebook
1
colaboratory
1
t2v
1
image-generation
1
multi-turn-dialogue
1
text-to-image-generation
1
aigc
1
content-production
1
fine-tuning-diffusion
1
video-generation
1
visual-art
1
hotshot
1
hotshot-xl
1
sdxl
1
text-to-gif
1
text-to-video-generation
1
assistant
1
language-model
1
machine-learning
1
python
1
speech
1
speech-synthesis
1
speech-to-text
1
speech-translation
1