{"id":51131278,"url":"https://github.com/Shubham2376G/NanoI2V","last_synced_at":"2026-07-13T20:00:37.597Z","repository":{"id":357074669,"uuid":"1235266237","full_name":"Shubham2376G/NanoI2V","owner":"Shubham2376G","description":"Building an Image-to-Video (I2V) Model from Scratch","archived":false,"fork":false,"pushed_at":"2026-07-03T07:06:15.000Z","size":16841,"stargazers_count":14,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-07-03T09:06:31.305Z","etag":null,"topics":["computer-vision","deep-learning","diffusion-models","flow-matching","generative-ai","huggingface","image-to-video","machine-learning","open-source","pytorch","text-to-video","transformer","video-diffusion","video-generation"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Shubham2376G.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-11T06:54:54.000Z","updated_at":"2026-07-03T07:06:20.000Z","dependencies_parsed_at":null,"dependency_job_id":"6ea4e915-2a1d-4e6f-8cb7-c8a8a803b175","html_url":"https://github.com/Shubham2376G/NanoI2V","commit_stats":null,"previous_names":["shubham2376g/nanoi2v"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Shubham2376G/NanoI2V","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Shubham2376G%2FNanoI2V","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Shubham2376G%2FNanoI2V/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Shubham2376G%2FNanoI2V/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Shubham2376G%2FNanoI2V/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Shubham2376G","download_url":"https://codeload.github.com/Shubham2376G/NanoI2V/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Shubham2376G%2FNanoI2V/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35434586,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-13T02:00:06.543Z","response_time":119,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","deep-learning","diffusion-models","flow-matching","generative-ai","huggingface","image-to-video","machine-learning","open-source","pytorch","text-to-video","transformer","video-diffusion","video-generation"],"created_at":"2026-06-25T13:00:24.777Z","updated_at":"2026-07-13T20:00:37.590Z","avatar_url":"https://github.com/Shubham2376G.png","language":"Python","funding_links":[],"categories":["Table of Contents \u003c!-- omit in toc --\u003e"],"sub_categories":["Open-source Toolboxes and Foundation Models"],"readme":"# NanoI2V: Building an Image-to-Video Model from Scratch\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/images/i2v.png\" width=\"100%\"\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://img.shields.io/badge/Python-3.10-blue\"\u003e\n  \u003cimg src=\"https://img.shields.io/badge/PyTorch-2.0-red\"\u003e\n  \u003cimg src=\"https://img.shields.io/badge/Transformers-DiT-purple\"\u003e\n  \u003cimg src=\"https://img.shields.io/badge/License-MIT-green\"\u003e\n  \u003cimg src=\"https://img.shields.io/github/stars/Shubham2376G/NanoI2V\"\u003e\n\u003c/p\u003e\n\n# Overview\n\nNanoI2V is a **from-scratch implementation** of an **Image-to-Video (I2V)** generation pipeline.\n\nThe project focuses on understanding and implementing the core concepts and building blocks behind modern video generation systems such as:\n\n- Variational Autoencoders (VAE)\n- Latent Video Modeling\n- Diffusion / Flow Matching\n- DiT Transformers\n- Cross-Attention Conditioning\n- Classifier-Free-Guidance\n\n\n---\n\n\u003e The series is published as a dedicated website with structured lessons, explanations, and code walkthroughs:\n\u003e **[shubham2376g.github.io/NanoI2V](https://shubham2376g.github.io/NanoI2V)**\n\u003e\n\u003e Each topic is a self-contained lesson - read in order or jump to what you need.\n\n---\n\n## What This Series Covers\n\nThis series explores the core building blocks behind modern image-to-video (I2V) models, including topics such as:\n\n| Area | Topics |\n|---|---|\n| VAE | Causal 3D convolutions, residual blocks, video encoders \u0026 decoders |\n| DiT | Rotary positional embeddings (RoPE), attention mechanisms, adaptive LayerNorm |\n| Flow \u0026 Diffusion | Flow matching, schedulers, denoising concepts |\n| Conditioning | Text conditioning, image conditioning, multimodal embeddings |\n| Training | End-to-end training pipeline, optimization, inference |\n\nAdditional topics and modules will be added as the series evolves.\n\n---\n\n## Repository Structure\n\n```text\nNanoI2V/\n├── vae/\n│   ├── conv.py                 # CausalConv3D implementation\n│   ├── blocks.py               # 3D ResBlocks and Spatial Attention modules\n│   ├── encoder.py              # VAE encoder\n│   ├── decoder.py              # VAE decoder\n│   └── vae.py                  # VAE model definition\n│\n├── dit/\n│   ├── rope.py                 # 3D Rotary Positional Embeddings (RoPE)\n│   ├── attention.py            # Self-Attention and Cross-Attention layers\n│   ├── blocks.py               # DiT blocks with Adaptive LayerNorm (adaLN)\n│   └── dit.py                  # Diffusion Transformer (DiT) architecture\n│\n├── flow/\n│   └── scheduler.py            # Flow Matching scheduler\n│\n├── conditioning/\n│   └── encoders.py             # Text and image conditioning encoders\n│\n├── data/\n│   ├── download_vidgen.py      # Dataset download utilities\n│   ├── prepare_vidgen.py       # Dataset preparation pipeline\n│   └── preprocess_vae.py       # VAE preprocessing scripts\n│\n├── docs/\n│   └── index.html              # Project website (GitHub Pages)\n│\n├── train_vae.py                # VAE training script\n├── train_dit.py                # DiT training script\n├── inference_dit.py            # Inference and video generation\n│\n└── README.md                   # Project documentation\n```\n\n---\n\n\n## High-Level DiT Pipeline\n\nNanoI2V follows the latent diffusion paradigm, where the Diffusion Transformer (DiT) operates entirely in the **latent space** produced by the VAE instead of directly on pixels.\n\nDuring each denoising step, the model takes:\n\n- **Noisy video latents** to be denoised\n- **Image latents** extracted from the conditioning image\n- **Text tokens** from the prompt encoder\n- **Image tokens** from the image encoder\n- **Timestep embeddings** indicating the current diffusion step\n\nThese inputs are processed by a stack of DiT blocks that iteratively refine the latent representation using attention and conditioning mechanisms. The final layer predicts the velocity field used by the flow-matching solver to progressively generate the target video latent.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/images/dit.png\" alt=\"High-Level DiT Architecture\" width=\"1000\"/\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n\u003ci\u003eSimplified overview of the NanoI2V generation pipeline. The figure emphasizes the flow of information between components rather than the exact implementation. Detailed architectural components are introduced in later chapters.\u003c/i\u003e\n\u003c/p\u003e\n\n---\n\n\n## Results\n\nThe following results were generated using the models implemented in this repository.\n\n### VAE Reconstruction Results\n\nThe VAE is trained to compress video clips into a latent representation and reconstruct them with minimal quality loss.\n\n\n![](docs/images/vae_recon_final.png)\n\n\n\n---\n\n### DiT Video Generation Results\n\nThe Diffusion Transformer (DiT) is trained in latent space and generates video sequences conditioned on an input image.\n\n#### Example 1\n\n\u003ctable\u003e\n  \u003ctr\u003e\n    \u003ctd align=\"center\"\u003e\u003cb\u003eInput Image\u003c/b\u003e\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e\u003cb\u003eGenerated Video\u003c/b\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd align=\"center\"\u003e\n      \u003cimg src=\"docs/images/s0.png\" width=\"300\"\u003e\n    \u003c/td\u003e\n    \u003ctd align=\"center\"\u003e\n      \u003cimg src=\"docs/images/s0.gif\" width=\"300\"\u003e\n    \u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd colspan=\"2\" align=\"center\"\u003e\n\n**Text Prompt**\n\n*Third-person follow shot of a Minecraft-style character running down a stone pathway. The character has brown hair, a grey shirt, and a sword strapped to their back, captured mid-stride from behind. The surrounding environment features textured cobblestone walls and building facades under natural daylight. Smooth animation, blocky voxel aesthetic.*\n\n\u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\n---\n\n#### Example 2\n\n\u003ctable\u003e\n  \u003ctr\u003e\n    \u003ctd align=\"center\"\u003e\u003cb\u003eInput Image\u003c/b\u003e\u003c/td\u003e\n    \u003ctd align=\"center\"\u003e\u003cb\u003eGenerated Video\u003c/b\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd align=\"center\"\u003e\n      \u003cimg src=\"docs/images/s1.png\" width=\"300\"\u003e\n    \u003c/td\u003e\n    \u003ctd align=\"center\"\u003e\n      \u003cimg src=\"docs/images/s1.gif\" width=\"300\"\u003e\n    \u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd colspan=\"2\" align=\"center\"\u003e\n\n**Text Prompt**\n\n*High-angle overhead shot of Formula 1 race cars navigating a sharp, wide turn on a grey asphalt track. The camera smoothly zooming and closing in on the bright green and yellow F1 car as it accelerates. The background shows a blurry race barrier and spectator area. Realistic lighting, high speed motion blur.*\n\n\u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\n\n\n---\n## Prerequisites\n\nA basic understanding of the following will help:\n\n- PyTorch basics\n- Transformer architecture and attention mechanisms\n- LLM fundamentals\n- Diffusion model intuition (helpful but not required)\n\nIf you've worked with LLMs before, many concepts here will feel familiar.\n\n---\n\n## ⭐ Support the Project\n\nIf you find this useful:\n- Star the repository ⭐\n- Share it with others interested in diffusion or video models\n\n\u003cp\u003e\n  \u003ca href=\"https://www.linkedin.com/in/shubham-aggarwal-a63b40276\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/Follow%20on-LinkedIn-0A66C2?style=for-the-badge\u0026logo=linkedin\u0026logoColor=white\" /\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FShubham2376G%2FNanoI2V","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FShubham2376G%2FNanoI2V","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FShubham2376G%2FNanoI2V/lists"}