{"id":19882998,"url":"https://github.com/modelscope/lite-sora","last_synced_at":"2025-05-02T14:32:59.206Z","repository":{"id":226708592,"uuid":"768488969","full_name":"modelscope/lite-sora","owner":"modelscope","description":"An initiative to replicate Sora","archived":false,"fork":false,"pushed_at":"2024-04-10T08:35:58.000Z","size":10084,"stargazers_count":70,"open_issues_count":2,"forks_count":4,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-04-11T08:41:15.083Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/modelscope.png","metadata":{"files":{"readme":"README-ZH.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2024-03-07T07:15:12.000Z","updated_at":"2024-04-08T16:54:32.000Z","dependencies_parsed_at":"2024-03-13T12:50:58.780Z","dependency_job_id":null,"html_url":"https://github.com/modelscope/lite-sora","commit_stats":null,"previous_names":["modelscope/lite-sora"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/modelscope%2Flite-sora","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/modelscope%2Flite-sora/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/modelscope%2Flite-sora/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/modelscope%2Flite-sora/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/modelscope","download_url":"https://codeload.github.com/modelscope/lite-sora/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224315686,"owners_count":17291088,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-12T17:19:07.037Z","updated_at":"2024-11-12T17:19:09.061Z","avatar_url":"https://github.com/modelscope.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Lite-Sora\n\n## 简介\n\nlite-sora项目是一个Sora技术复现的项目，它由华东师范大学和ModelScope社区共同发起，探索Sora背后的视频生成技术的最小复现和简洁实现，我们希望可以提供简洁易读的代码方便大家一起改进实验，不断探索提升开源视频生成技术的上限。\n\n## 技术路线\n\n* [x] 搭建基础架构\n  * [ ] 模型\n    * [x] Text Encoder（基于 Stable Diffusion XL 中的 [Text Encoder](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/blob/main/text_encoder_2/model.safetensors)）\n    * [x] VideoDiT（基于 [Facebook DiT](https://github.com/facebookresearch/DiT)）\n    * [ ] VideoVAE\n  * [x] Scheduler（基于 [DDIM](https://arxiv.org/abs/2010.02502)）\n  * [x] Trainer（基于 [PyTorch-lightning](https://lightning.ai/docs/pytorch/stable/)）\n* [x] 小规模数据集验证\n  * [x] [Pixabay100](https://github.com/ECNU-CILAB/Pixabay100)\n* [ ] 在大规模数据集上训练 Video Encoder \u0026 Decoder\n* [ ] 在大规模数据集上训练 VideoDiT\n\n## 使用\n\n### Python 环境搭建\n\n```\nconda env create -f environment.yml\nconda activate litesora\n```\n\n### 下载模型\n\n* `models/text_encoder/model.safetensors`: 来自 Stable Diffusion XL 的 Text Encoder，[下载链接](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/resolve/main/text_encoder_2/model.safetensors)\n* `models/denoising_model/model.safetensors`：我们在 [Pixabay100](https://github.com/ECNU-CILAB/Pixabay100) 数据集上训练的模型，该模型可以证明我们的训练代码能够正常拟合训练数据，分辨率为 64*64。**这个模型显然是过拟合的，由于训练数据少，尚不具备泛化能力，仅用于验证训练算法的正确性。** [下载链接](https://huggingface.co/ECNU-CILab/lite-sora-v1-pixabay100/resolve/main/denoising_model/model.safetensors)\n* `models/vae/model.safetensors`: Stable Video Diffusion 的 VAE. [下载链接](https://huggingface.co/stabilityai/stable-video-diffusion-img2vid-xt/resolve/main/vae/diffusion_pytorch_model.fp16.safetensors)\n\n### 训练\n\n```python\nfrom litesora.data import TextVideoDataset\nfrom litesora.models import SDXLTextEncoder2\nfrom litesora.trainers.v1 import LightningVideoDiT\nimport lightning as pl\nimport torch\n\n\nif __name__ == '__main__':\n    # dataset and data loader\n    dataset = TextVideoDataset(\"data/pixabay100\", \"data/pixabay100/metadata.json\",\n                               num_frames=64, height=64, width=64)\n    train_loader = torch.utils.data.DataLoader(dataset, shuffle=True, batch_size=1, num_workers=8)\n\n    # model\n    model = LightningVideoDiT(learning_rate=1e-5)\n    model.text_encoder.load_state_dict_from_diffusers(\"models/text_encoder/model.safetensors\")\n\n    # train\n    trainer = pl.Trainer(max_epochs=100000, accelerator=\"gpu\", devices=\"auto\", callbacks=[\n        pl.pytorch.callbacks.ModelCheckpoint(save_top_k=-1)\n    ])\n    trainer.fit(model=model, train_dataloaders=train_loader)\n```\n\n训练程序启动后，可开启 `tensorboard` 监视训练进度\n\n```\ntensorboard --logdir .\n```\n\n### 推理\n\n* 在像素空间生成一个视频。\n\n```python\nfrom litesora.models import SDXLTextEncoder2, VideoDiT\nfrom litesora.pipelines import PixelVideoDiTPipeline\nfrom litesora.data import save_video\nimport torch\n\n\n# models\ntext_encoder = SDXLTextEncoder2.from_diffusers(\"models/text_encoder/model.safetensors\")\ndenoising_model = VideoDiT.from_pretrained(\"models/denoising_model/model.safetensors\")\n\n# pipeline\npipe = PixelVideoDiTPipeline(torch_dtype=torch.float16, device=\"cuda\")\npipe.fetch_models(text_encoder, denoising_model)\n\n# generate a video\nprompt = \"woman, flowers, plants, field, garden\"\nvideo = pipe(prompt=prompt, num_inference_steps=100)\n\n# save the video (the resolution is 64*64, we enlarge it to 512*512 here)\nsave_video(video, \"output.mp4\", upscale=8)\n```\n\n* 把一个视频从像素空间编码到隐空间，然后解码它\n\n```python\nfrom litesora.models import SDVAEEncoder, SVDVAEDecoder\nfrom litesora.data import load_video, tensor2video, concat_video, save_video\nimport torch\nfrom tqdm import tqdm\n\n\nframes = load_video(\"data/pixabay100/videos/168572 (Original).mp4\",\n                    num_frames=1024, height=1024, width=1024, random_crop=False)\nframes = frames.to(dtype=torch.float16, device=\"cpu\")\n\nencoder = SDVAEEncoder.from_diffusers(\"models/vae/model.safetensors\").to(dtype=torch.float16, device=\"cuda\")\ndecoder = SVDVAEDecoder.from_diffusers(\"models/vae/model.safetensors\").to(dtype=torch.float16, device=\"cuda\")\n\nwith torch.no_grad():\n    print(frames.shape)\n    latents = encoder.encode_video(frames, progress_bar=tqdm)\n    print(latents.shape)\n    decoded_frames = decoder.decode_video(latents, progress_bar=tqdm)\n\nvideo = tensor2video(concat_video([frames, decoded_frames]))\nsave_video(video, \"video.mp4\", fps=24)\n```\n\n### 现阶段效果展示\n\n我们在 [Pixabay100](https://github.com/ECNU-CILAB/Pixabay100) 数据集上训练的模型，该模型可以证明我们的训练代码能够正常拟合训练数据，分辨率为 64*64。**这个模型显然是过拟合的，由于训练数据少，尚不具备泛化能力，仅用于验证训练算法的正确性。** [下载链接](https://huggingface.co/ECNU-CILab/lite-sora-v1-pixabay100/resolve/main/denoising_model/model.safetensors)\n\n|airport, people, crowd, busy|beach, ocean, waves, water, sand|bee, honey, insect, beehive, nature|coffee, beans, caffeine, coffee, shop|\n|-|-|-|-|\n|![](assets/airport_people_crowd_busy.gif)|![](assets/beach_ocean_waves_water_sand.gif)|![](assets/bee_honey_insect_beehive_nature.gif)|![](assets/coffee_beans_caffeine_coffee_shop.gif)|\n|fish, underwater, aquarium, swim|forest, woods, mystical, morning|ocean, beach, sunset, sea, atmosphere|hair, wind, girl, woman, people|\n|![](assets/fish_underwater_aquarium_swim.gif)|![](assets/forest_woods_mystical_morning.gif)|![](assets/ocean_beach_sunset_sea_atmosphere.gif)|![](assets/hair_wind_girl_woman_people.gif)|\n|reeds, grass, wind, golden, sunshine|sea, ocean, seagulls, birds, sunset|woman, flowers, plants, field, garden|wood, anemones, wildflower, flower|\n|![](assets/reeds_grass_wind_golden_sunshine.gif)|![](assets/sea_ocean_seagulls_birds_sunset.gif)|![](assets/woman_flowers_plants_field_garden.gif)|![](assets/wood_anemones_wildflower_flower.gif)|\n\n我们采用 [Stable-Video-Diffusion](https://huggingface.co/stabilityai/stable-video-diffusion-img2vid-xt) 中的 VAE 模型来做视频的编码和解码。我们的代码支持非常长的高分辨率视频！\n\nhttps://github.com/modelscope/lite-sora/assets/35051019/dc205719-d0bc-4bca-b117-ff5aa19ebd86\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmodelscope%2Flite-sora","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmodelscope%2Flite-sora","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmodelscope%2Flite-sora/lists"}