{"id":13493462,"url":"https://github.com/nateraw/stable-diffusion-videos","last_synced_at":"2025-05-14T04:08:07.367Z","repository":{"id":58838380,"uuid":"533451541","full_name":"nateraw/stable-diffusion-videos","owner":"nateraw","description":"Create 🔥 videos with Stable Diffusion by exploring the latent space and morphing between text prompts","archived":false,"fork":false,"pushed_at":"2024-09-21T05:28:49.000Z","size":10222,"stargazers_count":4579,"open_issues_count":54,"forks_count":441,"subscribers_count":55,"default_branch":"main","last_synced_at":"2025-05-10T09:39:31.435Z","etag":null,"topics":["ai-art","huggingface","huggingface-diffusers","machine-learning","stable-diffusion"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nateraw.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-09-06T18:21:50.000Z","updated_at":"2025-05-07T10:00:53.000Z","dependencies_parsed_at":"2024-01-07T16:13:55.561Z","dependency_job_id":"f7382188-7078-4520-bae9-878904c59447","html_url":"https://github.com/nateraw/stable-diffusion-videos","commit_stats":null,"previous_names":[],"tags_count":19,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nateraw%2Fstable-diffusion-videos","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nateraw%2Fstable-diffusion-videos/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nateraw%2Fstable-diffusion-videos/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nateraw%2Fstable-diffusion-videos/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nateraw","download_url":"https://codeload.github.com/nateraw/stable-diffusion-videos/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254069577,"owners_count":22009558,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-art","huggingface","huggingface-diffusers","machine-learning","stable-diffusion"],"created_at":"2024-07-31T19:01:15.426Z","updated_at":"2025-05-14T04:08:02.352Z","avatar_url":"https://github.com/nateraw.png","language":"Python","funding_links":[],"categories":["Python","Image Synthesis","HarmonyOS","其他_机器视觉","👑Stable Diffusion","Video \u0026 Animation"],"sub_categories":["Inbox: Stable Diffusion","Windows Manager","网络服务_其他","Python"],"readme":"# stable-diffusion-videos\n\nTry it yourself in Colab: [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/nateraw/stable-diffusion-videos/blob/main/stable_diffusion_videos.ipynb)\n\u003c!--\nTPU version (~x6 faster than standard colab GPUs): [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/nateraw/stable-diffusion-videos/blob/main/flax_stable_diffusion_videos.ipynb)\n--\u003e\n\n**Example** - morphing between \"blueberry spaghetti\" and \"strawberry spaghetti\"\n\nhttps://user-images.githubusercontent.com/32437151/188721341-6f28abf9-699b-46b0-a72e-fa2a624ba0bb.mp4\n\n## Installation\n\n```bash\npip install stable_diffusion_videos\n```\n\n## Usage\n\nCheck out the [examples](./examples) folder for example scripts 👀\n\n### Making Videos\n\nNote: For Apple M1 architecture, use ```torch.float32``` instead, as ```torch.float16``` is not available on MPS.\n\n```python\nfrom stable_diffusion_videos import StableDiffusionWalkPipeline\nimport torch\n\npipeline = StableDiffusionWalkPipeline.from_pretrained(\n    \"CompVis/stable-diffusion-v1-4\",\n    torch_dtype=torch.float16,\n).to(\"cuda\")\n\nvideo_path = pipeline.walk(\n    prompts=['a cat', 'a dog'],\n    seeds=[42, 1337],\n    num_interpolation_steps=3,\n    height=512,  # use multiples of 64 if \u003e 512. Multiples of 8 if \u003c 512.\n    width=512,   # use multiples of 64 if \u003e 512. Multiples of 8 if \u003c 512.\n    output_dir='dreams',        # Where images/videos will be saved\n    name='animals_test',        # Subdirectory of output_dir where images/videos will be saved\n    guidance_scale=8.5,         # Higher adheres to prompt more, lower lets model take the wheel\n    num_inference_steps=50,     # Number of diffusion steps per image generated. 50 is good default\n)\n```\n\n### Making Music Videos\n\n*New!* Music can be added to the video by providing a path to an audio file. The audio will inform the rate of interpolation so the videos move to the beat 🎶\n\n```python\nfrom stable_diffusion_videos import StableDiffusionWalkPipeline\nimport torch\n\npipeline = StableDiffusionWalkPipeline.from_pretrained(\n    \"CompVis/stable-diffusion-v1-4\",\n    torch_dtype=torch.float16,\n).to(\"cuda\")\n\n# Seconds in the song.\naudio_offsets = [146, 148]  # [Start, end]\nfps = 30  # Use lower values for testing (5 or 10), higher values for better quality (30 or 60)\n\n# Convert seconds to frames\nnum_interpolation_steps = [(b-a) * fps for a, b in zip(audio_offsets, audio_offsets[1:])]\n\nvideo_path = pipeline.walk(\n    prompts=['a cat', 'a dog'],\n    seeds=[42, 1337],\n    num_interpolation_steps=num_interpolation_steps,\n    audio_filepath='audio.mp3',\n    audio_start_sec=audio_offsets[0],\n    fps=fps,\n    height=512,  # use multiples of 64 if \u003e 512. Multiples of 8 if \u003c 512.\n    width=512,   # use multiples of 64 if \u003e 512. Multiples of 8 if \u003c 512.\n    output_dir='dreams',        # Where images/videos will be saved\n    guidance_scale=7.5,         # Higher adheres to prompt more, lower lets model take the wheel\n    num_inference_steps=50,     # Number of diffusion steps per image generated. 50 is good default\n)\n```\n\n### Using the UI\n\n```python\nfrom stable_diffusion_videos import StableDiffusionWalkPipeline, Interface\nimport torch\n\npipeline = StableDiffusionWalkPipeline.from_pretrained(\n    \"CompVis/stable-diffusion-v1-4\",\n    torch_dtype=torch.float16,\n).to(\"cuda\")\n\ninterface = Interface(pipeline)\ninterface.launch()\n```\n\n## Credits\n\nThis work built off of [a script](https://gist.github.com/karpathy/00103b0037c5aaea32fe1da1af553355\n) shared by [@karpathy](https://github.com/karpathy). The script was modified to [this gist](https://gist.github.com/nateraw/c989468b74c616ebbc6474aa8cdd9e53), which was then updated/modified to this repo. \n\n## Contributing\n\nYou can file any issues/feature requests [here](https://github.com/nateraw/stable-diffusion-videos/issues)\n\nEnjoy 🤗\n\u003c!--Commenting this section out for now, as there are issues with RealESRGAN python package--\u003e\n\u003c!--\n\n## Extras\n### Upsample with Real-ESRGAN\n\nYou can also 4x upsample your images with [Real-ESRGAN](https://github.com/xinntao/Real-ESRGAN)!\n\nIt's included when you pip install the latest version of `stable-diffusion-videos`! \n\nYou'll be able to use `upsample=True` in the `walk` function, like this:\n\n```python\npipeline.walk(['a cat', 'a dog'], [234, 345], upsample=True)\n```\n\nThe above may cause you to run out of VRAM. No problem, you can do upsampling separately.\n\nTo upsample an individual image:\n\n```python\nfrom stable_diffusion_videos import RealESRGANModel\n\nmodel = RealESRGANModel.from_pretrained('nateraw/real-esrgan')\nenhanced_image = model('your_file.jpg')\n```\n\nOr, to do a whole folder:\n\n```python\nfrom stable_diffusion_videos import RealESRGANModel\n\nmodel = RealESRGANModel.from_pretrained('nateraw/real-esrgan')\nmodel.upsample_imagefolder('path/to/images/', 'path/to/output_dir')\n```\n--\u003e\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnateraw%2Fstable-diffusion-videos","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnateraw%2Fstable-diffusion-videos","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnateraw%2Fstable-diffusion-videos/lists"}