{"id":26426490,"url":"https://github.com/paritoshk/cosmosscaling","last_synced_at":"2026-02-16T17:34:11.496Z","repository":{"id":281758994,"uuid":"942880389","full_name":"paritoshk/CosmosScaling","owner":"paritoshk","description":"Custom world creation with Cosmos Video to World model for frame prediction problem to create better simulated worlds  ","archived":false,"fork":false,"pushed_at":"2025-03-12T02:43:10.000Z","size":2183,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-08-01T18:27:36.781Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/paritoshk.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-03-04T20:34:24.000Z","updated_at":"2025-03-12T02:43:13.000Z","dependencies_parsed_at":"2025-03-18T03:32:32.128Z","dependency_job_id":null,"html_url":"https://github.com/paritoshk/CosmosScaling","commit_stats":null,"previous_names":["paritoshk/cosmosscaling"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/paritoshk/CosmosScaling","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paritoshk%2FCosmosScaling","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paritoshk%2FCosmosScaling/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paritoshk%2FCosmosScaling/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paritoshk%2FCosmosScaling/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/paritoshk","download_url":"https://codeload.github.com/paritoshk/CosmosScaling/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paritoshk%2FCosmosScaling/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29513994,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-16T09:05:14.864Z","status":"ssl_error","status_checked_at":"2026-02-16T08:55:59.364Z","response_time":115,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-03-18T03:32:23.299Z","updated_at":"2026-02-16T17:34:11.473Z","avatar_url":"https://github.com/paritoshk.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 🌌 CosmosScaling\n\n\u003e Custom world creation with NVIDIA's Cosmos Video2World model for advanced frame prediction and simulated environment generation.\n\n![Cosmos Banner](https://img.shields.io/badge/NVIDIA-Cosmos-76B900?style=for-the-badge\u0026logo=nvidia\u0026logoColor=white)\n![Model](https://img.shields.io/badge/Model-5B%20Parameters-blue?style=for-the-badge)\n![License](https://img.shields.io/badge/License-NVIDIA%20Open-orange?style=for-the-badge)\n![Education](https://developer.nvidia.com/blog/accelerate-custom-video-foundation-model-pipelines-with-new-nvidia-nemo-framework-capabilities/)\n\n## 🚀 Project Overview\n\nThis project harnesses NVIDIA's Cosmos-1.0-Autoregressive-5B-Video2World model for state-of-the-art video generation and prediction capabilities. The model delivers impressive results:\n\n- 📊 Extends 9-frame video inputs to 33 frames (generating 24 new frames)\n- 🖼️ Generates 32 frames from a single image input\n- 🎞️ Creates high-quality 3D videos at 1024×640 resolution\n\n## ⚙️ Environment Setup\n\n### 🖥️ RunPod Configuration\n\nFor optimal performance with the Cosmos-1.0-Autoregressive-5B-Video2World model, we recommend:\n\nUsing ```nvcr.io/nvidia/nemo:24.12``` version of NeMO since RunPod has outdated NVDA drivers. \n\n| Component | Specification |\n|-----------|---------------|\n| GPU | NVIDIA A40 (48GB) or A100 (80GB) |\n| vCPUs | 8-16 cores |\n| RAM | 64GB minimum |\n| Storage | 100GB+ SSD |\n| Container | `nvcr.io/nvidia/nemo:25.02.rc1` |\n\n\u003e ⚠️ **Note**: The model requires significant VRAM (41.3GB with offloading) as per NVIDIA's documentation.\n\n### 🔧 Container Configuration\n\n#### Basic Dependencies\n\n```bash\n# Update system and install prerequisites\napt-get update \u0026\u0026 apt-get install -y \\\n  ffmpeg \\\n  libsm6 \\\n  libxext6 \\\n  git \\\n  wget \\\n  \u0026\u0026 rm -rf /var/lib/apt/lists/\n```\n\n#### Python Dependencies\n\n```bash\n# Install required Python packages\npip install --no-cache-dir \\\n  transformers \\\n  huggingface_hub \\\n  opencv-python \\\n  numpy \\\n  python-dotenv \\\n  fastapi \\\n  uvicorn\n```\n\n#### Model Download\n\n```bash\n# Create directory for model\nmkdir -p /workspace/models/Cosmos\n\n# Download model weights\nhuggingface-cli download nvidia/Cosmos-1.0-Autoregressive-5B-Video2World \\\n  --local-dir /workspace/models/Cosmos \\\n  --include \".pt\" \".json\" \".md\"\n```\n\n## 📥 Repository Setup\n\n```bash\n# Clone this repository\ngit clone https://github.com/paritoshk/CosmosScaling.git\ncd CosmosScaling\n```\n\n## 🧠 Model Architecture\n\nOur investigation revealed:\n\n```python\n# Load model directly with PyTorch\nmodel_path = \"/workspace/models/Cosmos/model.pt\"\nmodel_dict = torch.load(model_path, map_location=\"cuda\")\n```\n\n**Key Findings:**\n- The model loads as a Python dictionary (8.54 GB on GPU)\n- No callable modules were found in the model dictionary\n- NVIDIA provides the architecture code separately via [Cosmos GitHub Repository](https://github.com/NVIDIA/Cosmos)\n\n### 📊 Model Specifications\n\n- 🧠 VRAM: 41.3GB with partial offloading\n- ⏱️ Generation Time: ~73 seconds on H100 GPU\n- 🎯 Error Rate: 2% with 9-frame video input\n\n## 🏗️ System Architecture\n\n![System Architecture](https://img.shields.io/badge/Architecture-4--Layer-success?style=for-the-badge)\n\n1. **Model Loading Layer** 📥\n   - Load model weights from file\n   - Integrate with NVIDIA's Cosmos architecture\n\n2. **API Layer** 🌐\n   - FastAPI service for inference requests\n   - Support for both image and video inputs\n\n3. **Video Processing Layer** 🎬\n   - Extract frames from input videos\n   - Process 9-frame sequences for best results\n   - Generate 24 new frames (extending to 33 total)\n\n4. **Parallelization Layer** ⚡\n   - Optimize with CUDA and batch processing\n   - Implement memory offloading strategies\n\n## 🛣️ Next Steps\n\n1. **Architecture Integration**\n   - Clone and integrate NVIDIA's Cosmos repository\n   - Match model weights with their architecture\n\n2. **Offloading Implementation**\n   - Implement recommended offloading strategy:\n     - Guardrails \u0026 T5 encoder \u0026 Diffusion decoder \u0026 Tokenizer\n   - Target 28.8GB VRAM usage\n\n3. **API Development**\n   - Create FastAPI endpoints for video/image processing\n   - Implement async processing for better throughput\n\n## 📋 Quick Start Guide for RunPod\n\nThis template uses NVIDIA's official NeMo container with all dependencies pre-installed:\n\n1. Clone the Cosmos repository:\n\n```bash\ncd /workspace\ngit clone https://github.com/NVIDIA/Cosmos.git\ncd Cosmos\n```\n\n2. Set up environment variables:\n\n```bash\nexport HF_TOKEN=\"your_hugging_face_token\"\nexport HF_HOME=\"/workspace/hf_cache\"\n```\n\n3. Run inference:\n\n```bash\ntorchrun --nproc-per-node=1 cosmos1/models/autoregressive/nemo/inference/video2world.py \\\n  --input_type video \\\n  --input_image_or_video_path /path/to/your/video.mp4 \\\n  --prompt \"A detailed and realistic scene\" \\\n  --ar_model_dir nvidia/Cosmos-1.0-Autoregressive-5B-Video2World \\\n  --video_save_name /workspace/generated_video.mp4\n```\n\n## 📚 References\n\n- [NVIDIA Cosmos GitHub](https://github.com/NVIDIA/Cosmos)\n- [Model Technical Paper](https://research.nvidia.com/publication/2025-01_cosmos-world-foundation-model-platform-physical-ai)\n- [NVIDIA Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license)\n\n---\n\n\u003cp align=\"center\"\u003e\n  \u003cb\u003eDeveloped with ❤️ for advanced AI video generation\u003c/b\u003e\n\u003c/p\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fparitoshk%2Fcosmosscaling","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fparitoshk%2Fcosmosscaling","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fparitoshk%2Fcosmosscaling/lists"}