{"id":47422858,"url":"https://github.com/OpenDriveLab/SparseVideoNav","last_synced_at":"2026-04-04T23:00:29.751Z","repository":{"id":336842395,"uuid":"1149404010","full_name":"OpenDriveLab/SparseVideoNav","owner":"OpenDriveLab","description":"Sparse Video Generation Model for Embodied Navigation conditioned on loose language guidance, 100% real world verification","archived":false,"fork":false,"pushed_at":"2026-03-31T01:54:41.000Z","size":8563,"stargazers_count":68,"open_issues_count":1,"forks_count":1,"subscribers_count":3,"default_branch":"main","last_synced_at":"2026-04-04T22:57:04.886Z","etag":null,"topics":["embodied-navigation","video-generation-model","vln"],"latest_commit_sha":null,"homepage":"https://opendrivelab.com/SparseVideoNav","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/OpenDriveLab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-02-04T04:27:22.000Z","updated_at":"2026-04-03T13:44:20.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/OpenDriveLab/SparseVideoNav","commit_stats":null,"previous_names":["opendrivelab/sparsevideonav"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/OpenDriveLab/SparseVideoNav","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenDriveLab%2FSparseVideoNav","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenDriveLab%2FSparseVideoNav/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenDriveLab%2FSparseVideoNav/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenDriveLab%2FSparseVideoNav/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/OpenDriveLab","download_url":"https://codeload.github.com/OpenDriveLab/SparseVideoNav/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenDriveLab%2FSparseVideoNav/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31418285,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-04T20:09:54.854Z","status":"ssl_error","status_checked_at":"2026-04-04T20:09:44.350Z","response_time":60,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["embodied-navigation","video-generation-model","vln"],"created_at":"2026-03-21T18:00:25.641Z","updated_at":"2026-04-04T23:00:29.735Z","avatar_url":"https://github.com/OpenDriveLab.png","language":"Python","funding_links":[],"categories":["largemodel"],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg src=\"asset/caption.png\" alt=\"SparseVideoNav Logo\"\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n\u003ch1 align=\"center\"\u003e\u003cstrong\u003eSparseVideoNav: Sparse Video Generation Propels Real-World Beyond-the-View Vision-Language Navigation\u003c/strong\u003e\u003c/h1\u003e\n\u003cp align=\"center\"\u003e\n    \u003ca href='https://betray12138.github.io/resume/' target='_blank'\u003eHai Zhang\u003c/a\u003e\u003csup\u003e*\u003c/sup\u003e\u0026emsp;\n    \u003ca href='https://github.com/stdcat' target='_blank'\u003eSiqi Liang\u003c/a\u003e\u003csup\u003e*\u003c/sup\u003e\u0026emsp;\n    \u003ca href='https://scholar.google.com/citations?user=ulZxvY0AAAAJ\u0026hl=en' target='_blank'\u003eLi Chen\u003c/a\u003e\u0026emsp;\n    \u003ca href='https://github.com/LiYuxian0306' target='_blank'\u003eYuxian Li\u003c/a\u003e\u0026emsp;\n    \u003ca href='https://github.com/PUFFERFISHTNT' target='_blank'\u003eYukuan Xu\u003c/a\u003e\u0026emsp;\n    \u003ca href='https://z-taylcr7.github.io/' target='_blank'\u003eYichao Zhong\u003c/a\u003e\u0026emsp;\n    \u003ca href='https://mars.hku.hk/people.html' target='_blank'\u003eFu Zhang\u003c/a\u003e\u003csup\u003e†\u003c/sup\u003e\u0026emsp;\n    \u003ca href='https://lihongyang.info/' target='_blank'\u003eHongyang Li\u003c/a\u003e\u003csup\u003e†\u003c/sup\u003e\u0026emsp;\n    \u003cbr\u003e\u003cbr\u003e\n    \u003ca href='https://hku.hk/' target='_blank'\u003e\u003cimg src=\"asset/HKU_logo.png\" alt=\"Project Page\" height=\"40\"\u003e\u003c/a\u003e\n    \u003cbr\u003e\n    The University of Hong Kong\u0026emsp;\n    \u003cbr\u003e\n  \u003c/p\u003e\n\u003c/p\u003e\n\n\u003cdiv id=\"top\" align=\"center\"\u003e\n\n\u003cdiv\u003e\n\u003ca href=\"https://opendrivelab.com/SparseVideoNav\" target=\"_blank\"\u003e\u003cimg src=\"https://img.shields.io/badge/Project_Page-green\" alt=\"Project Page\"\u003e\u003c/a\u003e\n\u003ca href=\"https://github.com/OpenDriveLab/SparseVideoNav\"\u003e\u003cimg alt=\"Repo\" src=\"https://img.shields.io/badge/github-repo-blue?logo=github\"/\u003e\u003c/a\u003e\n\u003ca href=\"https://arxiv.org/abs/2602.05827\" target=\"_blank\"\u003e\u003cimg src=\"https://img.shields.io/badge/arXiv-2602.05827-b31b1b\" alt=\"arXiv\"\u003e\u003c/a\u003e\n\u003ca href=\"https://creativecommons.org/licenses/by-nc-sa/4.0\" target=\"_blank\"\u003e\u003cimg src=\"https://img.shields.io/badge/License-CC_%20_BY--NC--SA_4.0-blue.svg\" alt=\"License\"/\u003e\u003c/a\u003e\n\u003c/div\u003e\n\n\n\u003c/div\u003e\n\n## 📖 Introduction\n\nSparseVideoNav introduces video generation models to real-world beyond-the-view vision-language navigation for the first time. It achieves sub-second trajectory inference with a sparse future spanning a 20-second horizon, yielding a remarkable 27× speed-up. Real-world zero-shot experiments show 2.5× higher success rate than state-of-the-art LLM baselines and mark the first realization in challenging night scenes.\n\n**Developers**: [Hai Zhang](https://zhanghenryhai12138.github.io/) and [Siqi Liang](https://github.com/stdcat)\n\n## 📢 News\n\n\u003e [!IMPORTANT]\n\u003e 🌟 Stay up to date at [opendrivelab.com](https://opendrivelab.com/#news)!\n\n- 🎉 **2026-02-05**: [Project Page](https://opendrivelab.com/SparseVideoNav) is now available!\n- 🎉 **2026-02-06**: [arXiv preprint](https://arxiv.org/abs/2602.05827) is now available!\n- 🎉 **2026-03-31**: Inference code and model checkpoint released!\n\n## 📌 Table of Contents\n- 📖 [Introduction](#-introduction)\n- 📢 [News](#-news)\n- 🔥 [Highlights](#highlights)\n- 🔧 [Installation](#-installation)\n- 📥 [Checkpoint](#-checkpoint)\n- 🚀 [Usage](#-usage)\n- 📝 [TODO List](#-todo-list)\n- 📬 [Contact](#-contact)\n- 📄 [License and Citation](#-license-and-citation)\n\n\n## 🔥 Highlights\n\n- We investigate beyond-the-view navigation tasks in the real world  by introducing video generation model to this field for the first time.\n- We pioneer a paradigm shift from continuous to sparse video generation for longer prediction horizon.\n- We achieve sub-second trajectory inference guided by a generated sparse future spanning a 20-second horizon. This yields a remarkable 27x speed-up compared to the unoptimized counterpart.\n- We achieve the first realization of beyond-the-view navigation in challenging night scenes with a 17.5% success rate.\n\n## 🔧 Installation\n\n**Requirements**\n- Linux (tested on Ubuntu)\n- Python 3.10\n- NVIDIA GPU with ≥ 16 GB VRAM\n- [uv](https://docs.astral.sh/uv/) ≥ 0.7\n\n```bash\n# 1. Clone the repository\ngit clone https://github.com/OpenDriveLab/SparseVideoNav.git\ncd SparseVideoNav\n\n# 2. Create virtual environment and install dependencies\nuv sync --all-groups\n\n# 3. Activate the environment\nsource .venv/bin/activate\n```\n\n\u003e `--all-groups` also installs `flash-attn`. Building it from source takes a few minutes on first install.\n\n## 📥 Checkpoint\n\nDownload the SparseVideoNav pipeline checkpoint and place it under `models/SparseVideoNav-Models/`:\n\n| Component | Download |\n|-----------|----------|\n| **SparseVideoNav pipeline checkpoint** | 🤗 [HuggingFace](https://huggingface.co/OpenDriveLab/SparseVideoNav_VGM) |\n\nExpected directory layout after download:\n\n```\nmodels/SparseVideoNav-Models/\n├── google/\n│   └── umt5-xxl/\n│       ├── special_tokens_map.json\n│       ├── spiece.model\n│       ├── tokenizer.json\n│       └── tokenizer_config.json\n├── models_t5_umt5-xxl-enc-bf16.pth\n├── Wan2.1_VAE.pth\n└── svn_ckpt/\n    ├── config.json\n    └── diffusion_pytorch_model.safetensors\n```\n\nIf you place the checkpoint elsewhere, update `ckpt_path` in `config/inference.yaml` or override it on the command line.\n\n## 🚀 Usage\n\n### 1. Command-line inference\n\n```bash\npython inference.py video_path=/path/to/input.mp4 'prompt=turn right'\n```\n\nResults are written to `outputs/\u003ctimestamp\u003e_\u003cvideo_name\u003e/`:\n- `predicted_video.mp4` — generated future video\n\nKey overrides:\n\n| Parameter | Default | Description |\n|-----------|---------|-------------|\n| `video_path` | — | Input video path (required) |\n| `prompt` | — | Language instruction (required) |\n| `output_path` | `outputs` | Root output directory |\n| `ckpt_path` | `models/SparseVideoNav-Models` | Pipeline checkpoint directory |\n| `inference.device` | `cuda:0` | Target device |\n| `inference.denoise_steps` | `4` | Denoising steps (higher → better quality) |\n\nExample with overrides:\n\n```bash\npython inference.py \\\n    video_path=/path/to/input.mp4 \\\n    'prompt=walk forward and turn left' \\\n    ckpt_path=/path/to/checkpoint \\\n    inference.device=cuda:0 \\\n    inference.denoise_steps=8\n```\n\n### 2. Gradio web demo\n\n```bash\npython gradio_interface.py\n```\n\nOpens a local demo at `http://0.0.0.0:7860`. Upload a video, enter a navigation instruction, and click **Run Prediction**.\n\nCommon options:\n\n| Flag | Default | Description |\n|------|---------|-------------|\n| `--ckpt_path` | from config | Override checkpoint directory |\n| `--device` | `cuda:0` | Target device |\n| `--port` | `7860` | Server port |\n| `--share` | `False` | Create a public Gradio share link |\n\n### 3. Python API\n\n```python\nfrom omegaconf import OmegaConf\nfrom inference import SVNPipeline\n\ncfg = OmegaConf.load(\"config/inference.yaml\")\ncfg.ckpt_path = \"/path/to/checkpoint\"\ncfg.inference.device = \"cuda:0\"\n\npipeline = SVNPipeline.from_pretrained(cfg)\n\n# Returns np.ndarray (T, H, W, C) uint8\nvideo = pipeline(video=\"/path/to/input.mp4\", text=\"turn right\")\n```\n\nFor direct access to the latent-space model:\n\n```python\nfrom sparseVideoNav.svn_model import SVNModel\n\nmodel = SVNModel.from_pretrained(\"/path/to/checkpoint/svn_ckpt\")\n```\n\n## 📝 TODO List\n- [x] SparseVideoNav Paper Release.\n  - [x] arXiv preprint is now available!\n- [x] SparseVideoNav Code Release.\n  - [x] Inference code of distilled video generation model and model checkpoint.\n  - [ ] Inference code of continuous action head and model checkpoint (Estimate 2026 Q3).\n- [ ] SparseVideoNav Dataset Release\n  - [ ] ~140h real-world VLN data (Estimate 2026 Q3).\n\n## 📬 Contact\n\nFor further inquiries or assistance, please contact [zhanghenryhai12138@gmail.com](mailto:zhanghenryhai12138@gmail.com) or [liangsiqi@connect.hku.hk](mailto:liangsiqi@connect.hku.hk)\n\n## 📄 License and Citation\n\nAll the data and code within this repo are under [CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/).\n\n- Please consider citing our work if it helps your research.\n```BibTeX\n@article{zhang2026sparse,\n  title={Sparse Video Generation Propels Real-World Beyond-the-View Vision-Language Navigation},\n  author={Zhang, Hai and Liang, Siqi and Chen, Li and Li, Yuxian and Xu, Yukuan and Zhong, Yichao and Zhang, Fu and Li, Hongyang},\n  journal={arXiv preprint arXiv:2602.05827},\n  year={2026}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FOpenDriveLab%2FSparseVideoNav","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FOpenDriveLab%2FSparseVideoNav","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FOpenDriveLab%2FSparseVideoNav/lists"}