{"id":28573372,"url":"https://github.com/opendrivelab/robodual","last_synced_at":"2025-06-10T21:17:36.534Z","repository":{"id":286928912,"uuid":"961703664","full_name":"OpenDriveLab/RoboDual","owner":"OpenDriveLab","description":"RoboDual: Dual-System for Robotic Manipulation","archived":false,"fork":false,"pushed_at":"2025-04-28T12:31:16.000Z","size":2280,"stargazers_count":80,"open_issues_count":0,"forks_count":2,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-06-06T20:15:50.513Z","etag":null,"topics":["dual-system","manipulation","robotics"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/OpenDriveLab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["OpenDriveLab"],"patreon":null,"open_collective":null,"ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"lfx_crowdfunding":null,"polar":null,"buy_me_a_coffee":null,"custom":null}},"created_at":"2025-04-07T03:07:23.000Z","updated_at":"2025-05-31T16:59:28.000Z","dependencies_parsed_at":"2025-04-09T03:38:04.013Z","dependency_job_id":null,"html_url":"https://github.com/OpenDriveLab/RoboDual","commit_stats":null,"previous_names":["opendrivelab/robodual"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenDriveLab%2FRoboDual","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenDriveLab%2FRoboDual/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenDriveLab%2FRoboDual/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenDriveLab%2FRoboDual/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/OpenDriveLab","download_url":"https://codeload.github.com/OpenDriveLab/RoboDual/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenDriveLab%2FRoboDual/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259152773,"owners_count":22813223,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dual-system","manipulation","robotics"],"created_at":"2025-06-10T21:17:35.612Z","updated_at":"2025-06-10T21:17:36.526Z","avatar_url":"https://github.com/OpenDriveLab.png","language":"Python","funding_links":["https://github.com/sponsors/OpenDriveLab"],"categories":[],"sub_categories":[],"readme":"# :gemini:RoboDual\nThe official implementation of our paper: \\\n**Towards Synergistic, Generalized and Efficient Dual-System for Robotic Manipulation**\n\u003cdiv id=\"top\" align=\"center\"\u003e\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"https://opendrivelab.github.io/RoboDual/resources/robodual/teaser_page.png\" width=\"1000px\" \u003e\n\u003c/p\u003e\n\u003c/div\u003e\n\n#### Overview of RoboDual:\n\u003e Our objective is to develop a synergistic dual-system framework which supplements the generalizability of large-scale pre-trained generalist with the efficient and task-specific adaptation of specialist. (a) The fast specialist policy obsesses real-time and accurate control by aid of the slow yet generalized outcome from the generalist one with large-scale data. (b) RoboDual exhibits significant improvement in terms of performance and efficiency over a single standalone option and surpasses previous state-of-the-arts in the real-robot setting.\n\n\n\u003e [Qingwen Bu](https://scholar.google.com/citations?user=-JCRysgAAAAJ\u0026hl=zh-CN\u0026oi=ao), [Li Chen](https://ilnehc.github.io/), _et al._\n\n\u003e #### 📝 [Paper](https://arxiv.org/pdf/2410.08001) | 🌍 [Project Page](https://opendrivelab.com/RoboDual/)\n\n\u003e :mailbox_with_mail: Point of contact: *Qingwen Bu* ( qingwen@opendrivelab.com ) or *Li Chen* ( ilnehc@opendrivelab.com )\n\n\n## :fire: Highlight\n\n\u003cdiv id=\"top\" align=\"center\"\u003e\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"assets/model_arch.png\" width=\"1000px\" \u003e\n\u003c/p\u003e\n\u003c/div\u003e\n\n- **[Auto-regressive Generalist + DIffusion Action Specialist**] We introduce a novel approach that integrates generalist and specialist policies into a synergistic framework, dubbed :gemini:RoboDual, following a **dual-system** spirit.\n- **[Decoupled Training \u0026 Input]** The framework facilitates the flexible integration of diverse modalities and allows for the deconstruction of the two models on the aspect of training data, thereby enhancing their individual strengths and capabilities.\n\n### Current Endeavors on Dual-systems\n\nThe trend of dual-systems for robotics is shown below. In particular, *Asynchronous* implementations include:\n- [Helix](https://www.figure.ai/news/helix) from Figure\n- [HiRT](https://arxiv.org/pdf/2410.05273) from Tsinghua\n- [LCB](https://arxiv.org/pdf/2405.04798) from UC Berkeley\n- **RoboDual** (This work)\n\n\u003cdiv id=\"top\" align=\"center\"\u003e\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"assets/dual_system_timeline.png\" width=\"1000px\" \u003e\n\u003c/p\u003e\n\u003c/div\u003e\n\n\u003e - Following RoboDual, the architecture of dual-systems in robotics converges to the 'VLM + Diffusion Transformer' paradigm.\n\u003e - Asynchronous inference with dual-system allows a more *decoupled* design and enables more flexible and scalable reasoning.\n\u003e - Beyond latents, explicit representations (*e.g.,* coarse action output from the System-2 as in RoboDual) should also be explored!\n\n\n\n## :loudspeaker: News\n\n- **[2025/04]** Code of RoboDual released. Check it out!\n- **[2024/10]** We released our paper on [arXiv](https://arxiv.org/abs/2410.08001).\n\n\n## :pushpin: TODO list\n\n- [x] Release checkpoints for reproduction (*Scheduled Release Date*: **Mid-April, 2025**)\n\n\n## :video_game: Getting Started \u003ca name=\"installation\"\u003e\u003c/a\u003e\n\n1. (Optional) We use conda to manage the environment.\n\n```bash\nconda create -n robodual python=3.10 -y\nconda activate robodual\n```\n\n2. Install dependencies.\n\n```bash\n# Install pytorch\n# Look up https://pytorch.org/get-started/previous-versions/ with your cuda version for a correct command\npip install torch torchvision torchaudio\n\n# Clone our repo and pip install to download dependencies\ngit clone git@github.com:OpenDriveLab/RoboDual.git\ncd robodual\npip install -e .\n\n# Install Flash Attention 2 for training (https://github.com/Dao-AILab/flash-attention)\npip install packaging ninja\nninja --version; echo $?  # Verify Ninja --\u003e should return exit code \"0\"\npip install \"flash-attn==2.5.5\" --no-build-isolation\n```\n\n3. Install CALVIN simulator.\n\n```bash\ngit clone --recurse-submodules https://github.com/mees/calvin.git\nexport CALVIN_ROOT=$(pwd)/calvin\ncd $CALVIN_ROOT\nsh install.sh\n```\n\n## :star: Model Checkpoints\n\n- Generalist Policy: [![Model on HF](https://huggingface.co/datasets/huggingface/badges/resolve/main/model-on-hf-sm.svg)](https://huggingface.co/qwbu/RoboDual-OpenVLA-Generalist)\n- Specialist Policy: [![Model on HF](https://huggingface.co/datasets/huggingface/badges/resolve/main/model-on-hf-sm-dark.svg)](https://huggingface.co/qwbu/RoboDual-Specialist)\n\n\n## Experiment on CALVIN\n\n### :ballot_box_with_check: Relevant Files:\n\u003e #### Training\n- vla-scripts/\n  - ```train_generalist_calvin.py```: Train OpenVLA on CALVIN dataset\n  - ```train_specialist_calvin.py```: Train DiT specialist with pre-trained generalist\n- prismatic/vla/datasets/\n  - ```calvin_dataset.py```: Data loader for CALVIN dataset\n\u003e #### Evaluation\n- vla-scripts/\n  - ```evaluate_calvin.py```: Initiate evaluation on CALVIN\n  - ```dual_sys_evaluation.py```: RoboDual-specific core implementation\n\u003e #### Model\n- prismatic/models/policy/:\n  - ```diffusion_policy.py```: Core implementation of our DiT action expert\n\n\n### :one: Generalist Training\n- Our generalist model is built upon OpenVLA, first change ```vla_path``` to your local path of OpenVLA model.\n- By default, we employ parameter efficient fine-tuning with LoRA rank 32.\n- Then initiate triaining with 8 GPUs:\n\n```bash\ntorchrun --standalone --nnodes 1 --nproc-per-node 8 vla-scripts/train_generalist_calvin.py \\\n                                 --dataset_name \"calvin\" \\\n                                 --run_root_dir \"run_log\" \\\n```\n\n### :two: Specialist Training\n- We do not train generalist with the specialist with an end-to-end manner and find it works equally well on CALVIN. To further train generalist, modify ```freeze_slow = False``` in the config.\n- Start training (100k steps) on CALVIN with 8 GPUs:\n```bash\ntorchrun --standalone --nnodes 1 --nproc-per-node 8 vla-scripts/train_spacialist_calvin.py \\\n                                 --num_inference_steps 5 \\       # sampling steps for DiT\n                                 --cond_drop_chance 0.1 \\        # condition drop chance for calssifier-free guidance\n                                 --with_depth True \\             # use depth input\n                                 --with_gripper True \\           # use gripper-view inputs (both RGB and depth)\n                                 --with_tactile True \\           # use visuo-tactile input\n                                 --batch_size 8 \\                # fine-tuning batch size\n                                 --learning_rate 1e-4 \\          # fine-tuning learning rate\n                                 --dataset_name \"calvin\" \\\n                                 --run_root_dir \"run_log\" \\\n```\n\n### :three: Evaluation \u003ca name=\"Evaluation\"\u003e\u003c/a\u003e\n\u003e First set your ```CALVIN_ROOT``` environment variable wtih:\n```bash\nexport CALVIN_ROOT=/path/to/your/calvin_root_path\n```\n- Start evaluation on CALVIN (multi-GPU is also supported):\n```bash\ntorchrun --standalone --nnodes 1 --nproc-per-node 1 vla-scripts/evaluate_calvin.py \\\n                                 --generalist_path \"/path/to/calvin_generalist\" \\\n                                 --specialist_path \"/path/to/calvin_specialist\" \\\n                                 --with_depth \\                 # use depth input\n                                 --with_gripper \\               # use gripper-view inputs (both RGB and depth)\n                                 --with_cfg \\                   # enable classifier-free guidance\n                                 --log_dir calvin\n```\n\u003e Please refer to ```vla-scripts/evaluate_calvin.py``` for all evaluation options.\n\n\n## :pencil: Citation\nIf you find our code or models useful in your work, please cite [our paper](https://arxiv.org/abs/2410.08001):\n\n```bibtex\n@article{bu2024robodual,\n  title={Towards Synergistic, Generalized, and Efficient Dual-System for Robotic Manipulation}, \n  author={Qingwen Bu and Hongyang Li and Li Chen and Jisong Cai and Jia Zeng and Heming Cui and Maoqing Yao and Yu Qiao},\n  journal={arXiv preprint arXiv:2410.08001},\n  year={2024}\n}\n```\n\n## Acknowledgements\n\nWe thank [OpenVLA](https://github.com/openvla/openvla) and [Latte](https://github.com/Vchitect/Latte) for their open-sourced work!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopendrivelab%2Frobodual","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopendrivelab%2Frobodual","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopendrivelab%2Frobodual/lists"}