{"id":20789481,"url":"https://github.com/real-stanford/busybot","last_synced_at":"2025-05-05T19:50:24.017Z","repository":{"id":67492630,"uuid":"550307451","full_name":"real-stanford/busybot","owner":"real-stanford","description":"[CoRL 2022] BusyBot: Learning to Interact, Reason, and Plan in a BusyBoard Environment","archived":false,"fork":false,"pushed_at":"2023-05-17T20:13:55.000Z","size":4441,"stargazers_count":18,"open_issues_count":0,"forks_count":1,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-03-30T23:11:27.266Z","etag":null,"topics":["manipulation","scene-graph","simulation","task-planning"],"latest_commit_sha":null,"homepage":"https://busybot.cs.columbia.edu/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/real-stanford.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2022-10-12T14:38:27.000Z","updated_at":"2024-08-08T15:32:15.000Z","dependencies_parsed_at":"2023-09-17T03:56:56.771Z","dependency_job_id":null,"html_url":"https://github.com/real-stanford/busybot","commit_stats":{"total_commits":6,"total_committers":3,"mean_commits":2.0,"dds":0.5,"last_synced_commit":"b2a955155546b74add255b9aa9b0d8fcd27f9b3f"},"previous_names":["real-stanford/busybot","columbia-ai-robotics/busybot"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/real-stanford%2Fbusybot","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/real-stanford%2Fbusybot/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/real-stanford%2Fbusybot/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/real-stanford%2Fbusybot/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/real-stanford","download_url":"https://codeload.github.com/real-stanford/busybot/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252565604,"owners_count":21768889,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["manipulation","scene-graph","simulation","task-planning"],"created_at":"2024-11-17T15:24:19.632Z","updated_at":"2025-05-05T19:50:24.001Z","avatar_url":"https://github.com/real-stanford.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# BusyBot: Learning to Interact, Reason, and Plan in a BusyBoard Environment\n\n[Zeyi Liu](https://lzylucy.github.io/), [Zhenjia Xu](https://www.zhenjiaxu.com/), [Shuran Song](https://www.cs.columbia.edu/~shurans/)\n\u003cbr\u003e\nColumbia University, New York, NY, United States\n\u003cbr\u003e\n[Conference on Robot Learning 2022](https://www.robot-learning.org/)\n\n### [Project Page](https://busybot.cs.columbia.edu/) | [Video](https://www.youtube.com/watch?v=EJ98xBJZ9ek) | [arXiv](https://arxiv.org/abs/2207.08192)\n\n\u003cimg style=\"left-margin:50px; right-margin:50px;\" src=\"teaser.png\"\u003e\n\n## Dependencies\nWe have prepared a conda YAML file which contains all the python dependencies.\n\n```\nconda env create -f environment.yml\n```\n\n\n## Data and Model\nDownload the pre-trained interaction and reasoning model from [Google Drive](https://drive.google.com/drive/folders/1fnQ_1I7wl9dykMarMxVPaaL_VLR7vaea?usp=sharing), and place the models under ```interact/pre-trained/``` and ```reason/pre-trained/``` respectively.\n\nDownload the interaction data for training the reasoning model and place the ```data``` directory under ```interact/```. The full dataset (~86G) can be downloaded from [here](https://busybot.cs.columbia.edu/data/data.zip), which contains all RGB images of the interaction sequences. We also provide a smaller alternative of the dataset with no images but pre-extracted image features on [Google Drive](https://drive.google.com/file/d/1WKdz6Tjx9_4Ga98iGX_ddR48dD6YiaQG/view?usp=sharing).\nYou can also generate your own interaction data by running the interaction module.\n\nThe full ```data``` directory is organized as follows:\n```\n.\n└── data/\n    ├── train/\n    │   ├── interact-data/  # 10,000 scenes\n    │   │   ├── 1/    # each scene contains 30 frames\n    │   │   │   ├── fig_0.png\n    │   │   │   ├── ...\n    │\t│   │   │   ├── fig_29.png\n    │   │   │   └── data.h5\n    │   │   ├── ...\n    │   │   ├── 10,000/\n    │   │   └── stats.h5\n    │   ├── plan-binary/  # 50 1-to-1 goal-conditioned tasks\n    │   │   ├── 1/\n    │   │   │   ├── fig_0.png\n    │   │   │   ├── ...\n    │   │   │   └── scene_state.pickle\n    │   │   ├── ...\n    │   │   ├── 50/\n    │   │   └── stats.h5\n    │   └── plan-multi/  # 50 1-to-many goal-conditioned tasks\n    │   │   ├── ...\n    │   │   ├── 50/\n    │   │   └── stats.h5\n    ├── valid/\n    │   ├── interact-data/  # 2,000 scenes\n    │   │   ├── ...\n    │   │   ├── 2,000/\n    │   │   └── stats.h5\n    │   ├── plan-binary/\n    │   └── plan-multi/\n    └── unseen/\n        ├── interact-data/  # 2,000 scenes\n        ├── plan-binary/\n        └── plan-multi/\n```\n```data.h5``` contains the image features, object states, actions, object positions, object bounding boxes, object types, and inter-object relations for each scene. ```stats.h5``` aggregates the information for each interaction sequence into a global file for faster loading.\n\n```interact/assets/objects/``` contains the URDF models for all objects. The Door, Lamp, and Switch category are selected from the PartNet-Mobility Dataset from [SAPIEN](https://sapien.ucsd.edu/). The Toy category are borrowed from [UMPNet](https://github.com/columbia-ai-robotics/umpnet). The ```interact/assets/objects/data.json``` file defines the train and test instances. Each object also corresponds to a ```object_meta_info.json``` file that contains basic object information: category, instance id, scale, moveable link, bounding box, cause/effect properties, etc. If you want to add new objects from the PartNet-Mobility Dataset, you can refer to ```interact/data_process.py``` on how to process the data.\n\n## Interaction Module\nTo train the interaction module, run the following command\n```sh\ncd interact\npython train.py --exp {exp_name}\n```\nYou can access the trained models and visualization under ```interact/exp/{exp_name}```.\n\n## Reasoning Module\nTo train the reasoning module, run the following command\n```sh\ncd reason\nbash scripts/train_board.sh --exp {exp_name}\n```\nYou can access the trained models under ```reason/model/{exp_name}```. \n\nFor future state prediction, we evaluate the state accuracy of objects. To map object features extracted from image to object states, we train a decoder for each object category and provide the pre-trained models under ```reason/decoders/```.\n\nTo run a demo of the trained reasoning model given a single interaction sequence, run the following command\n```sh\ncd reason\nbash scripts/demo_board.sh\n```\n\nIf you want to extract image features on your own interaction dataset, we provide a script to do that as well\n```sh\ncd reason\nbash scripts/feature_extract.sh\n```\n\n## Planning Module\nTo run evaluation on goal-conditioned tasks, run the following command\n```sh\ncd plan\nbash scripts/planning.sh\n```\n\n## Acknowledgement\nWe refer to [UMPNet](https://github.com/columbia-ai-robotics/umpnet) by Zhenjia Xu for the interaction module and [V-CDN](https://github.com/pairlab/v-cdn) by Yunzhu Li for the reasoning module when developing this codebase.\n\n## Citation\nIf you find this codebase useful, consider citing:\n\u003cdiv style=\"display:flex;\"\u003e\n\u003cdiv\u003e\n\n```\n@inproceedings{liu2023busybot,\n\ttitle={BusyBot: Learning to Interact, Reason, and Plan in a BusyBoard Environment},\n\tauthor={Liu, Zeyi and Xu, Zhenjia and Song, Shuran},\n\tbooktitle={Conference on Robot Learning},\n\tpages={505--515},\n\tyear={2023},\n\torganization={PMLR}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Freal-stanford%2Fbusybot","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Freal-stanford%2Fbusybot","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Freal-stanford%2Fbusybot/lists"}