{"id":20789492,"url":"https://github.com/real-stanford/universal_manipulation_interface","last_synced_at":"2025-04-14T22:14:53.208Z","repository":{"id":224491694,"uuid":"758406180","full_name":"real-stanford/universal_manipulation_interface","owner":"real-stanford","description":"Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots","archived":false,"fork":false,"pushed_at":"2024-12-18T08:03:22.000Z","size":28713,"stargazers_count":849,"open_issues_count":50,"forks_count":160,"subscribers_count":18,"default_branch":"main","last_synced_at":"2025-04-14T22:14:42.454Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://umi-gripper.github.io/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/real-stanford.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-02-16T08:53:11.000Z","updated_at":"2025-04-12T05:37:56.000Z","dependencies_parsed_at":"2024-04-03T07:26:06.963Z","dependency_job_id":"f0ef0ee7-26cc-43c9-b2ae-ecea40b0791d","html_url":"https://github.com/real-stanford/universal_manipulation_interface","commit_stats":null,"previous_names":["real-stanford/universal_manipulation_interface"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/real-stanford%2Funiversal_manipulation_interface","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/real-stanford%2Funiversal_manipulation_interface/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/real-stanford%2Funiversal_manipulation_interface/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/real-stanford%2Funiversal_manipulation_interface/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/real-stanford","download_url":"https://codeload.github.com/real-stanford/universal_manipulation_interface/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248968917,"owners_count":21191162,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-17T15:24:28.712Z","updated_at":"2025-04-14T22:14:53.189Z","avatar_url":"https://github.com/real-stanford.png","language":"Python","funding_links":[],"categories":["Rankings \u0026 Statistics","Python","Library","🕹️ Teleoperation \u0026 Digital Twins"],"sub_categories":["2022"],"readme":"# Universal Manipulation Interface\n\n[[Project page]](https://umi-gripper.github.io/)\n[[Paper]](https://umi-gripper.github.io/#paper)\n[[Hardware Guide]](https://docs.google.com/document/d/1TPYwV9sNVPAi0ZlAupDMkXZ4CA1hsZx7YDMSmcEy6EU/edit?usp=sharing)\n[[Data Collection Instruction]](https://swanky-sphere-ad1.notion.site/UMI-Data-Collection-Tutorial-4db1a1f0f2aa4a2e84d9742720428b4c?pvs=4)\n[[SLAM repo]](https://github.com/cheng-chi/ORB_SLAM3)\n[[SLAM docker]](https://hub.docker.com/r/chicheng/orb_slam3)\n\n\u003cimg width=\"90%\" src=\"assets/umi_teaser.png\"\u003e\n\n[Cheng Chi](http://cheng-chi.github.io/)\u003csup\u003e1,2\u003c/sup\u003e,\n[Zhenjia Xu](https://www.zhenjiaxu.com/)\u003csup\u003e1,2\u003c/sup\u003e,\n[Chuer Pan](https://chuerpan.com/)\u003csup\u003e1\u003c/sup\u003e,\n[Eric Cousineau](https://www.eacousineau.com/)\u003csup\u003e3\u003c/sup\u003e,\n[Benjamin Burchfiel](http://www.benburchfiel.com/)\u003csup\u003e3\u003c/sup\u003e,\n[Siyuan Feng](https://www.cs.cmu.edu/~sfeng/)\u003csup\u003e3\u003c/sup\u003e,\n\n[Russ Tedrake](https://groups.csail.mit.edu/locomotion/russt.html)\u003csup\u003e3\u003c/sup\u003e,\n[Shuran Song](https://www.cs.columbia.edu/~shurans/)\u003csup\u003e1,2\u003c/sup\u003e\n\n\u003csup\u003e1\u003c/sup\u003eStanford University,\n\u003csup\u003e2\u003c/sup\u003eColumbia University,\n\u003csup\u003e3\u003c/sup\u003eToyota Research Institute\n\n## 🛠️ Installation\nOnly tested on Ubuntu 22.04\n\nInstall docker following the [official documentation](https://docs.docker.com/engine/install/ubuntu/) and finish [linux-postinstall](https://docs.docker.com/engine/install/linux-postinstall/).\n\nInstall system-level dependencies:\n```console\n$ sudo apt install -y libosmesa6-dev libgl1-mesa-glx libglfw3 patchelf\n```\n\nWe recommend [Miniforge](https://github.com/conda-forge/miniforge?tab=readme-ov-file#miniforge3) instead of the standard anaconda distribution for faster installation: \n```console\n$ mamba env create -f conda_environment.yaml\n```\n\nActivate environment\n```console\n$ conda activate umi\n(umi)$ \n```\n\n## Running UMI SLAM pipeline\nDownload example data\n```console\n(umi)$ wget --recursive --no-parent --no-host-directories --cut-dirs=2 --relative --reject=\"index.html*\" https://real.stanford.edu/umi/data/example_demo_session/\n```\n\nRun SLAM pipeline\n```console\n(umi)$ python run_slam_pipeline.py example_demo_session\n\n...\nFound following cameras:\ncamera_serial\nC3441328164125    5\nName: count, dtype: int64\nAssigned camera_idx: right=0; left=1; non_gripper=2,3...\n             camera_serial  gripper_hw_idx                                     example_vid\ncamera_idx                                                                                \n0           C3441328164125               0  demo_C3441328164125_2024.01.10_10.57.34.882133\n99% of raw data are used.\ndefaultdict(\u003cfunction main.\u003clocals\u003e.\u003clambda\u003e at 0x7f471feb2310\u003e, {})\nn_dropped_demos 0\n````\nFor this dataset, 99% of the data are useable (successful SLAM), with 0 demonstrations dropped. If your dataset has a low SLAM success rate, double check if you carefully followed our [data collection instruction](https://swanky-sphere-ad1.notion.site/UMI-Data-Collection-Instruction-4db1a1f0f2aa4a2e84d9742720428b4c). \n\nDespite our significant effort on robustness improvement, OBR_SLAM3 is still the most fragile part of UMI pipeline. If you are an expert in SLAM, please consider contributing to our fork of [OBR_SLAM3](https://github.com/cheng-chi/ORB_SLAM3) which is specifically optimized for UMI workflow.\n\nGenerate dataset for training.\n```console\n(umi)$ python scripts_slam_pipeline/07_generate_replay_buffer.py -o example_demo_session/dataset.zarr.zip example_demo_session\n```\n\n## Training Diffusion Policy\nSingle-GPU training. Tested to work on RTX3090 24GB.\n```console\n(umi)$ python train.py --config-name=train_diffusion_unet_timm_umi_workspace task.dataset_path=example_demo_session/dataset.zarr.zip\n```\n\nMulti-GPU training.\n```console\n(umi)$ accelerate --num_processes \u003cngpus\u003e train.py --config-name=train_diffusion_unet_timm_umi_workspace task.dataset_path=example_demo_session/dataset.zarr.zip\n```\n\nDownloading in-the-wild cup arrangement dataset (processed).\n```console\n(umi)$ wget https://real.stanford.edu/umi/data/zarr_datasets/cup_in_the_wild.zarr.zip\n```\n\nMulti-GPU training.\n```console\n(umi)$ accelerate --num_processes \u003cngpus\u003e train.py --config-name=train_diffusion_unet_timm_umi_workspace task.dataset_path=cup_in_the_wild.zarr.zip\n```\n\n## 🦾 Real-world Deployment\nIn this section, we will demonstrate our real-world deployment/evaluation system with the cup arrangement policy. While this policy setup only requires a single arm and camera, the our system supports up to 2 arms and unlimited number of cameras.\n\n### ⚙️ Hardware Setup\n1. Build deployment hardware according to our [Hardware Guide](https://docs.google.com/document/d/1TPYwV9sNVPAi0ZlAupDMkXZ4CA1hsZx7YDMSmcEy6EU).\n2. Setup UR5 with teach pendant:\n    * Obtain IP address and update [eval_robots_config.yaml](example/eval_robots_config.yaml)/robots/robot_ip.\n    * In Installation \u003e Payload\n        * Set mass to 1.81 kg\n        * Set center of gravity to (2, -6, 37)mm, CX/CY/CZ.\n    * TCP will be set automatically by the eval script.\n    * On UR5e, switch control mode to remote.\n\n    If you are using Franka, follow this [instruction](franka_instruction.md).\n3. Setup WSG50 gripper with web interface:\n    * Obtain IP address and update [eval_robots_config.yaml](example/eval_robots_config.yaml)/grippers/gripper_ip.\n    * In Settings \u003e Command Interface\n        * Disable \"Use text based Interface\"\n        * Enable CRC\n    * In Scripting \u003e File Manager\n        * Upload [umi/real_world/cmd_measure.lua](umi/real_world/cmd_measure.lua)\n    * In Settings \u003e System\n        * Enable Startup Script\n        * Select `/user/cmd_measure.lua` you just uploaded.\n4. Setup GoPro:\n    * Install GoPro Labs [firmware](https://gopro.com/en/us/info/gopro-labs).\n    * Set date and time.\n    * Scan the following QR code for clean HDMI output \n    \u003cbr\u003e\u003cimg width=\"50%\" src=\"assets/QR-MHDMI1mV0r27Tp60fWe0hS0sLcFg1dV.png\"\u003e\n5. Setup [3Dconnexion SpaceMouse](https://www.amazon.com/3Dconnexion-SpaceMouse-Wireless-universal-receiver/dp/B079V367MM):\n    * Install libspnav `sudo apt install libspnav-dev spacenavd`\n    * Start spnavd `sudo systemctl start spacenavd`\n\n### 🤗 Reproducing the Cup Arrangement Policy ☕\nOur in-the-wild cup arragement policy is trained with the distribution of [\"espresso cup with saucer\"](https://www.amazon.com/s?k=espresso+cup+with+saucer) on Amazon across 30 different locations around Stanford. We created a [Amazon shopping list](https://www.amazon.com/hz/wishlist/ls/Q0T8U2N5U3IU?ref_=wl_share) for all cups used for training. We published the processed [Zarr dataset and](https://real.stanford.edu/umi/data/zarr_datasets) pre-trained [checkpoint](https://real.stanford.edu/umi/data/pretrained_models/) (finetuned CLIP ViT-L backbone).\n\n\u003cimg width=\"90%\" src=\"assets/umi_cup.gif\"\u003e\n\nDownload pre-trained checkpoint.\n```console\n(umi)$ wget https://real.stanford.edu/umi/data/pretrained_models/cup_wild_vit_l_1img.ckpt\n```\n\nGrant permission to the HDMI capture card.\n```console\n(umi)$ sudo chmod -R 777 /dev/bus/usb\n```\n\nLaunch eval script.\n```console\n(umi)$ python eval_real.py --robot_config=example/eval_robots_config.yaml -i cup_wild_vit_l.ckpt -o data/eval_cup_wild_example\n```\nAfter the script started, use your spacemouse to control the robot and the gripper (spacemouse buttons). Press `C` to start the policy. Press `S` to stop.\n\nIf everything are setup correctly, your robot should be able to rotate the cup and placing it onto the saucer, anywhere 🎉\n\nKnown issue ⚠️: The policy doesn't work well under direct sunlight, since the dataset was collected during a rainiy week at Stanford.\n\n### 🤗 Reproducing Policies on ARX X5 Robot Arms\nPlease follow [umi-on-legs](https://github.com/real-stanford/umi-on-legs) for hardware modification and [umi-arx](https://github.com/real-stanford/umi-arx) for detailed policy deployment instructions. \n\n\u003cimg width=\"90%\" src=\"assets/umi_cup_arx.gif\"\u003e\n\n## 🏷️ License\nThis repository is released under the MIT license. See [LICENSE](LICENSE) for additional details.\n\n## 🙏 Acknowledgement\n* Our GoPro SLAM pipeline is adapted from [Steffen Urban](https://github.com/urbste)'s [fork](https://github.com/urbste/ORB_SLAM3) of [OBR_SLAM3](https://github.com/UZ-SLAMLab/ORB_SLAM3).\n* We used [Steffen Urban](https://github.com/urbste)'s [OpenImuCameraCalibrator](https://github.com/urbste/OpenImuCameraCalibrator/) for camera and IMU calibration.\n* The UMI gripper's core mechanism is adpated from [Push/Pull Gripper](https://www.thingiverse.com/thing:2204113) by [John Mulac](https://www.thingiverse.com/3dprintingworld/designs).\n* UMI's soft finger is adapted from [Alex Alspach](http://alexalspach.com/)'s original design at TRI.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Freal-stanford%2Funiversal_manipulation_interface","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Freal-stanford%2Funiversal_manipulation_interface","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Freal-stanford%2Funiversal_manipulation_interface/lists"}