{"id":13439878,"url":"https://github.com/MarkFzp/mobile-aloha","last_synced_at":"2025-03-20T09:30:50.731Z","repository":{"id":215248747,"uuid":"661917172","full_name":"MarkFzp/mobile-aloha","owner":"MarkFzp","description":"Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperation","archived":false,"fork":false,"pushed_at":"2024-06-22T18:27:51.000Z","size":2400,"stargazers_count":4046,"open_issues_count":19,"forks_count":694,"subscribers_count":75,"default_branch":"main","last_synced_at":"2025-03-19T09:26:57.115Z","etag":null,"topics":["imitation-learning","robotics"],"latest_commit_sha":null,"homepage":"https://mobile-aloha.github.io/","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MarkFzp.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-07-04T01:07:37.000Z","updated_at":"2025-03-19T04:46:39.000Z","dependencies_parsed_at":"2024-01-03T11:29:46.179Z","dependency_job_id":"8fc1917f-cca1-48de-aad3-27840a47dc40","html_url":"https://github.com/MarkFzp/mobile-aloha","commit_stats":null,"previous_names":["markfzp/mobile-aloha"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MarkFzp%2Fmobile-aloha","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MarkFzp%2Fmobile-aloha/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MarkFzp%2Fmobile-aloha/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MarkFzp%2Fmobile-aloha/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MarkFzp","download_url":"https://codeload.github.com/MarkFzp/mobile-aloha/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244524860,"owners_count":20466515,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["imitation-learning","robotics"],"created_at":"2024-07-31T03:01:17.836Z","updated_at":"2025-03-20T09:30:50.708Z","avatar_url":"https://github.com/MarkFzp.png","language":"Jupyter Notebook","funding_links":[],"categories":["Jupyter Notebook","Repos","Projects"],"sub_categories":["Humanoids"],"readme":"# Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperation\n\n\n#### Project Website: https://mobile-aloha.github.io/\n\nThis codebase is forked from the [ALOHA repo](https://github.com/tonyzhaozh/aloha), and contains implementation for teleoperation and data collection with the Mobile ALOHA hardware.\nTo build ALOHA, follow the [Hardware Assembly Tutorial](https://docs.google.com/document/d/1_3yhWjodSNNYlpxkRCPIlvIAaQ76Nqk2wsqhnEVM6Dc) and the quick start guide below.\nTo train imitation learning algorithms, you would also need to install [ACT for Mobile ALOHA](https://github.com/MarkFzp/act-plus-plus) which is forked from [ACT](https://github.com/tonyzhaozh/act).\n\n### Repo Structure\n- ``config``: a config for each robot, designating the port they should bind to, more details in quick start guide.\n- ``launch``: a ROS launch file for all 4 cameras and all 4 robots.\n- ``aloha_scripts``: python code for teleop and data collection\n\n## Quick start guide\n\n### Software selection -- OS:\n\nCurrently tested and working configurations: \n- :white_check_mark: Ubuntu 18.04 + ROS 1 noetic\n- :white_check_mark: Ubuntu 20.04 + ROS 1 noetic\n\nOngoing testing (compatibility effort underway):\n- :construction: ROS 2\n- :construction: \u003e= Ubuntu 22.04\n\n### Software installation - ROS:\n1. Install ROS and interbotix software following https://docs.trossenrobotics.com/interbotix_xsarms_docs/\n2. This will create the directory ``~/interbotix_ws`` which contains ``src``.\n3. git clone this repo inside ``~/interbotix_ws/src``\n4. ``source /opt/ros/noetic/setup.sh \u0026\u0026 source ~/interbotix_ws/devel/setup.sh``\n5. ``sudo apt-get install ros-noetic-usb-cam \u0026\u0026 sudo apt-get install ros-noetic-cv-bridge``\n6. run ``catkin_make`` inside ``~/interbotix_ws``, make sure the build is successful\n7. go to ``~/interbotix_ws/src/interbotix_ros_toolboxes/interbotix_xs_toolbox/interbotix_xs_modules/src/interbotix_xs_modules/arm.py``, find function ``publish_positions``.\n   Change ``self.T_sb = mr.FKinSpace(self.robot_des.M, self.robot_des.Slist, self.joint_commands)`` to ``self.T_sb = None``.\n   This prevents the code from calculating FK at every step which delays teleoperation.\n### Hardware installation:\n\nThe goal of this section is to run ``roslaunch aloha 4arms_teleop.launch``, which starts\ncommunication with 4 robots and 3 cameras. It should work after finishing the following steps:\n\nStep 1: Connect 4 robots to the computer via USB, and power on. *Do not use extension cable or usb hub*.\n- To check if the robot is connected, install dynamixel wizard [here](https://emanual.robotis.com/docs/en/software/dynamixel/dynamixel_wizard2/)\n- Dynamixel wizard is a very helpful debugging tool that connects to individual motors of the robot. It allows\nthings such as rebooting the motor (very useful!), torque on/off, and sending commands.\nHowever, it has no knowledge about the kinematics of the robot, so be careful about collisions.\nThe robot *will* collapse if motors are torque off i.e. there is no automatically engaged brakes in joints.\n- Open Dynamixel wizard, go into ``options`` and select:\n  - Protocal 2.0\n  - All ports\n  - 1000000 bps\n  - ID range from 0-10\n- Note: repeat above everytime before you scan.\n- Then hit ``Scan``. There should be 4 devices showing up, each with 9 motors.\n\n\n- One issue that arises is the port each robot binds to can change over time, e.g. a robot that\nis initially ``ttyUSB0`` might suddenly become ``ttyUSB5``. To resolve this, we bind each robot to a fixed symlink\nport with the following mapping:\n  - ``ttyDXL_master_right``: right master robot (master: the robot that the operator would be holding)\n  - ``ttyDXL_puppet_right``: right puppet robot (puppet: the robot that performs the task)\n  - ``ttyDXL_master_left``: left master robot\n  - ``ttyDXL_puppet_left``: left puppet robot\n- Take ``ttyDXL_master_right``: right master robot as an example:\n  1. Find the port that the right master robot is currently binding to, e.g. ``ttyUSB0``\n  2. run ``udevadm info --name=/dev/ttyUSB0 --attribute-walk | grep serial`` to obtain the serial number. Use the first one that shows up, the format should look similar to ``FT6S4DSP``.\n  3. ``sudo vim /etc/udev/rules.d/99-fixed-interbotix-udev.rules`` and add the following line: \n\n         SUBSYSTEM==\"tty\", ATTRS{serial}==\"\u003cserial number here\u003e\", ENV{ID_MM_DEVICE_IGNORE}=\"1\", ATTR{device/latency_timer}=\"1\", SYMLINK+=\"ttyDXL_master_right\"\n\n  4. This will make sure the right master robot is *always* binding to ``ttyDXL_master_right``\n  5. Repeat with the rest of 3 arms.\n- To apply the changes, run ``sudo udevadm control --reload \u0026\u0026 sudo udevadm trigger``\n- If successful, you should be able to find ``ttyDXL*`` in your ``/dev``\n\nStep 2: Set max current for gripper motors\n- Open Dynamixel Wizard, and select the wrist motor for puppet arms. The name of it should be ```[ID:009] XM430-W350```\n- Tip: the LED on the base of robot will flash when it is talking to Dynamixel Wizard. This will help determine which robot is selected. \n- Find ``38 Current Limit``, enter ``300``, then hit ``save`` at the bottom.\n- Repeat this for both puppet robots.\n- This limits the max current through gripper motors, to prevent overloading errors.\n\n\nStep 3: Setup 3 cameras\n- You may use usb hub here, but *maximum 2 cameras per hub for reasonable latency*.\n- To make sure all 3 cameras are binding to a consistent port, similar steps are needed.\n- Cameras are by default binding to ``/dev/video{0, 1, 2...}``, while we want to have symlinks ``{CAM_RIGHT_WRIST, CAM_LEFT_WRIST, CAM_HIGH}``\n- Take ``CAM_RIGHT_WRIST`` as an example, and let's say it is now binding to ``/dev/video0``. run ``udevadm info --name=/dev/video0 --attribute-walk | grep serial`` to obtain it's serial. Use the first one that shows up, the format should look similar to ``0E1A2B2F``.\n- Then ``sudo vim /etc/udev/rules.d/99-fixed-interbotix-udev.rules`` and add the following line \n\n      SUBSYSTEM==\"video4linux\", ATTRS{serial}==\"\u003cserial number here\u003e\", ATTR{index}==\"0\", ATTRS{idProduct}==\"085c\", ATTR{device/latency_timer}=\"1\", SYMLINK+=\"CAM_RIGHT_WRIST\"\n\n- Repeat this for ``{CAM_LEFT_WRIST, CAM_HIGH}`` in additional to ``CAM_RIGHT_WRIST``\n- To apply the changes, run ``sudo udevadm control --reload \u0026\u0026 sudo udevadm trigger``\n- If successful, you should be able to find ``{CAM_RIGHT_WRIST, CAM_LEFT_WRIST, CAM_HIGH}`` in your ``/dev``\n\n\nStep 4: Setup the AgileX Tracer base\n- Connect the base to the computer via the stock CANBUS-to-USB cable, and power on.\n- Install SDK from AgileX\n    ```\n    pip3 install pyagxrobots\n    ```\n- Enable gs_usb kernel module\n   ```\n   sudo modprobe gs_usb\n   ```\n- Bring up the CAN device\n   ```\n   sudo ip link set can0 up type can bitrate 500000\n   ```\n- If no error occured in the previous steps, you should be able to see the can device now by using command\n   ```\n   ifconfig -a\n   ```\n- Install and use can-utils to test the hardware\n   ```\n   sudo apt install can-utils\n   ```\n- Testing commands:\n   ```\n   # receiving data from can0\n   candump can0\n   ```\n\nAt this point, have a new terminal\n    \n    conda deactivate # if conda shows up by default\n    source /opt/ros/noetic/setup.sh \u0026\u0026 source ~/interbotix_ws/devel/setup.sh\n    roslaunch aloha 4arms_teleop.launch\n\nIf no error message is showing up, the computer should be successfully connected to all 3 cameras, all 4 robot arms and the robot base.\n\n#### Trouble shooting\n- Make sure Dynamixel Wizard is disconnected, and no app is using webcam's stream. It will prevent ROS from connecting to\nthese devices.\n\n### Software installation - Conda:\n\n    conda create -n aloha python=3.8.10\n    conda activate aloha\n    pip install torchvision\n    pip install torch\n    pip install pyquaternion\n    pip install pyyaml\n    pip install rospkg\n    pip install pexpect\n    pip install mujoco\n    pip install dm_control\n    pip install opencv-python\n    pip install matplotlib\n    pip install einops\n    pip install packaging\n    pip install h5py\n    pip install tqdm\n    pip install wandb\n\n### Testing teleoperation\n\n**Notice**: Before running the commands below, be sure to place all 4 robots in their sleep positions, and open master robot's gripper. \nAll robots will rise to a height that is easy for teleoperation.\n\n    # ROS terminal\n    conda deactivate\n    source /opt/ros/noetic/setup.sh \u0026\u0026 source ~/interbotix_ws/devel/setup.sh\n    roslaunch aloha 4arms_teleop.launch\n    \n    # Right hand terminal\n    conda activate aloha\n    cd ~/interbotix_ws/src/aloha/aloha_scripts\n    python3 one_side_teleop.py right\n    \n    # Left hand terminal\n    conda activate aloha\n    cd ~/interbotix_ws/src/aloha/aloha_scripts\n    python3 one_side_teleop.py left\n\nThe teleoperation will start when the master side gripper is closed.\n\n\n## Example Usages\n\nTo set up a new terminal, run:\n\n    conda activate aloha\n    cd ~/interbotix_ws/src/aloha/aloha_scripts\n\n\nThe ``one_side_teleop.py`` we ran is for testing teleoperation and has no data collection. To collect data for an episode, run:\n\n    python3 record_episodes.py --dataset_dir \u003cdata save dir\u003e --episode_idx 0\n\nThis will store a hdf5 file at ``\u003cdata save dir\u003e``.\nTo change episode length and other params, edit ``constants.py`` directly.\n\nTo visualize the episode collected, run:\n\n    python3 visualize_episodes.py --dataset_dir \u003cdata save dir\u003e --episode_idx 0\n\nTo replay the episode collected with real robot, run:\n\n    python3 replay_episodes.py --dataset_dir \u003cdata save dir\u003e --episode_idx 0\n\nTo lower 4 robots before e.g. cutting off power, run:\n\n    python3 sleep.py\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMarkFzp%2Fmobile-aloha","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FMarkFzp%2Fmobile-aloha","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMarkFzp%2Fmobile-aloha/lists"}