{"id":20789497,"url":"https://github.com/real-stanford/flingbot","last_synced_at":"2025-05-05T19:45:32.329Z","repository":{"id":41573018,"uuid":"354453664","full_name":"real-stanford/flingbot","owner":"real-stanford","description":"[CoRL 2021 Best System Paper] This repository contains code for training and evaluating FlingBot in both simulation and real-world settings on a dual-UR5 robot arm setup for Ubuntu 18.04","archived":false,"fork":false,"pushed_at":"2021-10-13T13:25:48.000Z","size":20402,"stargazers_count":119,"open_issues_count":7,"forks_count":25,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-03-30T23:11:20.883Z","etag":null,"topics":["cloth-simulation","computer-vision","robotics"],"latest_commit_sha":null,"homepage":"https://flingbot.cs.columbia.edu/","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/real-stanford.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-04-04T04:17:42.000Z","updated_at":"2025-03-16T07:14:07.000Z","dependencies_parsed_at":"2022-08-04T02:30:24.027Z","dependency_job_id":null,"html_url":"https://github.com/real-stanford/flingbot","commit_stats":null,"previous_names":["real-stanford/flingbot"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/real-stanford%2Fflingbot","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/real-stanford%2Fflingbot/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/real-stanford%2Fflingbot/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/real-stanford%2Fflingbot/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/real-stanford","download_url":"https://codeload.github.com/real-stanford/flingbot/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252564327,"owners_count":21768635,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cloth-simulation","computer-vision","robotics"],"created_at":"2024-11-17T15:24:35.698Z","updated_at":"2025-05-05T19:45:32.303Z","avatar_url":"https://github.com/real-stanford.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ch1\u003e FlingBot: The Unreasonable Effectiveness of Dynamic Manipulations for Cloth Unfolding\u003c/h1\u003e\n\u003cdiv style=\"text-align: center;\"\u003e\n\n[Huy Ha](https://www.cs.columbia.edu/~huy/),  [Shuran Song](https://www.cs.columbia.edu/~shurans/)\n\nColumbia University, New York, NY, United States\n\n[Conference on Robot Learning 2021](https://www.robot-learning.org/) (Oral Presentation)\n\n[Project Page](https://flingbot.cs.columbia.edu/) | [Video](https://www.youtube.com/watch?v=T4tDy5y_6ZM\u0026ab_channel=ColumbiaArtificialIntelligenceandRobotics) | [Arxiv](https://arxiv.org/abs/2105.03655)\n\n\u003c/div\u003e\n\n\u003cimg style=\"left-margin:50px; right-margin:50px;\" src=\"assets/fling-teaser.png\"\u003e\n\n\u003cdiv style=\"margin:50px; text-align: justify;\"\u003e\nHigh-velocity dynamic actions (e.g., fling or throw) play a crucial role in our everyday interaction with deformable objects by improving our efficiency and effectively expanding our physical reach range.\nYet, most prior works have tackled cloth manipulation using exclusively single-arm quasi-static actions, which requires a large number of interactions for challenging initial cloth configurations and strictly limits the maximum cloth size by the robot's reach range.\nIn this work, we demonstrate the effectiveness of dynamic flinging actions for cloth unfolding with our proposed self-supervised learning framework, FlingBot.\nOur approach learns how to unfold a piece of fabric from arbitrary initial configurations using a pick, stretch, and fling primitive for a dual-arm setup from visual observations.\nThe final system achieves over 80% coverage within 3 actions on novel cloths,  can unfold cloths larger than the system's reach range, and generalizes to T-shirts despite being trained on only rectangular cloths.\nWe also finetuned FlingBot on a real-world dual-arm robot platform, where it increased the cloth coverage over 4 times more than the quasi-static baseline did.\nThe simplicity of FlingBot combined with its superior performance over quasi-static baselines demonstrates the effectiveness of dynamic actions for deformable object manipulation.\n\u003c/div\u003e\n\n\u003cbr\u003e\n\nThis repository contains code for training and evaluating [FlingBot](https://flingbot.cs.columbia.edu/) in both simulation and real-world settings on a dual-UR5 robot arm setup for Ubuntu 18.04.\nIt has been tested on machines with Nvidia GeForce 1080 Ti and GeForce RTX 2080 Ti.\n\n\n\nIf you find this codebase useful, consider citing:\n\n\n\u003cdiv style=\"display:flex;\"\u003e\n\u003cdiv\u003e\n\n```\n@inproceedings{ha2021flingbot,\n\ttitle={FlingBot: The Unreasonable Effectiveness of Dynamic Manipulation for Cloth Unfolding},\n\tauthor={Ha, Huy and Song, Shuran},\n\tbooktitle={Conference on Robotic Learning (CoRL)},\n\tyear={2021}\n}\n```\n\n\u003c/div\u003e\n\u003cimg src=\"assets/flingbot.gif\" width=160px height=160px/\u003e\n\u003c/div\u003e\n\nIf you have any questions, please contact [me](https://www.cs.columbia.edu/~huy/) at `huy [at] cs [dot] columbia [dot] edu`.\n\n\n# Table of Contents\n- 1 [Simulation](#simulation)\n  - 1.1 [Setup](#setup)\n  - 1.2 [Evaluate FlingBot](#evaluate-flingbot)\n  - 1.3 [Train FlingBot](#train-flingbot)\n  - 1.4 [Generating new tasks](#generating-new-tasks)\n- 2 [Real World](#real-world)\n  - 2.1 [Real world setup](#real-world-setup)\n  - 2.2 [Running Flingbot in the real world](#running-real-world-flingbot)\n\n# Simulation\n## Setup\n\nThis section walks you through setting up the CUDA accelerated cloth simulation environment.\nTo start, install [Blender](https://www.blender.org/download/), [docker](https://docs.docker.com/engine/install/ubuntu/) and [nvidia-docker](https://github.com/NVIDIA/nvidia-docker#quickstart).\n\n### Python Dependencies\n\nWe have prepared a conda YAML file which contains all the python dependencies.\n\n```\nconda env create -f flingbot.yml\n```\n\n### Compiling the simulator\n\nThis codebases uses a CUDA accelerated cloth simulator which can load any arbitrary mesh to train a cloth unfolding policy.\nThe simulator is a fork of [PyFlex](https://github.com/YunzhuLi/PyFleX) from [Softgym](https://github.com/Xingyu-Lin/softgym), and requires a GPU to run.\nWe have provided a Dockerfile in this repo for compiling and using this simulation environment for training in Docker.\n\n```\ndocker build -t flingbot .\n```\n\nTo launch the docker container, go to this repo's root directory, then run\n```\nexport FLINGBOT_PATH=${PWD}\nnvidia-docker run \\\n\t-v $FLINGBOT_PATH:/workspace/flingbot\\\n\t-v /path/to/your/anaconda3:/path/to/your/anaconda3\\\n\t--gpus all --shm-size=64gb  -d -e DISPLAY=$DISPLAY -e QT_X11_NO_MITSHM=1 -it flingbot\n```\nYou might need to change `--shm-size` appropriately for your system.\n\nAdd conda to PATH, then activate flingbot\n```sh\nexport PATH=/path/to/your/anaconda3/bin:$PATH\nconda init bash\nsource ~/.bashrc\nconda activate flingbot\n``` \nThen, at the root of this repo inside the docker container, compile the simulator with \n```\n. ./prepare.sh \u0026\u0026 ./compile.sh\n```\n*NOTE: Always make sure you're in the correct conda environment before running these two shell scripts*.\n\nThe compilation will result in a `.so` shared object file. \n`./prepare.sh` sets the environment variables needed for compilation and also tells the python interpreter to look into the build directory containing the compiled `.so` file.\n\nAfter this `.so` object file is created, you should be able to run experiments outside of docker as well as inside.\nIn my experience as well as [other's in the community](https://github.com/Xingyu-Lin/softgym/issues/5), docker is best used *only for compilation* and usually fails for running experiments.\nIf you experience this, try taking the compiled `.so` file and running the python commands in the sections to follow *outside of docker*.\nMake sure you set the environment variables correctly using `./prepare.sh`.\nNot setting `$PYTHONPATH` correctly will result in `ModuleNotFoundError: No module named 'pyflex'` when you try to `import pyflex`.\n\nYou can check out [Softgym's Docker guide](https://github.com/Xingyu-Lin/softgym/blob/master/docker/docker.md) and [Daniel Seita](https://danieltakeshi.github.io/)'s [blog post](https://danieltakeshi.github.io/2021/02/20/softgym/) on installing PyFlex with Docker for more information.\n\n## Evaluate FlingBot\n\nIn the repo's root, download the pretrained weights\n```sh\nwget https://flingbot.cs.columbia.edu/data/flingbot.pth\n```\nAs described in the paper, we evaluate FlingBot on 3 held-out different evaluation datasets.\nFirst are normal cloths, which contain rectangular cloths from the same distribution as the training dataset.\nSecond are large cloths, which also rectangular cloths like normal cloths, but larger than the system's reach range.\nThird are shirts, which are completely unseen during training.\n\n![](assets/sim-result.png)\n\nTo download the evaluation datasets\n```sh\nwget https://flingbot.cs.columbia.edu/data/flingbot-normal-rect-eval.hdf5\nwget https://flingbot.cs.columbia.edu/data/flingbot-large-rect-eval.hdf5\nwget https://flingbot.cs.columbia.edu/data/flingbot-shirt-eval.hdf5\n```\nTo evaluate FlingBot on one of the evaluation datasets, pass their respective paths\n\n```sh\npython run_sim.py --eval --tasks flingbot-normal-rect-eval.hdf5 --load flingbot.pth --num_processes 1 --gui\n```\nYou can remove `--gui` to run headless, and use more parallel environments with `--num_processes 16`.\nSince the simulator is hardware accelerated, the maximum `--num_processes` you can set will be limited by how much memory your GPU have.\nYou can also add `--dump_visualizations` to get videos of the episodes.\n\nThe output of evaluation is a directory whose name is prefixed with the checkpoint name (i.e.: `flingbot_eval_X` in the the example), which contains a `replay_buffer.hdf5`.\nYou can print the summary statistics and dump visualizations\n```\npython visualize.py flingbot_eval_X/replay_buffer.hdf5\ncd flingbot_eval_X\npython -m http.server 8080\n```\nThe last command starts a webserver rooted at `flingbot_eval_X` so you can view the visualizations on your web browser at `localhost:8080`.\n\n## Train Flingbot\n\nIn the repo's root, download the training tasks\n\n```sh\nwget https://flingbot.cs.columbia.edu/data/flingbot-rect-train.hdf5\n```\nThen train the model from scratch with\n```sh\npython run_sim.py --tasks_path flingbot-rect-train.hdf5 --num_processes 16 --log flingbot-train-from-scratch --action_primitives fling\n```\nMake sure to change `num_processes` appropriately for your GPU memory capacity.\nYou can also change `action_primitive` to any subset of `['fling', 'stretchdrag', 'drag', 'place']`. \n![](assets/primitives.png)\nFor instance, to train an unfolding policy which uses `fling` and `drag` at the same time, use `--action_primitive fling drag`.\n\n### Cloth renderer\n\nIn our paper, we use Blender to render cloths with domain randomization to help with sim2real transfer.\nHowever, training with Blender is much slower due to the over head of launching a rull rendering engine as a subprocess.\n\nWe also provide the option of rendering with opengl within PyFlex with `--render_engine opengl`.\nWe recommend using this option if domain randomization is not necessary.\n\n*Note: that the results reported in our paper were with `--render_engine blender`.*\n\nWe prefer using the Eevee engine over Cycles in Blender, since we require faster training time but do not need ray-traced images.\nHowever, because Eevee does not support headless rendering, you will need a virtual desktop environment if you plan to run the codebase on a headless server.\n\n## Generating new tasks\n\nYou can also generate new cloth unfolding task datasets.\nTo generate a normal rect dataset\n```sh\npython environment/tasks.py --path new-normal-rect-tasks.hdf5 --num_processes 16 --num_tasks 200 --cloth_type square --min_cloth_size 64 --max_cloth_size 104\n```\nwhere min and max cloth size is measured in number of particles.\nSince the default particle radius is `0.00625m`, `64-104` particles per edge gives a `0.4m-0.65m` edge length.\nSimilarly, to generate a large rect dataset\n```sh\npython environment/tasks.py --path new-large-rect-tasks.hdf5 --num_processes 16 --num_tasks 200 --cloth_type square --min_cloth_size 64 --max_cloth_size 120 --strict_min_edge_length 112\n```\nwhere a `strict_min_edge_length` of 112 ensures that at least one edge is greater than the system's physical reach range `112 * 0.00625m = 0.7m`.\nThis physical limit is imposed with a max lift height and max stretch distance.\nTo regenerate an unfolding task dataset with shirts similar to ours, download our version of Cloth3D, where all jackets have been filtered out and all meshes have been appropriately processed\n```sh\nwget  https://flingbot.cs.columbia.edu/data/cloth3d.tar.gz\ntar xvzf cloth3d.tar.gz\n```\nThen run the following command\n```sh\npython environment/tasks.py --path new-shirt-tasks.hdf5.hdf5 --num_processes 16 --num_tasks 200 --cloth_type mesh --cloth_mesh_path cloth3d/val\n```\n\nYou can replace `--cloth_mesh_path` with any directory containing *only quad meshes*.\nTo achieve the best simulation quality, make sure edge lengths in the meshes are roughly the particle radius (`0.00625m` by default).\n\n# Real World\n\nOur real world system uses 2 UR5 (CM3) arms, one equipped with an OnRobot RG2 gripper and one with a Schunk WSG50.\nWe modify the Schunk WSG50 finger tip with a rubber tip for better cloth pinch grasping.\nWe use 2 RGB-D cameras, an Azure Kinect v3 for the top down camera and an Intel Realsense D415.\n![](assets/real_world_setup.jpg)\nYou can setup the IP addresses of the robots and cameras inside `real_world/setup.py`\n\n## Real world setup \n### Cloth dataset\n\nThe real world testing cloth dataset is stored in a dictionary called `cloths` at the top of `real_world/realWorldEnv.py`.\nTo use your own cloths, add a dictionary item to `cloths` which contains `flattened_area` (as measured with `compute_coverage()`), `cloth_size` and `mass`.\nOne entry has been left in there as an example.\nIn each experiment, you can select the cloth you're currently using with `CURRENT_CLOTH`.\n\n### Workspace background segmentation \n\nIn FlingBot, we use domain randomization on the background texture in Blender to help with sim2real transfer.\nHowever, a simpler solution is to train a simulation model to work on only black backgrounds, then filter out the background in real.\nSince object-background segmentation is a solved problem, this hack feels acceptable in the context of getting high speed flinging to work.\nTo use this, you can set `replace_background=True` in `RealWorldEnv`'s constructor.\nThis allows a simulation model trained with an OpenGL rendering engine (such that background domain randomization is absent) to also be transferred to real.\nNote that for the results we report in the paper, we use `replace_background=False.` and Blender as our rendering engine.\n\n\n### Stretching primitive variables\n\nTo mitigate the use of expensive and inaccessible force sensors, we choose to implement our stretching primitive using vision only.\nThe stretching primitive and variables are defined in `real_world/stretch.py`.\nThe closed-loop procedure takes an RGB-D image at every iteration, determines whether the cloth has been stretched, then halting if it has and continues to stretch otherwise.\nThe stretching detector is implemented by finding the horizontal line at the top of the cloth mask then checking if it is straight.\n\nYou can set the 3 stretching variables as follows:\n - `FOREGROUND_BACKGROUND_DIST`: the depth distance from the front camera such that both arms and the cloth are closer than this distance after a grasp.\n - `GRIPPER_LINE`: The y (vertical) coordinate in the front camera image where the middle of the gripper is after a grasp.\n - `CLOTH_LINE`: The y (vertical) coordinate in the front camera image where the cloth is (just below the tips of the gripper) after a grasp.\nWhile setting up the stretching variables, you should set `debug=True` in `is_cloth_stretched()`.\n\nThis stretching primitive assumes the front camera is approximately `0.8m` away from the center of the arms, with a front view of the arms.\nMake sure the front camera is always centered as so. \nIf the distance of the front camera from the two arms is different, you can set  `FOREGROUND_BACKGROUND_DIST` accordingly.\n\n## Running Flingbot in the real world\n\n### Camera calibration\n\nFlingBot policies actions encoded as a pixel.\nTo find and grasp the 3D position in the real world setup of a chosen pixel in the RGB-D image, the camera pose relative to the arms' bases can be calibrated with\n```sh\npython calibrate_camera.py\n```\nThe output of this script are a depth scaling and the relative poses of the camera to the right arm and left arm, saved as `camera_depth_scale.txt`,  `top_down_left_ur5_cam_pose.txt`,  and `top_down_right_ur5_cam_pose.txt` respectively.\n\nI recommend rerunning this script everytime you run real world experiments.\n\n### Loading a simulation trained model\n\nTo load a simulation model whose path is at `./flingbot.pth` and run the real world setup,\n```sh\npython run_real_world.py --load flingbot.pth --warmup 128 --log logs/flingbot-real-world\n```\nwhere `--warmup` specifies the number of real world data points to collect before the model begins finetuning.\nLike all simulation commands, you'll need to run `. ./prepare.sh` before real world commands.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Freal-stanford%2Fflingbot","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Freal-stanford%2Fflingbot","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Freal-stanford%2Fflingbot/lists"}