{"id":13840738,"url":"https://github.com/google-research/ravens","last_synced_at":"2025-05-16T08:04:26.024Z","repository":{"id":37535066,"uuid":"321489256","full_name":"google-research/ravens","owner":"google-research","description":"Train robotic agents to learn pick and place with deep learning for vision-based manipulation in PyBullet. Transporter Nets, CoRL 2020.","archived":false,"fork":false,"pushed_at":"2024-07-30T21:24:12.000Z","size":5539,"stargazers_count":596,"open_issues_count":15,"forks_count":100,"subscribers_count":15,"default_branch":"master","last_synced_at":"2025-05-10T04:39:20.425Z","etag":null,"topics":["artificial-intelligence","computer-vision","deep-learning","imitation-learning","manipulation","openai-gym","pick-and-place","pybullet","rearrangement","reinforcement-learning","robotics","tensorflow","transporter-nets","vision"],"latest_commit_sha":null,"homepage":"https://transporternets.github.io","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/google-research.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-12-14T22:31:07.000Z","updated_at":"2025-05-08T07:38:12.000Z","dependencies_parsed_at":"2024-12-22T17:00:53.528Z","dependency_job_id":"45c024b4-5652-4ece-8c65-a3953495f042","html_url":"https://github.com/google-research/ravens","commit_stats":{"total_commits":28,"total_committers":7,"mean_commits":4.0,"dds":0.6428571428571428,"last_synced_commit":"4752ffab487ebd387781ad7890aa540ae2780970"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Fravens","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Fravens/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Fravens/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Fravens/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/google-research","download_url":"https://codeload.github.com/google-research/ravens/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254493378,"owners_count":22080126,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["artificial-intelligence","computer-vision","deep-learning","imitation-learning","manipulation","openai-gym","pick-and-place","pybullet","rearrangement","reinforcement-learning","robotics","tensorflow","transporter-nets","vision"],"created_at":"2024-08-04T17:00:54.734Z","updated_at":"2025-05-16T08:04:21.016Z","avatar_url":"https://github.com/google-research.png","language":"Python","funding_links":[],"categories":["[PyBullet](https://github.com/bulletphysics/bullet3)"],"sub_categories":[],"readme":"# Ravens - Transporter Networks\n\nRavens is a collection of simulated tasks in PyBullet for learning vision-based robotic manipulation, with emphasis on pick and place.\nIt features a Gym-like API with 10 tabletop rearrangement tasks, each with (i) a scripted oracle that provides expert demonstrations (for imitation learning), and (ii) reward functions that provide partial credit (for reinforcement learning).\n\n\u003cimg src=\"docs/tasks.png\"\u003e\u003cbr\u003e\n\n(a) **block-insertion**: pick up the L-shaped red block and place it into the L-shaped fixture.\u003cbr\u003e\n(b) **place-red-in-green**: pick up the red blocks and place them into the green bowls amidst other objects.\u003cbr\u003e\n(c) **towers-of-hanoi**: sequentially move disks from one tower to another—only smaller disks can be on top of larger ones.\u003cbr\u003e\n(d) **align-box-corner**: pick up the randomly sized box and align one of its corners to the L-shaped marker on the tabletop.\u003cbr\u003e\n(e) **stack-block-pyramid**: sequentially stack 6 blocks into a pyramid of 3-2-1 with rainbow colored ordering.\u003cbr\u003e\n(f) **palletizing-boxes**: pick up homogeneous fixed-sized boxes and stack them in transposed layers on the pallet.\u003cbr\u003e\n(g) **assembling-kits**: pick up different objects and arrange them on a board marked with corresponding silhouettes.\u003cbr\u003e\n(h) **packing-boxes**: pick up randomly sized boxes and place them tightly into a container.\u003cbr\u003e\n(i) **manipulating-rope**: rearrange a deformable rope such that it connects the two endpoints of a 3-sided square.\u003cbr\u003e\n(j) **sweeping-piles**: push piles of small objects into a target goal zone marked on the tabletop.\u003cbr\u003e\n\nSome tasks require generalizing to unseen objects (d,g,h), or multi-step sequencing with closed-loop feedback (c,e,f,h,i,j).\n\n**Team:** this repository is developed and maintained by [Andy Zeng](https://andyzeng.github.io/), [Pete Florence](http://www.peteflorence.com/), [Daniel Seita](https://people.eecs.berkeley.edu/~seita/), [Jonathan Tompson](https://jonathantompson.github.io/), and [Ayzaan Wahid](https://www.linkedin.com/in/ayzaan-wahid-21676148/). This is the reference repository for the paper:\n\n### Transporter Networks: Rearranging the Visual World for Robotic Manipulation\n[Project Website](https://transporternets.github.io/)\u0026nbsp;\u0026nbsp;•\u0026nbsp;\u0026nbsp;[PDF](https://arxiv.org/pdf/2010.14406.pdf)\u0026nbsp;\u0026nbsp;•\u0026nbsp;\u0026nbsp;Conference on Robot Learning (CoRL) 2020\n\n*Andy Zeng, Pete Florence, Jonathan Tompson, Stefan Welker, Jonathan Chien, Maria Attarian, Travis Armstrong,\u003cbr\u003eIvan Krasin, Dan Duong, Vikas Sindhwani, Johnny Lee*\n\n**Abstract.** Robotic manipulation can be formulated as inducing a sequence of spatial displacements: where the space being moved can encompass an object, part of an object, or end effector. In this work, we propose the Transporter Network, a simple model architecture that rearranges deep features to infer spatial displacements from visual input—which can parameterize robot actions. It makes no assumptions of objectness (e.g. canonical poses, models, or keypoints), it exploits spatial symmetries, and is orders of magnitude more sample efficient than our benchmarked alternatives in learning vision-based manipulation tasks: from stacking a pyramid of blocks, to assembling kits with unseen objects; from manipulating deformable ropes, to pushing piles of small objects with closed-loop feedback. Our method can represent complex multi-modal policy distributions and generalizes to multi-step sequential tasks, as well as 6DoF pick-and-place. Experiments on 10 simulated tasks show that it learns faster and generalizes better than a variety of end-to-end baselines, including policies that use ground-truth object poses. We validate our methods with hardware in the real world.\n\n## Installation\n\n**Step 1.** Recommended: install [Miniconda](https://docs.conda.io/en/latest/miniconda.html) with Python 3.7.\n\n```shell\ncurl -O https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh\nbash Miniconda3-latest-Linux-x86_64.sh -b -u\necho $'\\nexport PATH=~/miniconda3/bin:\"${PATH}\"\\n' \u003e\u003e ~/.profile  # Add Conda to PATH.\nsource ~/.profile\nconda init\n```\n\n**Step 2.** Create and activate Conda environment, then install GCC and Python packages.\n\n```shell\ncd ~/ravens\nconda create --name ravens python=3.7 -y\nconda activate ravens\nsudo apt-get update\nsudo apt-get -y install gcc libgl1-mesa-dev\npip install -r requirements.txt\npython setup.py install --user\n```\n\n**Step 3.** Recommended: install GPU acceleration with NVIDIA [CUDA](https://developer.nvidia.com/cuda-toolkit) 10.1 and [cuDNN](https://developer.nvidia.com/cudnn) 7.6.5 for Tensorflow.\n```shell\n./oss_scripts/install_cuda.sh  #  For Ubuntu 16.04 and 18.04.\nconda install cudatoolkit==10.1.243 -y\nconda install cudnn==7.6.5 -y\n```\n\n### Alternative: Pure `pip`\n\nAs an example for Ubuntu 18.04:\n\n```shell\n./oss_scipts/install_cuda.sh  #  For Ubuntu 16.04 and 18.04.\nsudo apt install gcc libgl1-mesa-dev python3.8-venv\npython3.8 -m venv ./venv\nsource ./venv/bin/activate\npip install -U pip\npip install scikit-build\npip install -r ./requirements.txt\nexport PYTHONPATH=${PWD}\n```\n\n## Getting Started\n\n**Step 1.** Generate training and testing data (saved locally). Note: remove `--disp` for headless mode.\n\n```shell\npython ravens/demos.py --assets_root=./ravens/environments/assets/ --disp=True --task=block-insertion --mode=train --n=10\npython ravens/demos.py --assets_root=./ravens/environments/assets/ --disp=True --task=block-insertion --mode=test --n=100\n```\n\nTo run with shared memory, open a separate terminal window and run `python3 -m pybullet_utils.runServer`. Then add `--shared_memory` flag to the command above.\n\n**Step 2.** Train a model e.g., Transporter Networks model. Model checkpoints are saved to the `checkpoints` directory. Optional: you may exit training prematurely after 1000 iterations to skip to the next step.\n\n```shell\npython ravens/train.py --task=block-insertion --agent=transporter --n_demos=10\n```\n\n**Step 3.** Evaluate a Transporter Networks agent using the model trained for 1000 iterations. Results are saved locally into `.pkl` files.\n\n```shell\npython ravens/test.py --assets_root=./ravens/environments/assets/ --disp=True --task=block-insertion --agent=transporter --n_demos=10 --n_steps=1000\n```\n\n**Step 4.** Plot and print results.\n\n```shell\npython ravens/plot.py --disp=True --task=block-insertion --agent=transporter --n_demos=10\n```\n\n**Optional.** Track training and validation losses with Tensorboard.\n\n```shell\npython -m tensorboard.main --logdir=logs  # Open the browser to where it tells you to.\n```\n\n## Datasets and Pre-Trained Models\n\nDownload our generated train and test datasets and pre-trained models.\n\n```shell\nwget https://storage.googleapis.com/ravens-assets/checkpoints.zip\nwget https://storage.googleapis.com/ravens-assets/block-insertion.zip\nwget https://storage.googleapis.com/ravens-assets/place-red-in-green.zip\nwget https://storage.googleapis.com/ravens-assets/towers-of-hanoi.zip\nwget https://storage.googleapis.com/ravens-assets/align-box-corner.zip\nwget https://storage.googleapis.com/ravens-assets/stack-block-pyramid.zip\nwget https://storage.googleapis.com/ravens-assets/palletizing-boxes.zip\nwget https://storage.googleapis.com/ravens-assets/assembling-kits.zip\nwget https://storage.googleapis.com/ravens-assets/packing-boxes.zip\nwget https://storage.googleapis.com/ravens-assets/manipulating-rope.zip\nwget https://storage.googleapis.com/ravens-assets/sweeping-piles.zip\n```\n\nThe MDP formulation for each task uses transitions with the following structure:\n\n**Observations:** raw RGB-D images and camera parameters (pose and intrinsics).\n\n**Actions:** a primitive function (to be called by the robot) and parameters.\n\n**Rewards:** total sum of rewards for a successful episode should be =1.\n\n**Info:** 6D poses, sizes, and colors of objects.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogle-research%2Fravens","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgoogle-research%2Fravens","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogle-research%2Fravens/lists"}