https://github.com/hyperplane-lab/rgbmanip

Official implementation of RGBManip (ICRA2024)
https://github.com/hyperplane-lab/rgbmanip

monocular-depth-estimation rl robotics

Last synced: 2 months ago
JSON representation

Official implementation of RGBManip (ICRA2024)

• Host: GitHub
• URL: https://github.com/hyperplane-lab/rgbmanip
• Owner: hyperplane-lab
• License: apache-2.0
• Created: 2024-03-25T08:35:35.000Z (10 months ago)
• Default Branch: main
• Last Pushed: 2024-08-29T13:19:20.000Z (5 months ago)
• Last Synced: 2024-08-29T14:48:12.579Z (5 months ago)
• Topics: monocular-depth-estimation, rl, robotics
• Language: Python
• Homepage: https://rgbmanip.github.io/
• Size: 6.24 MB
• Stars: 12
• Watchers: 1
• Forks: 1
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# Adapose

## Install

Install dependencies using `pip install -r requirements.txt`. The instalation is tested with `setuptools==65.6.3`. Some later versions may not be compatible.

*Potential Dependency Inssues:* Users have reported `Module not Found` error with mplib. This project is developed when the newest version of mplib is `mplib==0.0.8`. Newer versions if mplib may not be compatible with this project. We recommend using `mplib<=0.1.0`

## Download Dataset and Checkpoints

We provided scripts to fully automate dataset and checkpoint downloading from google drive. Just run `sh install.sh` to do the stuff and wait. The total file size to download is around 1GB.

You can also manually download them following the steps:

First, the dataset can be downloaded from [google drive](https://drive.google.com/file/d/154g8SzGFWOcLLjes40aTXFoREX47-ZPk/view?usp=sharing). Download it and decompress it into `downloads/dataset`

Second, download checkpoints for Pose Estimator from [google drive](https://drive.google.com/drive/folders/1WshZaRVllWxHfUFK1--e1hQd_64dAKMz?usp=sharing), and move all four checkpoints into `downloads/pose_estimator/.pth`.

Third, download checkpoints for Global Scheduling Policy from [google drive](https://drive.google.com/drive/folders/1YMI38jeLkPJQa_HJ-RnbQV36zPk4_OvH?usp=sharing), and save all checkpoints into `downloads/global_sheduling_policy/_0.pt`. Notice the `_0` is necessary since our code examines this number. Use `controller.load=CheckpointPath` option to load checkpoints.

## Run Experiments

To run experiments, use command line options: `python train.py dataset=?? task=?? pose_estimator=?? manipulation=?? controller=?? train=??`

Possible options include:

- dataset:

  - cabinet_train

  - cabinet_test

  - drawer_train

  - drawer_test

  - mug_train

  - mug_test

  - pot_train

  - pot_test

  - real_world

- task:

  - open_cabinet

  - open_cabinet_45

  - open_cabinet_no_dr

  - open_drawer

  - open_drawer_30

  - open_drawer_no_dr

  - open_pot

  - pick_mig

  - real_world

- pose_extimator:

  - adapose_cabinet

  - adapose_drawer

  - adapose_mug

  - adapose_pot

  - ground_truth

- manipulation:

  - open_cabinet

  - open_cabinet_open_loop

  - open_drawer

  - open_drawer_open_looop

  - open_pot

  - open_pot_open_loop

  - pick_mug

  - pick_mug_open_loop

- controller

  - gt_pose

  - heuristic_pose

  - rl

- train

  - controller

  - test

  - collect: Used to collect offline data for some baselines.

  - test_baselines: Test with offline actions generated by some baselines.

Some other possible arguments are:

- task.num_envs=X: Number of parallel environments.

- headless=True/False: Whether to show graphics.

- viewerless=True/False: Whether to disable on-hand camera to accelerate some non-vision tasks.

- exp_name=XXX: The name of the experiment (used for saving models and logging).

- controller.load=Checkpoint: Load the saved controller checkpoint.

All other configurations can be modified using the [hydra](https://hydra.cc/docs/intro/) style command line options.

## Miscs

The scripts for drawing the plots included in our paper is also provided in `miscs/plot`.

## Citations

To cite us at

```latex

@article{an2023rgbmanip,

  title={RGBManip: Monocular Image-based Robotic Manipulation through Active Object Pose Estimation},

  author={An, Boshi and Geng, Yiran and Chen, Kai and Li, Xiaoqi and Dou, Qi and Dong, Hao},

  journal={arXiv preprint arXiv:2310.03478},

  year={2023}

}

```