{"id":20731462,"url":"https://github.com/opendrivelab/mpi","last_synced_at":"2025-10-29T14:30:59.819Z","repository":{"id":240982527,"uuid":"800478564","full_name":"OpenDriveLab/MPI","owner":"OpenDriveLab","description":"[RSS 2024] Learning Manipulation by Predicting Interaction","archived":false,"fork":false,"pushed_at":"2024-08-18T06:45:05.000Z","size":1852,"stargazers_count":99,"open_issues_count":3,"forks_count":1,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-02-07T10:43:58.634Z","etag":null,"topics":["policy-learning","pre-training","robot-manipulation"],"latest_commit_sha":null,"homepage":"https://opendrivelab.com/MPI/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/OpenDriveLab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["OpenDriveLab"],"patreon":null,"open_collective":null,"ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"lfx_crowdfunding":null,"polar":null,"buy_me_a_coffee":null,"custom":null}},"created_at":"2024-05-14T12:12:04.000Z","updated_at":"2025-01-27T05:31:29.000Z","dependencies_parsed_at":"2024-06-04T07:40:21.881Z","dependency_job_id":"25fe7fbc-7d9a-4782-85c4-c62bb3aea76f","html_url":"https://github.com/OpenDriveLab/MPI","commit_stats":null,"previous_names":["opendrivelab/mpi"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenDriveLab%2FMPI","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenDriveLab%2FMPI/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenDriveLab%2FMPI/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenDriveLab%2FMPI/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/OpenDriveLab","download_url":"https://codeload.github.com/OpenDriveLab/MPI/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":238837897,"owners_count":19539080,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["policy-learning","pre-training","robot-manipulation"],"created_at":"2024-11-17T05:14:56.256Z","updated_at":"2025-10-29T14:30:59.806Z","avatar_url":"https://github.com/OpenDriveLab.png","language":"Python","funding_links":["https://github.com/sponsors/OpenDriveLab"],"categories":[],"sub_categories":[],"readme":"\u003e [!IMPORTANT]\n\u003e 🌟 Stay up to date at [opendrivelab.com](https://opendrivelab.com/#news)!\n\n# Learning Manipulation by Predicting Interaction (MPI)\n\n\n\u003ch3 align=\"center\"\u003e\n  \u003ca href=\"https://opendrivelab.com/MPI/\"\u003eProject Website\u003c/a\u003e |\n  \u003ca href=\"https://arxiv.org/abs/2406.00439\"\u003ePaper\u003c/a\u003e |\n  RSS 2024\n\u003c/h3\u003e\n\n\u003cimg width=\"1000\" alt=\"mpi\" src=\"assets/mpi_teaser.png\"\u003e\n\n## :fire: Highlight\n\n​**MPI** is an interaction-oriented representation learning method towards robot manipulation:\n\n- Instruct the model towards predicting transition frames and detecting manipulated objects with keyframes.\n- Foster better comprehension of “how-to-interact” and “where-to-interact”.\n- Acquire more informative representations during pre-training and achieve evident improvement across downstream tasks.\n\n## :movie_camera: Demo\nReal-world robot experiments on complex kitchen environment.\n\n\u003ctable class=\"center\"\u003e\n\u003ctr\u003e\n  \u003ctd style=\"text-align:center;\"\u003e\u003cb\u003eTake the spatula off the shelf (2x speed)\u003c/b\u003e\u003c/td\u003e\n  \u003ctd style=\"text-align:center;\"\u003e\u003cb\u003eLift up the pot lid (2x speed)\u003c/b\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n  \u003ctd\u003e\u003cvideo src=\"https://github.com/OpenDriveLab/MPI/assets/33364294/1652a279-6ec2-4150-97fd-4144b6a55914\" autoplay\u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cvideo src=\"https://github.com/OpenDriveLab/MPI/assets/33364294/818025c9-63fe-4889-af54-1a5e79fd5b8a\" autoplay\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n  \u003ctd style=\"text-align:center;\"\u003e\u003cb\u003eClose the drawer (2x speed)\u003c/b\u003e\u003c/td\u003e\n  \u003ctd style=\"text-align:center;\"\u003e\u003cb\u003ePut pot into sink (2x speed)\u003c/b\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n  \u003ctd\u003e\u003cvideo src=\"https://github.com/OpenDriveLab/MPI/assets/33364294/375edd93-cca6-447a-bc21-ed9d8d8bda77\" autoplay\u003e\u003c/td\u003e\n  \u003ctd\u003e\u003cvideo src=\"https://github.com/OpenDriveLab/MPI/assets/33364294/91697ec0-2414-424e-b1e8-821ffccc71b3\" autoplay\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\n## :rocket: News\n\n- **[2024/05/31]** We released the implementation of pre-training and evaluation on Referring Expression Grounding task. \n- **[2024/06/04]** We released our [paper](https://arxiv.org/abs/2406.00439) on arXiv.\n- **[2024/06/16]** We released the model weights.\n- **[2024/07/05]** We released the evaluation code on Franka Kitchen environment.\n\n## Getting Started \u003ca name=\"start\"\u003e\u003c/a\u003e\n- [Installation](#installation)\n- [Get representation](#representation)\n- [Pre-training](#pretraining)\n- [Evaluation](#evaluation)\n\n### Installation \u003ca name=\"installation\"\u003e\u003c/a\u003e\n\nStep 1. Clone and setup MPI dependency:\n```bash\ngit clone https://github.com/OpenDriveLab/MPI\ncd MPI\npip install -e .\n```\n\nStep 2. Prepare the language model, you may download DistillBERT from [HuggingFace](https://huggingface.co/distilbert/distilbert-base-uncased)\n\n### Get representation \u003ca name=\"representation\"\u003e\u003c/a\u003e\n\n#### :luggage: Checkpoints\nTo directly utilize MPI for extracting representations, please download our pre-trained weights:\n|Model|Checkpoint|Params.|Config|\n|:------:|:------:|:------:|:------:|\n|MPI-Small|[GoogleDrive](https://drive.google.com/file/d/1N7zCWi9ztrcCHsm4xhAA1hsnviv2gdvn/view?usp=drive_link)|22M|[GoogleDrive](https://drive.google.com/file/d/1zG9O9-F86hJowxCUrgpVFfanbIjwn9Tp/view?usp=sharing)|\n|MPI-Base|[GoogleDrive](https://drive.google.com/file/d/1JCpnxYGrrML8hdnMh0UeK6p_XuLNhmdm/view?usp=drive_link)|86M|[GoogleDrive](https://drive.google.com/file/d/1NYiPd72DEjVmErxg5lEHeRxWD0jJKz-0/view?usp=sharing)|\n\n\nYour directory tree should look like this: \n```\ncheckpoints\n├── mpi-small\n|   |—— MPI-small-state_dict.pt  \n|   └── MPI-small.json\n└── mpi-base\n    |—— MPI-base-state_dict.pt    \n    └── MPI-base.json\n```\n\n#### Obtain representation from pretrained MPI\nWe provide a example code [get_representation.py](./get_representation.py) to show how to obtain the pre-trained MPI features. The MPI encoder by default requires two images as input. In downstream tasks, we simply replicate the current observation to ensure compatibility.\n\nThe following diagram presents the composition and arrangement of the extracted tokens:\n\u003cimg width=\"400\" alt=\"tokens_mpi\" src=\"assets/tokens_mpi.jpg\"\u003e\n\n### Pre-training \u003ca name=\"pretraining\"\u003e\u003c/a\u003e\n#### Prepare Pre-training Dataset\nDownload [Ego4D](https://ego4d-data.org/docs/start-here/) Hand-and-Object dataset:\n```\n# Download the CLI\npip install ego4d\n# Select Subset Of Hand-and-Object\npython -m ego4d.cli.cli --output_directory=\u003cpath-to-save-dir\u003e --datasets clips annotations  --metadata --version v2 --benchmarks FHO\n```\nYour directory tree should look like this: \n```\n$\u003cpath-to-save-dir\u003e\n├── ego4d.json\n└── v2\n    |—— annotations  \n    └── clips\n```\nPreprocess dataset for pre-training MPI:\n```\npython prepare_dataset.py --root_path \u003cpath-to-save-dir\u003e/v2/\n```\n\n#### Pre-training script\n\u003cimg width=\"1000\" alt=\"mpi\" src=\"assets/pretrain_pipeline.png\"\u003e\nPre-train MPI on 8 Nvidia A100 GPUs:\n\n```bash\ntorchrun --standalone --nnodes 1 --nproc-per-node 8 pretrain.py\n```\n\n### Evaluation \u003ca name=\"evaluation\"\u003e\u003c/a\u003e\n#### Referring Expression Grounding\nStep 1. Prepare the OCID-Ref dataset following this [repo](https://github.com/lluma/OCID-Ref). Then put the dataset to \n\n```bash\n./mpi_evaluation/referring_grounding/data/langref\n```\n\nStep 2. Initiate evaluation with\n```bash\npython mpi_evaluation/referring_grounding/evaluate_refer.py test_only=False iou_threshold=0.5 lr=1e-3 \\\nmodel=\\\"mpi-small\\\" \\\nsave_path=\\\"MPI-Small-IOU0.5\\\" \\\neval_checkpoint_path=\\\"path_to_your/MPI-small-state_dict.pt\\\" \\\nlanguage_model_path=\\\"path_to_your/distilbert-base-uncased\\\" \\\n```\n\nor you can simply use \n```bash\nbash mpi_evaluation/referring_grounding/eval_refer.sh\n```\n\n#### Franka Kitchen\nFollowing the [guidebook](./mpi_evaluation/franka_kitchen/README.MD) to setup Franka Kitchen environment and download the expert demonstrations.\n\nEvaluating visuomotor control on Franka Kitchen environment with **25** expert demonstration.\n```bash\nCUDA_VISIBLE_DEVICES=0 PYTHONPATH=mpi_evaluation/franka_kitchen/MPIEval/core python mpi_evaluation/franka_kitchen/MPIEval/core/hydra_launcher.py hydra/launcher=local hydra/output=local env=\"kitchen_knob1_on-v3\" camera=\"left_cap2\" pixel_based=true embedding=ViT-Small num_demos=25 env_kwargs.load_path=mpi-small bc_kwargs.finetune=false job_name=mpi-small seed=125 proprio=9\n```\n\n## Citation\n\nIf you find the project helpful for your research, please consider citing our paper:\n\n```bibtex\n@article{zeng2024learning,\n  title={Learning Manipulation by Predicting Interaction},\n  author={Zeng, Jia and Bu, Qingwen and Wang, Bangjun and Xia, Wenke and Chen, Li and Dong, Hao and Song, Haoming and Wang, Dong and Hu, Di and Luo, Ping and others},\n  journal={arXiv preprint arXiv:2406.00439},\n  year={2024}\n}\n```\n\n## Acknowledgment\nThe code of this work is built upon [Voltron](https://github.com/siddk/voltron-robotics) and [R3M](https://github.com/facebookresearch/r3m). Thanks for their open-source work!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopendrivelab%2Fmpi","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopendrivelab%2Fmpi","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopendrivelab%2Fmpi/lists"}