{"id":50612121,"url":"https://github.com/UT-Austin-RPL/GROOT","last_synced_at":"2026-06-23T00:00:56.822Z","repository":{"id":202941620,"uuid":"654422158","full_name":"UT-Austin-RPL/GROOT","owner":"UT-Austin-RPL","description":"Official implementation of GROOT, CoRL 2023","archived":false,"fork":false,"pushed_at":"2023-11-04T02:23:40.000Z","size":87104,"stargazers_count":38,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2024-05-14T00:17:37.641Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/UT-Austin-RPL.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-06-16T05:14:36.000Z","updated_at":"2024-03-27T01:10:58.000Z","dependencies_parsed_at":null,"dependency_job_id":"bb1384c0-2f83-42cc-a37c-ae2bd5939729","html_url":"https://github.com/UT-Austin-RPL/GROOT","commit_stats":null,"previous_names":["ut-austin-rpl/groot"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/UT-Austin-RPL/GROOT","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/UT-Austin-RPL%2FGROOT","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/UT-Austin-RPL%2FGROOT/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/UT-Austin-RPL%2FGROOT/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/UT-Austin-RPL%2FGROOT/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/UT-Austin-RPL","download_url":"https://codeload.github.com/UT-Austin-RPL/GROOT/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/UT-Austin-RPL%2FGROOT/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34669839,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-22T02:00:06.391Z","response_time":106,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-06T05:00:21.752Z","updated_at":"2026-06-23T00:00:56.801Z","avatar_url":"https://github.com/UT-Austin-RPL.png","language":"Python","funding_links":[],"categories":["Robot Learning Policies with Visual Input"],"sub_categories":["Large Robot Foundation Models"],"readme":"# **GROOT: Learning Generalizable Manipulation Policies with Object-Centric 3D Representations** \n\n\n\u003c!-- \n\u003cp align=\"center\"\u003e\n\u003cimg src=\"./imgs/pull_figure.png\" height=\"100%\"\u003e\n\u003c/p\u003e\n --\u003e\n \n[Yifeng Zhu](https://www.cs.utexas.edu/~yifengz), [Zhenyu Jiang](https://zhenyujiang.me/), [Peter Stone](https://www.cs.utexas.edu/~pstone), [Yuke Zhu](https://www.cs.utexas.edu/~yukez/)\n\n\n[Project](https://ut-austin-rpl.github.io/GROOT) | [Paper](https://zhuyifengzju.github.io/files/GROOT_CoRL2023.pdf) | [Video](https://youtu.be/T_wW-Mzt49M?si=dFv-NU8itv_SZUpK)\n\n\nOur real robot experiments are based off of the control codebase [Deoxys](https://github.com/UT-Austin-RPL/deoxys_control) and its vision pipeline [Deoxys-Vision](https://github.com/UT-Austin-RPL/deoxys_vision) (Coming soon!).\n\n## 1. Installation\n\n```shell\npip install -r requirements.txt\n```\n\nIf you want to use this as a component, there is a easy way to allow you to access functions in groot. Simple do the following in your virtual environment:\n\n```shell\npip install -e .\n```\n\n### 1.1 Prerequisite\n\nWe need to install the vision models and checkpoints in order to run GROOT codebase. Here is a simple bash script that takes care of everything and also make sure that you can download the same commits of the repos as I used for my experiments. \n\n```\n./setup_vision_models.sh\n```\n\nThe resulting repo structure under `third_party` should be as follows:\n```\nthird_party/\n    dinov2\n    segment-anything\n    XMem\n    sam_checkpoints\n    xmeme_checkpoints\n```\n\n## 2. Data Collection\n\nThis repo relies on `deoxys_vision` and `deoxys_control` to collect images and control the robot (Franka Emika Panda). \n\n### 2.1 Reset robot states\n\nYou will first need to reset the robot to an initial joint configuration. This is to allow you collect data from a fixed iniital state distribution. \n\n``` \npython real_robot_scripts/deoxys_reset_joints.py\n```\n\n### 2.2 Collect data with spacemouse\n\nCollect demonstrations for a task. Specify the task name and consistently so that the code can successfully create the datasets later.\n\n```\npython real_robot_scripts/deoxys_data_collection.py --dataset-name DATASET_NAME\n```\n\n\n## 3. Curate Training Datasets\n\nTo curate the training datasets, there are three things to be done:\n1. create demonstrations\n2. annotate single-frame segmentation\n3. create point clouds to train policies fast\n\n### 3.1 Create Demonstration Dataset\n\n```\npython real_robot_scripts/create_dataset.py --folder DEMONSTRATION_FOLDER\n```\n\n### 3.2 Annotate Single-Frame Segmentation\n\n```\npython scripts/interactive_demo_from_datasets.py --dataset_path DATASET_PATH --num_objects NUM_OBJECTS\n```\n\nBy default, we assume you also annotate robotos during this process. This is for better performance of VOS without considering the robot as one of the objects, and also the GROOT policies assume that the input is including the robot point clouds, so the policy will exclude the robot's point clouds. Including this information is also beneficial for conducting ablation studies for your need. The process is adapted from the S2M GUI. \n\n\n![Annotating First Frame Example](docs/scribble_demo.gif)\n\n\nThis component is made possible by [S2M](https://github.com/hkchengrex/Scribble-to-Mask).\n\n\n### 3.3 Prepare Training set\n\nSimply run `python prepare_training_set.py dataset_path=DATASET_PATH` will do the work. The command is equivalent to the following one:\n\n```\npython prepare_training_set.py dataset_path=DATASET_PATH vos_annotation=true object_pcd=true pcd_aug=true pcd_grouping=true delete_intermediate_files=true\n```\n\nIf the the script is terminated in the middle and you do not want to create the whole set of intermediate files, you can specify one of the procedure from `true` to `false` and the script will skip the corresponding operation. After this process, you will get a `masks.hdf5` that stores the result from segmentation propagation, and a `*_training_set.hdf5` file that saves all the inputs that are used during training.\n\n\n## 4. Real Robot Evaluation\n\n\n### 4.1 Check if your cameras are properly configured and setup.\n\n```\npython verify_camera_observations.py\n```\n\n### 4.2 Evaluation Procedural\n\n1. Reset to initial positions\n\n``` \npython real_robot_scripts/deoxys_reset_joints.py\n```\n\n1. Run evaluation\n\n```\npython scripts/real_robot_eval_checkpoint.py \\\n        --checkpoint-dir CHECKPOINT_FOLDER \\\n        --checkpoint-idx 100 \\\n        --experiment-config real_robot_scripts/eval_new_instances/eval_new_instance_3.yaml\n```\n\n**Details:**\n\nIn the new instance case, the following script will be triggered: `real_robot_scripts/real_robot_sam_result.py` and subsequently `auto_annotation/sam_amg.py`.\n\nAnd there will be multiple files stored for the case of new instances:\n- The first frame captured by the camera: `evaluation_results/DATASET_NAME/first_frame`\n- The outputs of SAM: `evaluation_reulsts/DATASET_NAME/masks`\n- The output from cosine similarity: `annotations/DATASET_NAME/evaluation_{ID}`\n\nAnd the evaluation script will retrieve the image and annoation from `annotations/DATASET_NAME/evaluation_{ID}`.\n\n1. Run checkpoints:\n\n``` shell\npython scripts/real_robot_eval_checkpoint.py --checkpoint-dir experiments_real/VOS_3D_Real_Robot_Benchmark_pogba/stamp_paper_aug/VOS3DSingleTask/VOS3DRealRobotMaskedNoWristTransformerPolicy_seed10000/run_002 --checkpoint-idx 100 --experiment-config real_robot_scripts/eval_new_instances/eval_new_instance_3.yaml\n```\nSome common configs are pre-defined in a yaml file, and you only need\nto specify the yaml file in the argument `--experiment-config`.\n\nIf it's for testing new instances, you need to make sure that you specify the same experiment config for a single object. Otherwise the annotation used for propagating the VOS model will not match.\n\n## 5. Code Structures\n\nFor details, please refer to `code_structure.md`\n```\ngroot/\n\n    # Detailed jupyter notebook to go through important components of the repo, including:\n    #    - XMem tracking for robot data\n    #    - Backprojection to point clouds\n    #    - SAM result on robot data\n    #    - DINOv2 result on robot data\n    walkthrough_example.ipynb\n\n    # Scripts that include:\n    #    - Scribble to mask annotation\n    #    - process dataset for faster training\n    #    - Train robot policies\n    #    - Evaluate robot policies\n    scripts/\n       \n    groot_imitation/\n        # Groot algo repo that stores most of the policy components\n        groot_algo/\n            \n        # separate model for Segmentation Corresponde Model (Paper Section 3.4)\n        segmentation_correspondence_model/ # segmentation correspondence model\n\n        ## Configuration for third party models\n        vision_model_configs/\n\n\n    # Necessary scripts for real robot control\n    real_robot_scripts/\n\n\n    # third party models (Large vision models)\n    ## Third party code repo\n    third_party/\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FUT-Austin-RPL%2FGROOT","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FUT-Austin-RPL%2FGROOT","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FUT-Austin-RPL%2FGROOT/lists"}