{"id":19347284,"url":"https://github.com/sarthakjshetty/tfn-robot","last_synced_at":"2025-04-23T05:32:03.213Z","repository":{"id":75470371,"uuid":"595248410","full_name":"SarthakJShetty/tfn-robot","owner":"SarthakJShetty","description":"This repository hosts the physical robot code for ToolFlowNet. Published at CoRL '22.","archived":false,"fork":false,"pushed_at":"2024-04-16T22:10:23.000Z","size":34646,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-04-17T02:27:54.816Z","etag":null,"topics":["deep-learning","imitation-learning","learning-from-demonstration","machine-learning","robot-learning","robotics"],"latest_commit_sha":null,"homepage":" https://tinyurl.com/toolflownet","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SarthakJShetty.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2023-01-30T17:44:59.000Z","updated_at":"2023-03-01T21:52:51.000Z","dependencies_parsed_at":"2023-06-06T14:30:22.609Z","dependency_job_id":null,"html_url":"https://github.com/SarthakJShetty/tfn-robot","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SarthakJShetty%2Ftfn-robot","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SarthakJShetty%2Ftfn-robot/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SarthakJShetty%2Ftfn-robot/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SarthakJShetty%2Ftfn-robot/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SarthakJShetty","download_url":"https://codeload.github.com/SarthakJShetty/tfn-robot/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223909903,"owners_count":17223592,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","imitation-learning","learning-from-demonstration","machine-learning","robot-learning","robotics"],"created_at":"2024-11-10T04:15:24.414Z","updated_at":"2024-11-10T04:15:25.008Z","avatar_url":"https://github.com/SarthakJShetty.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ch1 align=\"center\"\u003e\n  ToolFlowNet: Robotic Manipulation with Tools via Predicting Tool Flow from Point Clouds\u003c/h1\u003e\n\n\u003cdiv align=\"center\"\u003e\n  \u003ca href=\"http://www.cs.cmu.edu/~dseita/\"\u003eDaniel Seita\u003c/a\u003e \u0026nbsp;•\u0026nbsp;\n  \u003ca href=\"https://yufeiwang63.github.io/\"\u003eYufei Wang\u003csup\u003e†\u003c/sup\u003e\u003c/a\u003e \u0026nbsp;•\u0026nbsp;\n  \u003ca href=\"https://sarthakjshetty.github.io/\"\u003eSarthak J. Shetty\u003csup\u003e†\u003c/sup\u003e\u003c/a\u003e \u0026nbsp;•\u0026nbsp;\n  Edward Y. Li\u003csup\u003e†\u003c/sup\u003e \u0026nbsp;•\u0026nbsp;\n  \u003ca href=\"https://zackory.com/\"\u003eZackory Erickson\u003c/a\u003e \u0026nbsp;•\u0026nbsp;\n  \u003ca href=\"https://davheld.github.io/\"\u003eDavid Held\u003c/a\u003e\n\u003c/div\u003e\n\n\u003ch4 align=\"center\"\u003e\n  \u003ca href=\"https://sites.google.com/view/point-cloud-policy/home\"\u003e\u003cb\u003eWebsite\u003c/b\u003e\u003c/a\u003e \u0026nbsp;•\u0026nbsp;\n  \u003ca href=\"https://arxiv.org/abs/2211.09006\"\u003e\u003cb\u003ePaper\u003c/b\u003e\u003c/a\u003e \n\u003c/h4\u003e\n\n\n![ToolFlowNet Success](assets/stitched_success.gif)\n\n:warning: **This is the repository containing the robot code to collect and process real world human demonstrator data for [ToolFlowNet](https://tinyurl.com/toolflownet)** :warning:\n\nThis is a `catkin` package that is used to collect human demonstration data in the format that ToolFlowNet expects. We use the generated `pkl` files at the end of the process to train ToolFlowNet on the real-world human demonstration data.\n\nTo use this package, first place this folder/repository inside the `src` folder of your `catkin` workspace.\n\nOnce you compile and build your `catkin` workspace, you should launch the necessary ROS nodes using:\n\n```\n    roslaunch mixed_media mixed_media.launch\n```\n\nThis command runs the point cloud processing node, that uses depth and RGB images from a Microsoft Kinect v2.\n\nIf there are no errors, you should be all set to collect demonstrations or run inference on a trained model.\n\n**Note:** [Here are additional notes] on our specific setup, which involves the [**ReThink Sawyer Arm**](https://www.rethinkrobotics.com/sawyer) and the [**Microsoft Azure Kinect**](https://azure.microsoft.com/en-us/products/kinect-dk/) that we used for the experiments reported in our [**CoRL '22 paper**](https://tinyurl.com/toolflownet).\n\n## Collecting demonstrations:\n\nTo collect demonstrations, run `robot.py` with the following flags:\n\n```\npython robot.py --policy hum_t_demo --n_targ 1 --num_demos 1\n```\n\n**Flags:**\n\n- ``--policy``: Indicates if we're collecting human  (`hum_t_demo`) or algorithimic (`alg_t_demo`) demonstrations or running inference (`run_inference`) with a trained model.\n- `--n_targ`: Specifies the number of targets that we're trying to scoop out of the water.\n- `--n_demos`: Specifies the number of demos that can be collected at a time. If you're constrained on memory set this to `1`.\n\n### Human Demonstrator:\n\nFor the human demonstrator setting, once the robot resets to the starting position, the joint impedance reduces significantly and the arm is compliant enough for a human operator to maneuver and scoop out the ball from the bin.\n\n### Algorithmic Demonstrator:\n\nFor the algorithmic demonstrator setting, make sure the environment bounds are correctly stored in `robot.py`.\n\nAt the end of each episode, the time-stamped data from the episode will be stored in appropriate folders under the `data/` directory.\n\n## Processing demonstration data:\n\nHere are the instructions for generating observation action pairs from the physical robot experiments.\n\n1. Once the demonstrations have been collected from the ```robot.py``` code, they are saved in the ```data/``` folder.\n\n2. To generate the observation, action pairs use the `visualization.py` code as follows:\n\n    ```\n    python visualization.py --folder_name data/policy_scripted_rotation_ntarg_01_ndist_00_maxT_10/data/ --encoding targ\n    ```\n\n3. When `visualization.py` runs, we get `pkl`s for each of the demonstrations. This is stored in the respective demonstration session's `pkl_dir_name_save` folder. To collect all the `pkl`s across the entire session, we use `pkl_check.py`. This script moves all the `pkl`s to a single folder and `tar`s them up as well into a `dataset_archive.tar` file.  This is so that we can move the data to a heavier machine for training. Here's the command for `pkl_check.py`:\n\n    ```\n    python pkl_check.py --src /data/sarthak/data_demo/data/policy_scripted_rotation_ntarg_01_ndist_00_maxT_10/data/ --dst /data/sarthak/v07_rotation_translation_variably_composed/ --k_steps 1 \n    ```\n\n`pkl_check.py` takes 3 arguments, 1. `src` which is the location of the `policy_data` folder. `pkl_check` recursively looks into these folders and finds the `pkl`s. 2. `dst` which is the location where you want to move all these `pkl`s to. 3. `k_steps` since a given `src` might contain `pkl`s from different `k_steps` as well.\n\n4. If you want to analyze some basic stats of the `pkl`s that you're about to train on, you can run `dataset_stats.py` which is available on `toolflownet-physical`. `dataset_stats.py` takes just one argument, `--dataset_loc` which should be the same as `dst` from the previous step. \n\n    ```\n    python dataset_stats.py --dataset_loc /data/sarthak/v07_rotation_translation_variably_composed/\n    ```\n\nOnce `dataset_stats.py` runs, it generates a simple plot called `dataset_stats.png` and stores in `dataset_loc`. You can visualize histograms of the `act_raw` component across all the `pkl`s in `dataset_loc`. Here is an example:\n![Dataset Stats](assets/dataset_stats.png)\n\n4. Here, we provide links to the demonstrator data (approximately 25GB) that we used to train the final variant of our model. You can download it using:\n\n    ```\n    gdown 1R1ZcdEA3WHr_V0fwOhJiRihiM4pPqhiE\n    ```\n\nUntar this zip file using:\n\n```\ntar -xvf corl_2022_human_demonstrator_data.tar.gz\n```\n\nThis should result in a folder called `v06_human_demonstrator_variable_composing_only_PKLs` which should contain the `pkl`s for 125 human demonstrations, necessary to train ToolFlowNet for the physical experiments.\n\n5. To train on this demonstrator, first make sure you're pointing to the right data directories in `launch_bc_mm.py`, including `DATA_HEAD_1` and the `suffix` variable in the main function in the same file. After checking the file variables are pointing to the right location, run:\n\n    ```\n    python experiments/bc/launch_bc_mm.py\n    ```\n\n\n## Running inference:\n\n1. To run inference, run `robot.py` with the `run_inference` argument for the `policy` flag:\n\n    ```\n    python robot.py --policy run_inference --n_targ=1\n    ```\n\n2. On the GPU machine, which contains the trained model, run `inference.py`, available on [`softagent_tfn`](https://github.com/DanielTakeshi/softagent_tfn/tree/physical):\n\n    ```\n    python -m bc.inference --exp_config SVD_POINTWISE_3D_FLOW --model_path data/local/BCphy_v06_human_fast_zero_lag_variable_composing_ntrain_0100_PCL_PNet2_svd_pointwise_acttype_flow_rawPCL_scaleTarg/BCphy_v06_human_fast_zero_lag_variable_composing_ntrain_0100_PCL_PNet2_svd_pointwise_acttype_flow_rawPCL_scaleTarg_2022_09_18_21_51_08_0001/model/ckpt_0340.tar --obs_dim 5 --max_n_points 1400 --scale_factor 100.0\n    ```\n\n3. The control PC running `robot.py` will now send point cloud observations to the GPU machine, which in-turn will run inference with the trained model and send back end-effector actions to the control PC.\n\n4. You can download the final variant of the model that we used to generate results for our CoRL '22 paper here, using:\n    ```\n    gdown --folder 163NsJJDxAuSpL6RSsVAEWnT0rbAyPMub\n    ```\n\nThe inference code uses the checkpoint directly in the ```tar``` format, and therefore you do not need to `untar` it. \n\n**Note:** This checkpoint is to be used after the `--model` flag in the `bc.inference` command above.\n\n## Contact:\nIf you run into any issues with the workflow, please contact the authors to correct/update this ```README```.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsarthakjshetty%2Ftfn-robot","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsarthakjshetty%2Ftfn-robot","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsarthakjshetty%2Ftfn-robot/lists"}