{"id":20207221,"url":"https://github.com/pathak22/hierarchical-imitation","last_synced_at":"2025-04-10T12:33:31.978Z","repository":{"id":85023067,"uuid":"218697688","full_name":"pathak22/hierarchical-imitation","owner":"pathak22","description":"[NeurIPS 2019] Code for Hierarchical Third-Person Imitation Learning","archived":false,"fork":false,"pushed_at":"2020-01-20T04:04:36.000Z","size":2942,"stargazers_count":59,"open_issues_count":1,"forks_count":10,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-03-24T11:21:17.498Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://pathak22.github.io/hierarchical-imitation/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pathak22.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-10-31T06:19:08.000Z","updated_at":"2025-03-18T08:16:41.000Z","dependencies_parsed_at":null,"dependency_job_id":"9eb97fbf-ba91-406c-ad60-204270971366","html_url":"https://github.com/pathak22/hierarchical-imitation","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pathak22%2Fhierarchical-imitation","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pathak22%2Fhierarchical-imitation/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pathak22%2Fhierarchical-imitation/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pathak22%2Fhierarchical-imitation/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pathak22","download_url":"https://codeload.github.com/pathak22/hierarchical-imitation/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248217145,"owners_count":21066633,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-14T05:28:02.681Z","updated_at":"2025-04-10T12:33:31.965Z","avatar_url":"https://github.com/pathak22.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Third-Person Visual Imitation Learning via Decoupled Hierarchical Controller ##\n### NeurIPS 2019\n#### [[Project Website]](https://pathak22.github.io/hierarchical-imitation/) [[Demo Video]](https://youtu.be/eWBkDuNFEKA)\n\n[Pratyusha Sharma](https://scholar.google.co.in/citations?user=RGiCLUgAAAAJ\u0026hl=en), [Deepak Pathak](https://people.eecs.berkeley.edu/~pathak/), [Abhinav Gupta](http://www.cs.cmu.edu/~abhinavg/)\u003cbr/\u003e\nCarnegie Mellon University\u003cbr/\u003e\nUniversity of California, Berkeley\u003cbr/\u003e\n\nWe study a generalized setup for learning from demonstration to build an agent that can manipulate novel objects in unseen scenarios by looking at only a single video of human demonstration from a third-person perspective. If you find this work useful in your research, please cite:\n\n    @inproceedings{sharma19thirdperson,\n        Author = {Sharma, Pratyusha and Pathak, Deepak\n                  and Gupta, Abhinav},\n        Title = {Third-Person Visual Imitation Learning via\n                  Decoupled Hierarchical Controller},\n        Booktitle = {NeurIPS},\n        Year = {2019}\n    }\n\n\u003cimg src=\"https://pathak22.github.io/hierarchical-imitation/resources/teaser.jpg\" width=\"800px\"/\u003e\n\nThe code for the paper consists of two modules:\n\n1. The Goal Generator: The goal generator takes in consecutive frames of a human video along with the present image of the table to hallucinate a possible next visual state of the robot's trajectory. It is contained inside the directory named 'pix2pix'.\n\n2. Low-level Controller: The low-level controller takes as input the current visual state and the predicted visual state and outputs an action.\n\nThe two models are trained independently and are run together in alternation at test time. \nThe code to run the models in alternation at test time is also in this repository.\n\n## Step 0: Installation and Prerequisites\n\n### Prerequisites\n- Python 3\n- Pytorch 0.4+\n- Linux or macOS\n- CPU or NVIDIA GPU + CUDA CuDNN\n\n### Installation\n- Clone this repo\n```\ngit clone https://github.com/pathak22/hierarchical-imitation.git\ncd hierarchical-imitation\n```\n\n## Step 1: Training the Goal Generator\nThe code for the goal generator is built using code from the wonderful [pix2pix](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix) repository. \n\n### a. Data pre-processing\nBefore training the goal generator the steps listed under [creating your own datasets for pix2pix](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/blob/master/docs/tips.md) need to be followed.\n\nSince we want to translate intent from human videos to robot videos the folders should be as follows:\nFolder A: Human demonstration frames\nFolder B: Robot demonstration frames\n\nThe code used to subsample the trajectories to equal lengths to roughly align them can be found in utils as subsample.py.\n\n### b. Training\nTrain the model:\n```\ncd pix2pix\npython train.py --dataroot /location/of/the/dataset --model pix2pix\n```\n\n### c. Evaluation\nEvaluating the model:\n```\npython test.py --dataroot /location/of/the/dataset --model pix2pix\n```\n\n## Step 2: Training the Low-level Controller / Inverse Model\n\n### a. Data pre-processing\nAt training time the inputs to the low-level controller are consecutive images from the robot trajectory and the joint angle of the robot at the end of the two frames.\nSubsample the robot trajectories using the code for subsample.py. \n\n### b. Training\nTrain the low-level controller using:\n```\npython controller_train.py --dataroot /location/of/the/dataset\n```\n\n### c. Evaluation\nEvaluate the low-level controller using:\n```\npython controller_test.py --dataroot /location/of/the/dataset\n```\n\n## Step 3: Running the models together on the robot\nTo finally test the controllers together on the robot use:\n```\npython run_on_robot.py --goal_generator /location/of/checkpoint --inverse_model /location/of/checkpoint --dataroot /location/of/the/humandemo \n```\n\n### Finally : Pointers\n1. Test how good the models are indivdually before running the joint run to get an estimate of how best can each of the models do in isolation\n2. Look at the predictions of the goal generator while running the final experiment on the robot\n3. A good place to start could be downloading the [[MIME Dataset]](https://www.dropbox.com/sh/hwgdkzbu488gwy4/AABPjRj_ZpNYn5WjbnkdoDKQa?dl=0). Alternatively, one could also collect their own dataset and follow the training protocol above.\n4. In case of a query, feel free to reach out Pratyusha Sharma at pratyuss@csail.mit.edu.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpathak22%2Fhierarchical-imitation","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpathak22%2Fhierarchical-imitation","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpathak22%2Fhierarchical-imitation/lists"}