{"id":38043342,"url":"https://github.com/xavierpuigf/watch_and_help","last_synced_at":"2026-01-16T19:56:06.184Z","repository":{"id":41107502,"uuid":"304482741","full_name":"xavierpuigf/watch_and_help","owner":"xavierpuigf","description":"Code for the paper Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration","archived":false,"fork":false,"pushed_at":"2022-07-15T15:42:14.000Z","size":473274,"stargazers_count":97,"open_issues_count":4,"forks_count":15,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-05-22T06:45:02.856Z","etag":null,"topics":["human-ai-teaming","multiagent","social-perception","virtualhome"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/xavierpuigf.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-10-16T00:54:18.000Z","updated_at":"2025-04-19T02:18:42.000Z","dependencies_parsed_at":"2022-08-10T01:35:34.622Z","dependency_job_id":null,"html_url":"https://github.com/xavierpuigf/watch_and_help","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/xavierpuigf/watch_and_help","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xavierpuigf%2Fwatch_and_help","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xavierpuigf%2Fwatch_and_help/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xavierpuigf%2Fwatch_and_help/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xavierpuigf%2Fwatch_and_help/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/xavierpuigf","download_url":"https://codeload.github.com/xavierpuigf/watch_and_help/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xavierpuigf%2Fwatch_and_help/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28482136,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-16T11:59:17.896Z","status":"ssl_error","status_checked_at":"2026-01-16T11:55:55.838Z","response_time":107,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["human-ai-teaming","multiagent","social-perception","virtualhome"],"created_at":"2026-01-16T19:56:05.453Z","updated_at":"2026-01-16T19:56:06.179Z","avatar_url":"https://github.com/xavierpuigf.png","language":"Python","funding_links":[],"categories":["Embodied Robotics"],"sub_categories":["Embodied Multi-Task \u0026 Household"],"readme":"# Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration\n\nThis is the official implementation of the paper [*Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration*](https://arxiv.org/abs/2010.09890). \n\nIn this work, we introduce Watch-And-Help (WAH), a challenge for testing social intelligence in agents. In WAH, an AI agent needs to help a human-like agent perform a complex household task efficiently. To succeed, the AI agent needs to i) understand the underlying goal of the task by watching a single demonstration of the human-like agent performing the same task (social perception), and ii) coordinate with the human-like agent to solve the task in an unseen environment as fast as possible (human-AI collaboration).\n\n![](assets/cover_fig_final.png)\n\nWe provide a dataset of tasks to evaluate the challenge, as well as different baselines consisting on learning and planning-based agents.\n\nCheck out a video of the work [here](https://youtu.be/lrB4K2i8xPI).\n\n## Cite\nIf you use this code in your research, please consider citing.\n\n```\n@inproceedings{\n      puig2021watchandhelp,\n      title={Watch-And-Help: A Challenge for Social Perception and Human-{\\{}AI{\\}} Collaboration},\n      author={Xavier Puig and Tianmin Shu and Shuang Li and Zilin Wang and Yuan-Hong Liao and Joshua B. Tenenbaum and Sanja Fidler and Antonio Torralba},\n      booktitle={International Conference on Learning Representations},\n      year={2021},\n      url={https://openreview.net/forum?id=w_7JMpGZRh0}\n}\n```\n\n## Setup\n### Get the VirtualHome Simulator and API\nClone the [VirtualHome API](https://github.com/xavierpuigf/virtualhome.git) repository one folder above this repository\n\n```bash\ncd ..\ngit clone --branch wah https://github.com/xavierpuigf/virtualhome.git\ncd virtualhome\npip install -r requirements.txt\n```\n\nDownload the simulator, and put it in an `executable` folder, one folder above this repository\n\n\n- [Download](http://virtual-home.org/release/simulator/v2.0/linux_exec.zip) Linux x86-64 version.\n- [Download](http://virtual-home.org/release/simulator/v2.0/macos_exec.zip) Mac OS X version.\n- [Download](http://virtual-home.org/release/simulator/v2.0/windows_exec.zip) Windows version.\n\n\n[IMPORTANT] Please make sure to use the [wah](https://github.com/xavierpuigf/virtualhome/tree/wah) branch of the VirtualHome repo and the v2.2.0 version of the executable.\n\n### Install Requirements\n```bash\npip install -r requirements.txt\n```\n\n### Download Checkpoints\nDownload the checkpoints for the weights to the hybrid help model and the watching model. Use\n\n```bash\nsh scripts/download_ckpts.sh\n```\n\n\n\n## Dataset\nWe include a dataset of environments and activities that agents have to perform in them. During the **Watch** phase and the training of the **Help** phase, we use a dataset of 5 environments. When evaluating the **Help** phase, we use a dataset of 2 held out environments.\n\nThe **Watch** phase consists of a set of episodes in 5 environments showing Alice performing the task. These episodes were generated using a planner, and they can be downloaded [here](http://virtual-home.org/release/watch_and_help/watch_data.zip). The training and testing split information can be found in `datasets/watch_scenes_split.json`. \n\nThe **Help** phase, contains a set of environments and tasks definitions. You can find the *train* and *test* datasets used in `dataset/train_env_set_help.pik` and `dataset/test_env_set_help.pik`. Note that the *train* environments are independent, whereas the *test* environments match the tasks in the **Watch** test split.\n\n\n### Create your own dataset \nYou can also create your dataset, and modify it to incorporate new tasks. For that, run\n\n```bash\npython gen_data/vh_init.py --num-per-apartment {NUM_APT} --task {TASK_NAME}\n```\nWhere `NUM_APT` corresponds to the number of episodes you want for each apartment and task and `TASK_NAME` corresponds to the task name you want to generate, which can be `setup_table`, `clean_table`, `put_fridge`, `prepare_food`, `read_book`, `watch_tv` or `all` to generate all the tasks.\n\nAfter creating your dataset, you can create the data for the **Watch** phase running the *Alice alone* baseline (see [Evaluate Baselines](#evaluate-baselines)).\n\nYou can then generate a dataset of tasks in a new environment where the tasks match those of the **Watch phase**. We do that in our work to make sure that the environment in the **Watch** phase is different than that in the **Help Phase** while having the same task specification. You can do that by running:\n\n```bash\npython gen_data/vh_init_gen_test.py\n```\n\nIt will use the tasks from the test split of the **Watch** phase to create a **Help** dataset.\n\n\n\n## Watch\nFirst, download the dataset for the **Watch** phase and put it under `dataset`. \nYou can train the goal prediction model for the **Watch** phase as follows:\n\n```bash\nsh scripts/train_watch.sh\n```\n\nTo test the goal prediction model, run:\n\n```bash\nsh scripts/test_watch.sh\n```\n\n## Help\nWe provide planning and learning-based agents for the Helping stage. The agents have partial observability in the environment, and plan according to a belief that updates with new observations.\n\n![](assets/collab_fig.gif)\n\n### Train baselines\n\n#### Hybrid Baseline\nWe train the hybrid baseline in 2 stages. One where we allow the agent to teleport to a location and one where the agent has to walk to the location. You can train the hybrid baseline as follows.\n\n``` bash\n# First stage\npython training_agents/train_a2c.py \\\n--max-num-edges 10 --max-episode-length 250 \\\n--batch_size 32 --obs_type partial --gamma 0.95 \\\n--lr 1e-4 --nb_episodes 100000 --save-interval 200\n--simulator-type unity --base_net TF --log-interval 1 \\\n--long-log 50 --logging --base-port 8681 \n--num-processes 5 --teleport \n--executable_file ../executable/linux_exec_v3.x86_64 \\\n--agent_type hrl_mcts --num_steps_mcts 24\n\n#Second stage\npython training_agents/train_a2c.py  \\\n--max-num-edges 10 --max-episode-length 250 \\\n--batch_size 32 --obs_type partial --gamma 0.95 \\\n--lr 1e-4 --nb_episodes 100000 --save-interval 200 \\\n--simulator-type unity --base_net TF --log-interval 1 \\\n--long-log 50 --logging --base-port 8681 \\\n--num-processes 5 \\\n--executable_file ../executable/linux_exec_v3.x86_64 \\\n--agent_type hrl_mcts --num_steps_mcts 50 \\\n--load-model {path to previous model}\n\n```\n### Evaluate baselines\nBelow is the code to evaluate the different planning-based models. The results will be saved in a folder called `test_results`. Make sure you create it first, one level above this repository. \n\n```bash\nmkdir ../test_results\n```\n\nHere is the code to evaluate the baselines proposed in the paper.\n```bash\n# Alice alone\npython testing_agents/test_single_agent.py\n\n# Bob planner true goal\npython testing_agents/test_hp.py\n\n# Bob planner predicted goal\npython testing_agents/test_hp_pred_goal.py\n\n# Bob planner random goal\npython testing_agents/test_hp_random_goal.py\n\n# Bob random actions\npython testing_agents/test_random_action.py\n```\n\nBelow is the code to evaluate the learning-based methods\n\n```bash\n# Hybrid Baseline\nCUDA_VISIBLE_DEVICES=0 python testing_agents/test_hybrid.py \\\n--max-num-edges 10 --max-episode-length 250 \\\n--num-processes 1 \\\n--agent_type hrl_mcts --num_steps_mcts 40 \\\n--load-model checkpoints/checkpoint_hybrid.pt\n\n```\n\n## Visualize results\n[Coming Soon]\nYou may want to generate a video to visualize the episode you just generated. Here we include a script to view the episodes you generate during the Help phase.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxavierpuigf%2Fwatch_and_help","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fxavierpuigf%2Fwatch_and_help","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxavierpuigf%2Fwatch_and_help/lists"}