{"id":13545948,"url":"https://github.com/xavierpuigf/virtualhome","last_synced_at":"2026-01-14T15:25:00.233Z","repository":{"id":37663924,"uuid":"177614859","full_name":"xavierpuigf/virtualhome","owner":"xavierpuigf","description":"API to run VirtualHome, a Multi-Agent Household Simulator","archived":false,"fork":false,"pushed_at":"2025-06-10T08:32:04.000Z","size":288323,"stargazers_count":581,"open_issues_count":54,"forks_count":86,"subscribers_count":11,"default_branch":"master","last_synced_at":"2025-11-09T18:28:03.299Z","etag":null,"topics":["computer-vision","deep-learning","graph","multi-agent","reinforcement-learning","simulator","unity"],"latest_commit_sha":null,"homepage":"http://virtual-home.org","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/xavierpuigf.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-03-25T15:34:12.000Z","updated_at":"2025-11-05T02:54:25.000Z","dependencies_parsed_at":"2023-02-10T00:45:48.717Z","dependency_job_id":"b9c4e89b-250c-4f85-b0d7-e08035df34bc","html_url":"https://github.com/xavierpuigf/virtualhome","commit_stats":{"total_commits":637,"total_committers":22,"mean_commits":"28.954545454545453","dds":0.5368916797488226,"last_synced_commit":"1d9ac2307dbb09d87e75046722336449e141039e"},"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"purl":"pkg:github/xavierpuigf/virtualhome","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xavierpuigf%2Fvirtualhome","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xavierpuigf%2Fvirtualhome/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xavierpuigf%2Fvirtualhome/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xavierpuigf%2Fvirtualhome/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/xavierpuigf","download_url":"https://codeload.github.com/xavierpuigf/virtualhome/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xavierpuigf%2Fvirtualhome/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28424374,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-14T13:30:50.153Z","status":"ssl_error","status_checked_at":"2026-01-14T13:29:08.907Z","response_time":107,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","deep-learning","graph","multi-agent","reinforcement-learning","simulator","unity"],"created_at":"2024-08-01T12:00:28.313Z","updated_at":"2026-01-14T15:25:00.214Z","avatar_url":"https://github.com/xavierpuigf.png","language":"Python","funding_links":[],"categories":["Papers","Python"],"sub_categories":["ICML 2022"],"readme":"\u003cdiv align=\"center\"\u003e\n\n[![docs](https://img.shields.io/badge/docs-updated-brightgreen)](http://virtual-home.org/documentation/)\n[![Discord](https://badgen.net/badge/icon/discord?icon=discord\u0026label)](https://discord.gg/JmzDYF3)\n[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/xavierpuigf/virtualhome/blob/master/demo/unity_demo.ipynb)\n[![PyPI - Downloads](https://img.shields.io/pypi/dm/virtualhome)](https://pypi.org/project/virtualhome/)\n\n# VirtualHome\n\n**VirtualHome** is an interactive platform to simulate complex household activities via programs. Key aspect of VirtualHome is that it allows complex interactions with the environment, such as picking up objects, switching on/off appliances, opening appliances, etc. Our simulator can easily be called with a Python API: write the activity as a simple sequence of instructions which then get rendered in VirtualHome. You can choose between different agents and environments, as well as modify environments on the fly. You can also stream different ground-truth such as time-stamped actions, instance/semantic segmentation, and optical flow and depth. The platform allows to simulate multi-agent activities and can serve as an environment to train agents for embodied AI tasks.\n\nCheck out more details of the environmnent and platform at [**virtual-home.org**](http://virtual-home.org). \n\n\u003cp align=\"center\"\u003e\n  \u003cimg width=\"460\" height=\"300\" src=\"assets/vh_intro.gif\"\u003e\n\u003c/p\u003e\n\n\u003c/div\u003e\n\n______________________________________________________________________\n\n## What is New\n\nVirtualHome 2.3 is out! Here are the latest updates:\n\n* Added Procedural Generation Capabilities to allow agents to explore a infinite possibility of unique environments!  \n* We added more custom designed environments!\n* Enhanced simulated physics with the environments\n* Time management system with synchronized day/night\n* New outdoor terrain with accurate sunlight/shadows\n* Improved indoor realtime lighting\n* More realistic rooms\n* Significant performance enhancements, asset optimizations, stability improvements\n* Updated documentation\n* General bug fixes for existing environments\n\n### Past updates\n* [August 2020] Added RL Environment Demo, and example to combine Environments with Ray.\n* [April 2020] We added new documentation! Check it out at http://virtual-home.org/documentation\n* We include OpenAI Gym like environments to train RL agents with VirtualHome. Check out the unity_environment.py class.\n* API to add characters in the scene and fixed cameras, and record from those cameras\n\n\n### Currently in Development\n\n* Enhancements for Procedural Generation Capabilities\n* Photorealism capabilities  \n* More actions and object interactions\n* Human interaction\n\n____________________________________________________________\n\n## Contents\n\n1. Overview\n2. Set Up\n3. Generating Videos/Keyframes\n4. VirtualHome RL Env\n5. Dataset \n6. Modify VirtualHome\n7. Citation\n8. Contributors\n\n______________________________________________________________________\n\n## Overview\n\nActivities in VirtualHome are represented through two components: *programs* representing the sequence of actions that compose an activity, and *graphs* representing a definition of the environment where the activity takes place. Given a program and a graph, the simulator executes the program, generating a video of the activity or a sequence of graphs representing how the environment evolves as the activity takes place. To this end, VirtualHome includes two simulators: the *Unity Simulator* and *Evolving Graph*. You can find a more complete documentation with examples and the different executables at http://virtual-home.org/documentation.\n\n#### Unity Simulator \n\nThis simulator is built in Unity and allows generating videos of activities. To use this simulator, you will need to download the appropiate executable and run it with the [Python API](virtualhome/simulation/unity_simulator/). You can check a demo of the simulator in [demo/unity_demo.ipynb](virtualhome/demo/unity_demo.ipynb)\n\n#### Evolving Graph\n\nThis simulator runs fully in python and allows to generate a sequence of graphs when a program is executed. You can run it in [simulation/evolving_graph](virtualhome/simulation/evolving_graph). Note that some of the objects and actions in this simulator are not supported yet in Unity Simulator.\n\n______________________________________________________________________\n\n## Set Up\n\n### Install\n\n```bash\n$ pip install virtualhome\n```\n\nWe also provide a [Jupyter](https://jupyter.org/) notebook with a demo and starting code. If you want to run the demo, [install Jupyter](https://jupyter.org/install.html) and run it on your host. If you are new to Jupyter, see [Running the Jupyter Notebook](https://jupyter-notebook-beginner-guide.readthedocs.io/en/latest/execute.html) for a walkthrough of how to use this tool.\n\n\n### Download Unity Simulator\n\nDownload the VirtualHome UnitySimulator executable and move it under `simulation/unity_simulator`.\n\n- [Download](http://virtual-home.org//release/simulator/v2.0/v2.3.0/linux_exec.zip) Linux x86-64 version.\n- [Download](http://virtual-home.org/release/simulator/v2.0/v2.3.0/macos_exec.zip) Mac OS X version.\n- [Download](http://virtual-home.org//release/simulator/v2.0/v2.3.0/windows_exec.zip) Windows version.\n\n\n### Test simulator\n\nTo test the simulator in a local machine, double click the executable, or run it via terminal. When running it via the temrinal, we recommend setting windowed mode (so that the simulator does not take the full screen), as such:\n\n```bash\n./path_to_exec -screen-fullscreen 0 -screen-quality 4 \n```\n\nOnce the simulator has started, run the demo in [demo/unity_demo.ipynb](virtualhome/demo/unity_demo.ipynb). \n\nIf you do not have a monitor or want to test the simulator remotely, you can either use [Docker](docker) or use an X server (find the installation instructions in [this medium post](https://towardsdatascience.com/how-to-run-unity-on-amazon-cloud-or-without-monitor-3c10ce022639)). When running the executable with an X server, use -batchmode. For Linux, you would do:\n\nFirst run the X server on a terminal. You will have to specify which display you want to use, and on which GPUs. By default it will use all the gpus available\n```bash\nsudo python helper_scripts/startx.py $display_num\n```\n\nOn a separate terminal, launch the executable\n```bash\nDISPLAY=:display_num ./{path_sim}/{exec_file}.x86_64 -batchmode\n```\n\nFor Linux, you can also launch the UnityCommunication specifying an executable file. This will directly open the executable on the right sceen. You can do it as follows:\n\nAfter running the X server, run:\n```python\nfrom simulation.unity_simulator import comm_unity\ncomm = comm_unity.UnityCommunication(file_name=file_name, port={your_port}, x_display={your_display})\n```\nIt will open an executable and create a communication object to render scripts or simulate actvities. You can open multiple executables at the same time, to train models or generate data using multiple processes.\n\n### Docker\nYou can also run Unity Simulator using Docker. You can find how to set it up [here](docker).\n\n______________________________________________________________________\n\n## Generating Videos and Snapshots\n\nVirtualHome *Unity Simulator* allows generating videos corresponding to household activities. In addition, it is possible to use *Evolving Graph* simulator to obtain the environment for each execution step and use *UnitySimulator* to generate snapshots of the environment at each step.\n\n\n### Generate videos\n\nOpen the simulator and run:\n\n```bash\ncd demo/\npython generate_video.py\n```\n\n\n### Generate snapshots\n\nOpen the simulator and run:\n\n```bash\ncd demo/\npython generate_snapshots.py\n```\nA grid of snapshots for the given script will be generated and saved in [demo/snapshot_test.png](virtualhome/demo/snapshot_test.png).\n\n______________________________________________________________________\n\n## VirtualHome RL Env\n\nVirtualHome can be used as an environment for Reinforcement Learning. We provide a base class `UnityEnvironment` in [simulation/environment/unity_environment.py](virtualhome/simulation/environment/unity_environment.py). You can test how the class works by running\n\n```bash\ncd demo\npython test_unity_environment.py\n```\n\nThe provided environment can be combined with [Ray](https://github.com/ray-project/ray), to run multiple environments in parallel, allowing to scale your Reinforcement Learning algorithms. You can test parallel environments by running:\n\n```bash\ncd demo\npython test_unity_environment_mp.py\n```\n\n______________________________________________________________________\n\n## Dataset\n\nWe collected a dataset of programs and augmented them with graphs using the Evolving Graph simulator. You can download them [here](http://virtual-home.org/release/programs/programs_processed_precond_nograb_morepreconds.zip). \nOnce downloaded and unzipped, move the programs into the `dataset` folder. You can do all this by executing the script\n```bash\n./helper_scripts/download_dataset.sh\n```\nThe dataset should follow the following structure:\n\n```\ndataset\n└── programs_processed_precond_nograb_morepreconds\n\t|── initstate\n\t├── withoutconds\n\t├── executable_programs\n\t|   ├── TrimmedTestScene7_graph\n\t|\t└── ...\n\t└── state_list\n\t\t├── TrimmedTestScene7_graph\n\t   \t└── ...\t\n```\n\nThe folders `withoutconds` and `initstate` contain the original programs and pre-conditions. \n\nWhen a script is executed in an environment, the script changes by aligning the original objects with instances in the environment. You can view the resulting script in `executable_programs/{environment}/{script_name}.txt`.\n\nTo view the graph of the environment, and how it changes throughout the script execution of a program, check   `state_list/{environment}/{script_name}.json`.\n\nYou can find more details of the programs and environment graphs in [dataset/README.md](virtualhome/dataset/README.md).\n\n\n### Script Augmentation\n\nIn *Synthesizing Environment-Aware Activities via Activity Sketches*, \nwe augment the scripts with two knowledge bases: `KB-RealEnv` and `KB-ExceptonHandler`.\nYou can download the augmented scripts in [KB-RealEnv](http://virtual-home.org/release/programs/augment_location.zip) and [KB-ExceptionHandler](http://virtual-home.org/release/programs/augment_exception.zip).\n\nHere, we provide the code to augment the sripts:\n\n\n#### Augment with `KB-RealEnv`\n\n```bash\ncd dataset_utils\npython augment_dataset_locations.py\n```\n\n\n#### Augment with `KB-ExceptionHandler`\n\n```bash\ncd dataset_utils\npython augment_dataset_exceptions.py\n```\n\n\n### Original Collected and Scripted Programs\n\nWe originally collected a set of programs to predict from language descriptions, and generated a larger of programs via a scripted language. Those programs are described [here](http://virtual-home.org/paper/virtualhome.pdf) as VirtualHome Activity (collected programs) ActivityPrograms (scripted programs). You can download them here:\n\n- [Download](http://virtual-home.org/release/programs/virtualhome_cvpr18/real.zip) VirtualHome Activity.\n- [Download](http://virtual-home.org/release/programs/virtualhome_cvpr18/synthetic.zip) ActivityPrograms.\n\n\n### Resources\n\nTo do the above generation and augmentation, some valuable resource files are used to set the properties of objects, set the affordance of objects, etc.\nCheck [resources/README.md](virtualhome/resources/README.md) for more details.\n\n______________________________________________________________________\n\n## Documentation\n\nTo learn more about VirtualHome, please check out [VirtualHome Docs](http://virtual-home.org/documentation/).\n\n______________________________________________________________________\n\n\n## Modify VirtualHome\n\nIf you would like to contribute to VirtualHome, or modify the simulator for your research needs. Check out the repository with the [Unity Source Code](https://github.com/xavierpuigf/virtualhome_unity). You will need to download the Unity Editor and build your own executable after having made the updates.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg width=\"300\" height=\"300\" src=\"assets/vh_overview.gif\"\u003e\n\u003c/p\u003e\n\n______________________________________________________________________\n\n## Citation\n\nVirtualHome has been used in:\n\n- VirtualHome: Simulating HouseHold Activities via Programs. [PDF](https://arxiv.org/pdf/1806.07011.pdf) \u003cbr/\u003e\nX. Puig*, K. Ra*, M. Boben*, J. Li, T. Wang, S. Fidler, A. Torralba.\u003cbr/\u003e\nCVPR2018.\n\n\n- Synthesizing Environment-Aware Activities via Activity Sketches.\u003cbr/\u003e\nA. Liao*, X. Puig*, M. Boben, A. Torralba, S. Fidler.\u003cbr/\u003e\nCVPR2019.\n\n\n- Watch-and-Help: A Challenge for Social Perception and Human-AI Collaboration.\u003cbr/\u003e\nX. Puig, T. Shu, S. Li, Z. Wang, J. Tenenbaum, S. Fidler, A. Torralba.\u003cbr/\u003e\nICLR2021, spotlight.\u003cbr/\u003e\nNeurIPS Cooperative AI Workshop 2020, Best Paper Award.\n\n- Pre-Trained Language Models for Interactive Decision-Making. [Project](https://shuangli-project.github.io/Pre-Trained-Language-Models-for-Interactive-Decision-Making/) | [PDF](https://arxiv.org/pdf/2202.01771.pdf)\u003cbr/\u003e\nS. Li, X. Puig, C. Paxton, Y. Du, C. Wang, L. Fan, T. Chen, D. Huang, E. Akyürek, A. Anandkumar, J. Andreas, I. Mordatch, A. Torralba, Y. Zhu.\u003cbr/\u003e\nNeurIPS 2022, Oral.\u003cbr/\u003e\n\nIf you plan to use the simulator, please cite both of the following papers (the first one introduced v1.0 and the second introduced v2.0, aka VirutalHome-Social):\n\n```\n@inproceedings{puig2018virtualhome,\n  title={Virtualhome: Simulating household activities via programs},\n  author={Puig, Xavier and Ra, Kevin and Boben, Marko and Li, Jiaman and Wang, Tingwu and Fidler, Sanja and Torralba, Antonio},\n  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},\n  pages={8494--8502},\n  year={2018}\n}\n```\n\n\n```\n@misc{puig2020watchandhelp,\n      title={Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration}, \n      author={Xavier Puig and Tianmin Shu and Shuang Li and Zilin Wang and Joshua B. Tenenbaum and Sanja Fidler and Antonio Torralba},\n      year={2020},\n      eprint={2010.09890},\n      archivePrefix={arXiv},\n      primaryClass={cs.AI}\n}\n```\n______________________________________________________________________\n\n\n## Contributors\n\nThe VirtualHome API and code have been developed by the following people.\n- Xavier Puig\n- Marko Boben\n- Kabir Swain\n- Jordan Ren\n- Kevin Ra\n- Zilin Wang\n- Shuang Li\n- Tianmin Shu\n- Andrew Liao\n\n______________________________________________________________________\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxavierpuigf%2Fvirtualhome","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fxavierpuigf%2Fvirtualhome","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxavierpuigf%2Fvirtualhome/lists"}