{"id":15160311,"url":"https://github.com/fcbg-platforms/vr-environment-genai-unity","last_synced_at":"2025-09-30T11:32:09.803Z","repository":{"id":254712379,"uuid":"847305182","full_name":"fcbg-platforms/VR-Environment-GenAI-Unity","owner":"fcbg-platforms","description":"Generative AI to create immersive VR environments. This is the Unity part (the client). ","archived":true,"fork":false,"pushed_at":"2024-09-17T16:41:46.000Z","size":53143,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-09-27T22:43:30.674Z","etag":null,"topics":["environment-creator","genai","mrtk","unity","vr"],"latest_commit_sha":null,"homepage":"","language":"C#","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fcbg-platforms.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-08-25T12:55:38.000Z","updated_at":"2024-09-25T13:39:27.000Z","dependencies_parsed_at":"2024-09-26T22:53:31.228Z","dependency_job_id":null,"html_url":"https://github.com/fcbg-platforms/VR-Environment-GenAI-Unity","commit_stats":{"total_commits":8,"total_committers":2,"mean_commits":4.0,"dds":0.25,"last_synced_commit":"c2c416971d4b23b75132dbe8d417dc12885c51d8"},"previous_names":["fcbg-hnp-vr/vr-environment-genai-unity","fcbg-platforms/vr-environment-genai-unity"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fcbg-platforms%2FVR-Environment-GenAI-Unity","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fcbg-platforms%2FVR-Environment-GenAI-Unity/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fcbg-platforms%2FVR-Environment-GenAI-Unity/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fcbg-platforms%2FVR-Environment-GenAI-Unity/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fcbg-platforms","download_url":"https://codeload.github.com/fcbg-platforms/VR-Environment-GenAI-Unity/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":234732482,"owners_count":18878416,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["environment-creator","genai","mrtk","unity","vr"],"created_at":"2024-09-26T22:43:55.820Z","updated_at":"2025-09-30T11:32:09.798Z","avatar_url":"https://github.com/fcbg-platforms.png","language":"C#","funding_links":[],"categories":[],"sub_categories":[],"readme":"# VR Environment creation by Generative AI, Unity client\n\nThis Unity project is a client for generating, modifying and exploring environments generated by AI from voice prompts.\n\n![Main VR scene](Documentation/resources/Main_view_VR.png \"View of the main VR screen.\")\n\nIt features two main scenes, one for desktop and one for virtual reality, that both let you explore new worlds.\nThe idea of this project is to let  you access all the AI features from an application, reducing the interruptions in the creation process.\n\n[Short complete demo.webm](https://github.com/user-attachments/assets/b8a97c92-ae75-4732-95af-c4fb64345bb0)\n\nThe project was developed at the [Fondation Campus Biotech Geneva](https://fcbg.ch/) ([GitHub](https://github.com/fcbg-hnp-vr/)),\nin collaboration with the [Laboratory of Cognitive Science](https://www.epfl.ch/labs/lnco/) (part of [EPLF](https://epfl.ch)).\n\n## Installation\n\nThe installation is simple, as we use standard Unity features.\nHere is the detailed guide.\n\n1. Download Unity Editor 2022.3.36+.\n2. Open the project and you should be able to use scenes, without the AI features.\n\nMost VR headsets compatible with OpenXR should work out of the box.\nThe project was developed with the HP Reverb G2 (a Windows Mixed Reality Headset).\n\n- If the XR assets do not work, you can try to reinstall them from the [Package Manager](https://docs.unity3d.com/Manual/Packages.html) (Package manager -\u003e XR Interaction Toolkit -\u003e Samples -\u003e Starter Assets -\u003e Reimport).\n- For Windows Mixed Reality Headsets, you may also try to reimport the plugins.\n\n\u003e [!Important]\n\u003e This is only the client,\nyou need a running server to actually use the AI features!\nThe repository to run the server can be found at: [VR-Environment-GenAI-Server](https://github.com/fcbg-hnp-vr/VR-Environment-GenAI-Server).\n\n## Usage\n\nThis Unity project features a scene generated by an AI from a text prompt.\nThe scene is dynamically created based on the input text and allows for unique and creative environments to be generated.\n\nTo generate a scene:\n\n1. Start the Python server, and edit the [configuration](#configuration) if necessary.\n2. Load the desired scene. On desktop load the \"Desktop Creator\" scene, and the \"VR Creator\" scene in VR.\n3. Enter a text prompt in the input field. In VR you can speak while holding the microphone.\n4. Press the \"Generate\" button.\n5. Watch as the AI creates a scene based on your text prompt. A progress bar indicates the remaining time.\n\n## Configuration\n\nYou may need to set a few configuration parameters for a smooth experience.\nMost configuration parameters are under `Assets/Configurations/api.json`.\n\nIn case you are running the project locally VR and AI on the same computer (the \"debug\" configuration), you should set the following.\n\n- `pythonFallbackApiFile`: file path to the Python ``api.json`` file, this file should be in the root folder of the Python project.\n\nIf you want to connect to a remote server, the important features to set:\n\n- `serverDefaultIp`: the IP to reach the AI server (IPv4 preferred).\n- `serverDefaultPort`: server default port number.\n\n\u003e **Note:** if both the server and the fallback api file are correctly specified, the server will be ignored and the connection will be local. You can replace the fallback api file path by any wrong path to prevent this behavior.\n\nSome options are considered legacy and where used for communication on the same file storage.\nYou can specify where Python should save the files, temporary locations are better.\n\n- `audioPath`: the path to the temporary audio file to save when using the speech-to-text feature.\n- `baseImage`: the path to the texture *that will be modified*.\n- `maskPath`: the path to the temporary mask texture that is used during the inpainting feature.\n\n## Features\n\nThe main features of this project are:\n\n- Text-to-Skybox generation.\n- Speech-to-text: speak naturally.\n- Mask drawing and inpainting: freely select and redraw elements.\n\n## Scenes\n\nAll the following scenes are in the ``Assets/Scenes`` folder.\n\n- \"Desktop Creator\" - Main scene for skybox creation in desktop mode.\n- \"VR Creator\" - Skybox creation with a VR headset. You can also use it when no headset is plugged in, but this is much of a debug feature.\n\nThe other scenes are for testing pruposes:\n\n- \"Speech Recognition\" - A simple scene that only integrates speech-to-text.\n- \"Experiments Scene\" - A scene containing various legacy experiments.\n\n## Documentation\n\nWe provide an [online documentation](https://fcbg-hnp-vr.github.io/VR-Environment-GenAI-Unity/), you can also access it in the `docs/` folder.\n\nThe documentation is generated with DocFX, you can regenerate it with [DocFX](https://github.com/dotnet/docfx).\n\n```bash\ncp README.md Documentation/index.md\n\n# Option 1: Offline, static documentation\ndocfx Documentation/docfx.json -t statictoc,custom_template\n\n# Option 2: on a web server\ndocfx Documentation/docfx.json\n```\n\nBoth commands will generate a documentation under `docs/`.\n\n## Legacy features\n\n### Skybox Import\n\nThis project allows for the import of custom skyboxes.\n\nThe skybox is automatically imported into the scene when the player hits enter.\n\nLaunch the scene:\n\n1. Maintain left click to start drawing.\n2. Use the scroll wheel to change the radius of the drawing area.\n3. Use right click to save the mask. It is saved in \"Assets/Temp/mask.png\" by default.\n\n### Breathing Experiment for Meditation\n\nAs a small bonus feature, this project includes a small breathing experiment for meditation.\n\nTo start the breathing experiment:\n\n1. Enable the breathing asset.\n2. Press the space bar on in-breathing and release on out-breath.\n\n## Links\n\nYou can find the active public repository for this project at: \u003chttps://github.com/HugoFara/speech-to-world-unity-client\u003e\n\nFor the official public AI server: \u003chttps://github.com/fcbg-hnp-vr/VR-Environment-GenAI-Server\u003e.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffcbg-platforms%2Fvr-environment-genai-unity","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffcbg-platforms%2Fvr-environment-genai-unity","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffcbg-platforms%2Fvr-environment-genai-unity/lists"}