{"id":22739832,"url":"https://github.com/smilingrobo/imagination-to-real","last_synced_at":"2025-04-12T20:52:57.337Z","repository":{"id":264300063,"uuid":"892978471","full_name":"SmilingRobo/imagination-to-real","owner":"SmilingRobo","description":"Train your robot to do whatever you want using Generative AI","archived":false,"fork":false,"pushed_at":"2025-02-06T15:59:30.000Z","size":40961,"stargazers_count":4,"open_issues_count":1,"forks_count":2,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-12T20:52:49.437Z","etag":null,"topics":["library","robot","robotframework","robotics"],"latest_commit_sha":null,"homepage":"https://www.smilingrobo.com","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SmilingRobo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-11-23T07:47:03.000Z","updated_at":"2025-02-06T15:59:33.000Z","dependencies_parsed_at":"2024-11-23T08:25:51.298Z","dependency_job_id":"6c742fb4-1aa8-495c-8534-2b02cd3cad92","html_url":"https://github.com/SmilingRobo/imagination-to-real","commit_stats":null,"previous_names":["smilingrobo/imagination-to-real"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SmilingRobo%2Fimagination-to-real","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SmilingRobo%2Fimagination-to-real/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SmilingRobo%2Fimagination-to-real/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SmilingRobo%2Fimagination-to-real/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SmilingRobo","download_url":"https://codeload.github.com/SmilingRobo/imagination-to-real/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248631725,"owners_count":21136560,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["library","robot","robotframework","robotics"],"created_at":"2024-12-10T23:06:17.971Z","updated_at":"2025-04-12T20:52:57.330Z","avatar_url":"https://github.com/SmilingRobo.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# **imagination_to_real** by SmilingRobo \n\n### [🌐 SmilingRobo](https://www.smilingrobo.com) | [📝 Paper](https://arxiv.org/abs/2411.00083) | [Open-Source Sprint](https://opensource.smilingrobo.com/projects/imagination-to-real/)\n\n**imagination-to-real** Train your robot to do whatever you want using Generative AI\n\n#### Description \nimagination-to-real empowers robotics developers by bridging the gap between generative AI and classical physics simulators. Our library prepares realistic, diverse, and geometrically accurate visual data from generative models. This data enables robots to learn complex and highly dynamic tasks, such as parkour, without requiring depth sensors.\n\n🚀 What It Does:\n\n⚪ Integrates generative models with simulators to create rich, synthetic datasets.\u003cbr\u003e\n⚪ Ensures temporal consistency with tools like Dreams In Motion (DIM).\u003cbr\u003e\n⚪ Offers compatibility with MuJoCo environments for seamless data preparation.\u003cbr\u003e\n\n🛠️ How to Use:\n\n⚪ Use Image_Maker for text-to-image generation tailored to your simulation needs.\u003cbr\u003e\n⚪ Combine the generated data with your preferred training framework to develop robust robot learning models.\u003cbr\u003e\n\n\u003e *We are creating SmilingRobo Cloud, which will allow you to train your robot using our innovative libraries and drag-and-drop facilities.*  \n\n---\n\n**Table of Contents**\n- [Install imagination_to_real](#installing-imagination_to_real-module)\n- [Image_Maker](#make-images-using-image_maker)\n  - [Installation](#installation)\n    - [Install ComfyUI + Dependencies](#install-comfyui)\n    - [Setting up Models](#setting-up-models)\n  - [Usage](#usage)\n    - [Running the Example Workflow](#running-the-example-workflow)\n    - [Adding Your Own Workflows](#adding-your-own-workflows)\n    - [Scaling Image Generation](#scaling-image-generation)\n- [Create Environment](#create-environment)\n  - [Installing Dependencies](#1️-installing-gym_dmc)\n  - [Usage](#usage)\n    - [Basic LucidSim Pipeline](#rendering-conditioning-images)\n    - [Full Rendering Pipeline](#full-lucidsim-rendering-pipeline)\n\n- [Citation](#citation)\n\n\n# Installing imagination_to_real module\n\n#### 1. Setup Conda Environment\n\n```bash\nconda create -n imagination_to_real python=3.10\nconda activate imagination_to_real\ngit clone https://github.com/SmilingRobo/imagination-to-real imagination_to_real\ncd imagination_to_real\npip install -e .\n\n```\n\n## Make Images using image_maker\n\n#### Install-ComfyUI\n\nFor consistency, we recommend\nusing [this version](https://github.com/comfyanonymous/ComfyUI/tree/ed2fa105ae29af6621232dd8ef622ff1e3346b3f) of\nComfyUI.\n\n```bash\n# Choose the CUDA version that your GPU supports. We will use CUDA 12.1\npip install torch==2.1.2 torchvision==0.16.2 torchaudio==2.1.2 --extra-index-url https://download.pytorch.org/whl/cu121\n\n# Installing ComfyUI\ngit clone https://github.com/comfyanonymous/ComfyUI\ncd ComfyUI\ngit checkout ed2fa105ae29af6621232dd8ef622ff1e3346b3f\npip install -r requirements.txt\n\n```\n\n#### Setting-up-Models\n\nWe recommend placing your models outside the `ComfyUI` repo for better housekeeping. For this, you'll need to link your\nmodel paths through a config file. Check out the `configs` folder for a template, where you'll specify locations for\ncheckpoints, controlnets, and VAEs. For the provided `three_mask_workflow` example, these are the models you'll need:\n\n- [SDXL Turbo 1.0](https://huggingface.co/stabilityai/sdxl-turbo/blob/main/sd_xl_turbo_1.0_fp16.safetensors): place\n  under `checkpoints`\n- [SDXL Depth ControlNet](https://huggingface.co/diffusers/controlnet-depth-sdxl-1.0): place under `controlnet`\n- [SDXL VAE](https://huggingface.co/stabilityai/sdxl-vae): place under `vae`\n\nAfter cloning this repository, you'll need to add ComfyUI to your `$PYTHONPATH` and link your model paths. We recommend\nmanaging these in a local `.env` file. Then, link the config file you just created.\n\n```bash\nexport PYTHONPATH=/path/to/ComfyUI:$PYTHONPATH\n\n# See the `configs` folder for a template\nexport COMFYUI_CONFIG_PATH=/path/to/extra_model_paths.yaml\n```\n\n## Usage\n\nimagination_to_real is organized by _workflows_. We include our main workflow called `three_mask_workflow`, which generates an image\ngiven a depth map along with three semantic masks, each coming with a different prompt (for example,\nforeground/background/object).\n\n#### Running the Workflow\n\n```bash\npython imagination_to_real/image_maker/scripts/demo_three_mask_workflow.py [--path-to-folder] [--seed] [--save]\n```\n\nwhere `path-to-folder` corresponds to your data to generate images, and the `save` flag writes the output to the corresponding `examples/three_mask_workflow/[example-name]/samples` folder. The script will randomly select one of our provided prompts.\n\nTo make your data, take this example as the reference `examples/image-maker/three_mask_workflow/ramps`\n\n##### Example\n\nWe provide example conditioning images and prompts for `three_mask_workflow` under the `examples/image-maker/three_mask_workflow` folder, grouped by scene. \n\nTo try it out, use:\n\n```bash\npython imagination_to_real/image_maker/scripts/demo_three_mask_workflow.py [--example-name] [--seed] [--save]\n```\n\nwhere `example-name` corresponds to one of the scenes in the `examples/image-maker/three_mask_workflow` folder.\n\n#### Adding Your Own Workflows\n\nThe graphical interface for ComfyUI is very helpful for designing your own workflows. Please see their documentation for\nhow to do this. By using this\nhelpful [workflow to python conversion tool](https://github.com/pydn/ComfyUI-to-Python-Extension.git), you can script\nyour workflows as we've done with `Image_Maker/workflows/three_mask_workflow.py`.\n\n#### Scaling Image Generation\n\nIn LucidSim, we use a distributed setup to generate images at scale. We utilize rendering nodes, launched independently\non many machines, that receive and fulfill rendering requests from the physics engine containing prompts and\nconditioning images through a task queue (see [Zaku](https://zaku.readthedocs.io/en/latest/)). We hope to release setup\ninstructions for this in the future, but we have included `Image_Maker/render_node.py` for your reference.\n\n---\n\n## Create Environment\n\n\u003ctable style=\"border-collapse: collapse; border: none; width: 100%;\"\u003e\n  \u003ctr\u003e\n    \u003ctd style=\"text-align: center; border: none;\"\u003e\n      \u003cimg src=\"assets/images/example_conditioning.png\" style=\"width: 500px; max-width: 100%;\" /\u003e\u003cbr\u003e\n    \u003c/td\u003e\n    \u003ctd style=\"text-align: center; border: none;\"\u003e\n      \u003cimg src=\"assets/images/example_imagen.png\" style=\"width: 500px; max-width: 100%;\" /\u003e\u003cbr\u003e\n    \u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\n#### 1.Installing gym_dmc\n\nThe last few dependencies require a downgraded `setuptools` and `wheel` to install. To install, please downgrade and\nrevert after.\n\n```bash\npip install setuptools==65.5.0 wheel==0.38.4 pip==23\npip install gym==0.21.0\npip install gym-dmc==0.2.9\npip install -U setuptools wheel pip\n```\n\n#### Usage\n\n**Note:** On Linux, make sure to set the environment variable ` MUJOCO_GL=egl`.\n\nLucidSim generates photorealistic images by using a generative model to augment the simulator's rendering, using\nconditioning images to maintain control over the scene geometry.\n\n#### Rendering Conditioning Images\n\nWe have provided an expert policy checkpoint under `checkpoints/expert.pt`. This policy was derived from that\nof [Extreme Parkour](https://github.com/chengxuxin/extreme-parkour). You can use this policy to sample an environment\nand visualize the conditioning images with:\n\n#### If you are using custom data\n\n\u003e if you are follwing example just run the script\n\n1. Create a `name.py` file in `imagination_to_real/specs` take `gaps.py` as reference, change line `5`.\n\n2. create your `name`.py and `name`.xml file in `imagination_to_real/lucidsim/tasks`. Take the `gaps.py` and `gaps.xml` as reference and just change the line `11` of .py and `3` of .xml\n\u003e if you are using your own robot then you have to change line `2` of xml too.\n\n3. Make you `name`.xml file and add it here `imagination_to_real/lucidsim/tasks/assets/terrains`. take the `gaps.xml` as reference\n\n\n```bash\n# example env-name: one of ['parkour', 'hurdle', 'gaps', 'stairs_v1', 'stairs_v2']\n!python imagination_to_real/lucidsim/scripts/play.py --save-path [--env-name] [--num-steps] [--seed]\n````\n\nwhere `save_path` is where to save the resulting video.\n\n#### Full Rendering Pipeline\n\nTo run the full generative augmentation pipeline, please also make sure the environment variables are still\nset correctly:\n\n```bash\nCOMFYUI_CONFIG_PATH=/path/to/extra_model_paths.yaml\nPYTHONPATH=/path/to/ComfyUI:$PYTHONPATH\n```\n\n\nYou can then run the full pipeline with:\n\n```bash\npython imagination_to_real/lucidsim/scripts/play_three_mask_workflow.py --save-path --prompt-collection [--env-name] [--num-steps] [--seed]\n```\n\nwhere `save_path` and `env_name` are the same as before. `prompt_collection` should be a path to a `.jsonl` file with\ncorrectly formatted prompts, as in the `imagination-to-real/tree/master/examples/image-maker/three_mask_workflow` folder.\n\n---\n\nWe thank the authors of [LucidSim](https://github.com/lucidsim/lucidsim) for their opensource code and [Extreme Parkour](https://github.com/chengxuxin/extreme-parkour) for their open-source codebase, which we used as a starting point for our library.\n\n\n\n## Citation\n\nIf you find our work useful, please consider citing:\n\n```\n@inproceedings{yu2024learning,\n  title={Learning Visual Parkour from Generated Images},\n  author={Alan Yu and Ge Yang and Ran Choi and Yajvan Ravan and John Leonard and Phillip Isola},\n  booktitle={8th Annual Conference on Robot Learning},\n  year={2024},\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsmilingrobo%2Fimagination-to-real","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsmilingrobo%2Fimagination-to-real","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsmilingrobo%2Fimagination-to-real/lists"}