{"id":31653110,"url":"https://github.com/boschresearch/image-render-generative","last_synced_at":"2026-01-20T16:35:53.991Z","repository":{"id":210019628,"uuid":"725495776","full_name":"boschresearch/image-render-generative","owner":"boschresearch","description":"Image rendering with generative AI methods","archived":false,"fork":false,"pushed_at":"2024-08-29T06:27:06.000Z","size":65,"stargazers_count":0,"open_issues_count":0,"forks_count":4,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-08-29T07:58:17.273Z","etag":null,"topics":["image-render"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/boschresearch.png","metadata":{"files":{"readme":"Readme.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-11-30T09:10:04.000Z","updated_at":"2024-08-29T06:27:11.000Z","dependencies_parsed_at":"2024-06-07T13:30:24.886Z","dependency_job_id":"3f6bc47d-561c-47b0-b047-9612622c1f24","html_url":"https://github.com/boschresearch/image-render-generative","commit_stats":null,"previous_names":["boschresearch/image-render-generative"],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/boschresearch/image-render-generative","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/boschresearch%2Fimage-render-generative","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/boschresearch%2Fimage-render-generative/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/boschresearch%2Fimage-render-generative/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/boschresearch%2Fimage-render-generative/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/boschresearch","download_url":"https://codeload.github.com/boschresearch/image-render-generative/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/boschresearch%2Fimage-render-generative/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278762925,"owners_count":26041444,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-07T02:00:06.786Z","response_time":59,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["image-render"],"created_at":"2025-10-07T10:41:16.964Z","updated_at":"2025-10-07T10:41:18.400Z","avatar_url":"https://github.com/boschresearch.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Image Render Generative\n\n**Check out the `stable` branch for the latest stable version**\n\nThis catharsys plugin adds functionality for using generative models to\nenhance or create new synthetic data based on renderings from catharsys.\n\n## Purpose of the project\n\nThis software is a research prototype, originally developed for use in rendering automation.\n\nThe software is not ready for production use. It has neither been developed nor tested for a specific use case. However, the license conditions of the applicable Open Source licenses allow you to adapt the software to your needs. Before using it in a safety relevant setting, make sure that the software fulfills your requirements and adjust it according to any applicable safety standards (e.g. ISO 26262).\n\n## Contributing to the project\n\nPlease read the conditions you need to comply with in order to contribute to this project in the [CONTRIBUTING](CONTRIBUTING.md) file. \n\n## License\n\nThe `image-render-generative` package is licenced under the Apache-2.0 license. See the [LICENSE](LICENSE.md) file for more details.\n\n\n## Overview\n\nThis catharsys plugin adds functionality for using generative models to\nenhance or create new synthetic data based on renderings from catharsys.\n\nCurrently, the following methods are available:\n- Takuma's Multi Control net for diffusers\n- (Control Net Nightly, deprecated)\n\nSince the dependencies of the methods differ from each other and from\ncatharsys, for each method a separate and new conda environment has\nto be created. To do so, call the respective scripts.\n\n\n## Install the Action\n\nTo use the generative actions, you need to install them into your Catharsys workspace. If you have a development installation change into the `image-render-generative` folder and execute the command `pip install -e .`. Otherwise, execute `pip install .` from within the folder. Now the actions are available from Catharsys configurations.\n\n\n## Takuma Multi Control Net\n\nThis model is based on the implementation of github user takuma104:\n\nhttps://github.com/takuma104/diffusers/tree/multi_controlnet\n\nThis implementation is a fork of the popular python package diffusers.\nTo setup the conda environment, download and install the repository\nand download all the needed model files, simply call\n\n    bash scripts/get_takuma_diffusers.sh\n\nThis also downloads additional networks into the ./models folder with\ngit lfs. Make sure git lfs is available before calling the script.\nAlso, consider to remove the .git folder in the models folder to save.\nAlso you can directly point to a different location in your catharsys\nconfig files.\n\nHowever, we suggest to perform the steps manually one after another and\nalso store the additional model files not directly in this module, but\nat a central location and reuse it from there.\n\nTo use takumas multi control net, you need to first add a pose painting\nstep to your actions and then add the generative step.\n\n### Create open pose images by paint pose action\n\nTo add pose painting to your config, add the following in the respective files:\n\nmanifest.json5:\n\n    // ...\n    paint_pose: {\n        sDTI: \"manifest/action:1\",\n        lConfigs: [\n            { sId: \"paint_pose_config\", sDTI: \"/catharsys/generative/paint_pose_data:1\", sForm: \"file/json\", bAddToPath: false }\n        ],\n        lDeps: [ \"label\" ] // \u003c- this should be the name of the action you use for  labeling\n    },\n    // ...\n\ntrial.json5:\n\n    // ...\n    mConfigs: {\n        ... \n        paint_pose_config: [ \"../generative/openpose\"],\n        ...\n    },\n    // ...\n\nlaunch.json5:\n\n    // ...\n    \"paint_pose\": {\n        sDTI: \"/catharsys/launch/action:1.0\",\n        sActionDTI: \"/catharsys/action/generative/paint_pose:1.0\",\n        mConfig: {\n            sDTI: \"/catharsys/launch/args:1.1\",\n            sInfo: \"Create an open pose pose image for processing with multi control nets\",\n            sExecFile: \"../exec/py_std_v1\",\n            iConfigGroups: 1,\n            iFrameGroups: 1\n        }     \n    },\n    // ...\n\nINPORTANT: Copy the file openpose.json5 from the examples folder from this repository to generative/openpose.json5 in your config.\n\n\nFinally, you can call the action to generate the pose image needed by the multi control net:\n\n    cathy ws launch -c config/your_workspace -a paint_pose\n\n\n### Add control net action to your catharsys config\n\nFor using Takuma's control net, you now have to add the following parts to your config files.\n\nmanifest.json5:\n\n    // ...\n    generative: {\n        sDTI: \"manifest/action:1\",\n        lConfigs: [\n            { sId: \"cnet_model\", sDTI: \"/catharsys/generative/model-id:1\", sForm: \"value\", bAddToPath: true },\n            { sId: \"cnet_config\", sDTI: \"/catharsys/generative/data:1\", sForm: \"file/json\", bAddToPath: true  }\n        ],\n        lDeps: [ \"render/std\" ] // \u003c- this should be your standard render action\n    },\n    // ...\n\ntrial.json5:\n\n    // ...\n    mConfigs: {\n        // ...\n        cnet_model: [\"Takuma-Diffusers\"],\n        cnet_config: [\"generative/example\" ],\n        // ...\n    },\n    // ...\n\nlaunch.json5:\n\n    \"generative\": {\n        sDTI: \"/catharsys/launch/action:1.0\",\n        sActionDTI: \"/catharsys/action/generative/apply:1.0\",\n        mConfig: {\n            sDTI: \"/catharsys/launch/args:1.1\",\n            sInfo: \"Perform generative action using control net.\",\n            // sExecFile: \"../exec/py_lsf_v1\", // \u003c- use this when using lsf\n            sExecFile: \"../exec/py_std_v1\",\n            iConfigGroups: 1,\n            iFrameGroups: 1\n        }     \n    },\n\nFinally, you need a config file generative/example.json5. Here, several control nets are added as examples. Use the ones you are interested in:\n\n    {\n        sDTI: \"/catharsys/generative/data:1\",\n\n        __locals__: {\n            sModelPath: \"/path/to/your/models\" // \u003c- add here the path to the models folder\n        },\n\n        mInputs: {\n            // here, all relative path to the outputs of previous results should be added\n            // these will be referenced in the sInput to the control nets below\n            Depth: \"Depth\",\n            OpenPose: \"AT_Label/asp_skeleton/PoseImage\",\n            SemSeg: \"AT_Label/semseg_only/Preview\"\n\n        },\n\n        mConfig: {\n            mParams: {\n                pretrained_model_name_or_path: \"${sModelPath}/stable-diffusion-v1-5\",\n                local_files_only: true,\n            },        \n            mControlNets: {\n                Depth: {\n                    // these parameters are directly passed to from_pretrained\n                    mParams: {\n                        pretrained_model_name_or_path: \"${sModelPath}/sd-controlnet-depth\",\n                        local_files_only: true,\n                    },\n                    fWeight: 0.6,\n                    sInput: \"Depth\",\n                    mPreprocess: [\n                        \"Normalize\",\n                        \"Invert\",\n                    ]\n                },\n                OpenPose: {\n                    mParams: {\n                        pretrained_model_name_or_path: \"${sModelPath}/control_v11p_sd15_openpose\",\n                        local_files_only: true,\n                    },\n                    fWeight: 1.0,\n                    sInput: \"OpenPose\",\n                },\n                SemSeg: {\n                    // these parameters are directly passed to from_pretrained\n                    mParams: {\n                        pretrained_model_name_or_path: \"${sModelPath}/control_v11p_sd15_seg\",\n                        local_files_only: true,\n                    },\n                    fWeight: 0.75,\n                    sInput: \"SemSeg\",\n                },               \n            },\n            sPrompt: \"men in black suits with ties and dark sunglasses (as in the film men in black or FBI agents)\",\n            sNegativePrompt: \"bad quality, low res\"\n        }\n    }\n\nNow, you have everything to perform the generative action:\n\n    cathy ws launch -c config/your_workspace -a generative\n\nHave fun!\n\n## Futher remarks\n\nPrompt engineering is the new art. For getting more realistic results, consider these sources:\n* https://stable-diffusion-art.com/realistic-people/\n\n\nPrompt Free Diffusion\n==============\nThis plugin provides an interface to use Prompt-free-Diffusion within Catharsys.\n\nInstallation. Run scripts/get_prompt_free_diffusion.sh \n\nUsing the PFD Action\n==============\nTo use this action, integrate it into your workspace accordingly.\n\nmanifest.json5:\n```\n    promptfreediffusion: {\n      std:{\n        sDTI: \"manifest/action:1\",\n        lConfigs: [       \n            // add the input image even if it is not used, mainly to get the frames etc\n            { sId: \"cnet_model\", sDTI: \"/catharsys/generative/model-id:1\", sForm: \"value\", bAddToPath: false },\n            { sId: \"cnet_config\", sDTI: \"/catharsys/generative/data:1\", sForm: \"file/json\", bAddToPath: true},          \n        ],\n        lDeps: [ \"render/std\" ]\n      },\n    }, \n```\ntrial.json5:\n\n```\n    cnet_model: [\"Prompt-Free-Diffusion\"],\n    cnet_config: [\"generative/example_pfd_config\"],\n```\n\nlaunch.json5:\n```\n    \"promptfreediffusion\": {\n        std: {\n          sDTI: \"/catharsys/launch/action:1.0\",\n          sActionDTI: \"/catharsys/action/promptfreediffusion/apply:1.0\",\n          mConfig: {\n              sDTI: \"/catharsys/launch/args:1.1\",\n              sInfo: \"Perform generative action using control net.\",\n              // sExecFile: \"../exec/py_lsf_v1\",\n              sExecFile: \"../exec/py_std_v1\",\n              // sExecFile: \"../exec/py_auto\",\n              iConfigsPerGroup: 40,\n              iFramesPerGroup: 1,\n              iRangeCount:1,\n              iRenderQuality: 4,\n          }     \n        },\n    }\n```\n\nFinally, a config file defining the arguments for prompt free diffusion we already referred to in the trial.json above is needed:\n\n```\n{\n    sDTI: \"/catharsys/generative/data:1\",\n\n    mInputs: {\n        Depth: \"Depth\",\n    },\n\n    mConfig: {\n        mParams: {\n            prompt_free_diffusion_root_path: \"path to the prompt free diffusion repo\",\n            reference_image: \"Path to the reference image to be used\"\n            // Diffusers supported by prompt free diffusion\n            sDiffuser_id: \"Deliberate-v2.0\",\n                // Possible Values:\n                // 'SD-v1.5'             \n                // 'OpenJouney-v4'       \n                // 'Deliberate-v2.0'     \n                // 'RealisticVision-v2.0'\n                // 'Anything-v4'         \n                // 'Oam-v3'              \n                // 'Oam-v2'\n            // Seecoders supported by prompt free diffusion\n            sSeecoder_id: \"SeeCoder\",\n                // 'SeeCoder'      \n                // 'SeeCoder-PA'   \n                // 'SeeCoder-Anime'             \n            iSeed: 20,\n            fCFGScale: 2.0\n        },        \n        mControlNets: {\n            Depth: {\n                sInput: \"Depth\",\n                mPreprocess: [\n                    \"Normalize\",\n                    \"Invert\",\n                ]\n            },\n        },\n    }\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fboschresearch%2Fimage-render-generative","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fboschresearch%2Fimage-render-generative","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fboschresearch%2Fimage-render-generative/lists"}