{"id":15146599,"url":"https://github.com/graemeniedermayer/3d-depth-gen","last_synced_at":"2026-02-23T12:08:02.927Z","repository":{"id":231582280,"uuid":"782116549","full_name":"graemeniedermayer/3d-depth-gen","owner":"graemeniedermayer","description":"auto1111 extension txt-to-3d. This uses monodepth maps + bpy (blender). WIP","archived":false,"fork":false,"pushed_at":"2024-08-27T03:04:37.000Z","size":7002,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-13T05:17:39.381Z","etag":null,"topics":["3d-reconstruction","automatic1111","monodepth","webxr"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/graemeniedermayer.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-04-04T17:06:30.000Z","updated_at":"2024-08-27T03:04:40.000Z","dependencies_parsed_at":"2024-11-02T00:41:13.311Z","dependency_job_id":"caaf02d1-4c80-40cd-94ab-b968e0faf9a2","html_url":"https://github.com/graemeniedermayer/3d-depth-gen","commit_stats":{"total_commits":11,"total_committers":1,"mean_commits":11.0,"dds":0.0,"last_synced_commit":"c3f2f4c78d23c4ef32109d956163f4a94ac2e54a"},"previous_names":["graemeniedermayer/3d-depth-gen"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/graemeniedermayer%2F3d-depth-gen","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/graemeniedermayer%2F3d-depth-gen/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/graemeniedermayer%2F3d-depth-gen/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/graemeniedermayer%2F3d-depth-gen/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/graemeniedermayer","download_url":"https://codeload.github.com/graemeniedermayer/3d-depth-gen/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247563930,"owners_count":20958971,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d-reconstruction","automatic1111","monodepth","webxr"],"created_at":"2024-09-26T12:03:08.878Z","updated_at":"2025-10-25T15:38:30.679Z","avatar_url":"https://github.com/graemeniedermayer.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 3d-depth-gen (WIP)\nCreate simple animated mesh from stable diffusion and monodepth.\n\nThis project has a secondary focus on the api to generate ai 3d objects for webxr.\n\n\u003cimg src=\"https://github.com/graemeniedermayer/3d-depth-gen/assets/38518120/62caf497-c64e-4923-abb8-2bcba1c9f955\" data-canonical-src=\"https://github.com/graemeniedermayer/3d-depth-gen/assets/38518120/62caf497-c64e-4923-abb8-2bcba1c9f955\" width=\"200\" height=\"200\" /\u003e\n\n## Installation\nRequires python3.10 + bpy (blender)\n\nRequires https://github.com/thygate/stable-diffusion-webui-depthmap-script\n\nShould work with install from URL \n\nShould look like\n![Screenshot from 2024-04-04 12-11-30](https://github.com/graemeniedermayer/3d-depth-gen/assets/38518120/22bb9093-0b90-45dd-bd9e-e310b268c9b9)\n\n## Goal List\nAspirational list for this project\n- lots of bugfixes (TODO put together install.py)\n- unity integration\n- docker image\n- tangent normal maps don't seem accurate\n- deeper integration for augmented reality\n- diffuse layer integration (alpha 3d objects?)\n\n## Quest 3 example\n- Must run off of a separate server (TODO maybe run off same hostname?).\n- Must be run off of ssl (webxr requirement).\n- Request URL will need to be replaced by the address pointing to server name.\n\n## JavaScript API Request example\nthreejs examples. hopefully this will be simpler in the future\n```\nlet fetch3d = (prompt)=\u003e{\n    const loader = new GLTFLoader();\n    const url_base = \"https://path_to_server.com:7860/\"\n    const url_depth = url_base + 'depth/generate'\n    const url_sd = url_base + 'sdapi/v1/txt2img'\n    const url_3d = url_base + 'depthcreation/generate'\n    const dic_sd = {\n        'prompt': prompt,\n        'negative_prompt': '',\n        'steps': 20,\n        'width': 1024,\n        'height': 1024, \n    }\n    fetch(url_sd,\n        {\n            method: 'POST',\n            body: JSON.stringify(dic_sd), \n            headers: {\n                'Content-Type': 'application/json'\n              },\n            mode:'cors',  \n        }\n    )\n    .then((e)=\u003ee.json())\n    .then((e)=\u003e{\n        const color_image = e['images'][0]\n        const dic_depth = {\n            \"depth_input_images\": [ color_image],\n            \"options\":{\n              \"compute_device\": \"GPU\",\n              \"model_type\": 7,\n              \"net_width\": 512,\n              \"net_height\":512,\n              \"net_size_match\": false,\n              \"boost\": false,\n              \"invert_depth\": false,\n              \"gen_rembg\": true,\n              \"rembg_model\": \"isnet-general-use\"\n            }\n        }\n    fetch(url_depth,\n        {\n            method: 'POST',\n            body: JSON.stringify(dic_depth), \n            headers: {\n                'Content-Type': 'application/json'\n              },\n            mode:'cors',  \n        }\n    )\n    .then((e)=\u003ee.json())\n    .then((e)=\u003e{\n        // use index 1 if background is removed\n        const depth_image = e['images'][1]\n\n        const dic_3d = {\n            'input_images': [color_image],\n            'depth_images': [depth_image],\n            'generate_options':{\n                'double_sided':true,\n                'net_width': 512,\n                'net_height':512,\n                'file_type':'glb'\n            }\n        }\n        fetch(url_3d,\n            {\n                method: 'POST',\n                body: JSON.stringify(dic_3d), \n                headers: {\n                    'Content-Type': 'application/json'\n                  },\n                mode:'cors',  \n            }\n        )\n        .then((e)=\u003ee.json())\n        .then((e)=\u003e{\n            const file_path = e['path']\n\n            // add 3d object\n            loader.load( url_base+'file=static/'+file_path, function ( gltf ) {\n                static_group.attach(gltf.scene.children[0])\n            }, undefined, function ( error ) {\n            \n                console.error( error );\n            \n            } );\n        })\n    })\n    })\n}\n```\n\n## Recognition\nThis uses lots of opensource tools. Thank you to everyone working on opensource. Y'all are the best.\n\nThis uses a lot of the code structure from https://github.com/thygate/stable-diffusion-webui-depthmap-script\n\nSpecial thanks to thygate and semjon00\n\n## Academic Projects used\n\nZoeDepth :\n\n```\n@misc{https://doi.org/10.48550/arxiv.2302.12288,\n  doi = {10.48550/ARXIV.2302.12288},\n  url = {https://arxiv.org/abs/2302.12288},\n  author = {Bhat, Shariq Farooq and Birkl, Reiner and Wofk, Diana and Wonka, Peter and Müller, Matthias},\n  keywords = {Computer Vision and Pattern Recognition (cs.CV), FOS: Computer and information sciences, FOS: Computer and information sciences},\n  title = {ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth},\n  publisher = {arXiv},\n  year = {2023},\n  copyright = {arXiv.org perpetual, non-exclusive license}\n}\n```\n\nMarigold - Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation:\n\n```\n@misc{ke2023repurposing,\n      title={Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation}, \n      author={Bingxin Ke and Anton Obukhov and Shengyu Huang and Nando Metzger and Rodrigo Caye Daudt and Konrad Schindler},\n      year={2023},\n      eprint={2312.02145},\n      archivePrefix={arXiv},\n      primaryClass={cs.CV}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgraemeniedermayer%2F3d-depth-gen","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgraemeniedermayer%2F3d-depth-gen","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgraemeniedermayer%2F3d-depth-gen/lists"}