{"id":15027256,"url":"https://github.com/shichenliu/softras","last_synced_at":"2025-05-16T12:12:39.098Z","repository":{"id":41086512,"uuid":"179163401","full_name":"ShichenLiu/SoftRas","owner":"ShichenLiu","description":"Project page of paper \"Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning\"","archived":false,"fork":false,"pushed_at":"2021-12-16T12:31:13.000Z","size":10983,"stargazers_count":1262,"open_issues_count":56,"forks_count":157,"subscribers_count":37,"default_branch":"master","last_synced_at":"2025-05-16T12:12:32.144Z","etag":null,"topics":["3d-reconstruction","computer-graphics","differentiable-rendering","geometry-processing"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ShichenLiu.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-04-02T21:44:11.000Z","updated_at":"2025-04-28T23:36:24.000Z","dependencies_parsed_at":"2022-08-10T01:42:43.550Z","dependency_job_id":null,"html_url":"https://github.com/ShichenLiu/SoftRas","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ShichenLiu%2FSoftRas","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ShichenLiu%2FSoftRas/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ShichenLiu%2FSoftRas/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ShichenLiu%2FSoftRas/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ShichenLiu","download_url":"https://codeload.github.com/ShichenLiu/SoftRas/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254527099,"owners_count":22085919,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d-reconstruction","computer-graphics","differentiable-rendering","geometry-processing"],"created_at":"2024-09-24T20:06:04.610Z","updated_at":"2025-05-16T12:12:34.084Z","avatar_url":"https://github.com/ShichenLiu.png","language":"Python","readme":"# Soft Rasterizer (SoftRas)\n\nThis repository contains the code (in PyTorch) for \"[Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning](https://arxiv.org/abs/1904.01786)\" (ICCV'2019 Oral) by [Shichen Liu](https://shichenliu.github.io), [Tianye Li](https://sites.google.com/site/tianyefocus/home), [Weikai Chen](http://chenweikai.github.io/) and [Hao Li](https://www.hao-li.com/Hao_Li/Hao_Li_-_about_me.html).\n\n## Contents\n\n1. [Introduction](#introduction)\n2. [Usage](#usage)\n3. [Applications](#applications)\n4. [Contacts](#contacts)\n\n## Introduction\n\nSoft Rasterizer (SoftRas) is a truly differentiable renderer framework with a novel formulation that views rendering as a **differentiable aggregating process** that fuses **probabilistic contributions** of all mesh triangles with respect to the rendered pixels. Thanks to such *\"soft\"* formulation, our framework is able to (1) directly render colorized mesh using differentiable functions and (2) back-propagate efficient supervision signals to mesh vertices and their attributes (color, normal, etc.) from various forms of image representations, including silhouette, shading and color images. \n\n\u003cimg src=\"https://raw.githubusercontent.com/ShichenLiu/SoftRas/master/data/media/teaser/teaser.png\" width=\"60%\"\u003e\n\n## Usage\n\nThe code is built on Python3 and PyTorch 1.6.0. CUDA (10.1) is needed in order to install the module. Our code is extended on the basis of [this repo](https://github.com/daniilidis-group/neural_renderer). `6/3/2021` update note: we add **testing models** and **recontructed color meshes** below, and also slightly optimized the code structure! Previous version is archived in the `legacy` branch.\n\n\nTo install the module, using\n\n```\npython setup.py install\n```\n\n## Applications\n\n### 0. Rendering\n\nWe demonstrate the rendering effects provided by SoftRas. Realistic rendering results (1st and 2nd columns) can be achieved with a proper setting of `sigma` and `gamma`. With larger `sigma` and `gamma`, one can obtain renderings with stronger transparency and blurriness (3rd and 4th column).\n\n![](https://raw.githubusercontent.com/ShichenLiu/SoftRas/master/data/media/demo/render/forward.gif)\n\n```\nCUDA_VISIBLE_DEVICES=0 python examples/demo_render.py\n```\n\n### 1. 3D Unsupervised Single-view Mesh Reconstruction\n\nBy incorporating SoftRas with a simple mesh generator, one can train the network with multi-view images only, without requiring any 3D supervision. At test time, one can reconstruct the 3D mesh, along with the mesh texture, from a single RGB image. Below we show the results of single-view mesh reconstruction on ShapeNet.\n\n![](https://raw.githubusercontent.com/ShichenLiu/SoftRas/master/data/media/demo/recon_rgb/02691156_150_000867_w_input.gif) ![](https://raw.githubusercontent.com/ShichenLiu/SoftRas/master/data/media/demo/recon_rgb/02958343_150_001155_w_input.gif) ![](https://raw.githubusercontent.com/ShichenLiu/SoftRas/master/data/media/demo/recon_rgb/03001627_150_002475_w_input.gif) ![](https://raw.githubusercontent.com/ShichenLiu/SoftRas/master/data/media/demo/recon_rgb/04090263_150_004755_w_input.gif) ![](https://raw.githubusercontent.com/ShichenLiu/SoftRas/master/data/media/demo/recon_rgb/04256520_150_004035_w_input.gif) ![](https://raw.githubusercontent.com/ShichenLiu/SoftRas/master/data/media/demo/recon_rgb/04379243_150_000843_w_input.gif)\n\nDownload shapenet rendering dataset provided by NMR:\n```\nbash examples/recon/download_dataset.sh\n```\n\nTo train the model:\n```\nCUDA_VISIBLE_DEVICES=0 python examples/recon/train.py -eid recon\n```\n\nTo test the model:\n```\nCUDA_VISIBLE_DEVICES=0 python examples/recon/test.py -eid recon \\\n    -d 'data/results/models/recon/checkpoint_0200000.pth.tar'\n```\n\nWe also provide our trained model here:\n- SoftRas trained with silhouettes supervision (62+ IoU): [google drive](https://drive.google.com/file/d/1GlZJVih5BMGp026mpxK2scWJXqT94VUx/view?usp=sharing)\n- SoftRas trained with shading supervision (64+ IoU, test with `--shading-model` arg): [google drive](https://drive.google.com/file/d/1r63AKNn3ecMho6RFE7gFefRv78Pmbe5h/view?usp=sharing)\n- SoftRas reconstructed meshes with color (random sampled): [google drive](https://drive.google.com/file/d/1gnSshn0k9JpVpoSTWIQoV2QFAlin3AUK/view?usp=sharing)\n\n### 2. Image-based Shape Deformation\n\nSoftRas provides strong supervision for image-based mesh deformation. We visualize the deformation process from a sphere to a car model and then to a plane given supervision from multi-view silhouette images.\n\n![](https://raw.githubusercontent.com/ShichenLiu/SoftRas/master/data/media/demo/deform/sphere_to_car.gif) ![](https://raw.githubusercontent.com/ShichenLiu/SoftRas/master/data/media/demo/deform/car_to_plane.gif)\n\n```\nCUDA_VISIBLE_DEVICES=0 python examples/demo_deform.py\n```\n\nThe optimized mesh is included in `data/obj/plane/plane.obj`\n\n### 3. Pose Optimization for Rigid Objects\n\nWith scheduled blurry renderings, one can obtain smooth energy landscape that avoids local minima. \nBelow we demonstrate how a color cube is fitted to the target image in the presence of large occlusions.\nThe blurry rendering and the corresponding rendering losses are shown in the 3rd and 4th columns respectively.\n\n![](https://raw.githubusercontent.com/ShichenLiu/SoftRas/master/data/media/demo/fitting/cubic.gif)\n\n### 4. Non-rigid Shape Fitting\n\nWe fit the parametric body model (SMPL) to a target image where the part (right hand) is entirely occluded in the input view.\n\n![](https://raw.githubusercontent.com/ShichenLiu/SoftRas/master/data/media/demo/fitting/smpl.gif)\n\n## Contacts\nShichen Liu: \u003cliushichen95@gmail.com\u003e\n\nAny discussions or concerns are welcomed!\n\n### Citation\n\nIf you find our project useful in your research, please consider citing:\n\n```\n@article{liu2019softras,\n  title={Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning},\n  author={Liu, Shichen and Li, Tianye and Chen, Weikai and Li, Hao},\n  journal={The IEEE International Conference on Computer Vision (ICCV)},\n  month = {Oct},\n  year={2019}\n}\n```\n\n```\n@article{liu2020general,\n  title={A General Differentiable Mesh Renderer for Image-based 3D Reasoning},\n  author={Liu, Shichen and Li, Tianye and Chen, Weikai and Li, Hao},\n  journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},\n  year={2020},\n  publisher={IEEE}\n}\n```\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshichenliu%2Fsoftras","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fshichenliu%2Fsoftras","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshichenliu%2Fsoftras/lists"}