{"id":47822883,"url":"https://github.com/pmh47/dirt","last_synced_at":"2026-04-03T19:15:34.444Z","repository":{"id":37848926,"uuid":"145043028","full_name":"pmh47/dirt","owner":"pmh47","description":"DIRT: a fast differentiable renderer for TensorFlow","archived":false,"fork":false,"pushed_at":"2022-03-16T17:46:25.000Z","size":334,"stargazers_count":307,"open_issues_count":33,"forks_count":62,"subscribers_count":14,"default_branch":"master","last_synced_at":"2023-11-07T18:57:59.827Z","etag":null,"topics":["differentiable-rendering","machine-learning","opengl","rendering","tensorflow","vision"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pmh47.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-08-16T21:58:25.000Z","updated_at":"2023-08-29T07:50:33.000Z","dependencies_parsed_at":"2022-08-08T22:01:50.653Z","dependency_job_id":null,"html_url":"https://github.com/pmh47/dirt","commit_stats":null,"previous_names":[],"tags_count":0,"template":null,"template_full_name":null,"purl":"pkg:github/pmh47/dirt","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pmh47%2Fdirt","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pmh47%2Fdirt/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pmh47%2Fdirt/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pmh47%2Fdirt/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pmh47","download_url":"https://codeload.github.com/pmh47/dirt/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pmh47%2Fdirt/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31372199,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-03T17:53:18.093Z","status":"ssl_error","status_checked_at":"2026-04-03T17:53:17.617Z","response_time":107,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["differentiable-rendering","machine-learning","opengl","rendering","tensorflow","vision"],"created_at":"2026-04-03T19:15:33.811Z","updated_at":"2026-04-03T19:15:34.433Z","avatar_url":"https://github.com/pmh47.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# DIRT: a fast Differentiable Renderer for TensorFlow\n\nDIRT is a library for TensorFlow, that provides operations for rendering 3D meshes.\nIt supports computing derivatives through geometry, lighting, and other parameters.\nDIRT is very fast: it uses OpenGL for rasterisation, running on the GPU, which allows\nlightweight interoperation with CUDA.\n\nThe following images illustrate the capabilities of DIRT; see\n[`samples`](https://github.com/pmh47/dirt/tree/master/samples) for source\ncode. The first uses simple monochromatic diffuse lighting calculated per-vertex and\ninterpolated with Gouraud shading; the others use per-pixel (deferred) lighting and\ntexture calculations.\n\nIn all cases, we can calculate gradients with respect to all inputs, including the geometry\n(vertex locations and normals), lighting parameters (e.g. colour and direction), and texture\n(the vertex UVs and the pixel values in the texture itself).\n\n![Three sample renderings](example_renderings.jpg \"Three sample renderings\")\n\n\n## Citation\n\nIf you use DIRT in your research, please cite: [*Learning Single-Image 3D Reconstruction by Generative Modelling of Shape, Pose and Shading*](https://doi.org/10.1007/s11263-019-01219-8) (P. Henderson and V. Ferrari, IJCV 2019).\n\nThe appropriate bibtex entry is:\n```\n@article{henderson19ijcv,\n  title={Learning Single-Image {3D} Reconstruction by Generative Modelling of Shape, Pose and Shading},\n  author={Paul Henderson and Vittorio Ferrari},\n  journal={International Journal of Computer Vision},\n  year={2019},\n  doi={10.1007/s11263-019-01219-8},\n  url={https://doi.org/10.1007/s11263-019-01219-8}\n}\n```\n\nThere is a brief description of how DIRT calculates gradients in Section 3.4 of my [PhD thesis](http://calvin.inf.ed.ac.uk/wp-content/uploads/Publications/theses/Henderson2019.pdf), for the case of per-face Lambertian shading without textures.\n\n## Why is DIRT useful?\n\nDrawing 3D (or 2D) shapes *differentiably* is challenging in TensorFlow. For example, you could create a tensor containing a white square on a black background using the following:\n```python\nimport tensorflow as tf\n\ncanvas_width, canvas_height = 128, 128\ncentre_x, centre_y = 32, 64\nsquare_size = 16\n\nxs, ys = tf.meshgrid(tf.range(canvas_width), tf.range(canvas_height))\n\nx_in_range = tf.less_equal(tf.abs(xs - centre_x), square_size / 2)\ny_in_range = tf.less_equal(tf.abs(ys - centre_y), square_size / 2)\npixels = tf.cast(tf.logical_and(x_in_range, y_in_range), tf.float32)\n```\nHowever, if you calculate gradients of the pixels with respect to `centre_x` and `centre_y`, they will always be zero -- whereas for most use-cases, they should be non-zero at the boundary of the shape.\n\nDIRT provides a single TensorFlow operation, `rasterise`, that renders shapes differentiably. Moreover, it includes helper code that supports 3D projection, lighting, etc.\nThis allows full 2D or 3D scenes to be assembled directly in TensorFlow, with gradients flowing through the geometry, lighting and surface parameters.\n\nUsing DIRT, the above example becomes:\n```python\nimport tensorflow as tf\nimport dirt\n\ncanvas_width, canvas_height = 128, 128\ncentre_x, centre_y = 32, 64\nsquare_size = 16\n\n# Build square in screen space\nsquare_vertices = tf.constant([[0, 0], [0, 1], [1, 1], [1, 0]], dtype=tf.float32) * square_size - square_size / 2.\nsquare_vertices += [centre_x, centre_y]\n\n# Transform to homogeneous coordinates in clip space\nsquare_vertices = square_vertices * 2. / [canvas_width, canvas_height] - 1.\nsquare_vertices = tf.concat([square_vertices, tf.zeros([4, 1]), tf.ones([4, 1])], axis=1)\n\npixels = dirt.rasterise(\n    vertices=square_vertices,\n    faces=[[0, 1, 2], [0, 2, 3]],\n    vertex_colors=tf.ones([4, 1]),\n    background=tf.zeros([canvas_height, canvas_width, 1]),\n    height=canvas_height, width=canvas_width, channels=1\n)[:, :, 0]\n```\n\n\n## Requirements\n\n- an Nvidia GPU; the earliest drivers we have tested with are v367\n- Linux; we have only tested on Ubuntu, but other distributions should work\n- a GPU-enabled install of TensorFlow, version 1.6 or later\n- python 2.7.9 / 3.5 or newer\n- cmake 3.8 or newer\n- gcc 4.9 or newer\n\n\n## Installation\n\n**Before** installing, you should activate a virtualenv with `tensorflow-gpu` installed (or ensure your system python has that package), as DIRT will use this to search for appropriate TensorFlow headers during installation.\n\nSimply clone this repository, then install with pip:\n```\ngit clone https://github.com/pmh47/dirt.git\ncd dirt\npip install .\n```\n\nIf you plan to modify the DIRT code, you may prefer to install in development mode:\n```\ncd dirt\nmkdir build ; cd build\ncmake ../csrc\nmake\ncd ..\npip install -e .\n```\n\nTo sanity-check your build, run `python tests/square_test.py`, which should produce the output `successful: all pixels agree`.\n\n\n### Troubleshooting\n\n- If the build cannot find `GL/gl.h` and `GL/glext.h`, you can get suitable versions of these by running the following from the `dirt` directory:\n    ```\n    mkdir external/GL ; cd external/GL\n    wget https://raw.githubusercontent.com/mesa3d/mesa/master/include/GL/gl.h\n    wget https://raw.githubusercontent.com/mesa3d/mesa/master/include/GL/glext.h\n    cd ../..\n    export INCLUDE=$PWD/external:$INCLUDE\n    ```\n\n- If the build cannot find `X11/Xlib.h`, install the system package `libx11-dev` or `libX11-devel`\n\n- You should ensure that libGL and libEGL are in a location on `LD_LIBRARY_PATH`, and that these are the versions shipped with your Nvidia driver. In particular, if you have installed Mesa or Hybris, their libGL or libEGL may be used (or may even have overwritten the Nvidia versions), and these will not work with DIRT\n\n- If you use a version of Ubuntu older than 18.04, and you use the Ubuntu-packaged Nvidia driver (i.e. installed with apt not Nvidia's runfile), then the correct GL libraries may not be found at runtime. Use `export LD_LIBRARY_PATH=/usr/lib/nvidia-XXX` (replacing XXX with your driver version). If that is not sufficient then also use `export LD_PRELOAD=/usr/lib/nvidia-XXX/libEGL.so:/usr/lib/nvidia-XXX/libOpenGL.so` to ensure that the Nvidia version of libEGL is used. If cmake fails to find OpenGL or EGL during setup, then also `export CMAKE_LIBRARY_PATH=/usr/lib/nvidia-XXX` before installing\n\n- If you are using Ubuntu 18.04 or newer, with the Ubuntu-packaged Nvidia drivers (i.e. installed with apt not Nvidia's runfile), and libOpenGL.so and/or libEGL.so is missing, then run `sudo apt install libglvnd-dev`\n\n- If you are using TensorFlow 1.14, there are some binary compatibility issues when using older versions of python (e.g. 2.7 and 3.5), due to compiler version mismatches. These result in a segfault at `tensorflow::shape_inference::InferenceContext::GetAttr` or similar. To resolve, either upgrade python to 3.7, or downgrade TensorFlow to 1.13, or build DIRT with gcc 4.8\n\n- You should ensure that graphics operations are enabled for your GPU (ALL_ON operation mode set by `nvidia-smi --gom=0`) -- this is the default, and does not need changing in most cases\n\n- If you see an error `cudaGraphicsGLRegisterImage failed: cudaErrorNotSupported`, this may be due to insufficient GPU memory. Note that DIRT allocates some memory through OpenGL outside of TensorFlow's allocator, so it may be necessary to reduce the memory reserved by TensorFlow (e.g. by using `allow_growth=True` in the session config)\n\n- If you see an error `Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR` when initialising cudnn after DIRT, this may again be due to insufficient GPU memory (see previous point).\n\n\n### Docker\n\n- Install [docker](http://docker.com)\n- Install [nvidia-docker](https://github.com/NVIDIA/nvidia-docker)\n- Configure the docker daemon to use the nvidia runtime\n\t- ```sudo vi /etc/docker/daemon.json```\n\t- Add \"default-runtime\": \"nvidia\" as the first entry in the JSON file\n\t- Run ```sudo service docker restart``` to restart the docker daemon\n- Example setup:\n\n```\n# clone dirt\ngit clone https://github.com/pmh47/dirt.git \u0026\u0026 cd dirt\n\n# build the image\nexport CUDA_BASE_VERSION=9.0\nexport UBUNTU_VERSION=16.04\nexport CUDNN_VERSION=7.6.0.64\ndocker build -t \u003cimage_name\u003e --build-arg CUDA_BASE_VERSION=$(echo $CUDA_BASE_VERSION) \\\n\t--build-arg UBUNTU_VERSION=$(echo $UBUNTU_VERSION) \\\n\t--build-arg CUDNN_VERSION=$(echo $CUDNN_VERSION) .\n\n# run the container and open a bash shell\nsudo docker run --runtime=nvidia \u003cimage_name\u003e /bin/bash\n```\n\n\n## Usage\n\nA simple, 2D example was given above.\nMore sophisticated examples rendering 3D meshes are in the `samples` folder.\n\nDIRT uses OpenGL for rasterisation, and uses OpenGL conventions for coordinate systems. In particular, the coordinates passed to `rasterise` are in OpenGL clip space, and the matrix helper functions assume that the camera points along the *negative* z-axis in world space.\nThe only exception is that rasterised images follow the TensorFlow convention of having the top row first.\n\nDIRT can be used in *direct* or *deferred* shading modes.\nDirect uses the rasterise operation directly to produce the final pixels, with simple Gouraud shading.\n Lighting calculations are performed per-vertex before rasterisation, and colours are interpolated between vertices linearly in 3D space).\nThis is very efficient and simple to work with, but limits certain lighting effects (e.g. specular highlights) and does not allow texturing.\nDeferred uses the rasterise operation to generate a G-buffer, that captures the scene geometry at each pixel (typically the underlying vertex location and normal).\nThen, lighting calculations are performed per-pixel in a second pass.\n\n\n## How does DIRT work?\n\n### Theory\n\nDIRT uses filter-based derivatives, inspired by OpenDR (Loper and Black, ECCV 2014).\nIt makes considerable effort to return correctly-behaving derivatives even in cases of self-occlusion, where other differentiable renderers can fail.\n\n\n### Implementation\n\nDIRT uses OpenGL for rasterisation, as this is fast, accurate, and very mature.\nWe use Nvidia's OpenGL / CUDA interop to allow the vertices and pixels to remain on the same GPU both for processing by TensorFlow and for rasterisation, thus minimising copying overhead compared with other approaches.\nTo avoid having to create an on-screen context for rendering, we use an Nvidia extension to EGL, that allows creating an OpenGL context bound to a GPU but not a physical display.\n\n\n## Alternatives to DIRT\n\nSeveral other differentiable renderers have been described and released in recent years:\n\n- [DEODR](https://github.com/martinResearch/DEODR) (de la Gorce et al., PAMI 2011) supports Gouraud shading with textures, using custom CPU/GPU rendering that has well-defined gradients even at occlusion boundaries. It has bindings for TensorFlow, PyTorch, and Matlab\n\n- [OpenDR](https://github.com/mattloper/opendr) (Loper and Black, ECCV 2014) supports Gouraud shading using Mesa CPU-based rendering, and uses filter-based derivatives similar to DIRT. It uses its own custom automatic differentiation framework written in python, hence does not integrate smoothly with TensorFlow\n\n- [Neural 3D Mesh Renderer](https://github.com/hiroharu-kato/neural_renderer) (Kato et al., CVPR 2018) supports similar functionality to DIRT, using a slightly different formulation for the approximate derivatives, but implements a custom rasterisation operation, rather than using OpenGL. It integrates with Chainer, but not TensorFlow (a PyTorch re-implementation is also available)\n\n- [tf_mesh_renderer](https://github.com/google/tf_mesh_renderer) (Genova et al., CVPR 2018) similarly uses custom rendering (on the CPU in this case), but integrates directly with TensorFlow\n\n- [tensorflow_mesh_renderer](https://github.com/ndrplz/tensorflow-mesh-renderer) (Palazzi et al., ECCV Workshops 2018) renders silhouettes using built-in TensorFlow ops, but does not support shading\n\n- [redner](https://github.com/BachiLi/redner) (Li et al., TOG 2018) is a differentiable path-tracer that can propagate gradients through indirect illumination, but which is much slower than methods like DIRT with only direct lighting\n\n\n## Contributing\n\nPull requests welcome!\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpmh47%2Fdirt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpmh47%2Fdirt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpmh47%2Fdirt/lists"}