{"id":28075414,"url":"https://github.com/nvpro-samples/vk_denoise_nrd","last_synced_at":"2025-05-13T00:57:07.660Z","repository":{"id":282981895,"uuid":"860666321","full_name":"nvpro-samples/vk_denoise_nrd","owner":"nvpro-samples","description":"Integration of NRD to an existing application","archived":false,"fork":false,"pushed_at":"2025-03-19T17:41:40.000Z","size":5069,"stargazers_count":6,"open_issues_count":0,"forks_count":0,"subscribers_count":9,"default_branch":"main","last_synced_at":"2025-05-13T00:57:02.061Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nvpro-samples.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-09-20T22:10:20.000Z","updated_at":"2025-04-15T07:29:06.000Z","dependencies_parsed_at":"2025-03-18T01:37:32.985Z","dependency_job_id":null,"html_url":"https://github.com/nvpro-samples/vk_denoise_nrd","commit_stats":null,"previous_names":["nvpro-samples/vk_denoise_nrd"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nvpro-samples%2Fvk_denoise_nrd","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nvpro-samples%2Fvk_denoise_nrd/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nvpro-samples%2Fvk_denoise_nrd/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nvpro-samples%2Fvk_denoise_nrd/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nvpro-samples","download_url":"https://codeload.github.com/nvpro-samples/vk_denoise_nrd/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253850883,"owners_count":21973672,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-05-13T00:57:07.026Z","updated_at":"2025-05-13T00:57:07.626Z","avatar_url":"https://github.com/nvpro-samples.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Integration of NRD to an existing application\n \n![A screenshot of the sample. It shows a path-traced Cornell box, denoised using NRD.](docs/nrd_denoiser.png)\n\nThis example demonstrates [NRD, \"NVIDIA Real-Time Denoisers\",\n](https://github.com/NVIDIAGameWorks/RayTracingDenoiser) in a simple path\ntracer rendering a glTF scene. NRD is a spatio-temporal post-processing library\nthat removes noise from Monte-Carlo based path tracers. NRD is not just a single\ndenoiser, in fact it is a collection of specialized denoisers for specific\nkinds of data, like diffuse and specular images, ambient occlusion, and shadow\ndata.\n\nNRD itself provides a comprehensive sample at \u003chttps://github.com/NVIDIAGameWorks/NRDSample\u003e,\nshowing all the bells and whistles of its various denoisers. This sample\ninstead focuses on an integration of the NRD library with a Vulkan renderer,\nshowing a simple pathtracer using NRD to denoise the rendered image.\n\nMore Resources:\n* Get more information at: \u003chttps://developer.nvidia.com/nvidia-rt-denoiser\u003e\n* NRD Sample: \u003chttps://github.com/NVIDIAGameWorks/NRDSample\u003e\n* NRD SDK: \u003chttps://github.com/NVIDIAGameWorks/RayTracingDenoiser\u003e\n\n## Building \n\nYou need the [nvpro_core](https://github.com/nvpro-samples/nvpro_core)\nrepository checked out next to the vk_denoise_nrd sample. At configuration time\nCMake will look for the nvpro_core repo and include it into the build. Follow\nthe general CMake process to configure and build the sample.\n\nThis project needs the NRD SDK. CMake for the sample project is configured to\nautomatically download the latest version of NRD from\n\u003chttps://github.com/NVIDIAGameWorks/RayTracingDenoiser\u003e.\n\n## NRD Denoising Methods\n\nNRD offers these denoising methods:\n\n* [ReBLUR](https://developer.nvidia.com/nvidia-rt-denoiser#REBLUR): Blur the information over multiple frames; good for low samples per pixel.\n* [ReLAX](https://developer.nvidia.com/nvidia-rt-denoiser#RELAX): Preserves lighting details better.\n* [SIGMA](https://developer.nvidia.com/nvidia-rt-denoiser#SIGMA): Works on all kinds of shadows.\n\nIn our sample, we test ReBLUR and ReLAX.\n\n\n## CMake Integration of NRD\n\nThe NRD SDK comes with direct CMake support. In this sample we use the\n'FetchContent' CMake module to automatically download and express a dependency\non the NRD SDK:\n\n```cmake\n# CMake module FetchContent\ninclude(FetchContent)\n\n# Tell CMake where to find NRDSDK, how to get it and where to place it locally.\n# CMake will automatically look for CMakeLists.txt in there\nFetchContent_Declare(\n    NRDSDK\n    GIT_REPOSITORY https://github.com/NVIDIAGameWorks/RayTracingDenoiser.git\n    GIT_SHALLOW\n    GIT_SUBMODULES\n    SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/externals/nrd\n)\n\n# Configure the NRD build to only produce SPIR-V binaries and embed them into the NRD library\noption (NRD_EMBEDS_DXIL_SHADERS \"NRD embeds DXIL shaders\" OFF)\noption (NRD_EMBEDS_DXBC_SHADERS \"NRD embeds DXBC shaders\" OFF)\nset(NRD_EMBEDS_DXIL_SHADERS OFF)\nset(NRD_EMBEDS_DXBC_SHADERS OFF)\n\n# Cause CMake to download and/or update our local copy of NRDSDK.\nFetchContent_MakeAvailable(NRDSDK)\n\n# Make our project link with NRD\ntarget_link_libraries(${PROJECT_NAME} NRD)\n```\n\nThe only NRD compile time options we set in this sample are\n`NRD_EMBEDS_DXIL_SHADERS` and `NRD_EMBEDS_DXBC_SHADERS` to disable building\nthese shader binaries. Refer to [externals/nrd/CMakeLists.txt](externals/nrd/CMakeLists.txt) for a list of\nsupported options, such as normal and roughness encoding. Our sample leaves\nthis at default (i.e. linear roughness and RGB10A2 normal encoding).\n\nAt CMake configuration time, NRD also creates a shader include file\n'nrd/Shaders/Include/NRDEncoding.hlsli' that will be used to communicate the\nchosen normal and roughness encoding to the user's shaders.\n\n## Integration of NRD\n\nNRD itself is platform-agnostic and thus can be used with DirectX and Vulkan.\nIts internal compute shaders are written in HLSL and typically compiled as part\nof the NRD SDK compilation process.\n\nThere are several ways to integrate NRD.\n* As a 'white-box library': the application is asked to do everything, including\n  allocating needed resources AND compiling NRD's shaders\n* As a 'black-box library': the application creates all resources, but the\n  shaders will be handed to the application as precompiled binary blobs\n  (SPIR-V, DXIL, etc)\n* Via NRD's integration layer: NRD SDK provides an integration wrapper based\n  on [NRI (NVIDIA Render Interface)](https://github.com/NVIDIAGameWorks/NRI)\n\nIn this sample we chose the 'black-box library' approach, using a thin wrapper\naround NRD that handles the integration of NRD with Vulkan.\n\n\n### NRD and GLSL\n\nEven though the the NRD-internal shaders are all precompiled, a few shader\nfragments will be needed in the user's shaders to encode data in a way NRD\nrequires. NRD is naturally written in HLSL, but can be instructed to emit\nGLSL-compatible syntax. We achieve this by defining `NRD_GLSL 1` at the top of\n`nrd.glsl` before including `nrd.hlsli`\n\nNRD build is using a SPIR-V-enabled DirectX shader compiler \"dxc\" to\ncompile its HLSL shaders into SPIR-V that Vulkan can digest. dxc comes with the\nVulkan SDK, which can be downloaded [here](https://vulkan.lunarg.com/sdk/home).\nAnother option is to download\n[https://github.com/microsoft/DirectXShaderCompiler](https://github.com/microsoft/DirectXShaderCompiler)\nand build it from source.\n\nIf NRD can't find dxc at CMake configuration time, consider providing CMake\nwith `-DNRD_DXC_CUSTOM_PATH=\u003ccustom/path/to/dxc\u003e` to point it to the dxc\nexecutable directly.\n\n### NRDWrapper\n\nThe NRD SDK comes with example code to integrate NRD with NRI (NVIDIA Render\nInterface). However, vk_denoise_nrd is targeted at integration of NRD with\nVulkan specifically, utilizing\n[nvpro_core's](https://github.com/nvpro-samples/nvpro_core) helper classes.\n\nNRD calls this integration variant _black-box library integration._ A small\nwrapper class, `NRDWrapper`, contains all the necessary code for that. Since\nNRD is graphics API agnostic (supporting D3D and Vulkan), it describes\nrequired resources in an API independent manner. NRDWrapper hides the\ncomplexities of interpreting these descriptions and creating all resources\n(buffers, textures) that NRD needs. The wrapper also interprets NRD's\ndescription of how to perform the denoising pipeline and translates this into\nexecuting a series of compute shader invocations with proper pipeline barriers\nin-between the passes. Code comments in NRDWrapper.cpp/.hpp describe the details\nof its API and implementation. Additionally, one can find a lot of information\nin the NRD headers themselves. Notably, `NRDDesc.h` and `NRD.hlsli` are worth\nlooking into to understand the various NRD inputs and outputs as well as\nlearning about what resource types are needed for each type of denoiser.\n\nNRDWrapper creates all NRD-internal textures on its own, with the the exception of\nthe \"user texture pool\". This is a pool of textures that both NRD and the application\noperate on. NRD calls these \"resources\". The user texture pool array may be sparsely\npopulated, depending on the used denoiser. Consult `NRDDescs.h` to see which denoisers\nrequire which types of resource (i.e. texture) and what format each resource needs\nto have.\n\nThis sample focuses on using the `REBLUR_DIFFUSE_SPECULAR` and\n`RELAX_DIFFUSE_SPECULAR` denoisers, which each take separate noisy\ndiffuse+hitdistance and specular+hitdistance buffers as input.\n\n## The path tracing pipeline\n\n1. The primary driver of the rendering is the `nrd.rgen` ray generation shader.\n   It'll cast the primary as well as all secondary ray segments.\n   This approach is preferred as it keeps the required \"stack memory\"\n   lower than recursively casting rays from closest-hit shaders.\n   \n2. The first step for each ray is to find the first non-mirrored surface, the _primary hit_.\n   At the primary hit we record all material properties (such as normals) that NRD needs and write\n   them into the appropriate buffers. If we did not hit anything, the ray missed all\n   geometry and thus will be recorded as hitting the skybox only. Mirrored surfaces\n   need special treatment ([PSR](#mirror-like-surfaces), described below).\n   \n3. Find direct lighting contributions. For this, we loop through\n   all lights, tracing a shadow ray for each. If the light is visible, we add its\n   contribution to the direct lighting. We also add the primary hit's material\n   light emission here. The direct light contributions are not noisy and thus\n   are written into a separate buffer which does not undergo denoising.\n   It will later be added back to the scene during compositing.\n\n4. Probe the environment map lighting at the primary hit. The environment map\n   has been preprocessed to provide a PDF and a randomized sampling pattern\n   according to that PDF, emphasizing sampling density around its bright spots. A\n   shadow ray tests if the primary hitposition is actually visible to the\n   environment map in the chosen direction. Since this sampling is randomized, we\n   later add its diffuse and specular environment map contribution to the noisy\n   diffuse and specular output images for the denoiser to deal with. Environment\n   map contributions via direct and indirect probing need to be carefully\n   weighted; see the [MIS weighting](#mis-weighting) section.\n\n5. From the hit position we perform path tracing to collect the incoming indirect light -\n   once along the material's diffuse BSDF, and once along its specular BSDF.\n   For this tracing we use the pathtrace.rchit shader. At each hit position we do the following:\n    - If the ray missed all geometry, we hit the environment map. Return its light contribution,\n      multiply it by a [MIS weight](#mis-weighting), and report the end of this ray via `payload.hitT = NRD_INF`.\n    - When hitting geometry, collect direct lighting contributions just as we did for the primary ray\n      (lights, envmap - see above).\n    - Sample the material's BSDF for a suggested direction to follow for the next path segment.\n    - In case of an absorption event, return `payload.hitT = NRD_INF` to end the path here.\n    - Otherwise, return a new ray segment (origin, direction and PDF value for the direction) and\n      the light contribution at the current hit position to `nrd.rgen`, which continues to accumulate\n      the light along the path.\n    - Repeat tracing paths until the ray ends or reaches the maximum number of bounces.\n      Pass the distance from the primary hit to the subsequent hit to NRD.\n    \n6. De-modulate diffuse and specular color and store them encoded for the denoiser. Also store enough information\n   to recompute the [(de)modulation values](#demodulation) in the composition shader.\n\n![A diagram of an application rendering pipeline using NRD. A ray tracer outputs\ndiffuse, specular, normal, ViewZ, and motion vectors, along with direct lighting.\nThe first 5 are passed to NRD, which outputs diffuse and specular. A Compose\nstep composites these together, and then a final TAA step performs antialiasing.](docs/NRD_pipeline2.png)\n\n## NRD Details\n\nAs mentioned above, in this sample, we're making use of two denoisers:\n`REBLUR_DIFFUSE_SPECULAR` and `RELAX_DIFFUSE_SPECULAR`.\n\nAccording to NRDDescs.h, `REBLUR_DIFFUSE_SPECULAR` needs the following inputs:\n\n* `IN_DIFF_RADIANCE_HITDIST`: a texture containing the 'demodulated' diffuse radiance at the primary hit point as well as the distance to the second hit (following the diffuse BSDF)\n* `IN_SPEC_RADIANCE_HITDIST`: a texture containing the 'demodulated' specular radiance at the primary hit point as well as the distance to the second hit (folloing the specular BSDF)\n* `MOTION`: a texture with 3D vectors of the previous pixel 3D position to the new position. The motion vectors can be set to 0, and set `isMotionVectorInWorldSpace` to true. This will do an approximation of the motion vectors without the need to provide them. This is what we have done.\n* `NORMAL_ROUGHNESS`: Normal, material roughness, and material ID packed into a 4D vector. The normal vector should be between [-1..1] and roughness in linear space. See the function `NRD_FrontEnd_PackNormalAndRoughness()` to see how this is done.\n* `VIEWZ`: This is the linear depth of the hit position in camera space. For example, `float z = (worldToView * hitPosition).z` yields the desired value.\n* It also takes a couple of optional inputs that this sample does not provide for simplicity's sake.\n\n![A diagram showing how ViewZ is computed from the primary hit point and the camera plane.](docs/viewz.png)\n\nThese are its output textures:\n * `OUT_DIFF_RADIANCE_HITDIST` - a texture containing the denoised diffuse image\n * `OUT_SPEC_RADIANCE_HITDIST` - a texture containing the denoised specular image\n\nThe denoiser `RELAX_DIFFUSE_SPECULAR` takes the same inputs, which makes it\neasier to switch between the two.\n\nConsult NRDDescs.h to see which inputs are required. For each input type\n(\"Resource Type\") it also prescribes the required image format and other\ndependencies.\n\nNRD can only work, if it is provided with the right data. NRD's [README.md](https://github.com/NVIDIAGameWorks/RayTracingDenoiser/blob/master/README.md)\ntalks about this in much detail. Below we will mention a few critical points to\nget right.\n\n\n### Splitting diffuse and specular signals\n\nNRD can be made work with combined diffuse and specular lighting in one\nchannel, but to improve denoising quality, it's better to separate out these two\nnoisy signals at the primary hit (or PSR as shown later).\n\n![A diagram of multiple paths traced through a scene, showing separate\ndiffuse/specular paths, the \"pathLength\" from the first to the second hit, and\ndirect light sampling at the first hit.](docs/NRD_sketch.png)\n\nSplitting the path traced image into a diffuse and specular image requires\nseparately following the diffuse and specular material's BSDF at the position\nof the first non-mirrored hit. This sample chooses to produce each signal's\nimage at full resolution, which also means we're doubling the amount of rays\nper pixel. Other approaches may used as well:\n\n  * a checkerboard pattern where every other pixel position follows either a diffuse or specular path.\n  * a probabilistic approach where following either a diffuse or specular path is a random choice.\n\nThere are certain requirements to make sure, diffuse and specular samples do\nexist in a specific neighborhood around each pixel. Follow the NRD README.md\nfile for more details.\n\nThe secondary paths to compute the indirect radiance coming from the diffuse\nBSDF's and specular BSDF's direction will then be performed in a regular\nfashion (i.e. after the first bounce, both kinds of light path sample the full BSDF).\nThis is expressed in `shaders/nrd.rgen` marked with `#DIFFUSE` and\n`#SPECULAR`. The path tracing code for gathering the indirect light contributions\ncan be found in `pathtrace.rchit`.\n\n### Hit distances\n\nNRD asks for a scalar 'hit distance' to be provided. This is typically the\npath length along the ray between the primary hit (or PSR) and the next hit.\nHit distances can also be extended to accumulate along the path, but this\nrequires special handling (refer to NRD's [README.md](https://github.com/NVIDIAGameWorks/RayTracingDenoiser/blob/master/README.md)).\nIn this example, we chose the easier method of just using the secondary path\nlength.\n\nSpecific hit distances are used under these circumstances:\n\n* if the ray is absorbed at the primary hit, we return a hit distance of 0.0.\n* if the secondary ray hits the environment, we return a hit distance of `NRD_INF`.\n\n\n### Mirror-like surfaces\n\nMirror-like surfaces work better with a special treatment. Instead of recording\nthe position and hit parameters at the surface of the mirror, we continue\nfollowing the mirrored ray until it hits something that is not a mirror - the\n_Primary Surface Replacement (PSR)_. It looks like as if we can see the\nmirrored objects \"behind\" the mirror as though the mirror acts as kind of a\nportal into a virtual world. The G-Buffer records the data of the PSR at its\nvirtual world space, such as Normal, Roughness and ViewZ. This helps the denoiser\ndenoise reflected objects much better.\n\nLook into `shaders/nrd.rgen` for `#PSR` to find the shader code that implements\nprimary surface replacement and consult NRD SDK's README.md for more details.\n\n### Demodulation\n\nDemodulation is the act of removing _intended_ 'noise' from the image before we\npresent it to the denoiser. These kinds of intended noise are typically details\ncoming from textures at the primary hit surface; for instance, grain in a wood texture. You certainly don't want the\ndenoiser recognizing these details as noise to remove. This would result in\nblurry, less detailed surfaces after denoising. In addition, we completely\nseparate out noise-free signals like:\n\n* light emissions of the primary surface;\n* perfect reflections of the environment map;\n* when the primary ray does not hit any geometry but instead hits the\n  environment map directly.\n\nWe follow NRD's README.md's suggestion for demodulating the diffuse and\nspecular signals in `nrd.rgen`.\n\nThe `composition.comp` shader will later multiply the denoised diffuse and\nspecular images by the inverse of the modulation factor and recombine all\nparts of the image (specular, diffuse, direct light) into a single one.\n\n### MIS weighting\n\nThe implemented path tracer uses importance sampling to estimate the\nintegral in the [rendering equation](https://en.wikipedia.org/wiki/Rendering_equation).\nCare has to be taken when the\npath tracer samples the same function multiple times, but uses different PDFs\nwhen doing so. This sample is built such that, at the primary hit and each\nsegment of the indirect path, we sample the environment map for its direct\nlight contribution at that point. It then follows the material's BSDF to probe\nfor indirect lighting contribution at the hit point. Thus, for any surface\npoint along the path it can happen that the environment is sampled twice: once\nvia directly probing the environment map (the envmap has its own precomputed\nsampling distribution) and once more when following the material's BSDF at each\nhit point. Both contributions need to be weighted accordingly. In this sample\nwe chose the \"power heuristic\" to compute the MIS weights for both\ncontributions.\n\n\n## Compositing\n\nThe final image will be composed of the denoised diffuse and specular color plus\ndirect lighting. Diffuse and specular are added only if there\nwas a hit and albedo is extracted from the base color and metalness information.\n\n\n```glsl\n  vec3 R = vec3(directLighting);\n  if(wasHit)\n  {\n    R += indirectDiff;\n    R += indirectSpec;\n  }\n```\n\n\n\n## Authors and Metadata\n\nTags:\n\n- raytracing, path-tracing, GLTF, HDR, tonemapper, picking, BLAS, TLAS, PBR\nmaterial, denoising, NRD\n\nExtensions:\n\n- VK_KHR_buffer_device_address, VK_KHR_acceleration_structure,VK_KHR_ray_tracing_pipeline, \nVK_KHR_ray_query, VK_KHR_push_descriptor, VK_KHR_shader_clock, VK_KHR_create_renderpass2\n\nAuthors:\n- [Martin-Karl Lefrançois](https://developer.nvidia.com/blog/author/mlefrancois/)\n- Mathias Heyer\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnvpro-samples%2Fvk_denoise_nrd","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnvpro-samples%2Fvk_denoise_nrd","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnvpro-samples%2Fvk_denoise_nrd/lists"}