{"id":18002984,"url":"https://github.com/achirkin/real-salient","last_synced_at":"2025-03-26T08:31:22.339Z","repository":{"id":92459176,"uuid":"266513237","full_name":"achirkin/real-salient","owner":"achirkin","description":"A GPU-only implementation of DenseCut for a RealSense camera","archived":false,"fork":false,"pushed_at":"2020-11-26T10:29:55.000Z","size":405,"stargazers_count":9,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"vr","last_synced_at":"2025-03-21T12:22:15.521Z","etag":null,"topics":["crf","cuda","gmm","grabcut","realsense"],"latest_commit_sha":null,"homepage":null,"language":"Cuda","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/achirkin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-05-24T09:58:10.000Z","updated_at":"2021-10-30T09:52:42.000Z","dependencies_parsed_at":"2023-06-02T12:45:11.936Z","dependency_job_id":null,"html_url":"https://github.com/achirkin/real-salient","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/achirkin%2Freal-salient","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/achirkin%2Freal-salient/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/achirkin%2Freal-salient/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/achirkin%2Freal-salient/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/achirkin","download_url":"https://codeload.github.com/achirkin/real-salient/tar.gz/refs/heads/vr","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245618763,"owners_count":20645064,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crf","cuda","gmm","grabcut","realsense"],"created_at":"2024-10-29T23:24:53.541Z","updated_at":"2025-03-26T08:31:22.333Z","avatar_url":"https://github.com/achirkin.png","language":"Cuda","funding_links":[],"categories":[],"sub_categories":[],"readme":"# real-salient\n\nThis is a header-only library that implements a modified version of\nthe [GrabCut algorithm](https://en.wikipedia.org/wiki/GrabCut).\nThe problem it solves is the two-class image segmentation (foreground/background);\nin other words, it detects a salient object in an RGB-D image.\nIt follows the logic described in the DenseCut paper by Cheng et al[[1]](#1),\nadapting it to GPU.\n\nIn short, the algorithm performs the following for every frame:\n\n  1. Get the color and depth buffers from a depth camera.\n  2. Assuming the salient object is in front, label the image pixels based on a simple treshold and the depth buffer\n     (i.e. just like in the [librealsense-grabcuts](https://github.com/IntelRealSense/librealsense/tree/master/wrappers/opencv/grabcuts) example).\n  3. Fit two Gaussian Mixture Models (GMM) onto the color frame to create the color models of background and foreground.\n  4. Use the trained models to label the image.\n  5. Use a Conditional Random Field model (CRF) to refine the labels.\n\n__Steps  2-5 are performed entirely on GPU, which allowed me to run the algorithm at steady 30 FPS.__\n\n#### Gaussian Mixture Models\n\nThe `gmm.cuh` module is a generic CUDA implementation of the\n[GMM](https://en.wikipedia.org/wiki/Mixture_model#Multivariate_Gaussian_mixture_model).\nIt can fit `M` GMMs, `K` components each, on a single image at once.\nThus, this module alone can be used for realtime `M`-class image segmentation.\n\nThe module uses the standard\n[EM-algorithm](https://en.wikipedia.org/wiki/Expectation%E2%80%93maximization_algorithm#Gaussian_mixture)\nfor estimation and\n[Cholesky decomposition](https://en.wikipedia.org/wiki/Cholesky_decomposition#LDL_decomposition_2)\nfor computing the covariance inverse and determinant.\n\n\n#### Conditional Random Fields\n\nThe `crf.cuh` module is an adaptation of the GPU implementation of CRF by\n[Jiahui Huang](https://github.com/heiwang1997/DenseCRF),\nwho used [Miguel Monteiro's](https://github.com/MiguelMonteiro/permutohedral_lattice)\nimplementation of fast gaussian filtering.\nThe theory for this implementation can be found in [[2]](#2) and [[3]](#3).\n\n## Examples\n\nThe examples make use of the core `real-salient` as well as of couple VR-related tricks.\nThey are hardcoded to use the Intel RealSense D415 camera and its SDK to\ncapture a color+depth video stream (`examples/vr-salient/include/cameraD415.hpp`).\n\n__VR bounds:__\nThe examples use [OpenVR](https://github.com/ValveSoftware/openvr) to improve the initial guess\nof the salient object position (step 2 in the algorithm above).\nI attach an extra tracker to the depth camera to locate its position in VR.\nThis allows me to find the position of the headset and hand controllers on the image via a simple coordinate transform.\nThis is implemented in `examples/vr-salient/include/vrbounds.hpp`.\n\n__VR depth stencil:__\nIn addition to the tracker positions, I employ a `Vulkan+OpenVR` combination to render the VR shaperone bounds\ninto a temporary buffer.\nThis allows me to cut-off all objects outside the user-defined play area from the scene.\nThis is implemented in `examples/vr-salient/include/vulkanheadless.hpp`.\n \n\n### vr-salient\n\n`vr-salient` is a standalone program.\nIn addition to the tweaks above, it uses OpenCV highgui library - only to display the window.\nThe VR tricks are optional in this example.\n\n### saber-salient\n\n`saber-salient` is a dynamic library to be used in my [BeatSaber plugin](https://github.com/achirkin/CameraPlus).\nIt functions the same as `vr-salient`, but does not require `OpenCV` and requires VR tracking.\n\n#### Demo videos:\n\n[\u003cimg src=\"examples/saber-salient/img/screen-1.jpg?raw=true\" width=\"260\" alt=\"360° Reason for Living by Morgan Page\" title=\"360° Reason for Living by Morgan Page\"/\u003e](https://youtu.be/1GdDrsxVWYE)\n[\u003cimg src=\"examples/saber-salient/img/screen-2.jpg?raw=true\" width=\"260\" alt=\"360° First of the Year (Equinox) by Skrillex\" title=\"360° First of the Year (Equinox) by Skrillex\"/\u003e](https://youtu.be/0zMn-zVGNNc)\n\n\n\n\n## References\n\n\u003ca id=\"1\" href=\"https://doi.org/10.1111/cgf.12758\"\u003e[1]\u003c/a\u003e\n[[pdf]](http://mftp.mmcheng.net/Papers/DenseCut.pdf)\nCheng, M.M., Prisacariu, V.A., Zheng, S., Torr, P.H.S. and Rother, C.\n_DenseCut: Densely Connected CRFs for Realtime GrabCut._\nComputer Graphics Forum, 34: 193-201. 2015.\n\n\u003ca id=\"2\" href=\"https://arxiv.org/abs/1210.5644\"\u003e[2]\u003c/a\u003e\nKrähenbühl, Philipp, and Vladlen Koltun.\n_Efficient inference in fully connected crfs with gaussian edge potentials._\nAdvances in neural information processing systems. 2011.\n\n\u003ca id=\"3\" href=\"https://graphics.stanford.edu/papers/permutohedral\"\u003e[3]\u003c/a\u003e\nAdams, Andrew, Jongmin Baek, and Myers Abraham Davis.\n_Fast high‐dimensional filtering using the permutohedral lattice._\nComputer Graphics Forum. Vol. 29. No. 2. Oxford, UK: Blackwell Publishing Ltd, 2010.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fachirkin%2Freal-salient","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fachirkin%2Freal-salient","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fachirkin%2Freal-salient/lists"}