{"id":19537908,"url":"https://github.com/scrn-vrc/point-pillars-in-unitycg","last_synced_at":"2026-03-07T09:33:23.638Z","repository":{"id":127937006,"uuid":"528298994","full_name":"SCRN-VRC/Point-Pillars-in-UnityCG","owner":"SCRN-VRC","description":"Implementation of PointPillars paper in Unity for VRChat","archived":false,"fork":false,"pushed_at":"2022-09-27T02:42:12.000Z","size":50983,"stargazers_count":18,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-26T15:41:31.938Z","etag":null,"topics":["machine-learning","point-cloud","pointpillars","shaders","unity3d","vrchat"],"latest_commit_sha":null,"homepage":"","language":"ShaderLab","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SCRN-VRC.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2022-08-24T06:43:59.000Z","updated_at":"2024-09-15T11:56:47.000Z","dependencies_parsed_at":"2024-01-18T16:13:09.056Z","dependency_job_id":null,"html_url":"https://github.com/SCRN-VRC/Point-Pillars-in-UnityCG","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/SCRN-VRC/Point-Pillars-in-UnityCG","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SCRN-VRC%2FPoint-Pillars-in-UnityCG","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SCRN-VRC%2FPoint-Pillars-in-UnityCG/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SCRN-VRC%2FPoint-Pillars-in-UnityCG/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SCRN-VRC%2FPoint-Pillars-in-UnityCG/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SCRN-VRC","download_url":"https://codeload.github.com/SCRN-VRC/Point-Pillars-in-UnityCG/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SCRN-VRC%2FPoint-Pillars-in-UnityCG/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30210852,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-07T09:02:10.694Z","status":"ssl_error","status_checked_at":"2026-03-07T09:02:08.429Z","response_time":53,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["machine-learning","point-cloud","pointpillars","shaders","unity3d","vrchat"],"created_at":"2024-11-11T02:29:53.190Z","updated_at":"2026-03-07T09:33:23.618Z","avatar_url":"https://github.com/SCRN-VRC.png","language":"ShaderLab","funding_links":["https://www.patreon.com/posts/tutorials-on-all-61128248"],"categories":[],"sub_categories":[],"readme":"# PointPillars in UnityCG\n\n\u003cimg src=\"https://i.imgur.com/JG3ICV4.png\" align=\"middle\"/\u003e\n\n### NOTE: This was built and tested with Unity 2019.4.31f1 using built-in render pipeline, there may be shader compatibility issues with other versions.\n\n## Table of Contents\n- [Overview](#overview)\n- [Problems](#problems)\n- [Setup](#setup)\n- [C++ Code](#c-code)\n- [Model Architecture](#model-architecture)\n- [GPU Implementation](#gpu-implementation)\n- [Resources](#resources)\n- [Datasets](#datasets)\n\n## Overview\n\nSimple implementation of [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://arxiv.org/abs/1812.05784) in Unity for VRChat using just fragment and geometry shaders without any additional dependencies. This is more of an educational tool than a practical implementation. This implementation takes ~40 frames to compute a single frame of input while running at ~20 FPS. The original runs around ~60 FPS in PyTorch.\n\n## Problems\n\nSlow computations aside, actual lidar data also returns a reflectance value. One of the key features used by the network during classification.\n\n\u003cimg src=\"https://i.imgur.com/7R79oj7.png\" align=\"middle\" width=\"500\"/\u003e\n\nThe table above was directly lifted from the PointPillars paper. It shows how reflectance raised the average precision from the base XYZ location information. This implementation supports reflectance, however, I modified it to a constant value of 0.15 because there is no easy way to estimate reflectance without extra information.\n\n## Setup\n\n1. Either clone the repo or download from [Release](https://github.com/SCRN-VRC/Point-Pillars-in-UnityCG/releases).\n2. Drag and drop the prefab in the `Prefabs` folder into the scene. Or open up the scene that came with the package.\n3. Run the network in Playmode.\n\nThe network outputs up to 100 predictions, only 33 of the bounding boxes are shown. Invalid predictions are returned as -1. All 100 predictions are rendered into `PointPillars\\RenderTextures\\Output3.renderTexture`. You can read it in a shader by importing the .cginc `PointPillars\\Shaders\\PointPillarsInclude.cginc`\n\n#### Shader properties:\n```C\n    Properties\n    {\n        _ControllerTex (\"Controller\", 2D) = \"black\" {}\n        _DataTex (\"Data Texture\", 2D) = \"black\"\n    }\n```\n`PPControllerBuffer.renderTexture` goes into _ControllerTex, `Output3.renderTexture` goes into _DataTex. `PPControllerBuffer.renderTexture` contains a count of all predictions.\n\n#### Includes:\n```C\n#include \"PointPillarsInclude.cginc\"\n```\n\n#### Functions:\n```C\nint count = getCount(_ControllerTex);\nfloat4 sizeRot = getPredictionSizeRotation(_DataTex, id);\nfloat3 pos = getPredictionPosition(_DataTex, id);\n```\n`id` is the index going from 0 to 99. An example of how to draw the bounding boxes with the following functions is in `PointPillars\\Display\\BBoxes\\BBoxDraw.shader`.\n\n\n## C++ Code\nThe C++ code included with the repo is just an exact CPU clone of how PointPillars would run on the GPU. No additional dependency is required to compile but it runs very slowly.\n\n## Model Architecture\n\u003cimg src=\"https://i.imgur.com/zSKsDQI.png\" align=\"middle\"/\u003e\n\nFigure from the original PointPillars paper. The network begins by voxelizing the lidar data into a 2D grid without bounding Z. Hence the name \"pillars\". This serves to condense the data into a dense matrix. Then it's fed into a classic 2D CNN classifier as the backbone, and ending with a single shot detector network structure, like YOLOv4.\n\n## GPU Implementation\n\u003cimg src=\"https://i.imgur.com/mnNYfS8.png\" align=\"middle\" width=\"500\"/\u003e\n\nThe GPU implementation for VRChat consists of 40+ cameras rendering to about 1GB of render textures. Moving points into pillar voxels required using a bitonic merge sort and d4rk's compact sparse texture code. One million particles were used in a geometry shader to scatter extracted features into the dense matrix for the 2D CNN.\n\nPointPillars spits out 321408 predictions towards the end. But because most of the outputs are 0, they can be filtered with d4rk's compact sparse texture method into a 32x32 render texture and sorted again.\n\nThe Non-maximum Suppression (NMS) method used at the end to remove duplicate bounding boxes was simplified in the GPU implementation because of how slow it already was. Instead of the normal [Rotation-robust Intersection over Union for 3D Object Detection](https://paperswithcode.com/paper/rotation-robust-intersection-over-union-for), I simply used a sphere intersection check on the shortest XY length of the bounding box. The original rotation-robust intersection implementation can be seen in the C++ code.\n\n## Resources\n- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://arxiv.org/abs/1812.05784)\n- [A Simple PointPillars PyTorch Implenmentation](https://github.com/zhulf0804/PointPillars)\n- [Compact Sparse Texture Demo](https://github.com/d4rkc0d3r/CompactSparseTextureDemo)\n- [Bitonic Merge Sort](https://en.wikipedia.org/wiki/Bitonic_sorter)\n- [Rotation-robust Intersection over Union for 3D Object Detection](https://paperswithcode.com/paper/rotation-robust-intersection-over-union-for)\n- [Quaternius Lowpoly Assets](https://www.patreon.com/posts/tutorials-on-all-61128248)\n\n## Datasets\n- [KITTI 3D Object Detection Evaluation 2017](https://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=3d)\n\nThanks to [d4rkpl4y3r](https://github.com/d4rkc0d3r/) for the Compact Sparse Texture Demo that made this possible.\n\nIf you have questions or comments, you can reach me on Discord: SCRN#8008 or Twitter: https://twitter.com/SCRNinVR","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscrn-vrc%2Fpoint-pillars-in-unitycg","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fscrn-vrc%2Fpoint-pillars-in-unitycg","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscrn-vrc%2Fpoint-pillars-in-unitycg/lists"}