{"id":19922846,"url":"https://github.com/seneral/makersvr","last_synced_at":"2025-09-21T07:00:08.515Z","repository":{"id":43481119,"uuid":"280957785","full_name":"Seneral/MakersVR","owner":"Seneral","description":"A cheap optical tracking system for room-scale VR and other applications","archived":false,"fork":false,"pushed_at":"2022-02-28T15:01:47.000Z","size":6885,"stargazers_count":59,"open_issues_count":2,"forks_count":9,"subscribers_count":11,"default_branch":"master","last_synced_at":"2025-05-03T07:37:29.956Z","etag":null,"topics":["camera","marker-tracking","optical-tracking","raspberry-pi","steamvr","stm32","vr"],"latest_commit_sha":null,"homepage":"https://www.seneral.dev/astertrack.html","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Seneral.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-07-19T21:58:45.000Z","updated_at":"2025-04-13T11:25:19.000Z","dependencies_parsed_at":"2022-09-12T01:01:16.800Z","dependency_job_id":null,"html_url":"https://github.com/Seneral/MakersVR","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Seneral/MakersVR","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Seneral%2FMakersVR","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Seneral%2FMakersVR/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Seneral%2FMakersVR/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Seneral%2FMakersVR/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Seneral","download_url":"https://codeload.github.com/Seneral/MakersVR/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Seneral%2FMakersVR/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":276204845,"owners_count":25602738,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-21T02:00:07.055Z","response_time":72,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["camera","marker-tracking","optical-tracking","raspberry-pi","steamvr","stm32","vr"],"created_at":"2024-11-12T22:12:15.908Z","updated_at":"2025-09-21T07:00:08.508Z","avatar_url":"https://github.com/Seneral.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# We Moved!\nThis project has been renamed and moved to \u003ca href=https://github.com/AsterTrack/AsterTrack\u003eAsterTrack\u003c/a\u003e after a long period of development. \u003cbr\u003e\nThis repository will remain only as an archive, as nearly every part of the project changed. \n\n\u003cp align=\"center\"\u003e\n  \u003cimg alt=\"Hardware Prototype of MakersVR\" src=\"Documentation/Media/Banner_AT_Baked.png\" width=\"100%\"/\u003e\n\u003c/p\u003e\n\n# MakersVR\nWelcome to MakersVR, an open source VR hardware project focused around a cheap optical tracking system. \u003cbr\u003e\nThe goal of this project is to create a cheap but capable tracking system that competes with existing commercial tracking systems like SteamVR Lighthouse, has the flexibility of a professional optical tracking system found in motion capture studios, but is cheap and easy to build. \u003c/br\u003e\nIt uses multi-cameras to triangulate individual small LEDs in space, so it is very easy to develop custom tracked DIY hardware, e.g. gun stocks, by adding LEDs on them in an arbitrary pattern, and it's also possible to develop fingertip tracking. \u003c/br\u003e\nSince the concept of such a system is decades old and still used in professional MoCap installations, although at a much higher price point, it could also integrate into existing MoCap focused solutions such as marker-based facial tracking and full body solvers. \u003c/br\u003e\nStandard 3-Point (or more) Full Body Tracking for VR is of course possible by default, and will likely be the most accurate next to SteamVR trackers, but by far not the cheapest solution for FBT. \n\n## How does it work?\nThe basis is an optical tracking systems with two or more cameras observing multiple active markers made with LEDs. \u003cbr\u003e\nThe \u003ca href=\"https://ar-tracking.com/technology/technical-details/\"\u003eART system\u003c/a\u003e is quite similar and provides a great overview of the general idea. \u003cbr\u003e\nEach tracking camera, called \u003ci\u003eMarker Detector\u003c/i\u003e, is a Raspberry Pi Zero plus camera. It is executing a Blob Detection algorithm to get the LEDs position in camera space. These are send over a serial connection to a central microcontroller, the \u003ci\u003eMarker Tracker\u003c/i\u003e. This serial connection is a CAT6 cable, but is currently used for power, camera sync and UART (might be switched to SPI later). \u003cbr\u003e\nThe centralized \u003ci\u003eMarker Tracker\u003c/i\u003e is currently a STM32F103 microcontroller, the cheap Blue Pill board. It is connected to up to three \u003ci\u003eMarker Detectors\u003c/i\u003e at once as well as the Host PC over a USB connection. It collects the Blob2D data from each \u003ci\u003eMarker Detector\u003c/i\u003e and currently immediately sends it to the Host PC using a isochronous Full-Speed USB 2.0 connection. It also times the frame of each \u003ci\u003eMarker Detector\u003c/i\u003e down to 0.1ms and sends synchronization feedback back over the serial connection. \u003cbr\u003e\nOn the Host PC, the \u003ci\u003eConfigurator\u003c/i\u003e program (and later the driver) connects to the USB device and 2D Blobs are streamed in. These 2D Blobs are converted to 3D rays using the position of the cameras in the room and the poses of the markers are inferred. This is still in debate on how exactly this will work, there are several approaches that I will be trying out. \u003cbr\u003e\nIn the Documentation you can find \u003ca href=\"https://github.com/Seneral/MakersVR/tree/master/Documentation/Design\"\u003emore detailed design documents\u003c/a\u003e, alongside a \u003ca href=\"Documentation/PreliminaryPartsList.ods\"\u003epreliminary parts list\u003c/a\u003e. This places the currently expected costs at below 200€ for a recommended system (3 cameras, 6 tracking targets) and 62€ for the absolute cheapest (2 cheap cameras, 3 tracking targets).\n\n## Current State\nI currently have a two-camera setup working as a prototype, and it works with my calibration marker prototype and tracking marker prototype. However I have not had luck with accurate intrinsic calibration yet, so the accuracy is not that great yet. Also since the cameras are not synced yet, quick movements disturb the marker greatly (and there's not filtering yet, either). \u003cbr\u003e\nHere's one of the \u003ci\u003eMarker Detector\u003c/i\u003e with the \u003ci\u003eMarker Tracker\u003c/i\u003e:\n\u003cp align=\"center\"\u003e\n  \u003cimg alt=\"Hardware Prototype of MakersVR\" src=\"Documentation/Media/HW_Prototype_Annotated.jpg\" width=\"50%\"/\u003e\n  \u003cbr\u003e\n  The \u003ci\u003eMarker Detector\u003c/i\u003e and \u003ci\u003eMarker Tracker\u003c/i\u003e prototypes\n\u003c/p\u003e\nThe actual tracking system on the host PC is functionally done and has been developed and successfully tested in testing environments. However it might need some adjustments to work practically. \u003cbr\u003e\nThe \u003ci\u003eConfigurator\u003c/i\u003e software currently contains a full testing mode and a device mode which connects to the actual hardware. In both modes you can do intrinsic calibration and extrinsic\u0026room calibration phases using the calibration marker, as well as marker calibration and tracking phases using a tracking marker. \u003cbr\u003e \n\u003cp align=\"center\"\u003e\n  \u003cimg alt=\"Software Configurator of MakersVR\" src=\"Documentation/Media/SW_Configurator_Annotated.png\" width=\"60%\"/\u003e\n  \u003cbr\u003e\n  The \u003ci\u003eConfigurator\u003c/i\u003e streaming blobs from the Prototype Hardware\n\u003c/p\u003e\nThe marker detection in the multi-camera marker tracking works similar to other professional tracking systems, with each 3D point identified based on the relations to its neighbours within the marker. This means markers can be easily read in using a calibration phase. \u003cbr\u003e\n\u003cp align=\"center\"\u003e\n  \u003cimg alt=\"Multi-Camera Tracking\" src=\"Documentation/Media/SW_Multi-Camera-Tracking.gif\" width=\"40%\"/\u003e\n  \u003cbr\u003e\n  Simulated marker tracking of a WMR-controller-like tracker. \n\u003c/p\u003e\nAny marker that has LEDs with (relatively) unique distances works fine. But a cheap and easy to produce marker is a huge challenge. The WMR-like ring simulated above is way too expensive for home production in low quantities, so a more traditional design is used, a base supporting several sticks with LEDs at the tip (see the \u003ca href=\"https://ar-tracking.com/products/markers-targets/targets/\"\u003eART targets\u003c/a\u003e to get an idea). The base is going to be 3D-printed and houses power distribution (1S LiPo), and can be adapted to support many different marker layouts - however the current prototype is made out of polymorph plastic, which allows for even quicker prototyping, \u003ca href=\"Documentation/Media/HW_Marker_PolymorphPlastic.jpg\"\u003eas seen here\u003c/a\u003e. The arms are made out of polypropylene straws, as they can survive quite a beating (for when you inevitably punch the walls) and can be easily replaced. The LED tips need to be spherical and well diffused, the current design is using a 3mm flat-top LED surrounded by sanded hot glue: \u003cbr\u003e\n\u003cp align=\"center\"\u003e\n  \u003cimg alt=\"Marker Prototype\" src=\"Documentation/Media/HW_Marker_Prototype.jpg\" width=\"71.875%\"/\u003e\n  \u003cbr\u003e\n  Marker Prototype powered by a 1S LiPo. Proportions and positioning can vary.\n\u003c/p\u003e\nOne hardware problem I'm still facing is the camera selection. The ecosystem of cameras for the Raspberry Pi is very limited, with the official modules having terribly low field of view, resulting in a smaller tracking volume, and most inofficial modules basing on the older camera module with much worse performance or a completely different chip alltogether with questionable support. Currently, I'm seeing if the Official Camera Module V2 with limited Field of View is viable: \u003cbr\u003e\n\u003cp align=\"center\"\u003e\n  \u003cimg alt=\"Software Configurator of MakersVR\" src=\"Documentation/Media/HW_CameraTrackingVolume.png\" width=\"60%\"/\u003e\n  \u003cbr\u003e\n  Tracking volume of three Marker Detectors using the Camera Module V2 in a small room\n\u003c/p\u003e\n\n## Going Forward\nAlthough having a functional second prototype, it is not ready for use yet, and there's a lot to improve still: \u003cbr\u003e\n1. Develop alternate continuous calibration algorithm in hopes of achieving better accuracy while being easier for the user\n2. Implement camera sync feedback (else fast movements are inaccurate)\n3. Add additional CPU blob detection pass to increase accuracy \n4. Add support for multiple properly tracked markers (currently simple frame-by-frame detection)\n5. Adapt SteamVR driver (SerialHMD) to general tracker use for FBT\n6. Improve QPU backend on the \u003ci\u003eMarker Detector\u003c/i\u003e to improve latency (can be reduced by up to 65% still)\n7. Develop power-management electronics to keep LiPo safe and develop PCB\n8. Develop final marker design(s), create 3D-print base\n9. Create a (3D-printed?) case for the electronics\n10. Add support for a more powerful microcontroller as the MarkerTracker: \u003c/br\u003e\n  Support more cameras at once, support chaining with more controllers, do onboard tracking, send of the final tracking data over bluetooth to standalone devices or over SPI port for embedded applications, all without requiring a host PC.\n\nAlong the way, depending on interest, I will expand the documentation on this project. \u003cbr\u003e\n\n## Rebuilding\nAlthough the software is not finalized yet, the first prototype can be build in its current state using just a soldering iron and it will stay the same for the forseeable future. All of the accuracy and performance improvements are purely software work. The required hardware components (with some parameters to adjust to budget) can be found \u003ca href=\"Documentation/PreliminaryPartsList.ods\"\u003ehere\u003c/a\u003e. If there's interest I'll add detailed instructions how to wire everything and tips for creating the calibration marker and tracking marker. \u003cbr\u003e\nThe software already has all required instructions for compiling in the respective subfolders, especially the testing mode of the \u003ci\u003eConfigurator\u003c/i\u003e software can be used without hardware.\n\n### Hardware\nThe Controller (MarkerTracker) is just a simple MicroController that doesn't do much right now, the code is only running on an STM32F103 right now though. \u003c/br\u003e\nFor each camera (MarkerDetector), only a Raspberry Pi Zero (or another model) is required, along with a MIPI camera. Currently, all standard camera modules are supported (V1 and V2), as well as all their derivatives, each with different drawbacks. Technically any monochromatic camera can be used, assuming a I2C driver implementation, as the RAW 8bit data from them is directly usable without an ISP at all. The \u003ca href=\"https://www.arducam.com/product/arducam-ov2311-mipi-2mp-global-shutter-camera-module-for-raspberry-pi-noir/\"\u003eArducam OV2311\u003c/a\u003e, with global shutter, monochromatic, 1600x1300 (2MP) at 60fps, no IR filter, and 100°hFoV support, it is a perfect fit, but costs 100$ per module, so I cannot recommend it as a camera to use right now. Also it requires custom software as it uses custom drivers, although I am very interested in getting them to work so if there is interest and a way to get them cheaply I will add support. There's also the \u003ca href=\"https://www.arducam.com/product/arducam-ov9281-1mp-global-shutter-noir-mono-mipi-camera-with-130deg-m12-mount-for-raspberry-pi/\"\u003eArducam OV9281\u003c/a\u003e with 1280×800 (1MP) at 60fps (arducam only expose 1 lane) for 42$ - \u003ca href=\"https://www.aliexpress.com/item/4001270692620.html\"\u003ethis cheaper model\u003c/a\u003e exposes 2 lanes and thus goes up to 144fps at full resolution, for just 32$. It might be a decent high-end high-framerate option that also might support hardware sync (supports external trigger, with pretty constant delay of \u003c 1ms, and aliexpress model exposes both as pins on the camera board).\n\n### Software\nRefer to each subprojects README for detailed instructions on setup.\n\n## Sub-Projects\n- Marker Detector: Software of the Raspberry Pi\n  - \u003ca href=\"https://github.com/Seneral/VC4CV\"\u003eVC4CV\u003c/a\u003e: The backends (GL and QPU) of the blob detection\n- Marker Tracker: Software for the STM32F103 that interfaces with the Marker Detectors (UART /w DMA) and the Host PC (USB)\n- Configurator: Test and future Configuration Software of the Host PC. Will be split later into tracking, configurator and driver\n\n## Tracking Quality\nAll I can deliver about tracking quality right now are preliminary results. Here are some numbers for the current prototype: \u003cbr\u003e\n### Latency\nThe newest WIP prototype uses the QPU backend of VC4CV, which is much, much faster than the GL backend that is currently in the repository.\n- The camera processing time before reaching my code is still unknown, but likely below 5ms\n- The frametime is 2-3ms for a 640x480 image and ~10ms for a 1640x1232 image - see below for more results.\n- The UART connection is currently only 115200 baud, so another ~8ms is added for just three markers\n- STM32 adds little to no latency, and considering the frame time and delay on host side, another 2-4ms can be added\n\nThis would add up to around 25ms, just to get it on the PC. \u003cbr\u003e\nFor the final version, I hope to speed up the serial connection, or use additional differential hardware.\nAssuming 1Mbaud (using either UART or SPI), it would only take around 1ms to transfer average data, so the final latency from movement to PC might drop as low as 10-15ms.\n\n#### Frametime results\nResults using WIP blob detection, from the moment the camera framework finishes a frame, to the moment the blobs are extracted. Note these are older, they do NOT include the frame copy fix which takes some more frametime. \u003c/br\u003e\nBut there are still some optimizations possible in ALL steps, but mostly Fetch, as the information where the blobs are could be passed from the QPU instead. In total, I expect I can cut this frametime down another 50-65%. \u003c/br\u003e\nHere's with a single blob with 10px radius:\n```\nZero, 1536x1232: 142fps; QPU 3.68 + Fetch 2.70 + CCL 0.12 =\u003e 6.5ms\nZero, 640x480: 482fps; QPU 1.01 + Fetch 0.46 + CCL 0.12 =\u003e 1.59ms\nZero, 512x480: 575fps; QPU 0.77 + Fetch 0.37 + CCL 0.12 =\u003e 1.26ms\n```\nAnd here's with a single blob with 40px radius:\n```\nZero, 1536x1232: 126fps; QPU 3.72 + Fetch 2.82 + CCL 0.79 =\u003e 7.33ms\nZero, 640x480: 345fps; QPU 1.02 + Fetch 0.59 + CCL 0.76 =\u003e 2.37ms\nZero, 512x480: 390fps; QPU 0.77 + Fetch 0.50 + CCL 0.75 =\u003e 2.02ms\n```\nQPU: Blob detection to mask on the QPU - Fetch: Fetching regions with blobs from the mask using CPU - CCL: Connected Component Labeling to extract distinct blobs using CPU\n\n### Accuracy\nLittle can be said about accuracy yet, as currently only the blobs binary average is taken as the center, and the size is also numerically estimated by the amount of dots. In the future, more sofisticated algorithms will be developed, like matching blobs to ellipsoids, or taking value averages for small blobs.\n\n## Contact\nDepending on what you want:\n- Create an issue on this repository for involved technical discussions\n- Catch me on Discord (Seneral#8110) for quick questions / discussions\n- Send me an email otherwise (contact@seneral.dev)\n\n## License\nRefer to each individual sub-project (MakersVR_\\*) for their licenses. Currently, all are licensed under MPL-2.0 \u003cbr\u003e\nThe rest of the materials in this repository are licensed under the MIT license.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fseneral%2Fmakersvr","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fseneral%2Fmakersvr","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fseneral%2Fmakersvr/lists"}